University of Southern California Dissertations and Theses

Teacher evaluation reform in situ: three essays on teacher evaluation policy in practice

(USC Thesis Other)

Teacher evaluation reform in situ: three essays on teacher evaluation policy in practice

PDF

Download

Open document

Flip pages

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content

Teacher Evaluation Reform in situ:
Three Essays on Teacher Evaluation Policy in Practice

by

Tracey L. Weinstein

A Dissertation Presented to the
FACULTY OF THE USC GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
(EDUCATION)

August 2014

Copyright 2014 Tracey L. Weinstein

ii

DEDICATION

To all my teachers, without whom I would have very little to say and no idea how to say it.

iii

ACKNOWLEDGEMENTS

There are so many people who have played a central role in my growth as person and
scholar and to whom I am forever indebted. They have served as advisors, colleagues, friends,
therapists, thoughtful critics, and the best cheerleaders. Many have served in each of these
capacities simultaneously, floating into and out of each role as the situation required. To
communicate the extent of my gratitude to each of these individuals will take a lifetime, but I
would like to begin by acknowledging them here.
To the teachers, administrators, and central office staff at the Los Angeles Unified School
District, thank you for taking time to generously share your experiences. I am inspired by the
devotion you have to your students and communities and admire your passion for learning and
growing as professionals. To the Talent Management Team, thank you for your partnership and
support in carrying out this work. Your commitment to understanding the impacts of your work
on teachers, administrators, and students is to be commended. I am especially thankful to Drew
Furedi and Donna Muncey for their patience in answering what I am sure felt like my endless
questions over the course of two years and their thoughtful and extremely useful feedback on
drafts of the papers that now make up my dissertation. Finally, I owe a huge thank you to Reino
Makkonen who was an invaluable partner in completing this work.
To my classmates, you have amazed and inspired me, angered and delighted me, pushed
me, and cheered for me over the last four years and I am thankful to each and every one of you.
To those in my cohort, thank you for making me a deeper thinker, a more engaged scholar, and a
committed advocate. I am particularly grateful to those of you who have become many of my
best friends and most trusted advisors. To the Sunday Night Dinner crew, thank you for making
sure I ate at least one square meal a week and for getting me through some of the toughest
iv

personal and professional times of my life. Kathryn and Keith, thank you for reading countless
papers with terrible grammar and reminding me to have faith in my ideas and to value divergent
perspectives. Alice and Cait, thank you for helping me stay the course when I wanted to give up,
and for pouring enough wine to help me forget why I wanted to quit in the first place. Stephani,
thank you for the late night work sessions and constant laughs, without which I would have
never finished my exams or dissertation. To Susan, thank you for your brilliant advice and
support and for being the best dog-sitter Ruby could ever ask for. Matt and Ayesha, thank you
for ensuring I maintained sanity over these last few years. Your brilliant feedback, and
unwavering support through the dissertation process has meant more than you will know. Your
intelligence amazes me and I am so lucky to call you both colleagues and lifelong friends.
To my committee members, Dr. Julie Marsh and Dr. Gary Painter, thank you. Your
thoughtful comments and advice pushed me to refine and improve many aspects of this work and
my dissertation is much stronger as a result. To my chair and advisor Katharine, thank you. It is
very rare that one person can have such a profound impact on another, but you have truly left a
mark on my life. Thank you for taking a chance on me as a bright-eyed nineteen year old with
more enthusiasm than actual capacity for scholarly work. Over the last six years you have been
an amazing guide, pushing me to think deeper, work harder, and care more than I ever thought I
could. You have led by example in demonstrating the commitment it takes to build the skills
necessary to produce high-quality research and have been an amazing coach in helping me to
develop them. You have taught me to have confidence in what I bring to the table and to always
ask questions that truly matter. Thank you for giving me the opportunity to engage in research
that plays a meaningful role in shaping policy. I am especially grateful for your generosity and
guidance during the dissertation process. Thank you for pushing me to make this work better
v

with each draft and for helping me look forward when I wanted to turn back. Thank you for
being my biggest advocate. I look forward to many more years of turning to you for sound
advice, thoughtful guidance, and a reassuring word.
It only makes sense to conclude these acknowledgements with the people who have been
with me since the beginning, my family. To my family by choice, the San Diego crew, you are
my heart and soul. Thank you for forcing me to act my age occasionally and for reminding me
that laughter is the best medicine. To my sisters Jill and Carly, and my brother and sister by
choice Armon and Alicia, thank you for listening to me complain, cry, laugh, and wonder over
the last four years. Thank you for feigning interest in my research and preventing me from giving
up on more than one occasion. To my parents, you are truly the best. Thank you for giving me
the confidence to venture down this uncertain path and for ensuring I stayed the course. Dad,
thank you for helping me keep perspective during it all and, Mom, thank you for reminding me
that by best is enough. To Grandma and Grandpa, thank you for teaching me that education is a
gift and for giving me the privilege of pursing it. I love you both so much. To Grami, thank you
for helping me see the humor in this insane endeavor. I am reminded each and every time we
speak that living life to the fullest means having passion for what you do and laughing hard and
often. And last but certainly not least, Tom, thank you. Thank you for believing in me to a fault.
Thank you for bouncing seamlessly between your roles as husband, copy editor, housekeeper,
cook, therapist, realtor, mover, dog walker, and friend, over the last few years without a single
complaint. Without your support I would be a far less better version of myself. Thank you.

vi

TABLE OF CONTENTS

DEDICATION............................................................................................................................... ii
ACKNOWLEDGEMENTS ........................................................................................................ iii
TABLE OF CONTENTS……………………………………………………………………….vi
CHAPTER ONE - Surveying the Landscape of Early Implementation and the Impacts of
Standards-Based, Multiple-Measure Teacher Evaluation Systems ......................................... 1
A Brief Review of the Extant Research on SBMMTES ............................................................. 4
The Initial Implementation Phase of the Educator Growth and Development Cycle ............... 18
Overview of the Dissertation .................................................................................................... 20
CHAPTER TWO - Making Sense to Make it Happen: How Teachers and Administrators
Understand a New Teacher Evaluation System and the Implications for Implementation 22
Introduction ............................................................................................................................... 22
Sensemaking Theory and a Cognitive Approach to Examining EGDC Implementation ......... 25
Individual Sensemaking ...................................................................................................... 26
Situated Cognition. .............................................................................................................. 27
The Role of Representations. .............................................................................................. 27
Data Sources and Methodological Approach ............................................................................ 28
Case Site and Participant Selection. .................................................................................. 29
Data Sources. ........................................................................................................................ 30
Analysis. ................................................................................................................................ 31
Limitations. .......................................................................................................................... 33
The EGDC Theory of Action and District Leaders’ Expectations for Implementation ............ 35
Expectations for Stakeholder Understanding of the Purpose of the EGDC. ................. 36
Expectations for Stakeholder Implementation of the EGDC. ......................................... 37
Findings ..................................................................................................................................... 41
Teacher and Administrator Understanding of the EGDC. ............................................. 41
Factors Shaping Stakeholder Sensemaking. ..................................................................... 43
The Relationship Between Stakeholder Understanding and Implementation. ............. 53
Discussion and Implications...................................................................................................... 57
CHAPTER THREE - Moving On and Moving Out? Examining Teacher Mobility in
Response to the Implementation of a Standards-Based, Multiple-Measure Teacher
Evaluation System ....................................................................................................................... 61
Introduction ............................................................................................................................... 61
vii

Research Questions ................................................................................................................... 63
Data and Sample........................................................................................................................ 64
Measuring Mobility. ............................................................................................................ 65
Analytic Approach .................................................................................................................... 70
The Difference-in-Difference Framework. ........................................................................ 70
Results ....................................................................................................................................... 82
School-Level Mobility.......................................................................................................... 82
Classroom-Level Mobility. .................................................................................................. 84
Discussion and Implications...................................................................................................... 85
CHAPTER FOUR - How Differential Access to Measures of Teacher Effectiveness Impact
Teacher Mobility in Response to Accountability in LAUSD .................................................. 89
Introduction ............................................................................................................................... 89
Research Questions ................................................................................................................... 92
Data and Sample........................................................................................................................ 94
Measuring Mobility. ............................................................................................................ 95
Analytic Approach .................................................................................................................... 97
The Difference-in Difference Framework. ........................................................................ 97
Descriptive Analyses. ......................................................................................................... 104
Limitations. ........................................................................................................................ 105
Results ..................................................................................................................................... 106
Elementary School-Level Mobility................................................................................... 107
Elementary Classroom-Level Mobility. ........................................................................... 109
Secondary School-Level Mobility. .................................................................................... 112
Secondary Classroom-Level Mobility. ............................................................................. 113
Descriptive Results. ........................................................................................................... 114
Discussion and Implications.................................................................................................... 120
References .................................................................................................................................. 124
Appendix A ................................................................................................................................ 135
Appendix B ................................................................................................................................ 152
Appendix C ................................................................................................................................ 170

1
CHAPTER ONE - Surveying the Landscape of Early Implementation and the Impacts of
Standards-Based, Multiple-Measure Teacher Evaluation Systems

Research has demonstrated that teachers matter. Teachers matter for student achievement
(e.g., Rivkin, Hanushek, & Kain, 2005; Rockoff, 2004; Sanders & Rivers, 1996), student
persistence and motivation (Croninger & Lee, 2001; Wentzel, 1997), student engagement and
satisfaction with school (Baker, 1999; Klem & Connell, 2004), and students’ long-term success
in life (Chetty, Rockoff, & Friedman, 2012, 2013). Given the consensus in the research that
teachers play a key role in helping students achieve important outcomes, the federal government
has moved teacher quality to the center of its reform agenda. Programs like Race to the Top,
School Improvement Grants, and the Teacher Incentive Fund tie much-needed federal funding to
the development and implementation of reforms that recruit, retain, evaluate, and develop high-
quality teachers locally (Perlman & Redding, 2011; U.S. Department of Education (DOE), 2009;
U.S. DOE, 2012a). More recently, waivers that provide states with flexibility from the
Elementary and Secondary Education Act (ESEA) also tie the opportunity to waive key pieces of
No Child Left Behind (NCLB) accountability to the implementation of more rigorous systems of
teacher evaluation and support (U.S. DOE, 2012b).
The current policy focus on identifying and cultivating “teacher quality” has prompted
states and districts across the country to design and implement new teacher evaluation systems
intended to better measure teacher effectiveness. As of 2013, 27 states required teacher
evaluations to be based on multiple measures and 35 required that student performance be a
significant factor in determining teachers’ evaluation scores (National Council on Teacher
Quality, 2013). Moreover, a multitude of districts, including New York City, Chicago,
Washington, D.C., and Los Angeles, have started to design and implement these new standards-
based, multiple-measure teacher evaluation systems (SBMMTES) locally.
2
These new systems differ markedly in form and content from existing systems of
evaluation, introducing a suite of new performance measures, including standards-based
classroom observations, measures of teachers’ “value-add” to student achievement on
standardized tests, surveys of students and parents, and in some cases, portfolios of instructional
materials. Together, these new measures are intended to address two primary flaws in existing
systems of teacher evaluation. First, SBMMTES are intended to provide teachers and
administrators with more diverse, and arguably better, information about areas of strength and
weakness in teacher practice, rather than the generic information produced from traditional
checklist-based measures. Second, the information produced from these new systems is intended
to inform more rigorous accountability for teacher performance, eradicating the perfunctory,
compliance-based traditions underlying current teacher evaluation practices (Donaldson, 2009;
Kauchak, Peterson & Driscoll, 1985; Weisberg, Sexton, Mulhern, & Keeling, 2009). In so doing,
these systems are expected to lead to improvements in individual teacher practice and more
informed human capital decisions (e.g., recruitment, retention, promotion, placement, and
compensation) that work together to increase the overall quality of the teacher workforce.
Despite their potential to improve the way in which states and districts assess and support
teacher quality, the implementation of SBMMTES across many states and districts has been met
with controversy and debate (Rado, 2012; Sanders, 2011; Sawchuck, 2013; Strauss, 2013).
Headlines from across the country demonstrate that teachers and administrators are struggling to
implement these systems as intended by policymakers, citing challenges with administrator time
and capacity to complete the required number of observations, among other barriers to
implementation (Garland, 2012; Fleischer, 2012; Paulson, 2012). Moreover, early evidence
suggests that these new systems are not necessarily doing a better job of differentiating teacher
3
quality with teacher ratings still clustered at the top end of the effectiveness distribution (Barge,
2012; Keesler & Howe, 2012; Florida Department of Education, 2013; Tennessee Department of
Education, 2012). There is also a growing debate about how to handle several important design
features associated with SBMMTES, including: how to combine multiple measures into a single
score, whether these scores should be a matter of public record or released privately to teachers
and administrators, and how, if at all, teacher evaluations should be linked to more summative
employment decisions. These early challenges and unanswered design questions suggest it may
be valuable to take a step back and examine more closely how these systems play out in practice
and the efficacy of SBMMTES in achieving their primary goals.
In the remainder of this chapter, I review the extant literature on the implementation and
impacts of SBMMTES, highlighting existing gaps in the knowledge base. Given the dearth of
empirical evidence on SBMMTES to date, I situate this discussion within a broader literature on
the implementation of other standards-based instructional reforms and teacher mobility in
response to increased test-based accountability. In so doing, I demonstrate how this broader
literature can inform our understanding of both the implementation and impacts of SBMMTES
on the teacher labor market. Throughout this discussion, I set the stage for a series of papers that
seek to fill important gaps in our knowledge on SBMMTES and introduce the research questions
that guide each paper. Next, I briefly introduce the context in which I answer these research
questions, the pilot implementation of a SBMMTES in the Los Angeles Unified School District.
I conclude with a brief overview of the remaining chapters in the dissertation.

4
A Brief Review of the Extant Research on SBMMTES

While states and districts are moving quickly to implement SBMMTES, extant research
on the implementation and impacts of these complex reforms remains limited. To date, most of
the research on SBMMTES has focused on examining the reliability and validity of the measures
used to identify teacher effectiveness (e.g., Chetty, Freidman, & Rockoff, 2011; Ho & Kane,
2013; Mertler, 1999; Peterson, Wahlquist, & Bone, 2000; Pianta, 2003; Tyler, Taylor, Kane, &
Wooten, 2010; Wenglinsky, 2002; Wright, Horn, & Sanders, 1997).
1
While extremely valuable,
this research fails to provide insight into how these measures might be integrated into a
comprehensive system of teacher evaluation and support. What evidence does exist on the
implementation of SBMMTES is limited in scope, focused primarily on identifying barriers to
implementation and cataloguing teacher and administrator perceptions of their experiences
implementing these systems (e.g., Heneman & Milanowski, 2003; Kimball, 2002; Sartain, 2009,
2011). Similarly, what we know about the impact of SBMMTES on relevant teacher outcomes is
restricted to only a handful of studies, offering minimal insight into their efficacy as a tool to
improve the overall quality of the teacher labor market (e.g., Dee & Wyckoff, 2013; Glazerman
& Seifullah, 2012; Sartain & Steinberg, 2014; Taylor & Tyler, 2012). Although limited in scope,
this research does lay the foundation necessary to begin to dig deeper into issues of
implementation with respect to SBMMTES and provides some insight into the efficacy of these
reforms for improving individual teacher practice and the quality of the teacher labor market.
The Implementation of SBMMTES. Research on the implementation of SBMMTES
demonstrates that teachers and administrators face a number of challenges carrying out these
new systems in practice (e.g., Firestone, et al., 2013; Kimball, 2002; Milanowski, 2001; Sartain,

11
For a comprehensive review of the reliability and validity of measures of teacher effectiveness see Goe, Bell, and
Little, 2008.
5
Stoelinga, & Brown, 2009). In their study of the pilot implementation of a new SBMMTES in
fourteen districts across Connecticut, Donaldson et al., (2014) found that these new systems were
implemented with relatively high degrees of fidelity across districts; however teachers and
administrators reported challenges with the amount of time required to complete the activities.
Similarly, in his examination of the implementation of a SBMMTES pilot in one Midwestern
district, Milanowski (2001) found that teachers and administrators struggled with time and
workload, reporting that the new evaluation process added additional work onto already
overburdened faculty without affording them additional time to carry out these responsibilities.
These findings are consistent across contexts and programs, with research on the implementation
of SBMMTES in New Jersey (Firestone, et al., 2013), Chicago (Sartain, Stoelinga, Brown,
2009), and Cincinnati (Heneman & Milanowski, 2003) finding that teachers and administrators
report significant challenges carrying out SBMMTES in practice largely because of the
additional time and work required to implement these complex reforms.
In addition to challenges with time and workload, the extant research suggests that
teachers and administrators may struggle to implement SBMMTES given limited capacity to
have meaningful conversations about instruction. In particular, Sartain et al. (2009, 2011) found
that the implementation of a new SBMMTES pilot in Chicago, was stifled by the weak
instructional coaching skills of administrators. According to the authors, administrators had a
difficult time adjusting to their new role of coach and evaluator, largely because of insufficient
capacity to offer specific, targeted strategies to teachers for improving instruction. Similarly,
Kimball (2002) found that while teachers reported having more instruction-focused
conversations with their administrators during their evaluation under the SBMMTES, these
6
conversations were characterized as relatively superficial and lacking specific feedback or
strategies for improvement that ultimately undermined the value of the SBMMTES for teachers.
Despite these initial challenges, existing evidence suggests that teachers and
administrators perceive SBMMTES to have an overall positive influence on their practice
(Donaldson et al., 2014; Heneman & Milanowksi, 2003; Kimball, 2002; Sartain, 2011). In
particular, Donaldson et al. (2014) found that 44% of teachers across the fourteen districts
participating in a SBMMTES in Connecticut, reported feeling that the information they received
from classroom observations led to positive changes in their practice. Moreover, the authors
found that 55% of administrators believed measures of student performance incorporated into the
SBMMTES contributed to positive changes in teacher practice. Similarly, Heneman and
Milanowski (2003) found that teachers participating in the Cincinnati SBMMTES reported
increased reflection on their practice as a result of the new evaluation process. Sartain et al.
(2011) found that teachers and administrators in Chicago reported a similar result after
participating in a SBMMTES pilot. In particular, these teachers and administrators noted that
their conversations about instruction during the post-observation conference were more
reflective, and better grounded in evidence about teacher practice than under the previous
evaluation system.
Although valuable in providing a picture of teacher and administrators perceptions of the
benefits and challenges associated with implementing a SBMMTES in practice, the research
described above fails to offer insight into the sources that may be driving teachers and
administrators to perceive these new evaluation systems as either beneficial or burdensome. This
oversight stems in part from the limited exploration of several assumptions embedded within the
theory of action underlying SBMMTES. One such assumption underlying the theory of action
7
for SBMMTES is that teachers and administrators will understand the processes and goals of
these new systems such that they implement them in the way intended by policymakers.
However, decades of implementation research suggests this assumption may be
misguided. This literature suggests that the meaning of a policy depends in large part on how
implementing agents negotiate policy directives with constraints in their environment and
knowledge of local needs (Berman & McLaughlin, 1978; Lipsky, 1980; Majone & Wildavsky,
1978; Mazmanian & Sabatier, 1989; Thomas, 1979). This process can lead to the adaptation of a
policy itself and contribute to variations in implementation (Datnow, Hubbard, & Mehan, 2002;
Honig, 2006; McLaughlin, 1976). Implementation research in education demonstrates that this
variation may be a function of individual cognition and how agents come to understand what
they are supposed to learn from policy ideas in order to change their practice (Cohen & Ball,
1990; Jennings, 1996; Spillane, 2004). This sensemaking process can shape how individuals
enact policy, helping to explain why observed implementation may differ from policymakers’
expectations (Coburn, 2001; Jennings, 1996; Spillane, Reiser, & Reimer, 2002).
In failing to explicitly examine the nature and source of stakeholder understanding and its
relationship to implementation, the existing research affords only a superficial understanding of
SBMMTES implementation and offers little insight into potential explanations for the variations
in their implementation observed to date. One exception to this is Donaldson et al. (2014), who
found that limited understanding contributed to the challenges faced by teachers and
administrators in implementing a SBMMTES. Although explicitly identifying understanding as
central to implementation, the authors do not examine how teacher and administrator
understanding is developed nor do they explore how the understandings constructed by these
stakeholders actually shapes implementation.
8
Research on the implementation of other standards-based instructional reforms
demonstrates the importance of examining the relationship between stakeholder understanding
and implementation (Coburn, 2001, 2005; Datnow & Castellano, 2000; Spillane, 1998; Spillane,
2004; Spillane & Callahan, 2000). For example, in a case study of two schools implementing
standards-based reading policies, Jennings (1996) finds that variations in implementation result
from differences among teachers in their understanding, which stems in part from individual
knowledge and beliefs about appropriate literacy instruction. Similarly, case studies of the
implementation of comprehensive school reform models in five schools demonstrate that
meaning-making by teachers varies, and is influenced largely by their emotions and the
connections they make between policy messages and existing professional values (Schmidt &
Datnow, 2005). Moreover, Coburn (2004) finds that teachers’ preexisting beliefs and practices
around reading instruction interact with messages about appropriate pedagogy from the
environment to shape their understanding and implementation of a new reading policy. These
examples demonstrate that regardless of the standards-based reform implemented, how
stakeholders understand certain policy ideas embedded within these complex reforms has
implications for their implementation.
To date, only one study has examined the process of sensemaking during the
implementation of a SBMMTES. Halverson, Kelley, and Kimball (2004) examine how school
leaders make sense of a SBMMTES in one district and find that how principals implement the
new process depends in large part on their existing knowledge, beliefs, and values with respect to
their role as instructional leader, their interpretation of the evaluation standards, and their school
context. This work demonstrates the importance of considering the mechanisms by which
individual implementing agents comes to understand SBMMTES, but leaves several areas
9
unexplored. Specifically, the authors do not examine sensemaking around the SBMMTES
process as a whole, focusing only on the Danielson-based evaluation framework. This provides
deep insight into administrator sensemaking with respect to new standards for effective practice
and classroom observations, but leaves a number of gaps in our understanding of administrator
sensemaking about these new systems more generally. Moreover, by focusing solely on school
leader sensemaking, the authors ignore teacher sensemaking and the role teacher and
administrator interaction may play in the understandings constructed by both stakeholder groups.
My first paper seeks to address this gap in the literature and contribute to the growing
knowledge base on the implementation of SBMMTES. Specifically, I ask: “How do teachers
and administrators make sense of a SBMMTES to construct an understanding of the reform
during early implementation?” and, “How does this understanding influence early
implementation of the SBMMTES by teachers and administrators?” In answering these research
questions, I draw on a theoretical framework grounded in sensemaking theory to provide much
needed evidence as to the mechanisms relevant for teacher and administrator understanding and
its relationship to implementation. By taking a theory-driven approach to examining
implementation, I move beyond cataloguing teacher and administrator reactions to SBMMTES
and dig deeply into one of the mechanisms that may be driving observed variations in
implementation. This paper also helps to advance theory; demonstrating how sensemaking
theory and a cognitive approach to examining implementation can be further developed to
examine standards-based reforms, like SBMMTES, that require the mutual engagement of
multiple implementing agents in order to achieve their intended goals.
The Impact of SBMMTES on the Teacher Labor Market. Another assumption
underlying the theory of action for SBMMTES is that implementing these new systems of
10
teacher evaluation and support will lead to positive changes in teacher and administrator
behavior with respect to the teacher labor market. As outlined above, SBMMTES are intended to
provide new and better information about teacher performance that is expected to facilitate
improvements in teacher practice and inform more meaningful human capital decisions like the
selective retention of high performers, differential attrition of low-performers, and strategic
staffing that ensures teachers are placed in schools and classrooms where they have the greatest
impact on student learning. As with other forms of accountability, however, targets of
SBMMTES may respond to potential sanctions in unintended ways (Booher-Jennings, 2005;
Crocco & Costigan, 2007; Jacob & Levitt, 2003). Concerned about additional accountability for
performance, teachers may move (or be moved) away from placements where additional
information about their practice is available (e.g., tested grades/subjects that have value-added
measures of effectiveness, VAMs) or in which it is perceived they are more likely to score
poorly (e.g., low-performing or highly disadvantaged schools/classrooms). Teacher response
may also vary based on how these scores are released –whether publicly, to provide parents and
communities with more information about teacher effectiveness, or privately, to provide teachers
and administrators with information to guide instructional improvement.
Whether intended or unintended, the potential for SBMMTES to induce teacher mobility
has several implications for the teacher labor market. If teacher mobility in response to a
SBMMTES leads to better teacher-site matches, these implications may actually be positive,
increasing teacher job satisfaction and productivity (Abelson & Baysinger, 1984; Jackson, 2010).
However, this mobility may also be disruptive, having adverse effects on school culture and staff
cohesion (Guinn, 2004), as well as negatively effecting the continuity of staff with the
institutional knowledge necessary to successfully carry out other reform initiatives designed to
11
increase student achievement at the impacted school site (Hess, 2003; Malen, Croninger,
Muncey, & Redmond-Jones, 2002; Malen & Rice, 2004; Rice & Croninger, 2005).
Changes in teacher mobility in response to SBMMTES and the new information about
teacher performance that they provide also have implications for how teachers are distributed
across schools and classrooms. Existing research on the effects of school- and test-based
accountability on teachers’ labor market behavior presents mixed evidence regarding teacher
response. Clotfelter, Ladd, Vigdor, and Diaz (2004) find that the introduction of North
Carolina’s state accountability system exacerbated teacher turnover in the state's lowest
performing schools, and Feng, Figlio, and Sass (2010) find a similar result in Florida after the
introduction of a school grading system. In contrast, Boyd, Lankford, Loeb, and Wyckoff (2008)
find that after the implementation of mandated state testing for 4
th
grade in New York, teacher
turnover decreased in 4
th
grade relative to other grades and in the years prior to mandated testing,
a result the authors posit could stem from the provision of additional resources to fourth grade to
offset the pressures of teaching for a high-stakes exam. While these studies suggest conflicting
patterns of teacher response, they demonstrate that teacher labor market behavior changes in
response to additional accountability.
Perhaps most importantly, this research demonstrates that changes in teachers’ labor
market behavior resulting from accountability policies has implications for the distribution of
high-quality teachers across schools and classrooms. For example, Feng, Figlio, and Sass (2010)
found that in addition to leading to increased turnover, changes in Florida’s accountability
system increased the average quality of teachers leaving the lowest-performing schools relative
to teachers in unaffected schools and in the years prior to changes in the school-accountability
system. Moreover, Clotfelter, Ladd, Vigdor, and Diaz (2004) found that North Carolina’s
12
school-grading system led to increases in the proportion of low-quality teachers in low-
performing, high-minority schools. New York again seemed to exhibit different patterns than
North Carolina and Florida, however; Boyd, Lankford, Loeb, and Wyckoff (2008) found that
after the introduction of test-based accountability in 4
th
grade in New York, teachers entering 4
th

grade were significantly less likely to be inexperienced than those moving into other elementary
grades, suggesting that the additional accountability had positive impacts on the level of
experience in New York’s 4
th
grade teacher labor market. Differences between Florida and North
Carolina relative to New York indicate that other factors that contribute to teacher mobility may
be at play, such as administrator discretion in teacher placement or the availability of open
positions and alternative labor market opportunities. Regardless, each study demonstrates that
teachers (and potentially administrators) respond to test-based accountability by changing
mobility patterns and that these responses have implications for how teachers are sorted across
schools and classrooms in each local context.
While valuable, the studies discussed above focus on school- and district-level
accountability, which serve only as indirect forms of accountability for teachers. SBMMTES,
especially those that include teacher-level VAMs, introduce a more direct form of accountability
for teachers by holding them individually responsible for the performance of their students.
Given patterns that have emerged in research about indirect accountability, we might expect
more direct forms of teacher accountability to have an even greater effect on teacher mobility
decisions. In addition, because SBMMTES are, at their core, an instructional reform designed to
provide better information about areas of strength and opportunities for growth in teacher
practice, we may expect quite different labor market responses than those observed from the
indirect accountability introduced by other test-based accountability policies. For example, we
13
may see that teachers are more likely to stay in their current schools and classrooms when
provided with concrete information about how to improve their practice. Moreover, we may see
that how this information is provided –either publicly or privately –leads to differences in teacher
response given the different consequences associated with having information about teacher
practice available to parents and communities rather than just teachers and administrators. Given
the unique nature of these reforms and the yet unsettled decisions about central design features
including how, and to whom, performance information should be released, how teacher mobility
may change in response to the implementation of SBMMTES remains an open question.
To date, only a few studies have explicitly examined the relationship between teacher
mobility and the implementation of SBMMTES that provide new information to teachers about
their performance through value-added and other measures (Dee & Wyckoff, 2013; Glazerman
& Seifullah, 2012; Sartain & Steinberg, 2014). These studies yield mixed evidence as to the
impact of these new systems on teacher mobility behavior and each suggests differing
implications for the composition of the teacher labor market in the respective local contexts.
Glazerman and Seifullah (2012) examine teacher retention in Chicago schools
participating in a comprehensive teacher evaluation system known as the Teacher Advancement
Program (TAP). TAP is a comprehensive school reform model that links a SBMMTES to
targeted professional development, career pathways for teachers, and performance-based
compensation. The authors find that teacher’s in schools randomly assigned to TAP have higher
retention rates overall in the first two years of program implementation relative to teachers in
non-TAP comparison schools. However, the authors are unable to disentangle whether the
SBMMTES specifically, or the SBMMTES in conjunction with the larger TAP system, drives
the mobility decisions of these teachers. The authors also hypothesize that mobility rates for
14
teachers in tested grades and subjects who also received VAM under TAP may differ from those
of teachers in similar placements within non-TAP schools, but find that this additional
information did not contribute to retention effects above and beyond those observed from
participating in the TAP pilot alone. Finally, the authors find no evidence that increased retention
resulting from TAP improved the quality of teachers in TAP schools, suggesting that
implementing a SBMMTES inclusive of a teacher-level VAM had no meaningful impact on the
composition of the teacher labor market in these Chicago schools.
Sartain and Steinberg (2014) also examine teacher mobility in response to a SBMMTES
in Chicago, drawing not on the TAP program, but on experimental evidence from a separate
SBMMTES pilot known as The Excellence in Teaching Project (EITP). Under EITP, teachers in
schools randomly selected for participation were evaluated multiple times a year using a
classroom observation tool based on Charlotte Danielson’s Framework for Teaching. Unlike the
TAP program, teacher evaluations under the EITP pilot were based solely on classroom
observations, but participants received detailed information about their practice according to this
new measure through post-conferences with their administrator. Sartain and Steinberg (2014),
exploit the random assignment of schools to EITP participation and examine the impact of
participating in this new evaluation process on the probability of a teacher switching schools
within the district or leaving the district altogether. They find that overall EITP participation did
not increase the probability of teachers switching schools or exiting the district. However, they
find that non-tenured, low-rated teachers who participated in EITP exited the district at
significantly higher rates than similar teachers in non-EITP schools. This suggests that different
types of teachers responded differently to EITP participation and did so in ways that actually
increased the overall quality of teachers in schools participating in the SBMMTES pilot.
15
Dee & Wyckoff (2013) also examine the impact of a SBMMTES on teacher retention.
Specifically, they use a regression discontinuity approach to compare the impact on teacher
retention of D.C. IMPACT –a new teacher evaluation system in the District of Columbia Public
Schools that links teacher dismissal and salary decisions to teacher performance. They find that
dismissal threats resulting from IMPACT scores increased voluntary attrition of low-performing
teachers by more than 50% in the following school year. Dee & Wyckoff (2013) also investigate
the possibility that IMPACT ratings may have contributed to teachers switching across tested
grades/subjects in order to improve their evaluation ratings from one year to the next. They find
that IMPACT did not induce switching between tested and non/tested subjects, but cannot reject
the null hypothesis that teachers left teaching for another job in the school or district not covered
under IMPACT, or to another job outside of teaching. This initial evidence demonstrates the
need to examine in greater detail, the implications of implementing a SBMMTES, inclusive of a
teacher-level VAM, on how teacher mobility may change from one year to the next both within
the district, but also outside of teaching. Finally, in contrast to Glazerman and Seifullah (2012),
who find no implications of increased retention resulting from TAP on the composition of the
teacher labor market in Chicago, Dee & Wyckoff (2013) find that IMPACT had and overall
positive impact on the quality of the teacher labor market in D.C.
Taken together, the existing research on teacher response to test-based accountability and
the implementation of SBMMTES, in both pilot and high-stakes settings, suggests that teacher
mobility behavior changes in response to the provision of additional information and
accountability for teacher performance. While some research finds that these changes lead to
increased mobility in disadvantaged schools and lowers the quality of those who remain in these
“hard-to-staff” contexts, others find that these changes lead to the selective retention of high-
16
performers and actually increase the overall quality of human capital within the teacher labor
market. These conflicting findings demonstrate the need for additional research on the impact of
SBMMTES on teacher labor market behavior across different contexts and types of systems.
To contribute to the growing evidence base on the broader effects of implementing
SBMMTES on teacher mobility, my second paper asks: “Are teachers who participate in an
SBMMTES pilot more likely to switch schools or classrooms or leave the district after the pilot’s
implementation relative to teachers who do not participate in the pilot reform? and, “Do these
effects differ among teachers in “hard-to-staff” contexts?” Moreover, to further our
understanding of how teacher participation in these new SBMMTES may impact the
composition of the teacher workforce I ask, “How, if at all, do changes in teacher mobility
resulting from participating in the SBMMTES pilot impact the composition of the teacher labor
market locally?” In answering these questions, I provide additional information to policymakers
about the relationship between participating in a SBMMTES pilot and the broader teacher labor
market, and how, if at all, these systems may work to address or exacerbate existing inequities in
access to high-quality teachers by changing the composition of the teacher labor market.
Finally, while some of the studies outlined above begin to examine how additional
accountability for performance may impact teacher mobility differently for different types of
teachers (e.g., those in tested grades/subjects or certain “hard-to-staff” contexts), we still know
very little about the differential effects of more direct forms of accountability on different types
of teachers. This is of particular policy relevance as SBMMTES redefine the type of information
available about teacher performance, which in itself may have implications for teacher school-
and classroom-level mobility. Moreover, as discussed above, many states and districts are
grappling with issues about how best to release this new performance information with some
17
arguing that it should be a matter of public record and available to parents and communities
(O’Donnell & Lieszkovszky, 2013), and others arguing that it should be privately released to
teachers and administrators so as not to undermine its utility as a tool for improving instruction
(American Institutes for Research, 2014; Strauss, 2012). Given that each approach to sharing this
new information about teacher performance has unique implications, we may expect that
teachers will respond differently within the labor market depending on how much information is
available about their performance and how this information is released. Yet we know very little
how either design choice will impact teacher mobility, if at all, and the implications for the
distribution of teachers across schools and classrooms.
In paper three, I examine each of these issues more closely. Specifically, I ask: “Are
teachers eligible to receive VAMs more likely to switch schools/classrooms or leave the district
after the public release of value-added scores or the implementation of a SBMMTES pilot
relative to teachers who are ineligible to receive VAMs?, and “Do these effects differ among
teachers in “hard-to-staff” contexts and/or based on how this new performance information is
released (e.g., publicly or privately)?” Moreover, to begin to build an understanding of the
implications of any differential effects on teacher mobility, I ask: “How, if at all, do changes in
teacher mobility resulting from access to additional information about teacher performance
through VAM impact the composition of the teacher labor market locally?” and “Do the
compositional effects differ based on how this new performance information is released?” In
answering these questions I further our understanding of the differential effects of SBMMTES
on different types of teachers with access to different information about their performance.
Moreover, I offer initial evidence to policymakers about the implications for teacher mobility of
publicly versus privately releasing teacher effectiveness data.
18

The Initial Implementation Phase of the Educator Growth and Development Cycle

To answer the research questions outlined above, I turn to the Los Angeles Unified
School District. At the beginning of the 2010-2011 school year, LAUSD began developing a
SBMMTES called the Educator Growth and Development Cycle (EGDC). The EGDC was
originally designed to include multiple measures of teacher effectiveness, including: classroom
observations of teacher practice by a site administrator and a second observer using protocols
aligned with the LAUSD Teaching and Learning Framework (adapted from Charlotte
Danielson’s Framework for Teaching)
2
; stakeholder feedback surveys of students and parents;
teacher- and school-wide value-added scores which capture a teachers’ contribution to student
achievement on standardized test scores (known as Academic Growth Over Time, AGT); and a
measure of teachers’ contribution to their school community.
Like many SBMMTES across the country the EGDC was intended to replace LAUSDs
current system of evaluation (known as the Stull) which is highly criticized for failing to
adequately assess teacher practice and encourage meaningful improvements in teacher quality
(The New Teacher Project, 2009). By relying on multiple measures of performance and
establishing a common language of instruction through the Teaching and Learning Framework,
LAUSD intended for the EGDC to better identify a teachers’ level of effectiveness and to
provide teachers and administrators with more detailed and diverse information about areas of
strength and weakness in teacher practice that can be used to inform more targeted growth and

2
The classroom observation measure incorporated teacher self-assessments and lesson planning activities, the actual
classroom observations and pre- and post-observation conferences between teachers and observers.
19
development opportunities, and eventually inform more summative human capital decisions
(e.g., retention and dismissal, placement, and compensation).
3

Serving as the backdrop to LAUSDs development of the EGDC was the controversial
and very public release of a database of teacher-level VAMs by the Los Angeles Times. In
August of 2010, at the start of the district’s first year of planning for the EGDC, the Los Angeles
Times released a database of self-generated value-added scores (i.e., not generated by LAUSD
for the purpose of assessing teacher efficacy) linked to teacher names for a subset of LAUSD
elementary teachers in grades three through five. The Times’ release of teacher value-added
received national attention as it was among the first ever searchable databases providing
information on individual teacher effectiveness (Felch, Son, & Smith, 2010; Roterham, 2010).
Although the Times’ value-added database was completely external to the district and unrelated
to its plans for the EGDC (which were already underway), the public release of VAMs for
LAUSD teachers by the Los Angeles Times foreshadowed the districts own plans to release a
separate set of value-added scores privately to teachers and administrators as part of the EGDC.
During the 2011-2012 school year, coming on the heels of the Times’ value-added
database release and LAUSDs internal planning, the district began its Initial Implementation
Phase (IIP) of the EGDC. The IIP piloted a reduced-form of the EGDC in a no-stakes setting
with a volunteer sample of 425 teachers and 118 administrators in 100 schools across the district.
Specifically, the IIP piloted the classroom observation cycle and stakeholder surveys of students
and parents for a subset of participating teachers. Moreover, as part of the IIP, LAUSD released
their own set of school-wide and individual-level VAMs for all schools and teachers in tested
grades/subjects, respectively. Importantly, this information was generated using a different

3
This theory of action for the EGDC is not LAUSDs formal theory of action, but rather one I derived based on
structured interviews with district leaders responsible for the design and implementation of the EGDC pilot.
20
formula than that used by the Los Angeles Times and was released privately to teachers and
administrators only. Although school-wide and individual VAMs were released as part of the
pilot, their distribution extended beyond just those teachers participating in the IIP. During the
2011-2012 school year, all teachers in tested grades and subjects (both IIP participants and non-
participants) received additional information about their effectiveness from VAM, while teachers
in non-tested grades and subjects did not receive this additional performance information.
Similarly, all teachers participating in the IIP received additional information about their
performance through classroom observation ratings, and if available, VAMs and stakeholder
surveys, while non-participants did not receive this additional performance information.

Overview of the Dissertation

The sections above demonstrate the need for additional research on the implementation
and efficacy of SBMMTES and outline the unique context in Los Angeles that provides a rich
setting in which to further examine these issues. The remainder of this dissertation is organized
around each set of research questions presented above. In Chapter 2, I answer the first set of
research questions which seek to provide insight into how teachers and administrators make
sense of a SBMMTES and the implications of the understandings they construct for
implementation. In Chapter 3, I answer the second set of research questions which examine the
impact of participating in a SBMMTES pilot on teacher school- and classroom-level mobility
and the implications for the composition of the teacher workforce in LAUSD. Finally, in Chapter
4 I answer the third set of research questions, providing early evidence about how teachers with
access to different levels of information about their practice may respond differently, in terms of
21
their school- and classroom-level mobility decisions, to the implementation of a SBMMTES, and
whether the public or private release of VAM leads to differences in these mobility decisions.
The structure of Chapters 2-4 follow a similar form. I begin with a brief introduction of
the paper and research questions that are focus of the chapter. Then, I move into a discussion of
the data and methodological approach used to answer the research questions, including a
discussion of the limitations of the data and their implications for the generalizability of my
results. Next, I provide the results. Finally, I conclude with a discussion of the implications of the
work for research, policy, and practice with respect to the design and implementation as well as
impacts of SBMMTES on the teacher labor market.

22
CHAPTER TWO - Making Sense to Make it Happen: How Teachers and Administrators
Understand a New Teacher Evaluation System and the Implications for Implementation

Introduction

As outlined in Chapter 1, states and districts across the country have moved quickly to
design and implement new teacher evaluation systems despite limited evidence about their
efficacy (National Council on Teacher Quality, 2013). These standards-based, multiple-measure
evaluation systems (SBMMTES) represent a significant departure from current systems of
evaluation; they introduce new standards for effective teaching and redefine teacher and
administrator work with respect to teacher evaluation as well as growth and development. By
simultaneously asking teachers and administrators to think differently about what good teaching
looks like, and to engage in new processes for measuring and supporting effective instruction,
the implementation of SBMMTES are at their core about teacher and administrator learning. To
successfully implement these systems, stakeholders must shift their thinking with respect to
teacher quality and evaluation, making sense of SBMMTES in the context of their own practice.
Despite the major shift in understanding required by teachers and administrators to
implement SBMMTES, we know very little about the process of teacher and administrator
learning embedded within these new reforms. The extant research on the implementation of
SBMMTES focuses primarily on cataloguing teacher and administrator reactions to these new
systems and outlining barriers to implementation (e.g., Donaldson, et al., 2014; Firestone, et. al,
2013; Heneman & Milanowski, 2003; Kimball, 2002; White et. al, 2013). While valuable, this
research assumes that teachers and administrators understand these reforms as intended by
policymakers, taking for granted the fact that agents do not simply accept or reject reform ideas
but rather negotiate their meaning based on what they already know and do in their daily work
(Berman & McLaughlin, 1978; Datnow, Hubbard, & Mehan, 2002; Lipsky, 1980; Majone &
23
Wildavsky, 1978). In failing to examine how stakeholders come to make sense of policy
messages as they learn about these new reforms, existing research provides little insight into the
role understanding may play in shaping the implementation of SBMMTES.
To address this gap in the literature and contribute to the growing knowledge base on the
implementation of SBMMTES, I turn to the pilot implementation of the Educator Growth and
Development Cycle (EGDC) in the Los Angeles Unified School District (LAUSD). Drawing on
the experiences of a subset of teachers and administrators participating in the EGDC pilot, I ask:
“How do teachers and administrators make sense of a SBMMTES to construct an understanding
of the reform during early implementation?” and “How does this understanding influence early
implementation of the SBMMTES by teachers and administrators?”
To answer the research questions I rely on a theoretical framework grounded in
sensemaking theory and a cognitive approach to examining implementation (Coburn 2001;
Coburn, 2005; Jennings, 1996; Spillane, Reiser, & Reimer, 2002; Spillane, 2004). This
perspective posits that implementing agents make sense of policy ideas through an active process
of interpretation; how they understand a policy ultimately shapes how it is implemented. This
theoretical lens is particularly valuable for examining the implementation of SBMMTES because
it provides a mechanism to explore how new ideas about teacher quality and evaluation are
processed by, and integrated into, teachers’ and administrators’ existing understandings of
effective teacher evaluation and support. Moreover, it connects this sensemaking process to
implementation, providing a model to carefully examine the nature and source of challenges
associated with understanding and implementing complex reforms like SBMMTES.
In taking a cognitive approach to examining the implementation of the EGDC, I find that
teachers and administrators draw heavily on their existing knowledge, beliefs, and experiences
24
when making sense of the new evaluation system. While these existing schemas dominate the
sensemaking process for administrators, teacher sensemaking with respect to the EGDC is also
largely shaped by their interactions with administrators around the new evaluation process. For
both groups, how understandings are constructed influence implementation of the EGDC–
contributing to either a superficial execution of the new evaluation process, or a more
instruction-focused implementation that stakeholders report has implications for improving
teacher practice. These findings demonstrate that ensuring SBMMTES are implemented in ways
that best serve to improve teacher practice requires structuring learning opportunities to support
the co-construction of understanding between teachers and administrators and the mutual
engagement of both stakeholder groups in implementing these new systems.
In what follows, I outline the theoretical framework that guides this work and affords a
careful examination of sensemaking and implementation in the context of the EGDC. Next, I
describe the data and methods used to answer the research questions. Then, to provide context
for my findings, I offer a more detailed description of the theory of action underlying the EGDC,
paying particular attention to district leaders’ expectations for teacher and administrator
understanding and implementation during the pilot. Following this discussion, I present my
findings, providing insight into how teachers and administrators made sense of the EGDC during
the pilot, relative to district leaders’ expectations, and the relationship between their
understanding and implementation. I conclude with a discussion of the central themes and their
implications for research, policy, practice, and theory with respect to implementing SBMMTES.

25
Sensemaking Theory and a Cognitive Approach to Examining EGDC Implementation
To examine how teachers and administrators made sense of the EGDC during the pilot
and the relationship between the understandings they constructed and implementation, I take a
cognitive approach to examining implementation. From this perspective, how a policy is
understood and subsequently implemented by local actors depends on the interaction of their
existing cognitive schemas –or knowledge, beliefs/values, and previous experiences –their
situation, or the environment in which they confront policy messages, and the ideas embedded
within a policy itself (Spillane, Reiser, & Reimer, 2002). A cognitive approach to examining
implementation is grounded in sensemaking theory, which posits that individuals construct
understanding through an active process of meaning making that requires new information be
interpreted through existing cognitive structures that shape what we know and understand about
a given phenomenon (Weick, 1995; Yanow, 1996). This lens has been widely used to examine
the implementation of other standards-based reforms –like state content/curriculum standards, as
well as reading interventions and comprehensive school reform (Coburn, 2005; Datnow &
Castellanos, 2000; Jennings, 1996; Spillane, 1998) –but has only recently been drawn on to study
the implementation of SBMMTES, specifically (Halverson, Kelley, & Kimball, 2004).
Using a cognitive approach to examine the role of teacher and administrator
understanding in shaping the implementation of a SBMMTES is valuable for several reasons.
First, this theoretical lens provides a mechanism to explore how teacher and administrator
learning takes place during the implementation of a SBMMTES. Second, this approach draws
useful attention to both the individual and social dimensions of meaning making. This is
particularly important because the implementation of SBMMTES relies on two classes of
implementing agents, teachers and administrators. Both have existing understandings of teacher
26
quality and evaluation that are influenced by myriad sources which they bring to bear when
making sense of the new process, but must also rely on interactions with one another to
implement this reform. Finally, by providing a framework to unpack the mechanisms by which
stakeholders come to understand these new evaluation processes, a cognitive approach to
examining implementation affords the opportunity to move beyond cataloguing implementation
challenges to identifying potential root causes. This information can be used to identify possible
solutions to the challenges currently faced by states and districts in implementing SBMMTES.
Figure 1 in Appendix A provides a conceptual framework (adapted from Spillane, Reiser,
& Reimer, 2002) for how teachers and administrators make sense of the EGDC and the
relationship between this process and implementation during the pilot. Based on a cognitive
approach to examining implementation the framework focuses our attention on three core
components: 1) individual sensemaking, 2) situated cognition, and 3) the role of representations.
According to this framework, these components interact to shape stakeholder understanding of
the EGDC which in turn, influences implementation of the new evaluation process.
Individual Sensemaking. The first component of the theoretical framework suggests a
focus on understanding the substance of implementing agents’ cognitive schemas –their existing
knowledge, beliefs/values, and previous experiences –to explore variations in meaning derived
from policy ideas. For example, when implementing the EGDC, teachers and administrators
bring existing understandings of teacher quality and evaluation –influenced by myriad sources
(e.g., the Stull evaluation, and/or serving as a Beginning Teacher Support and Assessment
(BTSA) support provider) –to bear to make sense of the EGDC. The sources of prior knowledge,
beliefs, and experiences drawn on by the implementing agent, and how he or she uses them to
interpret the EGDC, shapes understanding and has implications for implementation.
27
Situated Cognition. The second component focuses attention on the role of situation and
interaction in shaping sensemaking. This dimension suggests that implementing agents’
sensemaking about the EGDC is also constituted in the process of interacting with others and the
institutional context (Spillane, Reiser, & Reimer, 2002). Through these interactions,
implementing agents negotiate understandings of policy ideas from the environment. Because
teachers and administrators engage in implementation within the context of schools –each
characterized by a distinct set of norms and values, opportunities for interaction, and
organizational arrangements –it is important to directly examine the role of situation in shaping
individual sensemaking with respect to the EGDC. Exploring how situated cognition shapes
sensemaking helps identify potential reasons for similarities and differences in how teachers and
administrators come to understand and implement the EGDC during the pilot.
The Role of Representations. The final component concerns the role of artifacts and
policy representations in the sensemaking process. When implementing the EGDC teachers and
administrators are subject to a mélange of messages about teacher quality and evaluation, relying
on policy documents, training materials, data reports, and rubrics to make meaning of the new
system. The role of representations emphasizes that the nature, source, and structure of these
messages are an important part of how individuals make sense of the EGDC.
I use the framework in Figure 1, Appendix A to unpack how teachers and administrators
made sense of the EGDC during the pilot. For example, Teacher A may believe that evaluation
cannot be used to help teachers improve their practice, a belief reinforced by limited interactions
with her administrator when implementing the EGDC during the pilot. In contrast, Teacher B
may see evaluation and development as inextricably linked, a belief informed by years of
experience serving as a mentor in a new teacher induction program and reinforced by frequent
28
instruction-focused interactions with her administrator during the pilot. Differences in these
individual beliefs and experiences shape how Teacher A and Teacher B come to understand the
EGDC, with the former understanding the purpose of the EGDC as just another accountability
mechanism, and the latter understanding the EGDC as primarily about encouraging teacher
growth and development. From a cognitive perspective, variation in understanding between
Teacher A and Teacher B, may lead to differences in how they implement the new evaluation
system. Aligned with her previous evaluation experiences, Teacher A may engage in
implementation that is more compliance-driven, completing the required activities in a
superficial manner. In contrast, implementation by Teacher B may be more improvement-
focused, with reflection and improvement undergirding her completion of each required activity.
In the analyses presented below, I examine how the understandings constructed by
teachers and administrators of the EGDC influenced implementation during the pilot.
Specifically, I consider the factors that shape individual sensemaking, the role interactions and
school context play in the meaning making processes for each stakeholder group, and how, if at
all, the artifacts used to communicate policy ideas to teachers and administrators influenced
participant understanding. In so doing, I begin to develop our understanding of what, if anything,
the relationship between sensemaking and implementation tells us about the role of teacher and
administrator learning in implementing standards-based instructional reforms like SBMMTES.

Data Sources and Methodological Approach
Taking a cognitive approach to understanding teacher and administrator sensemaking
with respect to the EGDC and its relationship to early implementation requires a methodological
approach that allows for an investigation of individual sensemaking within the context of the
29
institutional environment. Therefore, I rely on a multiple-site (embedded) case study design with
individuals (teachers and administrators) nested within schools, serving as the first and second
unit of analysis, respectively (Yin, 2014). This design allows me to explore sensemaking and
early implementation of the EGDC at multiple levels and examine points of convergence and
divergence in these constructs within and across individuals, role groups, and schools.
Case Site and Participant Selection. I draw on the experiences of 16 teachers and 10
administrators in five schools across LAUSD, yielding a total of 5 school-level and 26
individual-level cases (See Appendix A, Figure 2).
4
The five case study schools were chosen via
purposeful, criterion-based selection from the sample of approximately 100 schools participating
in the pilot. The sample of schools was identified to obtain maximum variation across several
selection criteria including: grade-level (elementary/middle/high); student demographic makeup;
school performance as measured by Academic Performance Index (API) score (API is a
composite achievement measure that consists of standardized achievement scores and other
indicators); geographic region within LAUSD; and the level of experience and subject areas
taught by teachers participating in the pilot at each school site. These criteria were selected to
ensure data collection across a diverse set of contexts in which stakeholders from multiple
disciplines and points in their careers work to make sense of, and implement, the EGDC. Tables
1-3 in Appendix A show the school sites and individual participants across all selection criteria.
Within each school-level case, all participating teachers and administrators were asked to
participate in the study. At one site, one of the two participating administrators chose not to
participate in the study due to other commitments. However in all but this one case, all

4
As is shown in Figure 1, my initial sample included 18 teachers and 11 administrators. However, due to
insufficient data collected during interviews/focus groups I am unable to construct a clear picture of sensemaking
and implementation for two teachers and one administrator. I exclude these individuals from all analyses. I address
the issues this might pose for reliability of findings later in this section.
30
participating teachers and administrators at each site were part of the initial study sample.
5
It is
important to note that the sample of school sites and participants is not necessarily representative
of the larger EGDC pilot or of teachers and administrators in LAUSD. However, the focus of this
work is not to try and generalize from these individuals or sites to all others, but rather to better
understand the relationship between sensemaking and the implementation of a SBMMTES.
Data Sources. To understand teacher and administrator sensemaking and its relationship
to early implementation of the EGDC I draw primarily on 9 focus groups and 7 interviews with
16 teachers and 10 administrators across each case site (see Appendix A, Table 2 and Table 3,
respectively).
6
These data are supplemented by program documents that outline the EGDC
process, measures, and processes for implementation as well as other documents relevant to
supporting the current Stull evaluation system within LAUSD (see Appendix A, Figure 3). A
total of 10 interviews with central office administrators responsible for the design and
implementation of the EGDC, as well as with the Superintendent, and representatives from the
teacher and administrator unions, were also used to inform the theory of action underlying the
EGDC as well as district leaders’ expectations for teacher and administrator understanding and
implementation during the pilot.
All focus groups and interviews were conducted as part of a larger evaluation of the
EGDC pilot. Each of the focus groups and interviews lasted between 45 and 90 minutes and
were guided by semi-structured protocols designed to gather information about participant

5
Across all school sites participating in the IIP, several teachers dropped out of the process over the course of the
year. My sample of teacher cases is composed of all teachers within each case site who remained participating in the
EGDC for the full school year. Those who dropped out of the process were not contacted to participate in the study
as they could not be identified.
6
Due to scheduling challenges at the secondary level, some teachers (n=3) were unable to make the scheduled focus
group. For these teachers we conducted interviews using the teacher focus group protocol. For the remaining
teachers (n=15), all data were collected using teacher focus groups. Similarly, at the secondary level several
administrators at each site participated in the EGDC (n=8). When this occurred a single focus group with all
participating administrators was conducted using the administrator interview protocol. For the remaining
administrators (n=3), all data were collected using interviews.
31
understanding of the EGDC and their implementation experiences. All of the focus groups and
interviews, with the exception of one site administrator interview, were recorded and
subsequently transcribed.
7
As outlined in Table 4 in Appendix A, teacher and administrator focus
groups and interviews were conducted in March and April of 2012, towards the end of the
EGDC pilot, with additional follow-up with four teachers at two sites completed in March and
April of 2013. District leader interviews were conducted primarily at the start of the 2011-2012
school year; follow-up interviews with two of these leaders occurred in March and April of
2013.
8
Interviews with the Superintendent, President of the administrators' union (Associated
Administrators of Los Angeles), and a representative from the teachers' union (United Teachers
Los Angeles) were also conducted in March and April of 2013.

Analysis. To transform the rich evidence collected through focus groups, interviews, and
documents into a comprehensive picture of teacher and administrator sensemaking around the
EGDC, I took a primarily deductive approach to data analysis.
Content Analysis of Program and Related Documents. I employed content analysis to
extract relevant information from program documents (Patton, 2002). Because each document I
collected differed with respect to its purpose, I applied a general coding schema that catalogued
the source, intended audience, content, and implementation actions for teachers and
administrators (if applicable). After coding each document, I looked across codes to identify
points of interest and importance related to the purpose of the document for shedding light on the
goals of the EGDC as a system of teacher evaluation as well as growth and development.

7
One site administrator asked not to be recorded. To ensure all relevant information was captured during this
interview, the author conducted the interview while another researcher took notes.
8
The interview protocols used in each central office administrator interview were modified slightly depending on
the role of the individual in the design and implementation of the EGDC.
32
Coding Interview and Focus Group Transcripts. To capture the multiple understandings
and implementation experiences of teachers and administrators with respect to the EGDC, I
relied on a multi-step process for analyzing focus group and interview transcripts. I began with
extensive memoing and note taking, commenting and reflecting on the content of the data and
any initial patterns that emerged across sources (Stake, 1995; Creswell, 2007). This first step
helped me to generate an understanding of the individual teacher and administrator cases
specifically, and to determine the relevance of my initial codes for classifying the data. I
continued this memoing and note-taking process through all stages of data analysis, keeping a
running record of points of interest, possible themes, and areas for further examination.
After description and initial memoing, I classified my data according to a categorization
procedure derived from the theoretical framework. As outlined in Table 5 in Appendix A, the
main categories included: Individual Sensemaking, Situated Cognition, and Role of
Representations. These categories allowed for the sorting of data into the three components of
the theoretical framework. Within each of these categories sub-codes were developed to further
flesh out the dimensions within each component (e.g., prior knowledge, interactions). As the data
were reviewed and categorized into the initial codes, other relevant categories emerged and the
categories listed above were refined. For example, I added an explicit category for
implementation with the sub-codes, understanding, engagement, and changes in practice or
process. These codes were used to document reported implementation experiences and were
linked to codes capturing stakeholder sensemaking and understanding to examine relationships
between these constructs.
Given the multiple-case (embedded) design employed, the case analysis took place in
several stages. To construct the individual-level cases I used categorical aggregation to amass
33
related events and experiences with sensemaking about the EGDC and implementation during
the pilot in order to say something about them as a class for each individual (Stake, 1995). Next,
I aggregated the individual cases up to the role group and school-level in order to provide a clear
picture of sensemaking and implementation for each stakeholder group both within and across
sites. I chose to use this aggregation procedure in order to maximize my understanding of
similarities and differences in the sensemaking and implementation experiences for individuals,
role groups, and school sites and to highlight similarities and differences in these processes
across these multiple levels and contexts. Following both individual, role-group, and school-level
case analysis I employed a cross-case analysis looking across individual-level cases, as well as
across role groups (i.e. teacher or administrator) within and across school-level cases. Findings
presented below are derived primarily from the within and across role group analyses.
Throughout the analysis I applied several strategies to ensure the trustworthiness of my
findings including: systematic coding of data (Miles & Huberman, 1994), thick description
(Geertz, 1973), and (when possible) triangulation of data sources and methods (Patton, 2002;
Stake, 1995). Systematic coding of data and triangulation of findings across multiple sources
ensures my findings are credible, confirmable, and dependable (Lincoln & Guba, 1985). Thick
description and cross-case analysis allowed me to examine how similar themes emerged across
individuals, role groups, and sites. These strategies provided insight into reasons for convergence
or divergence in themes that helped to explain outliers to the trends observed in my data and
possible explanations for why certain findings may or may not be transferable across cases.
Limitations. Despite the systematic approach to ensuring the trustworthiness of my
findings, there are some important limitations to consider. First, I rely on self-reported data to
measure teacher and administrator sensemaking and implementation of the EGDC. Ideally,
34
sensemaking and implementation would be measured through a combination of self-reports and
direct observation. However, capacity constraints and access challenges necessarily limited my
data collection efforts to teacher and administrator focus groups and interviews as well as
document review only. Second, most of my conversations with teachers and administrators were
restricted to a single one-hour focus group or interview, necessitating that I examine their
sensemaking and implementation based on limited data. Third, in three cases (Teacher 2 at
School C, Teacher 2 at School D, Administrator 3 at School E), insufficient data were collected
on teacher and administrator experiences, limiting my ability to construct pictures of individual
sensemaking for these participants. The insufficient data prevented these individuals from
serving as information-rich cases so I exclude them from all analyses. To the extent that these
teachers and administrators are systematically different from other participants, the exclusion of
these individuals may limit the reliability of my findings.
Another important limitation to my data is the reliance on focus groups to capture teacher
and administrator sensemaking in a variety of cases. For example, at School D and School E all
administrator data collection was conducted in large focus groups, and at all sites the majority of
teacher data collection was conducted through small focus groups of two to four teachers.
Because focus groups afford more limited privacy than interviews and allow individuals to hear
other participants’ responses to questions of interest, there is a concern that the data collected in
focus groups may mask important distinctions in individual experience. I do find some
convergence in reported understanding and implementation of the EGDC among participants
whose experiences were captured using focus groups rather than interviews. While convergence
did not occur in all instances in which focus groups were used, it is important to consider the
generalizability of my findings to all participants in each focus group in light of this limitation.
35
Finally, the teachers and administrators who participated in the pilot were a sample of
volunteers that preliminary evidence suggests represent a select group of teachers within LAUSD
(Strunk, Weinstein, & Makkonnen, 2013). Of the IIP participants, those who participated in this
study represent an even more select group as they not only volunteered to participate in the pilot,
but continued to remain engaged in the pilot for the full school year and agreed to participate in
my study. Moreover, the majority of participating teachers, like most teachers in LAUSD during
the 2011-2012 school year, were relatively experienced, having 11 years of experience on
average.
9
Although likely not representative of all teachers and administrators in the pilot or
LAUSD, these teachers and administrators are well situated to serve as information-rich cases
given their prolonged engagement and commitment to participating in the SBMMTES pilot.

The EGDC Theory of Action and District Leaders’ Expectations for Implementation
Like many SBMMTES, the EGDC is grounded in a complex theory of action which
outlines how the district expects to move from their stated goals to measurable outcomes.
According to this theory of action, by relying on multiple measures of performance and
establishing a common language of instruction through the Teaching and Learning Framework,
the EGDC is intended to better identify a teacher’s level of effectiveness and to provide teachers
and administrators with more detailed and diverse information about areas of strength and
weakness in teacher practice. This information is expected to encourage improvements in teacher
quality by informing targeted professional development opportunities and eventually human
capital decisions (e.g., retention, compensation, promotion). To meet these goals, LAUSD

9
Between 2008-2009 and 2011-2012, LAUSD faced a series of budget shortfalls, forcing them to let go of a
significant proportion of the teacher workforce. Under state law, teacher layoffs are based on seniority which led to
the majority of novice teachers in LAUSD being laid off during this period. This left LAUSD with a more
experienced teacher workforce. During the 2011-2012 school year, the average experience of a teacher in LAUSD
was approximately nine years, suggesting that study participants were still slightly more senior than average.
36
expects teachers and administrators to complete several activities including: a self-assessment;
two formal observation cycles which include a lesson-design, classroom observation, and pre-
and post-observation conference; and an Individual Growth Plan, which includes reviewing
value-added measures of effectiveness when available. These activities are meant to encourage
teacher self-reflection and more instruction-focused interactions between teachers and
administrators. District leaders expect that together, these activities and associated actions will
generate improvements in teacher quality and ultimately student achievement.
10

Although LAUSD is currently in their first year of district-wide implementation of the
EGDC, this study focuses on teacher and administrator sensemaking and implementation with
respect to the EGDC as originally designed and implemented within the pilot during the 2011-
2012 school year.
11
Because educators implemented a reduced-form of the EGDC during the
pilot, the theory of action as well as district leaders’ expectations for teacher and administrator
engagement were slightly different than described above.
Expectations for Stakeholder Understanding of the Purpose of the EGDC. With
respect to understanding, district leaders described the purpose of the new evaluation process as
primarily about teacher growth and development, and to a lesser extent about improving
accountability for teacher performance. As outlined in Table 6 in Appendix A, five of six district

10
This theory of action does not necessarily reflect the district’s stated theory of action for the EGDC, but rather my
own interpretation, derived from interviews with central office administrators responsible for the design and
implementation of the EGDC during the IIP as well as from program documents used to train teachers and
administrators on the EGDC performance measures and implementation process.
11
The EGDC is currently being implemented with approximately 6,300 teachers in LAUSD. This revised process
includes two formal observation cycles with a self-assessment, initial planning conference, individual growth
planning inclusive of data-based objectives (similar to Student Learning Objectives), informal observations, and a
post-conference. All evidence from these observation cycles will be used to provide a final evaluation for teachers.
Value-added scores are not included in the current iteration of the EGDC. Student surveys will be utilized but will
not be included for stakes. LAUSD is also currently piloting the Contributions to School Community measure for a
subset of teachers. However, this measure will not be included in teachers’ formal evaluations for SY 2013-2014.
37
leaders were asked about the purpose of the EGDC and all five leaders noted that the EGDC was
intended to improve teachers’ instruction. For example, one district leader asserted:
[The theory of action underlying the EGDC is]…to really understand where each
individual [teacher] lands in terms of level of effectiveness and then, at that point,
it's a matter of meeting the teacher where they are and beginning that process to
kind of increase their effectiveness…So, I mean that's really – our theory of
action is that you identify areas where you need support. We will give you the
support…

By using the EGDC as a mechanism to improve teacher growth and development, district leaders
expected that the EGDC would not only lead to improvements in teacher practice, but ultimately
shift the culture in many schools towards one of continuous improvement:
So the end result of all of that [the EGDC] should be people who are way more
focused on kids as individuals in their classroom, parents as partners in the
process, teachers as co-collaborators or as members of the professional learning
community – I mean it's all intended to build the kind of school culture that is
pretty much associated with ongoing reflection and improvement.

Although all district leaders focused their discussion on the purpose of the EGDC as
primarily about teacher growth, two leaders did note that another goal of this process was to
eventually improve accountability for teacher performance:
When we do [implement the EGDC for stakes], there's two purposes to this whole
process. One is to identify quality practice and help people get there; and to
separate those who cannot get there from the system.

In both cases, district leaders asserted that this accountability goal, while important, was really
secondary to the growth and development goal. This was particularly the case during the pilot
since the EGDC was not fully implemented and teachers’ ratings were not yet attached to stakes.
Expectations for Stakeholder Implementation of the EGDC. In addition to expecting
teachers and administrators to understand that the EGDC is primarily about teacher growth and
development, district leaders had very specific expectations for how the new process would be
implemented during the pilot. All four district leaders asked about implementation during the
38
pilot reported two primary expectations for stakeholder implementation. First, district leaders
expected stakeholders to implement the required EGDC activities. Second, district leaders
expected that in completing the EGDC activities stakeholders would engage in increased
reflection about teacher practice and more instruction-focused interactions. The first expectation
for implementation is “process-focused” in that it centers on stakeholders completing just the
specific EGDC activities outlined in the new evaluation process. In contrast, the second
expectation for implementation is “instruction-focused” in that it centers on teachers and
administrators completing both the EGDC process and engaging in behaviors that district leaders
expect will encourage improvements in teachers’ instructional practice.
All four district leaders asked about implementation of the EGDC described process-
focused implementation as the first expectation for stakeholder engagement. In particular, these
district leaders reported that they expected teachers and administrators to implement at least one
full observation cycle during the pilot including: a teacher self-assessment, lesson design, pre-
and post-observation conference, the classroom observation, and an Individual Growth Plan.
Because district leaders intended for stakeholders to complete only this subset of EGDC
activities during the pilot, and targeted professional development was not yet online, district
leaders did not expect the process to encourage the dramatic improvements in teacher practice
and student achievement anticipated when the EGDC is fully implemented. However, during the
pilot, LAUSD did expect both teachers and administrators to engage in more instruction-focused
interactions and for teachers to reflect more on their practice according to the standards outlined
in the Teaching and Learning Framework.
In fact, all four district leaders described that they expected that completing the EGDC
activities would lead to instruction-focused implementation, with teachers and administrators
39
reflecting more on teacher practice and engaging in more interactions centered on instruction.
One district leader outlined the expectations for process-focused implementation and how it was
expected to lead to instruction-focused implementation:
So it starts with the self-assessment…and then, you go to the observation
process…and then the next piece is that conversation with your principal and the
teacher –that we have heard from teachers in the pilot hasn’t happened before –
and they are gaining so much from just having that collaborative relationship with
their principal to look honestly at how I rate myself, how you rate me, and then
let's have that conversation on where it's different.

The expectation that engaging in process-focused implementation –that is, simply executing the
EGDC activities –would then lead to instruction-focused implementation was further supported
by another district leader who stated:
[What we hope] is that you [the administrator] are able to say, ‘Okay you really
seem like you need help, and you identified in your self-assessment, that you need
help on this one portion of [whatever instructional dimension]…And wouldn’t
you know it, your colleague who teaches down the hall from you, is great at that.
She really is good,’ [so] what I’m going to do as your administrator, you all will
have given me time to go and watch her and then we’ll debrief it together…

Together these expectations demonstrate that policymakers had two primary intentions for
implementation during the pilot. The first was more superficial in terms of just having teachers
and administrators execute the EGDC activities and become familiar with this new process. The
second, and ultimate goal, was more substantial in that teachers and administrators were
expected to both execute the EGDC activities and, in so doing, engage in self-reflection and
more instruction-focused interactions, behaviors thought to be a pre-condition for improvement.
District leaders recognized that completing even this subset of EGDC activities during
the pilot required participating teachers and administrators to do more than implement new
procedures; the EGDC pilot forced each teacher and administrator to construct a new
understanding of teacher evaluation and support. As one district leader asserted:
40
A confirming piece that we learned, or that we confirmed, was in so many ways
how big a shift this is. Not just, yes, we’re doing robust framework-based
evaluation in the classroom observation piece, big shift there, obviously…but
[also a shift] in asking people to break old habits of how you do the evaluation, be
it writing, scripting, or whatever, to now you do it with a computer. So that’s also
a big shift at the same time as we are introducing a new paradigm…[with the
TLF].

This point was echoed by another district leader when discussing the new observation cycle:
…by using the framework to look at an observation experience, you’re
deconstructing the art of teaching and then trying to put it back together right?
Which is totally new for most people, I think, who have been trained over the last
10, 20, 30 years. So you know something kind of a new paradigm, on how you
think about dissecting a lesson. Then we’re saying, now do it all on a laptop, and
tag, and do all these new things, on a system that’s in its very first year’…so new
paradigm, new method of doing this, and you create, I think, disturbance…

The previous evaluation system, known in LAUSD as the Stull, was widely seen as a perfunctory,
compliance-based exercise focused on teacher accountability and divorced from growth and
development.
12
District leaders hoped that after participating in the EGDC teachers and administrators
would understand the new evaluation system to be a deliberate move away from the relatively
superficial Stull process, to a more meaningful accountability system focused on teacher growth and
development. However, they also recognized that this change was a learning process, as teachers and
administrators were being forced to make a “paradigm shift” with respect to their understanding of
teacher evaluation and its role in shaping teacher growth and development.

12
Under Stull, teachers experienced one classroom observation, during which an administrator rated the quality of
their instruction as “meets” or “below standards” in five areas of professional practice. Although teachers and
administrators were expected to meet to review the evaluation form, only when teachers received a below standards
rating were administrators required to provide specific, targeted feedback to guide improvement. In 2009, The New
Teacher Project (TNTP) conducted a study on teacher and administrator perceptions about the Stull process. They
found that 99 percent of teachers received a meets standards rating, yet only 64 percent of those surveyed reported
that evaluation provides them with information and strategies they could use to improve their practice. Moreover,
they found that only 64 percent of principals reported that the Stull evaluation system allows them to adequately
address issues of poor performance among their faculty (TNTP, 2009).
41
Findings
Teachers and administrators constructed multiple understandings of the EGDC that
aligned at least in part with district leaders’ expectations during the pilot. Variations in the
understandings constructed did appear to be associated with how stakeholders made sense of the
new evaluation process. Administrator understanding appeared to be influenced by previous
experiences conducting evaluation and existing beliefs about teachers as learners, while teacher
sensemaking was shaped by both the nature of previous evaluation experiences (e.g., positive,
negative, or neutral) and the substance and frequency of interactions with administrators around
the EGDC process. Although both stakeholder groups cited implementation challenges
associated with poor communication from LAUSD, how the EGDC was represented to
participants did not appear to play a major role in shaping participant understanding of the
purpose and goals of the new evaluation process. The understandings constructed by participants
of the EGDC did appear to influence implementation; however, teacher-administrator
interactions and the commitment of both stakeholders to completing the new evaluation process
largely determined how the EGDC was implemented during the pilot.
Teacher and Administrator Understanding of the EGDC. Teachers and administrators
reported understanding that the purpose of the EGDC was to: 1) encourage teacher growth and
development, 2) improve teacher accountability, or 3) accomplish both goals. Table 7 in
Appendix A provides the distribution of teacher and administrator reports across each type of
reported understanding of the purpose of the EGDC.
The top panel of Table 7 in Appendix A demonstrates that administrators understood the
purpose of the EGDC to be about improving teacher growth and development or both improving
growth and development as well as accountability. For example, one administrator noted, “I
42
really think that it [the EGDC] is a coaching process that allows teachers to improve on their
craft –and to get feedback so they can make changes.” Another described the EGDC as intended
to “improve accountability” but also to encourage “more of a detailed development…of teaching
skills as well as [administrator] observation abilities.” Interestingly, no administrators reported
understanding the purpose of the EGDC as solely about improving accountability, demonstrating
that the understandings constructed by this stakeholder group largely aligned with those intended
by district leaders during the pilot and outlined in Table 6 in Appendix A.
Teachers showed more variation in the understandings they constructed of the purpose of
the EGDC. The bottom panel of Table 7 in Appendix A shows that six teachers understood the
purpose of the EGDC as solely about teacher growth and development. One of these teachers
commented that the EGDC is intended, “…to make us [teachers] more reflective about our
practices,” while another commented that, “…it [the EGDC] is like self-reflection on your
practice… am I delivering instruction the right way, or how can I improve it?” Another four
teachers constructed a slightly different understanding of the EGDC, seeing the purpose as
encouraging teacher growth and development and improving accountability. For example, one
teacher asserted that the purpose of the EGDC is to “…reflect on what we’re doing, and to see
whether our teaching is really meaningful and relevant…But also to just—to separate the good
from the bad.” Although varying in the extent to which they saw accountability as a goal of the
EGDC, the majority of teachers constructed an understanding of the new systems purpose
aligned with district leaders.
However, six teacher participants constructed an understanding of the purpose of the
EGDC as solely about improving accountability. For example, one teacher noted:
Well I think, if you want to just get to the end thing it’s how do you get rid of
teachers that you perceive to be less competent…And I think the real intention is
43
not to reward excellence, so much as to build a progressive discipline procedure
where we have the paperwork…

This understanding of the EGDC as focused solely on increasing accountability for teacher
performance was echoed by five other teachers. Three of these participants agreed that the
purpose of the EGDC was to “separate the good from the bad” and “to prove that the good
teachers are wonderful and that the bad teachers are horrible.” This understanding differs from
that intended by those responsible for the design and implementation of the EGDC, who
emphasized the accountability goal as secondary to improving teacher growth and development.
Factors Shaping Stakeholder Sensemaking. The variations in teacher and administrator
understanding of the purpose of the EGDC appear to be associated with differences in how each
made sense of the policy ideas embedded within the new evaluation process. Below, I describe
the factors that appeared to shape sensemaking for each stakeholder group and demonstrate how
they contributed to variation in teacher and administrator understanding of the purpose and goals
of the EGDC during the pilot.
Administrators. Administrator sensemaking was primarily individual, constituted in the
interaction between the new procedures for executing the EGDC activities, and existing
cognitive schema with respect to teacher evaluation and support for instructional improvement.
In particular, administrators’ previous experiences with evaluation and existing beliefs about
teachers as learners influenced their understanding of the purpose and goals of the EGDC. In the
remainder of this section, I first describe these factors in greater detail and then conclude with an
examination of how they combined to shape administrator understanding.
Previous Evaluation Experiences. Administrator participants reported a variety of
different experiences conducting evaluations prior to the EGDC pilot, which appeared to shape
their sensemaking with respect to the EGDC. For some administrators these experiences were
44
limited to the current system of evaluation in LAUSD known as Stull, and for others these
experiences extended beyond Stull, to include evaluations of teacher practice using more
rigorous observation protocols and activities similar to those required by the EGDC.
13
For
example, when describing her previous evaluation experience, one administrator asserted:
We did the traditional Stull process [previously]. There was little pre-observation
discussion like what’s required with the EGDC, but I did the classroom
observations and wrote down my observations…many of our teachers reflected on
their practice under Stull, but it wasn’t as drill-down (detailed) as this EGDC…

Similarly, another administrator noted that:

...prior to last year I was looking at the rubrics and the materials that are used for
BTSA...since [last year] I have been using the work of Charlotte Danielson…so I
was already using a very similar process [to the EGDC] to help me begin to
focus more on an objective way of looking at teacher practice...

Each administrator, regardless of the nature of their previous experience, seemed to rely on what
they already knew about conducting evaluations when making meaning of the EGDC.
To facilitate a comparison of administrators’ previous evaluation experiences, I classified
administrators into high, medium, or low-levels of previous experience with similar evaluation
activities. Administrators who reported having previous experience completing all of the EGDC
processes –including using standards of effective teaching to evaluate teacher performance,
collecting evidence according to these standards during classroom observations of teacher
practice, conferencing with teachers to discuss the classroom observations, and using the
standards and evidence collected during observations to provide feedback to teachers about
specific areas of strength and opportunities for growth in their practice –were classified as having

13
It is important to note that within the focus group/interview administrators were asked to discuss how teachers
were provided with feedback prior to the EGDC. While this prompt did not explicitly ask administrators about the
previous teacher evaluation system it does suggest that a comparison be made between the EGDC and previous
feedback processes for teachers which may contribute to the prevalence of this factor in appearing to shape
administrator sensemaking. However, administrators drew on their previous experiences in evaluating and providing
feedback to teachers throughout the focus group both before and after this prompt was made indicating that previous
experiences likely played a role in shaping sensemaking for this stakeholder group.
45
“high” levels of previous experience. Administrators reporting experience with the majority of
these EGDC processes were classified as having “medium” levels of experience; and those who
reported experience with only one or two of these EGDC processes were classified as having
“low” levels of previous experience with similar evaluation activities.
Table 8 in Appendix A outlines the previous experience classifications for all case
participants and provides a sample of evidence to support the classification. The administrators at
School A and School C both reported high levels of previous experience with EGDC activities,
conducting standards-based evaluations using either the California Standards for the Teaching
Profession or other frameworks, including Charlotte Danielson’s Framework (the foundation for
the TLF), and spending time conferencing with teachers and providing feedback targeted to
specific areas of need identified during classroom observations. The administrators at School B
and School D described medium-levels of experience with similar evaluation activities,
conducting classroom observations and conferences to provide feedback on instruction, but
citing no experience using standards to guide these activities. In contrast, the administrators at
School E, reported low-levels of previous experience, conducting classroom observations with
teachers “when alarms go up,” but citing no explicit experience using standards to guide these
observations or providing targeted feedback to teachers based on these evaluations.
14

Beliefs about Teacher Learning. Existing beliefs about teachers as learners also appeared
to shape administrator sensemaking with respect to the EGDC. Although not prompted through
the interview and focus group questions, all ten administrator participants drew on their beliefs
about teachers as learners when describing their understanding of the purpose of the EGDC. For

14
As mentioned in the limitations section above, it is possible that the nature of focus groups led administrators at
School D and School E to report similar previous experiences with evaluation. My data suggest that this may have
been the case with respect to this specific question as the administrators tended to agree with a couple of respondents
in each focus group rather than each providing an individual response to how previous evaluations were conducted.
46
example, one administrator asserted, “… it’s a really different outlook [than under Stull] of
everybody has needs. Everybody has areas where they need to grow.” Similarly, when describing
his understanding of the purpose of the EGDC one administrator commented that although one
of the goals of the EGDC was to improve teacher practice, the majority of his faculty could not
reach that goal because, “...I’m talking 10, 15, maybe 20 percent of the faculty can pull this [the
EGDC] off...” While all administrators similarly drew on their beliefs about teachers as learners
to make sense of the EGDC, the nature of these beliefs differed across participants.
As above, to better understand the nature of administrator beliefs about teachers as
learners I classified each administrator into one of two categories. Administrators who reported
that all teachers have the capacity to reflect and grow comprised the first category, whereas
administrators who reported that not all teachers have the capacity (e.g., the skills and
knowledge) or willingness (e.g., desire or motivation) to improve their practice comprised the
second category.
15
Table 9 in Appendix A provides evidence as to the classifications for each
administrator based on beliefs about teachers as learners and provides evidence to support the
classification.
Administrators were evenly split with respect to their beliefs about teachers as learners.
Five administrators reported that all teachers have the capacity to improve, and five reported that
some teachers lack the capacity or willingness to improve. For example, the administrator at
School C reported that all teachers can continuously improve and that her responsibility as an
instructional leader is to push both low and high-performers towards more effective practice. The
administrators at Schools A and D also reported believing that all teachers can learn, noting that
helping all teachers to achieve growth can be challenging but is necessary and more easily

15
It is important to note that judging the extent to which administrators beliefs about teachers as learners are in fact
warranted is not the purpose of these classifications. Rather, my intent is to demonstrate that administrator beliefs
about teachers as learners fell into distinct categories that appeared to shape their sensemaking about the EGDC.
47
facilitated using the EGDC process than the previous evaluation system. In contrast, the
administrators at Schools B and E asserted a belief that only some teachers are actually willing to
reflect on their practice, and have the capacity to grow –those unwilling, they argued, will not
improve, regardless of the new evaluation process or administrator assistance.
16

The Relationship between Administrator Sensemaking and Understanding. Table 10 in
Appendix A demonstrates a clear relationship between how administrators’ previous evaluation
experiences and beliefs about teachers as learners combined to shape the understandings they
constructed of the purpose of the EGDC. Administrators with high- or medium-levels of
previous experience with similar evaluation activities, and who reported a belief that all teachers
have the capacity and willingness to improve, more often reported understanding the purpose of
the EGDC as primarily about improving teacher growth and development. In contrast,
administrators with lower levels of previous experience with similar evaluation activities and
who reported that only some teachers have the capacity or willingness to improve, were more
likely to report an understanding of the EGDC as at least in part about improving teacher
accountability. This second group of administrators typically reported understanding the EGDC
process as intended to encourage growth for those with the capacity or willingness to improve,
and about improving accountability for those without the capacity or willingness to work
towards improvement.
The relationship between administrator sensemaking and EGDC understanding
demonstrates that existing cognitive schema with respect to teacher evaluation and support are
central to how administrators make meaning of SBMMTES. This is perhaps not surprising, as

16
As mentioned in the limitations section above, it is possible that the nature of focus groups led administrators at
School D and School E to report similar beliefs about teachers as learners. My data suggest that this may have been
the case as the administrators at each of these sites tended to agree with a couple of respondents in each focus group
rather than each providing an individual response communicating their own beliefs about teachers as learners.
48
sensemaking theory suggests that things familiar to what one already knows serve as triggers for
what is noticed from the environment and the meaning made (Gentner, Ratterman, & Forbus,
1993; Gentner & Toupin, 1986; Ross, 1987). From this perspective, administrators who have
previous experience with similar evaluation activities and believe that all teachers can learn, are
best situated to construct an understanding of the EGDC that aligns with district leaders’ intent,
as limited restructuring of their existing schema for teacher evaluation and support is required.
For others, constructing an understanding of the EGDC as about growth and development first,
and accountability second, will likely require more significant shifts in their beliefs about
teachers as learners as well as in their understanding of the purpose of teacher evaluation.
Teachers. Much like administrators, teacher sensemaking was shaped by the interaction
of existing cognitive schema with respect to teacher evaluation and the new procedures outlined
by the EGDC. However, teacher sensemaking about the EGDC was also constituted largely in
their interactions with administrators, suggesting that situated cognition also played a role in
shaping how this stakeholder group constructed an understanding of the purpose of the EGDC.
In the remainder of this section, I describe these factors in greater detail, concluding with an
examination of how they combined to shape teachers’ understandings of the EGDC.
Previous Evaluation Experiences. Just as with administrators, teachers referenced their
previous evaluation experiences when making sense of the EGDC.
17
For example, at one site all
teachers agreed that the EGDC was “…more thorough, more reflective” than the previous
system. However, rather than describing their previous experiences in terms of the types of

17
It is important to note that within the focus group/interview teachers were asked to discuss their experiences with
each aspect of the EGDC (e.g., classroom observations). In some cases, teachers were asked how this experience
compared to their previous experiences. While this prompt did not explicitly ask teachers about the previous teacher
evaluation system it does suggest that a comparison be made between the EGDC and previous processes which may
contribute to the prevalence of this factor in appearing to shape teacher sensemaking. However, teachers drew on
their previous evaluation experiences throughout the focus group both before and after this prompt was made
indicating that previous experiences likely played a role in shaping sensemaking for this stakeholder group.
49
activities required to complete the process, participating teachers often described their previous
experiences more generally, focusing on the extent to which it was a positive experience overall.
For example, one teacher noted, “I wouldn’t want to go back to the Stull. It’s too dry, too short.
It had no significance,” while another teacher asserted, the previous system was “a joke” in terms
of providing any meaningful insight into improving practice.
To examine how teachers’ previous evaluation experiences were associated with
understanding, I classified them into positive, negative, or neutral categories. Teachers’ who
described their previous evaluation experiences as “good” or “helpful” to some extent, were
classified as having “positive” previous evaluation experiences. Teachers whose reported
experiences with previous evaluation systems were not necessarily good or bad, were classified
as “neutral”, and teachers who referenced frustration or anger with the previous evaluation
procedures were classified as having “negative” previous evaluation experiences. Table 11 in
Appendix A outlines the classifications for all participating teachers.
Only four teachers, all at School B, described their previous experiences with teacher
evaluation as positive, noting that in the past their administrators had conducted Stull evaluations
with fidelity including observations and conferences with targeted feedback on instruction. These
teachers noted that their current principal was “unique” in her dedication to making evaluations
under Stull a meaningful process. In contrast, the three teachers at School E reported strictly
negative experiences with their previous evaluations, asserting that they rarely took place and
often did not include any feedback. The remaining nine teachers at School C and School D
described their experiences in relatively neutral terms, noting that they were short but
collaborative, although not necessarily helpful for improving instruction.
18

18
As mentioned above, it is possible that the nature of focus groups led teachers at each site to report similar
previous evaluation experiences. My data suggest that this may have been the case at some sites with teachers
50
Teacher-Administrator Interactions. In addition to previous evaluation experiences,
teachers drew on interactions with their administrators to make sense of the EGDC. For example,
Teacher 1 at School D described several interactions with their administrator over the course of
the pilot. This teacher met with their administrator to discuss his instruction during a post-
conference as well as several other times to discuss the EGDC process as well as how the
feedback conversations were structured. In contrast, Teacher 4 at School D commented that they
had only one “quick” meeting with their administrator that did not include any specific feedback
on instruction. Across the board, teachers described either frequent or infrequent interactions
with their administrator that fell into one of three categories: 1) “procedural interactions” or
those that involved figuring out how, or when, to execute the EGDC activities, 2) “substantive
interactions” or interactions that centered on identifying areas of strength and opportunities for
growth in teacher practice, or 3) “both,” interactions that were procedural and substantive.
Table 12 in Appendix A outlines the distribution of teacher reports about the frequency
and nature of their interactions with administrators during the pilot and provides a sample of
evidence to support each classification. The first thing to highlight is the variation both within,
and across schools, in terms of the nature and frequency of teachers reported interactions with
administrators. In particular, the reported experiences of teachers at School C, School D, and
School E, demonstrate that the nature of interactions varied across teacher-administrator pairs,
even within the same school context. Overall, six teachers described infrequent, substantive
interactions with their administrator. For example, Teacher 1 at School A noted:

agreeing with a couple of respondents in each focus group rather than each providing an individual response
communicating their own previous evaluation experiences. However, in other cases, teachers did report differing
experiences within a single focus group. Also important to note is that teachers were not asked to restrict their
comments to previous experiences at their current site and in many cases teachers reports focused on evaluation
experiences from across their careers (at multiple sites both inside and outside LAUSD).
51
…I think the opportunity to reflect on purposeful groups changed my practice and
that came about as a result of the post-observation conference. So the opportunity
to discover things in one’s practice versus being told is very important…

In contrast, eight teachers reported having only procedural interactions with their
administrators, with the majority of those occurring infrequently. For example, Teacher 4
at School D noted that interactions with her administrator were focused primarily on
discussing the observation process and trying to figure out the appropriate steps:
Well, they [the district] didn't know what they were doing…at this time it was the
beginning…both of us going ‘well, we don't know what's going on, but this is
what I'm presenting and this is what you should see,’ and they [the observers]
walked in and…they both said, well, it looked good…

Finally, two teachers reported having frequent interactions with their administrators that were
both procedural and substantive. For example, Teacher 4 in School C and Teacher 1 in School D
both described having frequent opportunities to discuss the EGDC with their administrator. In
both cases, participating teachers described the process by which they “negotiated” meaning of
the reform together with their administrators, challenging one another’s ideas about the purpose
and goals of the EGDC and how the reform should be implemented. Moreover, each reported a
number of substantive interactions with focused discussions on strategies to improve instruction.
The Relationship between Teacher Sensemaking and Understanding. Just as with
administrators, the combination of factors shaping teacher sensemaking appeared to contribute to
variations in understanding of the purpose of the EGDC. Table 13 in Appendix A provides the
classifications for all participating teachers on both previous evaluation experiences and teacher-
administrator interactions, by the understandings they constructed of the EGDC. Teachers who
reported neutral or positive experiences with their previous evaluations and had any substantive
interactions with their administrator –regardless of how frequently they occurred –most often
understood the EGDC as intended to improve teacher growth and development. In contrast,
52
teachers who reported experiencing only procedural interactions, regardless of frequency or
whether they had positive, neutral, or negative, experiences with evaluation in the past,
constructed an understanding of the EGDC as solely designed to improve teacher accountability,
seeing no connection between this new evaluation process and teacher growth and development.
The strong relationship between the understandings constructed by teachers and the
nature of their interactions with administrators suggests that situated sensemaking, or the
negotiation of meaning between actors and their environment, appeared to play the primary role
in shaping teacher understanding of the purpose of the EGDC. This aligns with evidence about
the role of administrators in shaping teacher sensemaking during the implementation of other
standards-based reforms. This literature suggests that principals influence teacher sensemaking
by serving as gatekeepers of policy messages, partners in negotiation of reform ideas, and key
players in constructing environments that either help or hinder teacher sensemaking (Coburn,
2005; Halverson, Kelley, & Kimball, 2004). Although I do not find evidence that administrators
shaped teacher sensemaking in all of these ways, they did appear to serve as partners for teachers
in negotiating an understanding of the EGDC during the pilot.
Despite the strong relationship between teacher and administrator interactions and
understanding, I do find two outliers to the general trend. Teacher 3 and Teacher 4 in School D
reported no substantive interactions with their administrator around the EGDC during the pilot,
but both constructed an understanding of the new evaluation system as primarily about teacher
growth and development. As mentioned above, each of these teachers also reported experience
serving as a mentor for beginning teachers through BTSA, which provided them with experience
using standards to help struggling teachers improve their skills. Moreover, each reported that
they reflected on their practice regularly, both in the context of the Stull process and outside of
53
any evaluation system. These previous experiences suggest an alignment between their existing
schema for teacher evaluation and support and the ideas embedded within the EGDC, which may
help to explain how they constructed an understanding of the EGDC as about teacher growth and
development even without experiencing substantive interactions with their administrators.
The Relationship Between Stakeholder Understanding and Implementation. As
outlined above, district leaders had two primary expectations for teacher and administrator
implementation of the EGDC during the pilot. First, they expected these stakeholders to execute
the required EGDC activities, completing the new evaluation process. Second, district leaders
expected that by completing all of the EGDC activities, teachers would reflect more on their
instructional practice and engage in more interactions around instruction with their administrator.
Teachers and administrators could either adopt both goals, which I call “instruction-focused”
implementation, or just adopt the first but not the second, which I call “process-focused”
implementation. Process-focused implementation requires participants to learn new procedures
for evaluating teacher performance and participating in evaluation. Instruction-focused
implementation, requires teachers and administrators to both complete new procedures for
evaluation, and shift their thinking away from an understanding of teacher evaluation as separate
from growth and development, towards one in which these two processes are inextricably linked.
To examine the relationship between understanding and implementation, I classified
teachers into one of these two implementation categories. Teachers who reported completing all
the EGDC activities, and who explicitly reported participating in more self-reflection and
interactions with their administrators around instruction, were classified as engaging in
“instruction-focused” interaction. In contrast, teachers who reported completing only the EGDC
activities without increased reflection, or interactions with their administrator focused on
54
instruction, were classified as engaging in process-focused implementation only. Table 14 in
Appendix A demonstrates that the understandings constructed by teachers and administrators
during the EGDC did in fact shape how they implemented the reform during the pilot and the
extent to which this implementation aligned with district leaders’ expectations.
Seven of the sixteen teacher-administrator pairs engaged in “instruction-focused”
implementation, executing the required EGDC activities and engaging in increased reflection and
instruction-focused interactions that teachers reported often led to positive changes in practice. In
all of these cases, one or both stakeholders described an understanding of the EGDC as at least in
part about teacher growth and development, and, in almost all of these cases, the administrator
held this view. Another seven teacher-administrator pairs engaged in “process-focused”
implementation. These teacher-administrator pairs executed the EGDC activities but reported no
increases in teacher self-reflection or interactions around instruction. Teachers and
administrators who engaged in process-focused implementation, all understood the purpose of
the EGDC to be at least in part about improving accountability, and in all of these cases
administrators understood the purpose of the EGDC as both about improving teacher
development as well as accountability.
Although clear patterns between teacher and administrator understanding and the nature
of implementation during the EGDC pilot exist, there are outliers to this general pattern. In two
cases, teacher-administrator pairs failed to engage in any form of implementation during the pilot
(Administrator 1 and Teacher 2, School E, and Administrator 1 and Teacher 3, School C), and in
another two cases both teachers and administrators understood the purpose of the EGDC to be
about improving teacher growth and development but each engaged in only process-focused
implementation, suggesting that despite understanding the EGDC as intended to generate teacher
55
growth and development they engaged in none of the behaviors expected to facilitate improved
teacher practice (Administrator 1 and Teacher 5, School D, and Administrator 2 and Teacher 4,
School D). These outliers demonstrate that while understanding did appear to influence
implementation of the EGDC during the pilot, other factors also appeared to play a role in
determining how teachers and administrators implemented the new evaluation process.
Factors Mediating the Relationship between Understanding and Implementation. To
better explore how other factors may have mediated the relationship between understanding and
implementation, I reexamined the factors shaping sensemaking for each of the outlier cases.
Each of these cases demonstrates that in addition to understanding, the nature of teacher-
administrator interactions and the commitment of both stakeholders to completing the new
evaluation process also shape how teachers and administrators implemented the EGDC.
For both Administrator 1 and Teacher 2 at School E, and Administrator 1 and Teacher 3
at School C, implementation stalled because of a lack of commitment to the EGDC process.
Teacher 2 at School E reported that her administrator was not committed to implementing the
EGDC process with her during the pilot:
…once I did the step [the observation], my next step is then encumbered upon
others to do things. And of course I cannot do my next steps, if the others are not
stepping along with me. So my steps stopped a long time ago…We were told, you
know, here is a meeting we are going to go too, and we will contact you about
future meetings. Well first of all, I’ve never been contacted about future meetings.
I’ve been contacting them, trying to keep in touch.

Without an administrator committed to completing the EGDC process, Teacher 2, was unable to
move forward with the required activities, despite herself being committed to the EGDC process.
Teacher 3 at School C further underscored this point. This teacher asserted that she felt her
administrator was demanding too much from her with respect to the process, asking to see more
“bells and whistles” on her lesson design before agreeing to do the classroom observation.
56
Teacher 3 reported that she refused to “jump through hoops” for the pilot, and wanted to submit
a traditional lesson design rather than the one required by the EGDC process. The lack of
commitment by Teacher 3 to executing the EGDC activities ultimately stalled implementation.
Together, the experiences of Teacher 2 at School D and Teacher 3 at School C underscore that
regardless of how teachers and administrators understand the EGDC, implementation requires a
mutual commitment from both actors to completing the new evaluation process (McLaughlin,
1987; Pressman & Wildavsky, 1984; Weatherly & Lipsky, 1977).
The reported implementation experiences of Teacher 4 and Teacher 5 at School D shed
light on another important factor mediating the relationship between understanding and
implementation –the nature of teacher-administrator interactions. Despite understanding the
EGDC to be about improving teacher growth and development, Teacher 4 and Teacher 5
reported engaging in only procedural interactions with their administrator, limiting their
conversations to negotiating expectations for executing the EGDC process. In contrast, all other
teachers who understood the EGDC to be about improving teacher growth and development
engaged in instruction-focused implementation. These teachers reported at least some
substantive interactions with their administrators (see Appendix A, Table 13), suggesting
exposure to the conversations district leaders expected to lead to improvements in teacher
practice. The experiences of Teacher 4 and Teacher 5 at School D demonstrate that in some
cases, the nature of teacher-administrator interactions mediated the relationship between
understanding and implementation of the EGDC. This difference is particularly interesting as
neither Teacher 4 nor Teacher 5 in School D relied on teacher-administrator interactions to make
sense of the EGDC. Thus while interactions may not be central to how all teachers make sense of
the EGDC, they do appear to play a primary role in how teachers implement SBMMTES.
57
Discussion and Implications
To date existing research on the implementation of SBMMTES has focused primarily on
identifying barriers and facilitators to implementation, taking an atheoretical approach to
understanding the implementation of these complex reforms. In this exploratory study I take a
cognitive approach to examining implementation and provide initial evidence to suggest that
how teachers and administrators make sense of a SBMMTES contributes to variations in their
understanding as well as implementation of the reform. However, I also find that teacher and
administrator sensemaking is not the only factor shaping implementation; teacher-administrator
interactions and the mutual commitment of both stakeholders to the SBMMTES process also
play a large part in determining how the reform is implemented. This early evidence has several
implications for research, policy, practice, and theory with respect to SBMMTES.
The extent to which teachers and administrators’ existing cognitive schema for teacher
evaluation and support aligned with that underlying the EGDC, mattered for the understanding
teachers and administrators constructed of the new evaluation system. This suggests that
individual cognitive schema play a central role in shaping teacher and administrator perceptions
of the purpose of a SBMMTES. Given this finding, ensuring SBMMTES act as a mechanism to
improve teacher practice will likely require structuring learning opportunities that help teachers
and administrators make sense of these new reforms as intended by policymakers. In particular,
these learning opportunities should: 1) make explicit for stakeholders how SBMMTES are both
similar and different from previous evaluation systems, 2) educate stakeholders on how to
execute new evaluation procedures as well as the underlying purpose of these new systems as
compared to the old, and 3) equip both teachers and administrators with the tools necessary to
have substantive, instruction-focused conversations. For example, Administrator 1 and
58
Administrator 3 at School D as well as Administrator 1 and Administrator 2 at School E asserted
that many teachers feel threatened when discussing improvement with their administrator since
they also conduct teacher evaluations. Providing specific training to administrators on how to
break down these barriers with their faculty and build a strong culture of learning in their school,
may help to alleviate this concern and ensure teachers and administrators have productive
conversations about improving teacher practice in the context of a SBMMTES.
The findings presented here also demonstrate that administrators play a central role in the
implementation of SBMMTES, both in terms of how they shape teacher sensemaking about
these reforms and also how they interact with teachers to ensure successful implementation. This
suggests that policymakers should consider providing teachers and administrators with
opportunities to build understanding of these new systems together, rather than relying on wholly
separate trainings. During the EGDC pilot, LAUSD held separate trainings for teachers and
administrators, which stakeholders reported led to confusion in the implementation process
particularly with respect to the required activities and appropriate timeline (Strunk, Weinstein,
Makkonen, 2013). Although these issues did not appear to influence teacher and administrator
understanding of the purpose of the EGDC directly, it is possible that it indirectly stifled
communication which may have had implications for implementation. Because the successful
implementation of SBMMTES requires a shared commitment on behalf of both teachers and
administrators, ensuring that these stakeholders have the opportunity to make sense of
SBMMTES in the context of their joint work may prove important for ensuring that these
systems are implemented in a way that leads to the meaningful improvements in teacher practice.
The central role of teacher-administrator interaction in directly shaping the
implementation of the EGDC also demonstrates the need to pay particular attention to situated
59
cognition when examining SBMMTES relative to other standards-based reforms. Extant research
demonstrates that teachers rely on interactions with their administrators and colleagues to make
sense of new content standards (Spillane, 2004) or reading curriculum (Coburn, 2005). However,
in the context of these reforms, interaction between these two groups is not required for
implementation, and the role of these interactions in shaping implementation is indirect. For
example, a teacher might benefit from interactions with their administrator in terms of their
depth of understanding about a given set of math standards, but this interaction does not directly
shape how the teacher implements these standards in their classroom. In the case of
implementing the EGDC, teacher-administrator interactions not only benefited teacher
sensemaking, but were actually required to successfully implement the new process. This
suggests that when examining SBMMTES, situated cognition may play a more direct and
prominent role in shaping implementation than in other standards-based reforms.
Finally, although the findings presented here have several implications for research,
policy, practice, and theory, future research should build on this initial evidence to more fully
explore teacher and administrator sensemaking and implementation with respect to these new
evaluation systems. In particular, future research should work to build a richer picture of teacher
and administrator sensemaking as it relates to the implementation of SBMMTES over time, to
better understand how this may change the nature of teacher and administrator learning with
respect to SBMMTES. In addition, because the findings presented here are derived from a small
group of teachers and administrators in a single context, additional research across a diversity of
contexts and types of SBMMTES is necessary. The fact that sensemaking alone did not explain
all the variation in the implementation of the EGDC, also suggests that future research may
60
benefit from relying on additional theoretical lenses that better capture the role of interaction in
shaping teacher and administrator learning during the implementation of a SBMMTES.

61
CHAPTER THREE - Moving On and Moving Out? Examining Teacher Mobility in
Response to the Implementation of a Standards-Based, Multiple-Measure Teacher
Evaluation System

Introduction

In addition to requiring teachers and administrators to think differently about their
practice, SBMMTES change the landscape of teacher accountability in ways that may influence
teacher and administrator behavior with respect to the teacher labor market. Some of these
changes in behavior may be intended by policymakers who anticipate that SBMMTES will help
encourage more informed decisions about placing teachers in the schools and classrooms where
they are needed most. However, as discussed above, teachers and administrators may respond to
the additional accountability for performance embedded within SBMMTES in unintended ways,
systematically moving (or being moved) away from placements where they feel they may
receive a lower score (e.g., low-performing schools). Whether intended or unintended, or
induced by a teacher or administrator, changes in teacher mobility in response to SBMMTES
have implications for who teaches and where, raising questions about how, if at all, the
implementation of these new systems may influence the composition of the teacher labor market.
As outlined in Chapter 1, only a few studies have explicitly examined the impact of
SBMMTES on the teacher labor market (Dee & Wyckoff, 2013; Glazerman & Seifullah, 2012;
Sartain & Steinberg, 2014). This research finds mixed results about the relationship between
SBMMTES and teacher mobility and is concentrated on evidence collected from three evaluation
systems in the context of only two large urban districts. This limited research base provides
policymakers with little clarity about the implications of implementing different types of
SBMMTES on the teacher labor market across a diversity of local contexts.
62
In this paper, I contribute to the growing knowledge base on the broader effects of
implementing SBMMTES by examining teacher mobility in response to the implementation of a
SBMMTES pilot in the Los Angeles Unified School District (LAUSD). As described earlier,
during the 2011-2012 school year, LAUSD began the Initial Implementation Phase of the
Educator Growth and Development Cycle (EGDC), piloting this new multiple-measure
evaluation system with a small group of volunteer teachers and administrators. With LAUSD
making clear its intentions to ramp up the EGDC for districtwide implementation in the coming
year (Llanos, 2011; Singh, 2012), and the teachers’ union vehemently opposing the new
evaluation system (Llanos, 2011; Fletcher, 2013), the EGDC pilot garnered significant local and
national media attention. These actions made clear to pilot participants that LAUSD would be
rolling out the EGDC –for stakes –in the proximate future, and ensured that these same teachers
felt the threat of increased accountability for performance brought about by the pilot.
19

The differential exposure of teachers to the EGDC pilot in LAUSD provides the
opportunity to examine how teacher mobility may change in response to the threat of additional
accountability brought about by these new systems. This is of particular policy relevance as
states and districts continue to grapple with the design of SBMMTES and move forward with
their implementation despite limited evidence about their efficacy. In particular, this study
provides new evidence about how teachers might respond to this potential increase in
accountability even in the earliest stages of implementing these new reforms. Given that states
and districts across the country are currently in these early stages, the results presented here are
particularly valuable as they can help inform critical discussions about the best approaches to

19
During the 2011-2012 school year, all teachers in tested grades and subjects also received VAM and therefore had
additional information about their performance from this new measure. However, only 2% of individuals who
received VAM also participated in the pilot. In this paper, I focus specifically on individuals who experienced the
new evaluation process under the SBMMTES pilot, rather than those who received additional information through a
single new performance measure. The impact on teacher mobility of receiving VAM, is the topic of Chapter 4.
63
designing and implementing SBMMTES while these new systems are still in their nascent stages
and can be more easily modified to ensure they best achieve their goals.
In what follows, I review the research questions that guide a formal investigation of the
relationship between the implementation of a SBMMTES and teacher mobility, using LAUSD as
a case. Then, I describe the data and sample as well as the methodological approach used to
generate my findings, and provide the results. I conclude with the implications of this work for
research, policy, and practice with respect to the design and implementation of SBMMTES.

Research Questions

In this paper, I examine how the threat of additional accountability brought about by the
implementation of a SBMMTES pilot impacts teacher mobility. Specifically, I ask: Are teachers
who participate in an SBMMTES pilot more likely to switch schools or classrooms or leave the
district after the pilot’s implementation relative to teachers who do not participate in the pilot?
and, How, if at all, do changes in teacher mobility resulting from participating in a SBMMTES
pilot impact the composition of the teacher labor market locally? In answering these questions, I
provide additional information to policymakers about the relationship between the
implementation of SBMMTES and teacher mobility and how, if at all, these systems may work
to change the composition of the teacher labor market locally.
I find that teachers who participated in the EGDC pilot were no more or less likely than
teachers who did not participate to switch schools or classrooms or leave the district in response
to the threat of additional accountability. These null findings remain consistent across multiple
comparison groups, suggesting that even after adjusting for selection bias to the treatment
condition, participating in the EGDC pilot had no impact on teacher mobility. The results suggest
64
that participating in a SBMMTES may not necessarily lead to the positive labor market outcomes
intended by policymakers, and that other policy mechanisms may be required to ensure
SBMMTES achieve their goal of improving the overall quality of the teacher workforces
The results presented here must be considered in light of several limitations. Perhaps
most importantly, the findings are based on teacher mobility in response to the implementation
of a SBMMTES pilot in a no-stakes setting and therefore represent teacher response to the threat
of additional accountability only. In particular, we may expect that the findings presented here
represent a lower bound as the system has not yet been attached to stakes and teachers arguably
face less of a threat than might be felt when the system is linked to summative employment
decisions. Moreover, although the district made clear that the EGDC would be scaled up in the
following school year, it is possible that this threat was undermined by the vocal opposition to
the new evaluation system from the teachers’ union, leading teachers to believe the system
would likely not come online in the next year. Finally, only 371 teachers participated in the
SBMMTES pilot, of which only a small proportion switch schools/classrooms or leave LAUSD
each year. This necessarily leads to limited statistical power in many of my analyses, making it
difficult to observe an effect even if one truly exists. Despite these limitations, when considered
as part of the broader literature on SBMMTES, my findings can provide policymakers with
additional insight into the impact of implementing these complex systems on teacher mobility
and the implications for the composition of the teacher labor market in another local context.

Data and Sample

To answer the research questions outlined above I rely on a five-year panel of teacher,
student, and school-level administrative data from LAUSD. The data contain demographic and
employment information for all elementary and secondary classroom teachers in first through
65
twelfth grades in traditional schools in LAUSD, linked to the demographic and performance
information of their students, and the schools in which they worked in the years before (2007-
2008 to 2010-2011) and after the EGDC pilot was implemented (2011-2012). The full panel
includes approximately 118,000 teacher-year observations with 31,500 unique teachers.
Measuring Mobility. The primary outcomes of interest in the analyses outlined below
are teachers’ annual school and classroom mobility decisions. Teacher mobility comes in two
forms: 1) “forced” mobility which represents a decision made for a teacher based on external
factors (e.g., layoffs, reorganization, etc.), or 2) “teacher-initiated” mobility which arguably
represents a choice made by a teacher to change placements (e.g., requesting a transfer, accepting
a position in another district). Forced mobility is likely not caused by the provision of additional
performance information as it is related to factors beyond a teacher’s control. If I include forced
mobility in the analyses below, I risk attributing mobility resulting from factors unrelated to
teacher (or administrator) response to the EGDC pilot and accountability threat, to the treatment
itself, biasing my estimates of the observed relationship.
It is difficult, however, to separate teacher-initiated from forced mobility decisions within
administrative data. For example, it is possible that teachers or administrators utilize
standardized test results and other information to place teachers into specific positions (e.g.,
Boyd, et al, 2008; Cohen-Vogel, 2011). Given limitations in the administrative data I cannot
observe the extent to which this type of forced mobility occurs. At minimum, the LAUSD data
allow me to observe whether a teacher is laid off at the end of the year or receives a reduction-in-
force (RIF) notice that is then rescinded. Teachers who receive RIF notices and have the
appropriate certifications, may “bump” a less senior teacher from their position in order to
remain employed. Because teachers do not choose to be laid off and are effectively forced to
66
activate the bump provision when they receive a RIF notice, I consider both types of mobility
“forced” and exclude teachers who experience mobility resulting from either of these actions
from the analysis. The administrative data also allow me identify whether a teacher retires at the
end of the year, a decision likely unrelated to the implementation of the pilot. So as not to
attribute mobility resulting from retirements to the EGDC pilot, I exclude teachers who retire at
the end of each year and consider all other mobility as “teacher-initiated.”
20

To capture a teacher’s school- and classroom-level mobility, I must be able to observe a
teacher’s school site and/or classroom placement in the current year as well as in the following
year.
21
As such, I restrict the school-level sample to teachers in traditional public schools linked
to school sites in the current and following year for school year 2007-2008 through 2011-2012
(stayers and switchers), as well as teachers who exit the dataset and do not return (leavers).
22

Similarly, for the classroom-level analysis I restrict the sample to teachers in traditional schools
linked to classroom placements in the current and following year over the same period. This
leaves approximately 62,000 elementary teacher-year observations with 16,000 unique
elementary teachers, and approximately 55,000 secondary teacher-year observations with 15,500

20
It is possible that a teacher could choose to retire as a result of the EGDC. However, over the study period
LAUSD experienced extreme budget shortfalls, forcing them to issue furloughs. These furloughs meant that teacher
salaries were effectively diminished. These declines in salary have significant implications for teachers’ final
pension benefits as they are determined in part by their salary in the last three years of employment. This pension
structure likely diminished the incentive for teachers close to retirement to do so voluntarily over the study period.
Data limitations do not allow me to address the possible impact of these delayed retirements on teacher mobility but
do provide support for the assumption that all retirements observed were likely for reasons other than the
implementation of the EGDC pilot (e.g., poor health) and can therefore be excluded from the analyses.
21
My data are right censored at 2012-2013 so teacher mobility decisions at the end of this school year cannot be
observed. I use the 2012-2013 school year of data to observe teacher mobility decisions at the end of the 2011-2012
school year only and exclude all observations in the 2012-2013 school year from my analysis.
22
Given the special contexts in which teachers in alternative schools work (e.g., teaching across multiple campuses
or teaching multiple grade-levels within a single class) it is difficult to identify their primary school or classroom
placement. As such, I choose to exclude them from the analysis. I also exclude teachers in affiliated charter schools
as the contexts in which these teachers work may differ systematically from the contexts in which teachers in
traditional schools work and in ways that influence teacher mobility.
67
unique secondary teachers, for the school and classroom analyses (See Appendix B, Table 1 for
sample sizes at the school- and classroom-level for elementary and secondary teachers).
I examine changes in teacher mobility at the school- and classroom-level, separately, as
each suggests a distinct response to the implementation of the EGDC pilot. School-level mobility
captures a teacher’s decision to switch schools or leave LAUSD, relative to staying in his or her
current school, while classroom-level mobility captures a teacher’s decision to switch from one
grade or subject placement into another. It is important to note then, that the switch and leave
decisions of teachers at the school-level are qualitatively different from those at the classroom-
level because they are estimated relative to staying in a school versus staying in a grade-level or
subject placement. To capture these differences, I rely on different definitions of the mobility
outcome at the school- and classroom-level in all analyses.
School-Level Mobility. At the school-level, I am interested in understanding how, if at
all, the threat of additional accountability for teacher performance brought on by the
implementation of the EGDC pilot differentially impacts the mobility of teachers who
participated in the pilot across schools within the district. To capture teachers’ school-level
mobility decisions in response to this event of interest, I would ideally measure whether a teacher
stays in his or her current school, switches to another school within the district, or leaves LAUSD
at the end of each year. However, given the small sample of teachers who participated in the
EGDC pilot and switch schools or leave the district at the end of each year, I am unable to
examine teachers’ decisions to switch schools or leave LAUSD, separately.
23
Instead, I look at
teacher turnover generally and examine the probability that a teacher stays in their current
school, or moves, defined as either switching schools or leaving LAUSD at the end of each year.

23
As discussed in greater detail below, this is particularly the case in two of the comparison groups as I am looking
at a reduced sample of comparison teachers in addition to the limited sample of treated teachers. In order to compare
results across all three comparison groups I rely on this collapsed mobility outcome across all comparison groups.
68
This outcome is still critical to the theory of action behind SBMMTES because it demonstrates
the extent to which implementing these systems increases turnover generally, in the local teacher
labor market, and the implications of any observed disruption for who teaches and where.
Classroom-Level Mobility. At the classroom-level, measuring teacher mobility is more
complex. I am primarily interested in whether or not a teacher changes classroom placements
from one that is eligible to receive a value-added measure (VAM) of teacher performance to one
that is not, or vice versa. I choose to look at classroom mobility in this way because it captures a
teacher’s decision to move into, or out of, a placement where the accountability threat is
arguably greater. This is because, teachers in tested grades and subjects are eligible to receive
VAM and can therefore be held directly accountable for student achievement, whereas teachers
in non-tested grades and subjects are not eligible to receive VAM and subsequently cannot be
held accountable for student achievement in the same way. In fact, during the EGDC pilot,
LAUSD had not developed a measure to hold teachers in non-tested grades and subjects
accountable for student achievement, and made no public announcements that they had plans to
do so in the near future. This left teachers in non-tested placements facing less accountability
threat after the implementation of the EGDC pilot than their counterparts in tested placements.
At the elementary-level whether a teacher is eligible to receive a VAM score or not is a
direct function of the grade they teach. Since ELA and math VAM scores at the elementary-level
are calculated for all teachers in grades 3-6 but are not calculated for teachers in grades K-2, the
former are considered teachers in tested placements and the latter non-tested placements.
24
At the
secondary-level, VAM scores are calculated for only a subsample of courses. Teachers who

24
In some cases an elementary teacher may be assigned to a “combination” class, teaching both 2
nd
and 3
rd
graders.
This occurs for 5.4% of elementary teachers in my sample. In these instances this teacher is simultaneously teaching
a tested grade and a non-tested grade requiring a decision be made about their placement. For the purposes of the
analysis presented here I consider any teacher as “tested” if they teach ANY tested grade in a given year.
69
instruct these courses receive VAM and are therefore in tested placements while teachers who do
not instruct these courses do not receive VAM and are therefore in non-tested placements.
However, at the secondary-level 54% of teachers in LAUSD are the instructor of record in both
tested and non-tested courses simultaneously. Because all teachers who teach a tested course
receive VAM for at least that course, I consider all teachers who teach at least one tested course
as in a tested placement and all teachers who teach no tested courses as in non-tested placements.
At the classroom-level, a teacher chooses to stay in their current placement (i.e., teaches
in a tested grade or non-tested grade and remains in a tested or non-tested grade, respectively, the
following year), switch classroom placements (i.e., moves from a tested grade into a non-tested
grade or vice versa either within the same school or at a different school, the following year), or
leave LAUSD altogether (i.e., drops from the dataset in the following year) at the end of each
year. However, as above, I am limited by sample size and am forced to look at teacher turnover
more generally. Therefore, I define the classroom-level mobility outcome as whether or not a
teacher stays in their current classroom placement, or moves placements, defined as either
switching placements within or across schools in the district or leaving LAUSD altogether.
It is important to note that the opportunity to move from a tested to a non-tested
placement differs in important ways for elementary relative to secondary teachers. Specifically,
certification poses less of a barrier for elementary teachers, as all elementary grades require that
teachers hold a multiple-subject credential. A teacher interested in switching from a tested-
placement like Grade 3, to a non-tested placement like Grade 1, already holds the appropriate
certification. For secondary teachers, however, certification is more specialized and teachers
must hold a single-subject credential associated with a particular content area (e.g., Single-
Subject ELA or Single-Subject Math). A teacher interested in switching from a tested-placement
70
like Grade 10 ELA to a non-tested placement like Grade 11 Fine Arts would need to hold a
credential in this additional area, or obtain a credential in this area, before switching placements.
This necessarily constrains the opportunity for secondary teachers to switch across tested and
non-tested placements. The limited opportunity for classroom-level mobility faced by secondary
teachers has implications for the results produced from the classroom-level analyses for
secondary teachers, a point I return to later when discussing the findings.

Analytic Approach

To determine whether participating in a SBMMTES and subsequently facing the threat of
additional accountability for performance differentially impacts teacher mobility, I employ a
series of bivariate logistic regressions in a difference-in-difference framework. This approach
allows me to compare the mobility of teachers who participated in the EGDC pilot and
subsequently felt the threat of additional accountability, to a comparison group of teachers who
did not participate in the pilot and did not feel this same threat. Because how teachers respond to
the threat of additional accountability for performance may influence who teaches and where
they choose to teach, I am also interested in understanding the implications of any observed
changes in teacher mobility on the characteristics of those who teach in LAUSD.
The Difference-in-Difference Framework. To answer the research questions above I
can compare mobility rates for pilot participants before and after the EGDC pilot was
implemented in LAUSD. This approach will tell me if mobility rates have changed at all in the
period after teachers felt the threat of additional accountability for performance. However,
several other reforms were also going on in LAUSD during 2011-2012 when the EGDC pilot
was implemented, making it likely that other unobservable factors (e.g., new curriculum,
71
changes in district policy) impacted all teachers over this same period. To the extent that these
unobservable factors also impact teacher mobility, then simple pre-post comparisons could
attribute changes in teacher mobility to participation in the EGDC pilot, rather than the secular
trend in mobility rates across LAUSD over this same period.
25

In order to establish unbiased estimates of the treatment effect, I utilize a difference-in-
difference (DID) approach that compares pilot participants to non-participants, both before
(2007-2008 to 2010-2011) and after (2011-2012) the EGDC pilot was implemented in LAUSD.
The DID approach relies on the assumption that the treated and non-treated groups are
comparable in how they are affected by forces influencing the outcome but different in how they
are affected by the treatment. In the study proposed here, this assumption –known as parallel
trends –suggests that the mobility decisions of teachers who did not volunteer to participate in
the pilot are expected to be “comparable” to those that would have been made by teachers who
did volunteer to participate in the pilot in the absence of the accountability threat (Wooldridge,
2010). If this assumption is met then the comparison group can be used to control for secular
trends in the labor market impacting all teachers over the same period, ultimately purging the
estimated effects of factors unrelated to the implementation of the EGDC pilot.
Treatment and Comparison Groups. During the EGDC pilot, teachers were not
randomly assigned to treatment, but rather volunteered to participate in the new evaluation
process and therefore self-selected into the treatment group. This makes it difficult to identify a
comparison group that meets the parallel trends assumption, as teachers who self-
selected/volunteered for the treatment group (pilot participants) likely have certain observable
(e.g., demographics, years of experience, employment status) as well as unobservable (e.g.,

25
I use bivariate logistic regressions given the binomial distribution of my mobility outcome at both the school and
classroom-level. This binomial distribution requires logistic regression as it does not meet the assumptions necessary
for using traditional OLS which assumes a normal distribution of the outcome variable.
72
motivation, alignment with the philosophy of EGDC) characteristics. In an effort to mitigate this
selection bias, I rely on three separate comparison groups that each provide valuable information
about the relationship between teacher mobility and the threat of additional accountability
brought on by the implementation of the EGDC pilot. These groups include: 1) all teachers in
EGDC pilot schools who did not themselves participate, 2) all teachers in LAUSD who did not
participate in the EGDC pilot, and 3) a group of like-teachers who did not participate in the
EGDC pilot that were selected using a propensity score matching (PSM) technique.
I begin by using a comparison group of all teachers in EGDC pilot schools who did not
participate in the pilot (CG1). We might expect these teachers will be most similar to pilot
participants as they make mobility decisions in the same school contexts but did not experience
the treatment. However, as outlined in Table 2 and Table 3 in Appendix B, at both the
elementary and secondary levels, pilot participants are significantly more likely than non-
participants in pilot schools to be female, have more experience, have National Board
Certification, and receive fewer needs improvement or below standard ratings on their previous
evaluation (p<0.10 to p<0.001). Moreover, at the secondary level participating teachers are
significantly more likely than non-participants in pilot schools to hold an advanced degree and
teach in a tested placement (p<0.05 and p<0.001, respectively). This indicates that the pilot
participants are significantly different from, and seemingly more highly qualified and perhaps
more effective than, non-participating teachers within EGDC pilot schools, which may bias my
results. In theory, this selection bias should be alleviated by the DID framework as long as there
are parallel trends in the outcome variable before the intervention.
Figures 2 and 4 in Appendix B show that the classroom-level mobility rates for
participating teachers at both the elementary and secondary level do not appear to differ
73
substantially from those of non-participants in pilot schools prior to the implementation of the
EGDC. However, Figures 1 and 3 in Appendix B show that the school-level mobility trends of
elementary and secondary teachers in pilot schools who did not participate in the pilot do differ
from those of participating teachers in the early years of the panel, with noticeable dips in rates
of moving in 2008-2009. Despite these dips in the earlier years, the trends return back to parallel
in 2009-2010 and 2010-2011, the years just prior to the EGDC pilot. This suggests that while
participating teachers are significantly different from non-participating teachers within EGDC
pilot schools on observable characteristics, their mobility trends are similar just before the onset
of treatment and the parallel trend assumption holds.
26

In order to triangulate my findings across multiple comparison groups, I also rely on a
second comparison group that includes all teachers in LAUSD that did not participate in the
SBMMTES pilot (CG2). This comparison allows me to draw on a much larger sample of
comparison teachers, increasing the statistical power of my analyses. As outlined in Table 2 and
Table 3 in Appendix B, elementary and secondary teachers in LAUSD who did not participate in
the EGDC pilot also differ significantly from pilot participants on observable characteristics
including: employment status, experience, highest degree earned, and NBC certification (p<0.10
to p<0.001). However, teachers in this comparison group are not significantly different from
participants in terms of previous evaluation ratings, indicating they may be more similar in terms
of performance, than are non-participants in pilot schools.
27

26
As discussed in greater detail below, because these trends are not smooth across the board, particularly in the early
years (2007-2008 to 2008-2009), I also run a series of specification checks looking at the difference-in-difference
estimates between the post-treatment period and each pre-treatment year separately, to ensure these slight
differences in trends in the early year of my panel do not bias my results.
27
It is important to note that Stull ratings are not necessarily the best measure of teacher performance.
Approximately 99% of teachers are identified as meeting standards on their Stull each year. However, while limited,
this is the only available measure of teacher quality available in LAUSD, so I use it here noting its limitations.
74
As above, the school and classroom-level mobility rates for pilot teachers at both the
elementary and secondary level do not appear to differ substantially from those of non-
participating teachers in LAUSD prior to the implementation of the EGDC pilot, with a few
exceptions (see Appendix B, Figures 5-8). Most notably, the classroom-level mobility of
participating elementary teachers and the school-level mobility of participating secondary
teachers does diverge from that of non-participating teachers in LAUSD in the early years of the
panel. However, as above, these trends return back to parallel in 2009-2010 and 2010-2011, the
years just prior to the EGDC pilot.
28
Since there are fewer observed differences between pilot
participants and all non-participants in LAUSD than observed when looking just within pilot
schools, and the parallel trends assumption holds, using all non-participants in LAUSD as the
primary comparison group may provide a better approximation of the treatment effect. However,
each group provides a slightly different interpretation of the treatment effect, so I use both in all
analyses and check to see whether the results are sensitive to changes in the comparison group.
Although non-participants in pilot schools and non-participants in LAUSD provide
information-rich comparison groups given parallel trends in mobility prior to treatment, they do
not sufficiently address the issue of selection bias in the treatment group. In an attempt to better
address this issue, I generate a third non-treated group (CG3) using propensity score matching
(PSM). PSM is an analytic technique that helps to address the non-random sorting of individuals
into a treatment or control condition by identifying a comparison group that is similar to the
treatment group based solely on their observable characteristics (Dehejia & Whaba, 2002;
Murnane & Willett, 2011). This approach assumes that using observable characteristics to
identify a matched sample provides a better chance of selecting a comparison group that also

28
Again, I run specification checks to ensure my results are not impacted by differences in mobility between
treatment and comparison teachers in the early years of the panel.
75
matches on unobservable characteristics. In the case of the EGDC pilot, if we assume that self-
selection into participation is, in part, a function of the observable characteristics of teachers,
than PSM can be used to purge the estimated treatment effect of this selection bias (Imbens &
Wooldridge, 2009; Shadish, Cook, & Campbell, 2002). It is important to note that while PSM
helps to address the selection problem it cannot fully account for the bias in the treatment group
because treated teachers are matched to non-treated teachers only on observable characteristics.
It cannot account for unobservable factors like teachers’ motivation or buy-in to the EGDC pilot,
which may in fact impact their decision to participate. However, the estimates obtained when
using the PSM-selected comparison group can be interpreted as a rigorous estimate of the
treatment effect, and further facilitate a triangulation of the results across different samples.
To identify the appropriate comparison group using PSM, I first isolate a teacher’s
propensity to participate in the EGDC pilot using a logistic regression that estimates participation
in the pilot as a function of available teacher, student, and school covariates. I estimate this
selection model separately for elementary and secondary teachers based on the 2010-2011 school
year of data since teachers volunteered to participate in the EGDC pilot at the end of this year.
Table 4 in Appendix B provides a list of all covariates included in the selection models at the
elementary and secondary level.
After estimating the propensities, I identified treatment and control pairs using nearest
neighbor matching with no replacement, which selects from the sample of non-participants the
“nearest neighbor” in terms of propensity to participate in the EGDC pilot for each participant
(Murnane & Willett, 2011).
29
To employ the matching algorithm the distributions of the
estimated propensities must have sufficient overlap such that a nearest neighbor can be identified

29
I choose to implement nearest neighbor matching with no replacement in order to ensure sufficient sample size for
the CG3 analyses. Allowing no replacement ensures a slightly larger sample size for all CG3 analyses, but also
introduces the possibility that a better match could be found were replacement allowed.
76
from the region of common support (Caliendo & Kopeinig, 2005). Figures 9 and 10 in Appendix
B provide the distribution of estimated propensity scores for treated and non-treated teachers at
the elementary and secondary levels, respectively. It is important to note that at both levels the
distribution for non-treated teachers is right skewed, with a large proportion of teachers falling in
the lower half of the estimated propensity scores. In addition, there is a limited sample from
which to draw within the region of common support as evidenced by the density among control
teachers in this region. Despite the limited density, there is sufficient overlap to select matches
from within the region of common support. To ensure the most valid matches are identified, I
draw nearest neighbors from within this region of common support only.
After identifying the nearest neighbor matches, it is important to ensure that conditioning
on the propensity score sufficiently reduced bias in the matched control group. I test group mean
differences in the characteristics of teachers both before and after employing the matching
algorithm to ensure that this is in fact the case (Rosenbaum & Rubin, 2005). Tables 5 and 6 in
Appendix B provide the results of these balancing tests and demonstrate that in most cases there
were no significant differences on observable characteristics between pilot participants and non-
participants prior to matching. In all cases in which significant differences did exist prior to
matching, bias was sufficiently reduced and the differences were completely eliminated. Given
the results of the balancing tests, I proceed with the identified matches.
30

By definition, the comparison teachers identified using PSM do not differ from pilot
teachers on observable characteristics. However, it is still important to ensure parallel trends in
the mobility of pilot teachers and comparison teachers selected using PSM prior to the
implementation of the SBMMTES pilot. Figures 11-14 in Appendix B provide the mobility rates

30
It is important to note that the use of PSM limits my sample size to just teachers treated by the SBMMTES pilot
and a single nearest neighbor match in each year. This further reduces statistical power in all of the analyses using
the third comparison group presented below.
77
for pilot and comparison teachers selected using PSM over the five year panel. Here we see that
while pilot teachers are most similar to teachers selected using PSM in terms of observable
characteristics, they demonstrate less consistent trends in mobility over time. The parallel trends
assumption appears to hold for elementary teachers’ school-level mobility and secondary
teachers’ classroom-level mobility (see Appendix B, Figures 11 and 14, respectively). However,
this assumption does not appear to hold for elementary teachers at the classroom-level, where
mobility differs greatly over time between pilot and PSM-selected comparison teachers (see
Appendix B, Figure 12). Moreover, trends in mobility are less consistent for secondary teachers’
school-level mobility where we see parallel trends in the earlier years, and a noticeable
divergence in trends in 2009-2010 that continues into the post-intervention period (see Appendix
B, Figure 13). In these latter two cases the lack of parallel trends is concerning and limits the
validity of any conclusions drawn from these comparisons.
School-Level Models. At the school-level I compare the mobility rates of pilot
participants to that of non-participants, separately for each comparison group outlined above,
both before and after the threat of additional accountability for teacher performance was
introduced by the EGDC pilot. This is formalized in model (1) which estimates the probability
that a teacher will move (i.e., switch schools or leave LAUSD) relative to the base outcome of
staying in their current school:
(1) l

!

"
At the school-level,

is a binary variable that equals zero if teacher i, in
classroom j, in school s, in year t, stays in his/her school, and equals one if he or she switches
schools or leaves the district. To measure the differential impact of participating in the EGDC
pilot on teacher mobility in response to the threat of additional accountability for performance, I
78
specify a series of explanatory variables in all specifications. Specifically, PILOT
t
is an indicator
for time that equals one in 2011-2012 when the pilot was implemented and equals zero in all
other years. TREATED
ijst
is the treatment indicator, which equals one in each year if the teacher
participated in the SBMMTES pilot in 2011-2012, and

is the interaction between time and
treatment (PILOT
t
* TREATED
ijst
). Finally, X
ijst
is a vector of individual teacher and classroom-
level characteristics and Z
st
is a vector of school characteristics.
In particular, X
ijst
includes controls for: teacher gender, race/ethnicity, years of
experience
31
, education level (e.g., bachelors, masters, or doctorate), and employment status
(e.g., permanent, probationary, temporary), as well an indicator for whether or not the teacher is
National Board Certified (NBC).
32
In addition, because teachers in tested grades and subjects
who receive VAM arguably face greater accountability threat than teachers in non-tested
placements, I include a control for whether or not a teacher is in a tested grade or subject. To
account for classroom characteristics, I include indicators for the percent of students taught who
are English learners (ELL), eligible for free or reduced-price lunch (FRL), and high-performing
(i.e., who score proficient or advanced in ELA or Math, on the California Standards Test [CST]).
At the school-level, Z
st
includes controls for school-level percent ELL, percent FRL, and
performance, measured by a schools’ current year Academic Performance Index (API) score.
33

31
LAUSD collects years of experience data based on teacher placement on the district salary schedule. Because the
salary schedule maxes out at 10 years of experience, the years of experience variable is truncated at 10+. A teacher
with 10+ years of experience may actually have ten years or may have 25 years of experience.
32
Although National Board Certification serves as only a rough proxy for teacher quality, it is the best available
measure within my data. I do not control for teacher quality using AGT for several reasons. First, AGT is only
available for approximately 50% of teachers in LAUSD leaving some teachers without these data. Second, AGT was
first produced for teachers in 2010-2011 and therefore is not available in earlier years.
33
I use API score as my measure of school performance for several reasons. First, I do not have school-wide value-
added in pre-treatment years making it impossible to include this as my measure of school performance. Second,
API score provides more information about school performance than CST proficiency rates alone as it is a
composite measure based on CST scores, high school exit exam scores, graduation rates, and attendance. Finally,
API score is a widely used and easily interpretable measure of relative school performance that parents, teachers,
and administrators rely on to judge school quality, making it a useful proxy for the signal a school sends to teachers
about the quality of the school context.
79
Finally, the implementation of the SBMMTES pilot occurred simultaneously with other reform
initiatives in LAUSD. Several schools in LAUSD were part of the Public School Choice
Initiative (PSCI) and others received School Improvement Grants (SIG), both of which may
influence teacher mobility. To control for this, I include indicators for whether or not the teacher
is in a school that participated in PSCI or SIG. These indicators equal one in the year that the
reform is expected to impact teacher mobility (e.g., in the first year the school is identified for, or
treated by PSCI, or receives SIG funds to implement a turnaround reform).
Across all models, the probability of a teacher staying in his or her current school, or
moving at the end of the year is a function of time (measured as pre –or post-EGDC pilot), the
school (Z
st
) and teacher/classroom-level (X
ijst
) controls, TREATED
ijst
and the interaction of time
and treatment

(PILOT
t
* TREATED
ijst
). The coefficient of interest,

, captures the differential
effect of the threat of high-stakes accountability for teacher performance brought on by the
EGDC pilot, on the mobility decisions of teachers who participated in the pilot relative to those
who did not participate after controlling for individual teacher, classroom, and school
characteristics as well as secular trends in the teacher labor market. However, when using non-
linear regression models like bivariate logistic regression, interactions provide the effect of
interest in multiplicative terms (Buis, 2010). Therefore, to obtain more interpretable estimates I
compute the marginal differences for every model specification using the following approach:
[#Turnover Rate #Participant, Post-Pilot$-Turnover Rate #Participant, Pre-Pilot$-
[Turnover Rate#Non-Participant, Post-Pilot$-(Turnover Rate(Non-Participant, Pre-Pilot]
As outlined above, in a few cases the mobility rates of teachers in the treatment and
comparison groups do not meet the parallel trends assumption in the early years of the panel
(2007-2008 to 2008-2009). This raises concerns about the extent to which comparing teachers in
these groups over time will produce unbiased estimates of the treatment effect. Moreover, it is
80
possible that teachers felt the threat of additional accountability for performance in the year prior
to the implementation of the EGDC pilot and subsequently responded by switching schools or
classrooms or leaving the district. This is because in May of the 2010-2011 school year LAUSD
announced that the SBMMTES pilot would be implemented in the following year and
confidentially released VAM scores for teachers in tested grades and subjects. This came on the
heels of a very public release of teacher-level VAM by the Los Angeles Times in August 2010.
34

To ensure that these events, along with the lack of parallel trends in early years of the
panel do not influence the estimated effects, I run a variation of model (1) that estimates the
probability that a pilot participant –as compared to a non-participant –will move relative to the
base outcome of staying in the same school across all pre –and post-treatment years, separately:
(2) l

∑
&
'

(
&)

*

∑
&
'

&)+

!

"
This model is identical to model (1) except with respect to time. Replacing PILOT
t
is a
vector of year indicators for 2007-2008 to 2011-2012 (∑
&
(
&)
'

) that equal one in each
year of data. This vector of year indicators is fully interacted with the treatment (∑
&
'

&)+

) to produce the coefficients of interest; each interaction between time and treatment
captures the mobility rates of participants relative to non-participants in that year. This means we
can explicitly examine how teacher mobility differs between pilot participants and non-
participants in each year separately. In this specification, we would ideally see no significant
differences in the mobility rates of pilot participants relative to non-participants in all years prior
to the implementation of the pilot and the associated accountability threat. This would suggest

34
It is important to note that the confidential release of VAM by LAUSD as well as the public-release of teacher-
level VAM by the Los Angeles Times only provided teachers in tested grades and subjects with access to additional
information about their performance. Since the models presented are concerned with the mobility of pilot
participants relative to non-participants controlling for whether or not a teacher is in a tested grade or subject, this
potential threat to internal validity is less of a concern. However, how teachers in tested grades and subjects respond
to the public release of teacher performance information when faced with the threat of additional accountability is an
interesting policy question with respect to SBMMTES and is the topic of Chapter 4.
81
that any significant effects observed in 2011-2012 are likely associated with teachers’ response
to the threat of additional accountability in light of participating in the EGDC pilot.
To determine if the difference in observed mobility rates for pilot participants relative to
non-participants in the year the EGDC pilot was implemented (2011-2012) and those observed in
prior years are in fact significantly different, I run Wald tests. These tests produce DID estimates
for each pre- and post-treatment year combination separately, rather than generating a single DID
estimate based on the average trend over all pre-treatment years; they also generate DID
estimates between the pre-pilot year (2010-2011), when we might anticipate an early response to
the threat of additional accountability, and all earlier years. In so doing, this approach allows for
a more careful examination of the DID estimate of the treatment effect separately in the years
where we do not observe parallel trends and those in which we do. Moreover, it affords the
opportunity to examine the extent to which teachers responded to the threat of additional
accountability in the year prior to the implementation of the EGDC pilot. If the Wald tests return
significant differences, than the effect observed in the post-period can in fact be attributed to
differences in teacher response to the treatment itself. If they are not significant, we cannot reject
the null hypothesis that factors unrelated to the treatment did not induce differential mobility for
participants relative to non-participants over the same period.
Classroom-Level Models. For the classroom-level analysis I rely on the same models
used in the school-level analysis and described above. The primary difference in the classroom-
level analysis is the interpretation of the outcome variable in both model (1) and model (2). At
the classroom-level

is a binary variable that equals zero if teacher i, in classroom j, in
school s, in year t, stays in his/her current tested or non-tested placement, and equals one if he or
she moves, either switching placements (i.e., moving from a tested to non-tested placement or
82
vice versa) or leaving LAUSD. Across all models, I estimate the extent to which teachers move
relative to the base outcome of staying in their current tested or non-tested placement.

Results
The results presented below demonstrate that implementation of the EGDC pilot in
LAUSD did not differentially impact teacher mobility. These findings remain consistent across
all samples, suggesting that the estimated effects are robust to changes in the comparison group.
These null results suggest that simply implementing a SBMMTES may not lead to changes in
teacher labor market behavior –either positive or negative –in light of increased accountability
for performance, and that other policy mechanisms may be necessary to induce the positive
changes in the composition of the teacher labor market that SBMMTES are expected to produce.
School-Level Mobility. Figures 1, 3, 5, 7, 11, and 13 in Appendix B illustrate the school-
level mobility of elementary and secondary teachers who participated in the pilot relative to non-
participants in pilot schools (CG1), non-participants in LAUSD (CG2), and non-participants
selected using PSM (CG3), respectively. Across all figures we see that after the implementation
of the SBMMTES pilot in 2011-2012, teacher mobility increases for both participants and non-
participants, with mobility rising more rapidly for non-participants in each of the comparison
groups.
35
This provides suggestive evidence that teachers who participated in the EGDC pilot,
and subsequently felt the threat of increased accountability, may have been less likely than those
who did not participate in the pilot to move away from their current school.
Tables 7 and 8 in Appendix B present the DID estimates produced from model (1) which
formalizes the descriptive comparisons in the figures above. These marginal effects are presented

35
It is important to note that as mentioned earlier, the parallel trends assumption does not hold when examining the
school-level mobility of treated teachers relative to those in CG3. As such, all results produced from this estimate
must be interpreted with caution and cannot be interpreted as casual.
83
as predicted probabilities and can be interpreted as percentages. For example in the far left
column of Table 7 we see that elementary teachers who participated in the pilot and subsequently
received additional performance information were 0.3% less likely than those who did not
participate to move away from their current school, relative to staying, after the implementation
of the EGDC pilot. In fact, at both the elementary and secondary level, across all comparison
groups, pilot participants are less likely to switch schools after the implementation of the EGDC
pilot than are non-participants, but these differences are not significant.
To ensure the findings obtained from model (1) are not impacted by the lack of parallel
trends in mobility between treatment and comparison teachers in early years, or by the potential
early response of teachers to the threat of additional accountability for performance, I estimate
model (2) which produces the difference in teacher mobility between pilot participants and non-
participants for each year separately. Tables 9 and 10 in Appendix B provide Wald tests of the
DID estimates of treated teachers’ school-level mobility relative to that of non-treated teachers in
each comparison group produced from model (2). The top three rows each provide a single DID
estimate of the proportion of teachers who moved relative to the base outcome of staying in the
same school for participants relative to non-participants between 2007-2008 to 2010-2011 (prior
to the implementation of the SBMMTES pilot) and 2011-2012 (after the SBMMTES pilot). The
bottom two rows provide the same for the year prior to the implementation of the EGDC pilot
(2010-2011) and all earlier years (2007-2008 to SY 2009-2010).
Looking at the top two rows of Tables 9 and 10, we see that even when examining the
DID estimates between the post-pilot period and all prior years separately, we see no significant
differences in the mobility of teachers who participated in the EGDC pilot and felt the threat of
increased accountability for performance, and those who did not. Moreover, the bottom two rows
84
provide no evidence of any threat to the internal validity of our DID estimates; participants were
no more or less likely than non-participants to experience turnover in the year prior to
participating in the SBMMTES pilot (2010-2011) relative to earlier years (2008-2009 to 2009-
2010). These results confirm that participating in the EGDC pilot had no differential impact on
the school-level mobility of elementary or secondary teachers.
Classroom-Level Mobility. Turning to the classroom-level, Figures 2, 4, 6, 8, 12, and 14
in Appendix B graphically demonstrate the classroom-level mobility of pilot participants relative
to non-participants across all comparison groups. Here we again see that after the
implementation of the EGDC pilot in 2011-2012, the mobility of elementary and secondary
teachers who participated in the pilot, and their comparison group counterparts, increased. In
addition, we see that these increases occurred more rapidly for teachers in each of the
comparison groups, relative to those who participated in the pilot.
36
This evidence parallels that
observed at the school-level and suggests that teachers who participated in the SBMMTES pilot
may be more likely than those who did not participate to stay in their current classroom
placement when faced with the threat of additional accountability for performance.
Tables 11 and 12 in Appendix B formalize this descriptive evidence and present the DID
estimates produced from model (1) for elementary and secondary teachers classroom-level
mobility. As above, the results represent marginal effects and are reported as predicted
probabilities. Here we see that across the board there again appears to be no differential impact
on teacher mobility of participating in the EGDC pilot. At both the elementary and secondary
level, across all comparison groups, teachers who participated in the pilot, were no more or less
likely than that those who did not to move away from their classroom placement after the

36
There is one exception to this trend – the classroom-level mobility of elementary teachers in CG3 relative to those
who participated in the pilot. The parallel trends assumption does not hold in this sample so we are unable to draw
any conclusions from either the descriptive evidence or regression results when looking at this group of teachers.
85
implementation of the SBMMTES pilot. Although we observe no effect, the fact that classroom-
level mobility increased generally over the same period is interesting. While these increases may
not represent a response to the EGDC pilot, they do suggest that there was general upheaval in
LAUSD at the time the EGDC was implemented.
Tables 13 and 14 in Appendix B provide Wald tests of the DID estimates of elementary
and secondary treated teachers’ classroom-level mobility relative to that of non-treated teachers
in each comparison group produced from model (2). Similar to the school-level results, we again
see that the null findings are substantiated by the DID estimates produced for each pre- and post-
pilot comparison, respectively. Moreover, the bottom two rows of each table again demonstrate
that teachers did not respond prematurely to the threat of additional accountability for
performance as we see no significant differences in the mobility of pilot participants relative to
non-participants in the year prior to the EGDC pilot relative to earlier years. Taken together,
these results demonstrate that participating in the new evaluation process also had no differential
impact on teachers’ classroom-level mobility regardless of the comparison group used.

Discussion and Implications
The theory of action underlying SBMMTES assumes that teachers and administrators
will respond to SBMMTES by making more informed human capital decisions that increase the
overall quality of the teacher workforce. However, the extant research on the implementation of
SBMMTES provides mixed evidence about how the implementation of these new evaluation
systems actually impact teacher mobility. The results presented above contribute to this limited
knowledge base and demonstrate that the implementation of the EGC pilot did not differentially
impact teacher mobility. These findings suggest that while policymakers intend for SBMMTES
86
to contribute to positive changes in the composition of the teacher labor market, simply requiring
teachers and administrators to participate in these new systems may not induce these changes.
The findings presented here are somewhat surprising given the context in which the
SBMMTES pilot under study was implemented. During the 2011-2012 school year LAUSD
made clear to teachers and administrators participating in the pilot that the EGDC would be
implemented district-wide in the following school year, introducing the threat of additional
accountability for teacher performance. In discussions with pilot participants, many noted that
they wanted to participate in the EGDC pilot because they knew that full-scale implementation
was inevitable, suggesting that this threat was in fact felt by participating teachers and
administrators. However, because none of the new information about teacher performance
generated from the EGDC pilot in the current year was linked to summative employment
decisions, it is possible that the threat was not sufficient to induce teacher mobility.
Sartain and Steinberg (2014) also examine teacher mobility in response to a no-stakes
SBMMTES pilot in Chicago and find no effects of the pilot on teacher turnover overall within
treated relative to comparison schools; however, they do find evidence of heterogeneous effects
on the turnover of low-performing, non-tenured teachers. Although I am unable to look at
heterogeneous effects here, given the small sample of pilot participants, the lack of overall
effects in both local contexts suggests that implementing a SBMMTES in a low-stakes setting,
may not contribute to changes in teacher mobility generally, regardless of the strength of the
accountability threat felt by teachers and administrators. It is also possible that teacher mobility
in response to a SBMMTES may differ over the long term. Both the results presented here and
those found by Sartain and Steinberg (2014) represent short-term effects, as each study relies on
only a single year of post-treatment data. Future research would benefit from examining how, if
87
at all, teacher response to the threat of additional accountability within a SBMMTES might differ
over time, and the implications for the quality of the teacher labor market in the long-term.
The results presented here also suggest that differences in teacher response to the
implementation of a SBMMTES may depend, in part, on how the SBMMTES is designed. In
many states and districts across the country SBMMTES are implemented in conjunction with
other human capital reforms including: performance-based compensation, targeted professional
development, and career pathways. For example, the IMPACT evaluation system in D.C. was
implemented in conjunction with financial rewards for outstanding performance and targeted
professional development aligned to teachers’ needs. Similarly, in Chicago, the implementation
of the Teacher Advancement Program (TAP) introduced a SBMMTES alongside performance-
based compensation, targeted professional development, and career pathways. Although,
LAUSD intended for the EGDC to eventually sit alongside other human capital reforms –most
notably targeted professional development linked to teachers’ needs –this was not the case
during the pilot. Rather, teachers who participated in the pilot experienced the new evaluation
process in the absence of these other human capital reforms.
The fact that the implementation of the EGDC pilot in LAUSD did not occur
simultaneously with other human capital reforms may have led teachers to respond differently to
the threat of additional accountability for performance than they would have had these other
human capital reforms been in place. The extant research on the impact of SBMMTES –
implemented in conjunction with other human capital reforms –further supports this point. For
example, Dee and Wyckoff (2013) find that the implementation of DC IMPACT increased the
voluntary attrition of low-performing teachers by more than 50% from DC Public Schools.
Similarly, Glazerman and Seifullah (2012) find that when implemented in conjunction with other
88
human capital reforms under TAP, the SBMMTES pilot in Chicago had a significant impact on
teacher mobility, increasing teacher retention in treated relative to comparison schools. In
contrast, I found that a SBMMTES pilot –when implemented as a stand-alone reform –did not
induce changes in teacher school- or classroom-level mobility. Considered together, these results
suggest that the design of SBMMTES, and the types of reforms they are coupled with, may lead
teachers to respond differently within the teacher labor market to their implementation.
The results presented here also suggest that how SBMMTES are implemented may have
implications for the extent to which they induce the mobility behavior intended by policymakers.
As outlined in Chapter 2, the implementation of the SBMMTES pilot in LAUSD was
inconsistent across school sites. In particular, Chapter 2 demonstrates that across five sites
participating in the SBMMTES pilot there was widespread variation in the extent to which
teachers and administrators engaged with the new information about teacher performance
produced from the EGDC process. To the extent that this superficial engagement in the
implementation of the SBMMTES pilot was widespread, than we may not expect the provision
of additional performance information to differentially influence teacher mobility, regardless of
the strength of the accountability threat felt by teachers and administrators.

89
CHAPTER FOUR - How Differential Access to Measures of Teacher Effectiveness Impact
Teacher Mobility in Response to Accountability in LAUSD

Introduction

In addition to changing the landscape of teacher accountability, SBMMTES redefine the
type of information available about teacher practice, creating an environment of differentiated
accountability for teacher performance. As outlined in Chapter One, many SBMMTES are
inherently designed to provide different types of teachers with different information about their
practice. For example, by including value-added measures (VAMs), which are only available for
teachers in tested grades and subjects, SBMMTES ensure that teachers in these placements
receive more information about their performance than their colleagues in non-tested placements.
Moreover, by producing new information about teacher effectiveness, SBMMTES raise the issue
of who should have access to this new information about teacher performance. While some
believe this teacher effectiveness data should be shared publicly with parents and communities,
others argue that it should only be released confidentially to teachers and administrators for
formative use. Because what information is available about teacher performance, and who has
access to it, has implications for teacher accountability, we may expect that each of these design
choices will have implications for teacher behavior within the local teacher labor market.
However, as described in Chapter One, the extant research on the impact of SBMMTES
only provides insight into the implications of these new systems on teacher mobility generally,
leaving several areas unexplored. In particular, the existing evidence does not speak to how
teachers with access to different information about their performance may respond differently to
the implementation of a SBMMTES. This is of particular policy relevance as the disequilibrium
in access to information is a common design feature in many new evaluation systems, yet we
know very little about the implications this may have for the teacher labor market.
90
Moreover, the existing research does not examine how teachers might react to the public
release of teacher performance information relative to the private release of this same
information. Given that publicly releasing information about teacher performance allows
teachers to be held accountable by parents and communities, rather than just their schools and
districts, we may expect teachers to respond differently depending on who has access to new
information about their effectiveness. This issue is also central to the current policy debate as
states and districts across the country continue to grapple with balancing public pressure for
increased transparency about teacher quality with a desire to ensure that this new performance
information is best utilized to ensure teacher growth and development.
To begin to address these gaps in the knowledge base with respect to the impact of
SBMMTES on the teacher labor market, I turn to the unique context for teachers in the Los
Angeles Unified School District during the 2010-2012 school years. As described earlier, during
the 2011-2012 school year, LAUSD simultaneously implemented the EGDC pilot and released
value-added scores for all teachers in tested grades and subjects regardless of their participation
in the pilot process. With the Superintendent making clear his intentions to implement the EGDC
–inclusive of teacher-level VAM –districtwide in the coming year, and the local teachers union
vocally opposing this choice, both the EGDC pilot and the release of teacher-level VAM by
LAUSD garnered significant media attention (Llanos, 2011; Singh, 2012). These actions ensured
that all teachers in LAUSD felt the threat of increased accountability in light of the EGDC pilot,
but only some teachers –those eligible to get VAMs –received any additional test-based
information about their practice. In this environment, we may expect teachers with access to
VAMs to react differently within the teacher labor market than those who similarly felt the
accountability threat from the EGDC pilot, but lacked access to this same information.
91
Serving as the backdrop to LAUSD’s release of VAM was the controversial and very
public release of a separate database of teacher-level VAM scores by the Los Angeles Times. In
2010-2011, the Times hired a consultant to calculate individual teacher VAMs for elementary
teachers in Grade 3 through Grade 5 and reported them, linked to teachers’ names and schools, in
an online searchable database.
37
Although the release of the Times’ VAM database was
completely unauthorized by the district, and carried no weight in teacher performance decisions,
the value-added scores produced by the Times were well known among teachers in LAUSD.
38

The Times’ release of VAM scores at the beginning of the 2010-2011 school year provided a
form of highly public accountability for teachers who received VAMs and their schools, whereas
the district’s private release of VAM scores alongside the EGDC pilot the next year, provided
these teachers with a very different form of accountability. This form of accountability was much
more private given that teachers’ scores were not linked to their names and were never shared
publicly, but was also more direct as teachers in tested grades and subjects as well as their
administrators now had access to much more information about their performance than ever
before. Given the differences in the nature of accountability introduced by the public versus
private release of VAM, we may expect teachers to respond differently to the two VAM releases.
The environment in Los Angeles during the 2010-2012 school years provides the
opportunity to examine the relationship between SBMMTES and the teacher labor market in two
unique ways. First, the LAUSD case affords an explicit examination of how teachers with access
to different types of information about their performance may respond differently to additional
accountability brought about by the implementation of a SBMMTES pilot and the implications,

37
It is important to note that soon after the initial release, the Los Angeles Times updated this database to include
teacher-level VAM for teachers in Grade 3 through Grade 8.
38
Interviews with district leaders responsible for training teachers on the districts own teacher-level VAM noted
significant concerns among teachers about the Times’ database.
92
if any, for the quality of teachers in schools and classrooms within LAUSD. Second, because the
release of VAM in light of the EGDC pilot succeeded a more public release of value-added
scores for elementary teachers in LAUSD by the Los Angeles Times, the context for teachers in
LAUSD provides the opportunity to examine how the relationship between teacher mobility and
the provision of additional performance information, may differ based on whether that
information is a matter of public record or provided confidentially to teachers and administrators.
Moreover, it affords an examination of the implications of any differences in teacher response to
these two types of accountability for the composition of the teacher labor market. This has
important implications for policy as states and districts across the country are currently designing
and implementing SBMMTES with little insight into the implications of the choices they make
with respect to each of these design features on the broader teacher labor market.
In what follows, I review the research questions that guide a formal investigation of these
issues in LAUSD. Then, I describe the data and sample as well as the methodological approach
used to generate these findings, and provide the results. I conclude with the implications of this
work for research, policy, and practice, with respect to the design of SBMMTES inclusive of a
value-added measure of teacher performance.

Research Questions

In this paper, I examine the differential impact of receiving additional performance
information on teacher mobility in response to the threat of increased accountability for
performance. Moreover, I examine how, if at all, teacher mobility might differ given how this
additional performance information is released –whether publicly to parents and communities, or
privately to teachers and administrators in the context of a SBMMTES pilot. Specifically, I ask:
93
1. Are teachers eligible to receive VAMs more likely to switch schools/classrooms
or leave the district after the public release of value-added scores, or the
implementation of a SBMMTES pilot, relative to teachers who are ineligible to
receive VAMs?
2. Does this relationship differ among teachers in “hard-to-staff” contexts and/or
based on how this new performance information is released (e.g., publicly to
parents and communities or privately to teachers and administrators)?
Moreover, to begin to build an understanding of the implications of any differential
trends in teacher mobility, I ask:
1. How, if at all, do changes in teacher mobility associated with access to additional
information about teacher performance through VAM influence the composition
of the teacher labor market locally?
2. Do the compositional implications differ based on how this new performance
information is released?
39

Overall, I find that teachers’ school- and classroom-level mobility does differ among
those with access to VAM, relative to those without access to this additional performance
information. Moreover, I find suggestive evidence that whether this additional information is
released publicly or privately in the context of a SBMMTES matters, with elementary teachers
demonstrating different mobility behavior after the Times’ VAM release relative to the LAUSD
release. Finally, descriptive evidence suggests that at least at the elementary level, how teacher
performance information is released may have different implications for the composition of the

39
Because the public release of teacher-level VAM in the Times’ database applied only to teachers in Grades 3-5, I
restrict any comparisons between teacher mobility in response to the public versus private release of teacher-level
VAM to elementary teachers, and examine the relationship between receiving additional performance information
on teacher mobility in response to a SBMMTES pilot for elementary and secondary teachers.

94
teacher labor market in LAUSD. Together, these findings provide early evidence that teachers
with access to additional performance information do respond to the threat of additional
accountability, and that the nature of this threat may matter for who teaches and where.
It is important to note from the outset, that I examine teacher mobility in response to the
threat of increased accountability in the context of a SBMMTES pilot implemented in a pilot/no-
stakes environment. As such, the results provided here may differ from those observed when
considering teacher response to similar reforms in a high-stakes environment. Despite this
context, the results still provide critical signals of a likely lower bound of how access to
additional performance information may impact the mobility of teachers in response to
accountability within a SBMMTES when fully implemented for stakes. Given the prevalence of
SBMMTES pilots inclusive of value-added measures of teacher effectiveness across the country,
and the limited information available on how teachers might respond differently to this
information based on how it is released, the evidence provided here can help policymakers
anticipate potential responses by teachers (and possibly administrators) to several critical design
features embedded within SBMMTES, even in the pilot stages of these new reforms.

Data and Sample

To answer the research questions outlined above I rely on the same five-year panel of
teacher, student, and school-level administrative data from LAUSD, outlined in Chapter Three.
However, given differences in the research questions I rely on a different sample of treatment
and comparison teachers as well as slightly different definitions of the outcome variables in all
analyses. In what follows, I describe these differences in greater detail.
95
Measuring Mobility. Similar to Chapter Three, the primary outcomes of interest in the
analyses outlined below are “teacher-initiated” mobility decisions at the school- and classroom-
level.

Therefore, I exclude from the analysis sample all teachers who were laid off or retired at
the end of each year, as well as any teacher who experienced mobility resulting from a RIF.
Similarly, I restrict the sample to teachers in traditional public schools linked to school sites
and/or classroom placements in the current and following year for school year 2007-2008
through 2011-2012 (stayers and switchers), as well as teachers who exit the dataset and do not
return (leavers). This leaves approximately 62,000 elementary teacher-year observations with
16,000 unique teachers, and approximately 55,000 secondary teacher-year observations with
15,500 unique teachers, for all analyses (See Appendix C, Table 1 for sample sizes).
School-Level Mobility. At the school-level, I am interested in understanding whether or
not a teacher with access to additional performance information is more or less likely than a
teacher without access to this same information, to move schools within LAUSD when faced
with the threat of additional accountability for performance. Moreover, for elementary teachers, I
am interested in understanding how teacher mobility might differ when teacher performance
information is released publicly as opposed to privately within the context of a SBMMTES. To
capture teachers’ school-level mobility decisions in response to each of these events of interest, I
measure whether a teacher stays in his or her current school, switches to another school within
the district, or leaves LAUSD at the end of each year. Unlike Chapter Three, I am not limited by
sample size and can examine each of these distinct mobility choices separately, rather than
combining switch and leave decisions into a single mobility event.
Classroom-Level Mobility. At the classroom-level I am again primarily interested in
whether or not a teacher changes classroom placements from one that is eligible to receive
96
additional performance information (e.g., VAM) to one that is not, or vice versa. Given that
eligibility to receive VAM is based on a teacher’s grade-level or subject placement I consider all
elementary teachers in tested grades (3-6), and all secondary teachers who teach at least one
tested course, as in a tested placement, and all other teachers as in non-tested placements.
For all elementary analyses, the classroom-level mobility outcome captures whether or
not a teacher stays in their current placement (i.e., teaches in a tested grade or non-tested grade
and remains in a tested or non-tested grade, respectively, the following year), switches classroom
placements (i.e., moves from a tested grade into a non-tested grade or vice versa either within the
same school or at a different school, the following year), or leaves LAUSD altogether (i.e., drops
from the dataset in the following year). At the secondary level, I collapse switching and leaving
into a single “moving” category. I take this approach for a two reasons. First, as outlined in
Chapter Three, when making a decision about whether or not to stay in their current classroom
placement, secondary teachers face a credentialing constraint not faced by elementary teachers.
Second, because secondary teachers often teach both tested and non-tested courses
simultaneously, and I am forced to assign a teacher to a single tested or non-tested designation,
examining the extent to which secondary teachers switch classroom placements masks important
variations in the contexts in which these teachers work.
40
In an effort to partially address these
barriers to mobility for secondary teachers at the classroom-level, I choose to examine the
probability that they will stay in their current placement, or move placements (either switching
or leaving) at the end of each year.

40
For example, in my data only 3% of secondary teachers teach only a tested subject, leaving a sample size that does
not permit meaningful analyses. Moreover, 54% of secondary teachers teach courses in both tested and non-tested
grades/subjects simultaneously (e.g., teaching Art and US History). This makes it difficult to capture switching into
and out of tested placements as the majority of secondary teachers do both simultaneously.
97
Analytic Approach

As outlined above, the purpose of this analysis is to determine whether or not tested
teachers, who receive additional information about their performance through VAM, respond
differently to the threat of increased teacher-level accountability than do non-tested teachers,
who do not receive additional information about their performance through VAM. In addition, I
am interested in understanding how teacher mobility might differ when teacher performance
information is released publicly versus privately. Because how teachers respond to each of these
events has implications for how teachers are distributed across schools and classrooms, I am also
concerned with changes in the mobility patterns of all teachers within LAUSD as well as with
teachers in “hard-to-staff” contexts, including those in low-performing, high-poverty, or high-
English Learner schools and/or classrooms. In addition, I am interested in the implications of any
observed changes on the characteristics of those who teach in LAUSD.
The Difference-in Difference Framework. To answer the research questions, I rely on
a similar analytic approach as outlined in Chapter Three. In particular, I rely on separate
difference-in-difference (DID) analyses for elementary and secondary teachers at both the
school- and classroom-level that account for differences in the outcome variable.
For elementary teachers, I use a series of multinomial logistic regression models within a
difference-in-difference (DID) framework to examine the differential impact of receiving
additional performance information (through VAM) on teachers’ school- and classroom-level
mobility. Because I am also interested in how teacher mobility may differ depending on how
teacher performance information is released, I also compare elementary teacher mobility in the
year the Times’ VAM database went online (2010-2011), and in the year LAUSD privately
shared teacher-level VAM (2011-2012), to the period in which value-added information about
98
teacher performance did not exist (2009-2010). For secondary teachers, I use a similar approach
at the school-level, but at the classroom-level I rely on a series of bivariate logistic regression
models in a DID framework that account for the binomial distribution of the outcome variable.
Because secondary teachers were not part of the Times’ VAM release, at this level, I am
primarily concerned with the differential impact of receiving additional performance information
on teacher mobility in response to the accountability threat introduced by the EGDC pilot.
Treatment and Comparison Group. As described in Chapter Three, the DID approach
relies on the use of a comparison group in order to parse out the effect of a treatment on a given
outcome of interest. Identifying a valid comparison group relies on the parallel trends assumption
which in the case of this study suggests that the mobility decisions of those in the comparison
group are expected to be “comparable” to those that would have been made by teachers in the
treated group in the absence of treatment (Wooldridge, 2010). For the analyses below, the
specific treatment under study is the extent to which receiving additional information about
teacher performance differentially influences teacher response to the threat of increased
accountability brought on by the EGDC pilot, and for elementary teachers, how teacher mobility
might differ when performance information is released publicly as compared to privately.
Therefore, I consider all teachers in tested grades and subjects in LAUSD (who are eligible to
receive VAM) as in the treatment group.
41

Teachers in non-tested grades and subjects are not eligible to receive VAM and therefore
respond to the threat of additional accountability for performance and the public release of VAM
in the absence of additional performance information, making them a conceptually strong

41
It is important to note that in 2011-2012 LAUSD did not release individual-VAM for teachers in Grade 5 Science,
Grade 8 Social Science, and Grades 10 and 11 ELA. However, because these teachers were under the impression
that they would receive VAM at the end of the year, I can assume that they still expected to receive this additional
performance information and that the treatment is unaffected by this event.
99
comparison group. However, we may expect that teachers in non-tested grades and subjects
differ systematically from tested teachers, particularly with respect to their mobility behavior.
This is because a teacher’s mobility options may differ based on a number of factors, including
the grade/subject in which he or she teaches, and or his or her credential area.
As outlined in Tables 2 and 3 in Appendix C, across all years both elementary and
secondary teachers in tested grades and subjects are significantly different from teachers in non-
tested grades and subjects across a number of observable characteristics. Specifically, elementary
and secondary teachers in tested grades and subjects are significantly more likely than those in
non-tested grades and subjects to be in the earlier years of their career (p<0.01 to p<0.001).
Moreover, at the elementary-level, teachers in tested grades and subjects are significantly less
likely to be female and minority than are those in non-tested grades and subjects (p<0.001). In
contrast, at the secondary-level tested teachers are significantly more likely to be female and
minority than are their non-tested counterparts (p<0.01 to p<0.001). I also observe some
systematic differences in quality among teachers in tested relative to non-tested grades and
subjects. In some years tested elementary teachers are significantly more likely to have National
Board Certification [NBC] (p<0.10 to p<0.001) than are non-tested teachers. In addition, with
the exception of 2008-2009, secondary teachers in tested grades and subjects are significantly
more likely to have higher numbers of needs improvement or below standard ratings on the Stull
evaluation than are those in non-tested grades and subjects (p<0.01 to p<0.001).
Although teachers in tested grades and subjects differ on observable characteristics from
those in non-tested grades and subjects, the mobility rates of teachers in each group at both the
school- and classroom-level have relatively similar trends in the years prior to the release of the
Times’ database and the implementation of the EGDC pilot in the full sample (see Appendix C,
100
Figures 1 to 16). As we might expect, these trends are slightly more variable at the classroom-
level but are generally parallel in the years prior to treatment, with a few exceptions. For
elementary teachers with students in the bottom third of proficiency we see noticeable
differences in mobility for teachers in tested grades and subjects relative to those in non-tested
grades and subjects in the years leading up to the treatment period, raising concerns about the
validity of this particular comparison group within this sample (see Appendix C, Figure 6).
Similarly, for secondary teachers, we see more variable trends across the board, suggesting that
in these samples this comparison is more tenuous (see Appendix C, Figures 9 to 16).
Given the parallel trends in mobility for teachers in tested relative to non-tested grades
and subjects in the majority of our samples of interest, I rely on the latter as the comparison
group and include appropriate controls in all models to account for any observable differences
between tested and non-tested teachers over time. Moreover, I consider all results produced from
the samples in which the parallel trends assumption does not hold as non-causal estimates of the
treatment effect and interpret them with caution when discussing the results below.
Model Specifications for Elementary Analyses. As described above, at the elementary-
level I run separate analyses that compare the school-and classroom-level mobility rates of
teachers in tested grades relative to those of teachers in non-tested grades both before and after
the threat of additional accountability for performance went into effect with the implementation
of the EGDC pilot. Moreover, I examine how mobility rates for these teachers might differ after
the Times’ database publicly released teacher-level VAM compared to LAUSDs private release
of teacher-level VAM in the context of the EGDC pilot. This is formalized below:
(1) l

∑
&
'

(
&)

*
,

∑
&
'

&)+
,

!

"
At the school-level,

is a categorical variable that equals zero if elementary
teacher i, in classroom j, in school s, in year t, stays in his/her school, equals one if he or she
101
switches schools, and equals two if he or she leaves the district. At the classroom-level,

is a categorical variable that equals zero if elementary teacher i, in classroom j, in school s, in
year t, stays in his/her current tested or non-tested placement, equals one if he or she switches
placements (i.e., moves from a tested to non-tested placement or vice versa), and equals two if he
or she leaves LAUSD. In both cases, the model is estimated relative to the base outcome of
staying in the same school or classroom placement, respectively.
Across all models, the probability of a teacher staying in his or her current
school/classroom placement, switching schools/classroom placements, or leaving LAUSD at the
end of the year is a function of time -measured by a vector of year indicators for 2007-2008 to
2011-2012 (∑
&
(
&)
'

), the same school (!

) and teacher/classroom (

) controls as used
in Chapter Three
42
, treatment –measured by an indicator equal to one for teachers in tested
grades and equal to zero for teachers in non-tested grades (,

), and the interaction of
time and treatment (∑
&

&)+
'

,

) which captures the differential mobility of
tested relative to non-tested teachers in each year after controlling for individual teacher,
classroom, and school characteristics as well as secular trends in the teacher labor market.
In this model, the coefficients of interest are the interaction between SY 2010-2011 and
,

, which provides an estimate of the differential impact of the public release of VAM
by the Times on the mobility of teachers in tested grades and subjects, as well as the interaction
between SY 2011-2012 and ,

, which provides an estimate of the differential impact of
the receiving additional performance information –through the private release of VAM –on the

42
There is one small difference in the controls used here relative to those used in Chapter Three. In this set of
analyses I remove the control for whether or not a teacher is in a tested grade or subject, replacing it with an
indicator for whether or not a teacher participated in the EGDC pilot. I do this because teachers who participated
directly in the pilot may have received other new information about their practice, or had a different experience
consuming the information within a teacher-level VAM.
102
mobility of teachers in tested grades and subjects in response to the threat of increased
accountability brought about by the EGDC pilot. As in Chapter Three, to account for the fact that
interactions are calculated in multiplicative terms when using non-linear models, I compute the
marginal differences for every model specification.
In the school-level models using the full sample of LAUSD teachers, an increase or
decrease in switching schools or leaving LAUSD for elementary teachers in tested grades
indicates the differential relationship between a tested teacher’s decision to move away from his
or her current school context relative to staying, in light of the accountability threat introduced
by the EGDC pilot or the public release of VAM by the Times. For the “hard-to-staff” analyses I
run conditioned models that compare the mobility of tested and non-tested teachers in the bottom
third of the “hard-to-staff” school samples including: 1) low-performing schools as measured by
API, 2) high % ELL schools, and 3) high %FRL schools.
43
In these samples, any increase or
decrease in switching schools or leaving LAUSD can be interpreted as the differential impact of
receiving additional performance information on the mobility of teachers in tested grades and
subjects within these “hard-to-staff” schools relative to teachers in non-tested grades and subjects
teaching in similarly “hard-to-staff” school contexts.
44

In the classroom-level models using the full sample of LAUSD teachers, an increase or
decrease in switching placements or leaving LAUSD for elementary teachers in tested grades
indicates the differential relationship between a tested teacher’s decision to move away from his
or her current tested or non-tested placement relative to staying, in light of the accountability

43
I also ran specifications of all models using different definitions of “hard-to-staff” contexts for low-performing
school and high-proportion ELL schools including top and bottom quartile as well as top and bottom 50% and find
that my results remain robust across specifications. I am unable to run similar specification checks for the “hard-to-
staff” high-poverty schools as there is not sufficient variation in this variable across years, with the distribution
heavily skewed by the large number of elementary schools in LAUSD with 100% students in poverty.
44
Given the small sample of teachers who are both tested and in hard-to-staff contexts I am unable to model these
differential effects explicitly by including an indicator for whether or not the teacher is in a hard-to-staff context and
then interacting this indicator with the post-treatment period and the tested indicator.
103
threat introduced by the EGDC pilot, or the public release of VAM by the Times. For the “hard-
to-staff” analyses I run conditioned models that compare the mobility of tested and non-tested
teachers in the bottom third of the hard-to-staff classroom samples including: 1) low-performing
as measured by CST proficiency rates, 2) high % ELL schools, and 3) high %FRL schools.
45
In
these samples, any increase or decrease in switching classroom placements or leaving LAUSD
can be interpreted as the differential impact of receiving additional performance information on
the mobility of teachers in tested grades and subjects within these “hard-to-staff” classrooms
relative to non-tested teachers teaching in similar contexts.
46

Across all specifications at both the school- and classroom-level, we would ideally see no
significant differences in the mobility rates of teachers in tested relative to non-tested grades in
all years prior to each period of interest and the treatment period itself. This would suggest that
any significant differences in the mobility rates of teachers in tested relative to non-tested grades
observed between SY 2010-2011, the year the Times publicly released teacher-level VAM, and
all pre-treatment years (SY 2007-2008 to SY 2009-2010) are likely a function of teachers’
response to the public release of teacher-level VAM by the Times. Similarly, significant
differences in the mobility rates of teachers in tested relative to non-tested grades between SY
2011-2012, the year LAUSD implemented the EGDC pilot and privately released teacher-level
VAM, and all pre-treatment years (SY 2007-2008 to SY 2009-2010) would indicate that any
differential relationship observed in 2011-2012 captures tested teachers’ mobility in response to
the threat of additional accountability for performance under the EGDC pilot.

45
I also ran specifications of all models using different definitions of “hard-to-staff” contexts for low-performing
school and high-proportion ELL schools including top and bottom quartile as well as top and bottom 50% and find
that my results remain robust across specifications. I am unable to run similar specification checks for the “hard-to-
staff” high-poverty schools as there is not sufficient variation in this variable across years, with the distribution
heavily skewed by the large number of elementary schools in LAUSD with 100% students in poverty.
46
As above, I also ran specifications of all models using different definitions of “hard-to-staff” contexts for low-
performing classrooms, classrooms with high-proportions ELLs, and high-proportions of students in poverty
including top and bottom quartile% as well as 50% and find that my results remain robust across specifications.
104
To determine if the difference between the observed mobility rates for tested relative to
non-tested teachers in the years of interest (2010-2011 and 2011-2012) and those observed in the
years prior to the release of any teacher-level VAM are in fact significantly different, I run Wald
tests. As explained in Chapter Three, these tests produce DID estimates for each pre- and post-
treatment year combination, separately. If they return significant differences, than the effect
observed post-treatment can in fact be attributed to differences in teacher response to the
treatment itself. If they are not significant, we cannot reject the null hypothesis that factors
unrelated to the treatment of interest did not induce differential mobility for teachers in tested
grades and subjects relative to their non-tested counterparts over the same period.
Model Specifications for Secondary Analyses. At the school-level, the secondary
analyses are identical to those used in the elementary analyses; I simply replace the elementary
sample with the sample of secondary teachers. However, at the classroom-level, the secondary
analyses differ somewhat. The primary difference is the interpretation of the outcome variable.
For the secondary classroom-level analyses,

is a binary variable that equals zero if
teacher i, in classroom j, in school s, in year t, stays in his/her current tested or non-tested
placement and equals one if he or she moves away from his/her tested or non-tested placement
(i.e., switches from a tested to a non-tested placement or vice versa, or leaves LAUSD in the
following year). At both the school- and classroom-level, I run separate specifications of Model
1 using the full sample of secondary teachers as well as those in “hard-to-staff” schools and
classrooms, and run Wald tests to determine if the difference between the observed relationship
in each period of interest and those observed in prior years are in fact significantly different.
Descriptive Analyses. Although the above analyses provide insight into the relationship
between teacher mobility and the threat accountability brought on by the EGDC pilot, as well as
105
the public versus private release of teacher performance information, they provide little insight
into the implications this may have for the teacher labor market. To shed light on this issue, I
supplement the above analyses with descriptive analyses. For elementary teachers at both the
school- and classroom-level I examine who stays, switches, and leaves LAUSD at the end of SY
2010-2011 when the Times’ database was released, and at the end of SY 2011-2012 when
LAUSD privately released teacher-level VAM in the context of the EGDC pilot. Differences in
the characteristics of stayers, switchers, and leavers at the end of each of these years will provide
insight into the implications of providing different types of teachers with access to different
information about their performance on the composition of the teacher labor market, and how
these implications might differ if teacher effectiveness data is publicly versus privately released.
For secondary teachers, I examine the characteristics of those who stay in the same school,
switch schools, or leave LAUSD at the end of each treatment year, and do the same at the
classroom-level for those who stay in the same placement or move placements.
Limitations. It is important to note, that the findings presented below are not without
limitations. Specifically, these results provide initial evidence about the relationship between
implementing a SBMMTES inclusive of a value-added measure of performance and teacher
school- and classroom-level mobility within LAUSD, and how mobility behavior might differ
when teacher performance information is released publicly to parents and the community, as
opposed to privately to teachers and administrators. Perhaps most importantly, because the
parallel trends assumption is violated in a number of samples, the results presented here do not
provide causal evidence of these impacts across all contexts.
Moreover, because teacher-level VAM was released by LAUSD as a formative measure
of teacher performance during the period under study, and the VAM released by the Times had
106
no bearing on teacher performance decisions, I cannot confidently generalize the findings
presented below to contexts in which VAM is released as part of a comprehensive evaluation
system for the purposes of teacher accountability. Finally, the findings presented below are also
limited in the extent to which they can speak to the differential effect of the threat of high-stakes
accountability for teacher performance on the mobility of secondary teachers in tested grades and
subjects across different types of school and classroom contexts within LAUSD. Because I was
forced to classify secondary teachers into a tested or non-tested designation, I necessarily mask
the true contexts in which these teachers work. This could help to partially explain the lack of
parallel trends between teachers in tested grades and subjects and those in non-tested grades and
subjects across all samples at the secondary level. Future research would benefit from a more
nuanced exploration of secondary teacher course loads that better captures secondary teachers’
“true” opportunity for transfer between tested and non-tested classroom placements.

Results

The results presented below demonstrate that the threat of additional accountability for
teacher performance introduced through the implementation of the EGDC pilot did not
differentially influence the mobility of elementary teachers in tested grades at the school- or
classroom-level but was associated with differential mobility among secondary teachers in tested
grades and subjects at the school-level. Moreover, they show that elementary teachers eligible to
receive VAM scores exhibit different classroom-level mobility behavior when this information is
released publicly than when this information is released privately. While the public release of
VAM increased teacher turnover out of tested grades and subjects in hard-to-staff contexts, the
private release of this same information actually increased retention of these same elementary
107
teachers at both the school and classroom-level. Interestingly, although secondary teachers were
not directly impacted by the Times’ VAM database, these teachers demonstrated reduced
turnover at the school-level after the public release of teacher-level VAM. Together, these
findings suggest the need for careful consideration of the design of SBMMTES that provide
different types of teachers with different information about their practice, as both what
information is available, and how it is released, may contribute to the differential mobility of
different types of teachers across schools and classrooms, with implications for the quality of
teachers in the local teacher labor market.
Elementary School-Level Mobility. Figures 1 to 4 in Appendix C graphically illustrate
the school-level mobility decisions of all elementary teachers in tested grades relative to those in
non-tested grades within LAUSD overall, and for just those teachers in “hard-to-staff” schools.
Looking more closely at teacher mobility in SY 2010-2011 –after the public release of teacher-
level VAM –and SY 2011-2012 –the year the EGDC pilot was implemented –we also see
relatively consistent trends between the treatment and control groups. However, in the full
sample we do see some small deviations in the proportion of teachers leaving in tested relative to
non-tested grades after the implementation of the EGDC pilot. This suggests that threat of high-
stakes accountability brought on by the EGDC pilot may have contributed to small differences in
tested teachers’ decisions to leave LAUSD altogether. It also provides suggestive evidence that
that the public release of teacher-level VAM may not have differentially influenced the school-
level mobility of tested teachers, overall or in “hard-to-staff” schools.
Table 4 in Appendix C formalizes this descriptive evidence, providing the Wald tests of
the DID estimates of tested teachers’ school-level mobility relative to those of non-tested
teachers produced from Model 1. The top panel of Table 4 provides the DID estimates of the
108
proportion of teachers who switched schools relative to the base outcome of staying in his or her
current school for teachers in tested relative to non-tested grades between SY 2007-2008 to SY
2009-2010 (prior to the implementation of the SBMMTES pilot) and SY 2011-2012 (after the
SBMMTES pilot), and between SY 2007-2008 to SY 2009-2010 (prior to the Times’ release)
and SY 2010-2011 (after the Times’ release). The bottom panel provides the same information
for the proportion of teachers who left LAUSD relative to staying in their current school. For
both switching and leaving, I exclude the DID estimate between SY 2011-2012 and SY 2010-
2011 as teacher-level VAM was already available to teachers in this period, muddying the
treatment effects. All estimates are reported as predicted probabilities and can be interpreted as
percentages. For example, Column 1 in the top panel of Table 4 shows teachers in tested relative
to non-tested grades were 0.03% less likely to switch schools after the implementation of the
SBMMTES pilot than before (2009-2010), although this difference is not significant.
Across all samples we see that elementary teachers in tested grades relative to non-tested
grades appear to be slightly less likely to switch schools or leave LAUSD after the
implementation of the EGDC pilot than in the year when no VAM was available (2009-2010),
although none of these differences are statistically significant (see top panel of Appendix C,
Table 4). We do see that in low-performing schools, tested relative to non-tested teachers were
1.7% less likely to switch schools after the implementation of the EGDC pilot than in the year
before VAM was introduced. However, this difference is only significant at the p<0.10 level.
In contrast, we do see some evidence that the public-release of teacher-level VAM by the
Los Angeles Times differentially influenced the mobility of elementary teachers in tested relative
to non-tested grades, at least among teachers in “hard-to-staff” schools. Specifically, in the top
panel of Table 4 we see that among teachers in low-performing schools those in tested grades
109
relative to those in non-tested grades were 2.1% less likely to switch schools after the Times’
database release (p<0.05) than in the year prior (2009-2010). Similarly, in schools with high
proportions of ELLs tested elementary teachers, relative to their non-tested counterparts, were
1.8% less likely to switch schools after the public release of teacher-level VAM than in the
previous school year, although this is only significant at the p<0.10 level.
Together these results provide suggestive evidence that teachers with access to additional
performance information did not respond differently to the threat of accountability embedded
within the EGDC pilot. However, the public release of VAM did influence teacher mobility at
least in “hard-to-staff” schools, decreasing the rates at which tested teachers switched away from
these low-performing, high-ELL school contexts. This finding is somewhat surprising as we may
expect that teachers would be more likely to switch away from hard-to-staff schools after the
public release of VAM. This is because teachers may perceive that these contexts make it more
difficult to receive a high-VAM given the challenges associated with bringing lower-performing
students to grade-level and differentiating instruction to English Learners. However, if teachers
in these schools received high-VAM under the Times’ database, or were higher performing to
begin with, then perhaps we may expect them to be more likely to stay given their success in
these contexts. While the analyses presented here do not afford insight into the extent to which
the former is in fact the case, I explore this second possibility in the descriptive analyses below.
Elementary Classroom-Level Mobility. Turning to the classroom-level, Figures 5 to 8
in Appendix C provide descriptive evidence of the classroom-level mobility decisions of
elementary teachers in tested grades relative to those in non-tested grades both within LAUSD
overall, and for just those teachers in “hard-to-staff” classrooms. Here we see noticeable jumps
in the likelihood of switching placements, among teachers in tested grades in 2010-2011, the
110
year the Times released teacher-level VAM publicly. In addition, we see small but noticeable
differences in the rates of switching, and leaving for teachers in tested grades relative to those in
non-tested grades after the implementation of the EGDC pilot.
Table 5 in Appendix C provides the Wald tests of the DID estimates of tested teachers’
classroom-level mobility relative to that of non-tested teachers produced from Model 1. The top
panel of Table 5 provides the DID estimates of the proportion of teachers who switched
placements (either from a tested to non-tested placement or vice-versa) relative to the base
outcome of staying in his or her current placement for teachers in tested relative to non-tested
grades between SY 2007-2008 to SY 2009-2010 (prior to the implementation of the EGDC pilot
or release of teacher-level VAM) and SY 2011-2012 (after the EGDC pilot), and between SY
2007-2008 to SY 2009-2010 (prior to the Time’s release) and SY 2010-2011 (after the Times’
release). The bottom panel provides the same for the proportion of teachers who left LAUSD
relative to staying in their current classroom placement. As above, the estimates are reported as
predicted probabilities and therefore can be interpreted as percentages.
Similar to the school-level results, we see that in the full sample, tested relative to non-
tested elementary teachers appear to be no more or less likely to switch out of tested placements
after the implementation of the EGDC pilot than in the year before any VAM was available. We
do see some evidence of a differential response to the threat of increased accountability by tested
teachers in low-performing and high-ELL schools. However, given the concerns raised above
with the comparison groups in these samples I do not explore these further.
Interestingly, we also see that in the full sample and high-proportion ELL sample, after
the Times publicly released VAM, elementary teachers in tested grades relative to non-tested
grades were 2% and 3% more likely to switch classroom placements than in the prior year (2009-
111
2010) (p<0.05). This suggests that overall and in classrooms with high-proportions of ELLs,
tested teachers may have responded to the Times’ VAM release by switching into non-tested
placements in the next year. However, as above we must be cautious in interpreting the latter of
these two results given concerns raised earlier with comparing the mobility of tested and non-
tested teachers in the hard-to-staff classroom sample.
Turning to the bottom panel of Table 5, we see a much different picture for tested
teachers relative to non-tested teachers in their decisions to leave LAUSD versus stay in their
classroom placement after the implementation of the EGDC pilot and private release of VAM.
Specifically, we see that in the full sample, elementary teachers in tested grades relative to those
in non-tested grades were 1.0% less likely to leave LAUSD after the implementation of the
EGDC pilot than in the pre-treatment period (SY 2009-2010), a difference that is significant at
the p<0.05 level. Similarly, while not causal, we see that teachers in high-proportion ELL
classrooms were 2.1% less likely to leave LAUSD after the private release of VAM (p<0.05) and
1.4% less likely than non-tested teachers to leave after the Times’ database release than in the
year prior (SY 2009-2010), although this difference is only significant at the p<0.10 level.
Together, these results demonstrate that the public release of VAM by the Los Angeles
Times appeared to differentially increase the probability that tested teachers in the full sample
would switch classroom placements, but had no differential influence on the leave decisions of
these same teachers. In contrast, the implementation of the EGDC pilot appeared to have no
differential influence on the switching behavior of teachers in tested grades relative to those in
non-tested grades in the full sample, but did appear to decrease their likelihood of leaving the
district altogether. This suggests that elementary teachers responded differently to the public
release of VAM relative to the private release of VAM. In particular, we see that while the public
112
release of VAM appeared to induce turnover out of tested placements, the private release
increased retention in these same placements, at least in the full sample of teachers in LAUSD.
Secondary School-Level Mobility. Turning to the secondary-level, Figures 9 to 12 in
Appendix C graphically illustrate the school-level mobility decisions of all secondary teachers in
tested grades and subjects relative to those in non-tested grades and subjects within LAUSD
overall, and for just those teachers in “hard-to-staff” school contexts. As discussed above, in
almost all samples, the parallel trends assumption does not hold. This variability in mobility
patterns among tested relative to non-tested secondary teachers prior to the implementation of
the SBMMTES pilot, indicates that the results presented below must be interpreted with caution
as this comparison appears to be more problematic than at the elementary-level.
Table 6 in Appendix C provides the Wald tests of the DID estimates of tested secondary
teachers’ school-level mobility relative to that of their non-tested counterparts produced from
Model 1 for the full and “hard-to-staff” school samples. As above, the top panel of Table 6
provides the DID estimates of the proportion of teachers who switched schools relative to the
base outcome of staying in their current school for teachers in tested relative to non-tested grades
between SY 2007-2008 to SY 2009-2010 (prior to the implementation of the EGDC pilot or
Times’ VAM release) and SY 2011-2012 (after the EGDC pilot), and between SY 2007-2008 to
SY 2009-2010 (prior to the Times’ VAM release) and SY 2010-2011 (after the Times’ release),
while the bottom panel provides the same for the proportion of teachers who left LAUSD
relative to staying in their current school. As in the elementary analyses, I exclude the 2010-2011
and 2011-2012 DID comparison and report all estimates as predicted probabilities.
Turning to the top panel of Table 6, we see that across the full sample, and sample of
teachers in high-proportion ELL and FRL schools, secondary teachers in tested grades and
113
subjects relative to those in non-tested grades and subjects are significantly less likely, on
average, to switch schools after the implementation of the SBMMTES pilot and the Times’
VAM release than in the pre-treatment period (2009-2010). Similarly, turning to the bottom
panel of Table 6, we see that across the full and low-performing schools samples tested relative
to non-tested teachers are significantly less likely to leave LAUSD after the implementation of
the EGDC pilot than in the pre-treatment period. This is not the case after the public release of
VAM which appears to have had no differential impact on the leave decisions of teachers in
tested relative to non-tested grades and subjects. Together, these results suggest that secondary
teachers with access to additional performance information did respond differently to the
accountability threat brought about by the implementation of the EGDC pilot than did their non-
tested counterparts, and that this response was similar to that observed after the public release of
teacher level VAM at least in terms of switching schools. However the lack of parallel trends
makes drawing any substantive conclusions from these results challenging.
Secondary Classroom-Level Mobility. Turning to the classroom-level, Figures 13 to 16
in Appendix C outline the classroom-level mobility decisions of secondary teachers in tested
grades and subjects relative to those in non-tested grades and subjects both within LAUSD
overall, and for just those teachers in “hard-to-staff” classrooms. Across all samples, we see
similar variation in the classroom-level mobility trends between these two groups as observed at
the school-level. Importantly, we see noticeable differences in the classroom-level mobility of
tested relative to non-tested secondary teachers over time, with particular divergence beginning
in 2009-2010 and evening out by 2011-2012. This again generates concern with comparing
tested and non-tested secondary teachers, and suggests the results be interpreted with caution.
114
Table 7 in Appendix C provides the Wald tests of the DID estimates of secondary tested
teachers’ classroom-level mobility relative to that of non-tested teachers produced from Model 1.
Specifically, Table 7 provides the DID estimates of the proportion of tested relative to non-tested
secondary teachers who moved placements (either switching placements or leaving LAUSD),
relative to the base outcome of staying in their current placement, between SY 2007-2008 to SY
2009-2010 (prior to the implementation of the SBMMTES pilot or the Times’ VAM release) and
SY 2011-2012 (after the SBMMTES pilot), and between SY 2007-2008 to SY 2009-2010 (prior
to the Times’ VAM release) and SY 2010-2011 (after the Times’ release). Again, the estimates
are reported as predicted probabilities and can be interpreted as percentages.
Here we see no evidence that the implementation of the EGDC pilot differentially
influenced the classroom-level mobility of secondary teachers with access to additional
performance information. Specifically, we see that tested secondary teachers are no more or less
likely to move placements than are their non-tested counterparts in the year after the
implementation of the EGDC pilot (SY 2011-2012) relative to the pre-treatment period (SY
2009-2010). Moreover, we see no evidence of a differential response among tested secondary
teachers to the public release of teacher-level VAM. This suggests that secondary teachers’
classroom-level mobility decisions were not differentially influenced by the threat of additional
accountability for performance or the public release of teacher performance information.
However, given the concerns raised above about comparing the mobility of tested relative to
non-tested secondary teachers, these results must be interpreted with caution.
Descriptive Results. Although interesting in themselves, the results discussed above do
not provide insight into the implications of any observed changes in teacher mobility for the
composition of the teacher labor market in LAUSD. Understanding these implications for the
115
types of teachers in LAUSD is of significant policy relevance as two of the primary goals
underlying the theory of action for SBMMTES are to improve the overall pool of human capital
within the local teacher labor market, and subsequently to ensure all students have access to
high-quality instruction. This is particularly the case in LAUSD, where under Superintendent
Deasy, the mission of his Talent Management Division is “…to ensure that every classroom is
led by an effective teacher…” (LAUSD, 2013). Below I provide the results from a descriptive
analysis of the characteristics of teachers who stay, switch, or leave at both the school- and
classroom-level after the public release of teacher-level VAM and the implementation of the
EGDC pilot. These results are limited to all elementary analyses and the school-level secondary
analyses as no significant differences in mobility were observed between tested and non-tested
secondary teachers at the classroom-level.
Implications for the Elementary Teacher Labor Market. Table 8 in Appendix C
provides descriptive statistics of the characteristics of elementary teachers in tested grades who
stay in their current schools relative to tested teachers who switch schools or leave LAUSD in
the year after the implementation of the SBMMTES pilot, both for the full sample as well as the
“hard-to-staff” school samples. The results presented above demonstrate that after the
implementation of the EGDC pilot, tested teachers in low-performing schools were significantly
less likely to switch schools than non-tested teachers compared to prior years, suggesting an
increase in retention for teachers with access to additional information about their performance.
Although this difference was small in magnitude and only significant at the p<0.10 level,
Column 2 of Table 8 demonstrates that the elementary teachers in tested grades who were
retained at their schools were significantly more likely than those who switched schools to be
mid-career (7-9 years of experience) and permanent employees (p<0.10 and p<0.001,
116
respectively). In addition, these tested stayers were significantly more likely than those who
chose to switch schools to have higher 5
th
Grade Math VAM (p<0.01).
47
Although descriptive,
this suggests that the small increases in retention observed after the implementation of the EGDC
pilot may have had an overall positive influence on the quality of tested teachers in in these low-
performing school contexts, particularly in 5
th
Grade.
At the school-level, the public-release of teacher-level VAM by the Los Angeles Times
also appeared to have a positive influence on the quality of tested teachers in “hard-to-staff”
schools. The results presented above suggest that after the Times’ VAM release, elementary
teachers in tested grades who taught in low-performing schools and schools with high-
proportions of ELLs, were less likely to switch schools than teachers in non-tested grades
teaching in similar contexts. Turning to Columns 2 and 3 of Table 9 in Appendix C, we see that
among tested teachers in low-performing and high-ELL schools, those who stayed were less
likely to be novice teachers (p<0.001), and more likely to be permanent employees (p<0.001)
compared to those who switched schools. In the low-performing sample, tested teachers who
stayed were not significantly different in terms of observed quality than those who switched, but,
in the high-ELL sample, we do see that those who stayed were significantly more likely to have
fewer needs improvement or below standards ratings on their Stull than were those who switched
(p<0.01). Together this evidence, suggests that reductions in switching schools among tested
teachers relative to non-tested teachers after the public-release of VAM had a small positive
influence on the quality of tested teachers in schools serving high-proportions of ELLs.

47
It is important to note that Stull ratings are not necessarily the best measure of teacher performance.
Approximately 99% of teachers are identified as meeting standards on their Stull each year. However, while limited,
this is the only available measure of teacher quality available across years in LAUSD, so I use it here in conjunction
with over measures available post-AGT release noting its limitations.
117
At the classroom-level, the findings above suggest that after the public release of VAM
tested elementary teachers in the full sample and classrooms with high-proportions of ELL
students, were significantly more likely to switch classroom placements, relative to staying in
their current classroom placement, than were their non-tested counterparts. However, they were
no more or less likely to leave after the same period. In contrast, the EGDC pilot appeared to
have no differential influence on the switching behavior of tested teachers in these same
contexts, but differentially decreased the probability that tested teachers in the full sample would
leave LAUSD altogether. Table 10 and Table 11 in Appendix C provide descriptive evidence of
the implications of these changes on the composition of teachers in tested placements in
LAUSD. Turning to Column 1 of Table 10, we see that after the implementation of the EGDC
pilot, tested elementary teachers in the full sample who stayed in their placement were
significantly more likely than those who left LAUSD to be mid-career, permanent teachers
(p<0.001) with fewer needs improvement or below Stull ratings (p<0.001), and higher 4
th
Grade
ELA and 6
th
Grade Math VAM (p<0.05 and p<0.10, respectively). This suggests that the
increased retention observed after the EGDC pilot may have contributed to the selective
retention of more experienced and higher-quality teachers in these tested placements in LAUSD.
Column 1 and Column 3 of Table 11 in Appendix C show a slightly different set of
implications for the trends observed after the public release of teacher-level VAM in the prior
year. In particular, descriptive evidence suggests that the public release of teacher-level VAM,
may have increased the churn of less experienced and lower-quality teachers out of tested and
into non-tested placements in the full and high-ELL samples. Specifically, among tested teachers
in the full sample and high-proportion ELL classrooms, those who switched placements were
significantly more likely than those who stayed in their placements to be inexperienced, non-
118
permanent employees (p<0.001). Moreover, in the full sample, tested teachers who switched
placements had more needs improvement and below standard Stull ratings (p<0.001), and lower
3
rd
and 5
th
Grade Math and ELA VAM (p<0.01 to p<0.001) than those who stayed. Similarly, in
the high-ELL sample, tested teachers who switched into non-tested placements had lower 3
rd

Grade ELA, and 3
rd
and 4
th
Grade Math VAM relative to those who stayed (p<0.05).
Implications for the Secondary Teacher Labor Market. Tables 12 and 13 in Appendix C
provide descriptive results of the characteristics of secondary teachers in tested grades who
stayed in their current school relative to tested teachers who switched schools or left LAUSD in
the year after the implementation of the EGDC pilot and the public release of teacher-level
VAM, respectively, for both the full as well as the “hard-to-staff” school samples.
After the implementation of the EGDC pilot, I found that secondary teachers in tested
grades and subjects in the full sample of schools, as well as those with high-proportions of ELLs
and students in poverty, were significantly less likely than those in non-tested grades and
subjects to switch schools or leave LAUSD. Table 12 demonstrates that this increased retention
may have had an overall positive impact on the teacher labor market. In particular, within the full
and high-poverty schools samples, tested teachers who stayed in their school after the
implementation of the pilot were significantly more likely than those who switched schools or
left LAUSD to be experienced, permanent employees with fewer needs improvement ratings
(p<0.10 to p<0.001). Interestingly, in the case of the high-poverty schools, we see that these
same teachers were also of higher quality according to VAM but that in the full sample and high-
ELL schools, this differential mobility did little to the distribution of teacher quality in tested
grades and subjects across these secondary schools in LAUSD.
119
The results above also provide some evidence that after the public release of teacher-level
VAM, relative to the prior year, tested secondary teachers were significantly less likely to switch
schools compared to their non-tested counterparts, suggesting increased retention in the full
sample of schools as well as those with high-proportions of ELLs and students in poverty. Table
13 demonstrates that overall, this increased retention may also have had positive implications for
the secondary teacher labor market in these schools. For example, tested teachers who stayed
were more experienced and of higher-quality, on average, relative to those who switched schools
after the public release of VAM. In particular, turning to Column 1 of Table 13, we see that in
the full sample of schools, tested teachers who stayed were significantly more likely than those
who switched to be permanent employees (p<0.001) with fewer needs improvement or below
standards Stull ratings (p<0.001) and higher VAM in US and World History (p<0.01 and p<0.05,
respectively). Column 2 of Table 13 demonstrates that in high-proportion ELL schools, tested
teachers who stayed were also of higher quality than those who left, having fewer low Stull
ratings (p<0.001), and higher VAM in ELA and US history (p<0.10 and p<0.05, respectively).
However, we see that in high-poverty schools, the retention effects observed appeared to have no
implications for the quality of tested teachers in these schools. Together, these results suggest
that the increased retention of secondary tested teachers relative to their non-tested counterparts
observed after the public-release of teacher-level VAM had positive implications for the quality
of tested secondary teachers in LAUSD overall, and in schools with high-proportions of ELLs.
Although helpful in demonstrating potential implications of post-reform mobility on the
distribution of teacher quality within LAUSD, it is important to note that these findings are
entirely descriptive. As such, I cannot rule out the possibility that the changes in the composition
of the elementary and secondary teacher labor market observed after the public release of
120
teacher-level VAM and the implementation of the EGDC pilot in LAUSD may be due to other
influences on teacher mobility over the same period. However, the results do provide suggestive
evidence of the implications of these changes in mobility and offer insight into how mobility
might change among teachers with access to additional performance information after the
implementation of a SBMMTES pilot, or the public release of teacher-level VAM.

Discussion and Implications
States and districts across the country are making sweeping changes to how they manage
human capital, from the implementation of SBMMTES, to performance-based classroom
assignments, and career pathways. However, policymakers and practitioners are being forced to
move forward with these comprehensive approaches to improving human capital with little
information about how, if at all, these strategies can work together in practice.
By examining the impact of the threat of additional accountability for teacher
performance on teacher mobility decisions given differences in access to information about
teacher performance, this paper contributes to this knowledge base and provides policy makers
and practitioners with information about how implementing SBMMTES inclusive of value-
added measures of teacher performance may influence the composition of the teacher labor
market. Overall, I find that secondary teachers in tested grades and subjects are significantly less
likely than their non-tested colleagues to switch schools in response to the threat of additional
accountability for performance. Moreover, I find that those who stay in their schools are
significantly more likely to be experienced and higher quality than their counterparts who switch.
This suggests that providing secondary teachers with additional information about their practice
may actually have positive implications for the composition of the teacher labor market.
121
Interestingly, I do not find any evidence that this differential access to information
matters for elementary teacher mobility in response to this same accountability threat. This
suggests that elementary and secondary teachers with access to additional information about their
performance may respond in different ways to the increased accountability brought about by a
SBMMTES. It is possible that this observed difference between elementary and secondary
teachers is driven by how teachers at each of these levels use VAM, or the extent to which
teachers at these levels feel the accountability threat embedded within a SBMMTES. Although it
is beyond the scope of this paper, future research would benefit from exploring these differences
further to understand why elementary and secondary teachers might respond differently to the
threat of additional accountability. Understanding these differences may help to shed light on
what is driving the different behavior of these actors at each level and what, if any, implications
this may have for how teachers are sorted across schools and classrooms within a given district.
The findings presented here also offer suggestive evidence to policymakers about how
teacher mobility decisions may differ when additional information about their performance is
released publicly as compared to privately within the context of a SBMMTES. Overall, I find
that after the public release of VAM, elementary and secondary teachers in tested grades and
subjects were significantly less likely to switch schools than were their non-tested counterparts
overall, and in some “hard-to-staff” schools. This result is surprising for a number of reasons.
First, we might expect teachers in “hard-to-staff” contexts to be more likely to move away from
challenging placements when faced with additional accountability as it is perceived they may be
more likely to receive low-VAM in these contexts. I find that this is not the case; however, we do
see that those who stay in these schools are more experienced and higher-quality on average,
suggesting that they have demonstrated success in these contexts and may not have seen value in
122
changing schools given their current performance. Second, because the public release of VAM
by the Times did not include any information for secondary teachers, it is a bit surprising that we
actually see increased retention for this group of teachers in this period. It could be that
secondary teachers received other forms of feedback about their instruction or experienced other
events during this period that contributed to their increased retention relative to their non-tested
counterparts. Regardless, we see that this increased retention had an overall positive influence on
the quality of secondary teachers staying in their schools, suggesting an overall net benefit of
secondary teacher mobility after the public release of VAM.
At the classroom-level, the results present a slightly different story. Ultimately, I find no
evidence that the implementation of the EGDC pilot nor the public release of VAM differentially
impacted the mobility of tested secondary teachers out of tested into non-tested placements. As
discussed above, this could be a function of how teachers were assigned to tested and non-tested
placements or reflect the limited opportunity for mobility at the secondary level. Despite the null
effects observed for secondary teachers, I do find that for elementary teachers, the public release
of VAM appeared to induce turnover out of tested placements while the private release increased
retention in these same placements, at least in the full sample of teachers. Moreover, I find that
the increased turnover observed after the public release of VAM actually contributed to the
“churn” of lower-performing and less-experienced elementary teachers out of tested and into
non-tested grades in the following year. To the extent that these teachers have more success in
non-tested placements, this may have an overall positive effect on the composition of the teacher
labor market in LAUSD; however, if these teachers continue to perform poorly than this increase
in mobility is simply shuffling poor performers into the lower elementary grades.
123
Taken together, these early findings suggest that providing teachers and administrators
with a measure of their contribution to student learning may contribute to the selective retention
of more experienced and higher-quality teachers overall. These results are encouraging for
policymakers and practitioners seeking to use value-added measures in teacher evaluations or as
part of a broader human capital management strategy. Specifically, it suggests that providing this
new information about teacher performance may actually increase retention in schools and
classrooms, and have positive net effects on the characteristics of educators within the local
teacher labor market. However, this early evidence also suggests that policymakers should
carefully consider how teacher performance information generated within SBMMTES is
released. The fact that teacher mobility after the public release of teacher-level VAM increased
the likelihood that less-experienced, low-performing elementary teachers switched from tested
grades into non-tested grades suggests that publicly releasing teacher effectiveness information
may contribute to the unintentional sorting of low-quality teachers into early elementary grades.

124
References

Abelson, M. A., & Baysinger, B. D. (1984). Optimal and dysfunctional turnover: Toward an
organizational level model. The Academy of Management Review, 9(2), 331–341.

Agresti, A. (2007). An introduction to categorical data analysis, 2
nd
edition. Hoboken, New
Jersey: John Wiley & Sons, Inc.

American Institutes for Research. (7 February 2014). Considerations regarding release of
individual teacher value-added scores. Commentary. Retrieved from
http://www.air.org/resource/considerations-regarding-release-individual-teacher-value-
added-scores

Baker, J. (1999). Teacher-student interactions in urban at-risk classrooms: Differential behavior,
relationship quality, and student satisfaction with school. The Elementary School Journal,
100, 1, 57-70.

Barge, J.D. (2012). Teacher keys and leader keys effectiveness systems: 2012 Pilot evaluation
report. Georgia Department of Education.

Berman, P., & McLaughlin, M.W. (1978). The RAND change agent study, Vols 1-8. Santa
Monica, CA: The RAND Corporation.

Booher-Jennings, J. (2005). Below the bubble: "Educational triage" and the Texas accountability
system. American Educational Research Journal, 42, 2, 231-268.

Boyd, D., Lankford, H., Loeb, S., & Wyckoff, J. (2008). The impact of assessment and
accountability on teacher recruitment and retention: Are there unintended consequences?
Public Finance Review, 36, 1, 88-111.

Boyd, D., Lankford, H., Loeb, S., Wyckoff, J. (2002). Initial matches, transfers, and quits: Career
decisions and the disparities in average teacher qualifications across schools. Working
Paper.

Buis, M. L. (2010). Stata tip 87: Interpretation of interactions in non-linear models. The Stata
Journal, 10(2), 305-308.Boyd, Lankford, Loeb, & Wyckoff, 2002

Caliendo, M., & Kopeinig, S. (2005). Some practical guidance for the implementation of
propensity score matching. Discussion paper No. 1588. Retrieved from
http://ssrn.com/abstract=721907.

Chetty, R., Friedman, J.N., & Rockoff, J.E. (2011). The long-term impacts of teachers: Teacher
value-added and student outcomes in adulthood. Cambridge, MA: National Bureau of
Economic Research. Working Paper No. 17699.
125
Chetty, R., Friedman, J.N., & Rockoff, J.E. (2013). Measuring the impacts of teachers II:
Teacher value-added and student outcomes in adulthood. Cambridge, MA: National
Bureau of Economic Research. Working Paper No. 19424.

Clotfelter, C. T., Ladd, H. F., Vigdor, J. L., & Diaz, R. A. (2004). Do school accountability
systems make it more difficult for low-performing schools to attract and retain high-
quality teachers? Journal of Policy Analysis and Management, 23, 2, 251-271.

Coburn, C. (2005). Shaping teacher sensemaking: School leaders and the enactment of reading
policy. Educational Policy, 19(3), 476-509.

Coburn, C. E. (2004). Beyond decoupling: Rethinking the relationship between the institutional
environment and the classroom. Sociology of Education, 77, 211-244.

Coburn, C.E. (2001). Collective sensemaking about reading: How teachers mediate reading
policy in their professional communities. Educational Evaluation and Policy Analysis,
23, 2, 145-170.

Cohen, D.K., & Ball, D.L. (1990). Policy and practice: An overview. Educational Evaluation
and Policy Analysis, 12, 3, 233-239.

Cohen-Vogel, L. (2011). Staffing to the test: Are today’s school personnel practices evidence
based? Educational Evaluation and Policy Analysis, 33, 4, 483-505.

Creswell, J.W. (2007). Qualitative inquiry and research design: Choosing among five traditions.
Thousand Oaks, CA: Sage.

Crocco, M. S., & Costigan, A. T. (2007). The narrowing of curriculum and pedagogy in the age
of accountability: Urban educators speak out. Urban Education, 42, 6, 512-534.

Croninger, R.G. & Lee, V.E. (2001). Social capital and dropping out of high school: Benefits to
at-risk students of teachers’ support and guidance. Teachers College Record, 103, 4, 548-
581.

Datnow, A., Castellano, M. (2000). Teachers’ responses to Success for All: How beliefs,
experiences, and adaptations shape implementation. American Educational Research
Journal, 37(3), 775-799.

Datnow, A., Hubbard, L., & Mehan, H. (2002). Extending educational reform: From one school
to many. New York: Routledge Falmer Press.

Dee, T., Wyckoff, J. (2013). Incentives, selection, and teacher performance: Evidence from
IMPACT. National Bureau of Economic Research. Working Paper #19529. Boston, MA:
National Bureau of Economic Research.
126

Dehejia, R.H., Wahba, S. (2002). Propensity score matching methods for nonexperimental causal
studies. The Review of Economics and Statistics, 84, 1, 151-161.

Donaldson, M.L. (June 2009). So long, Lake Wobegon? Using teacher evaluation to raise
teacher quality. Center for American Progress.

Donaldson, M., Cobb, C., LeChasseur, K., Gabriel, R., Gonzales, R., Woulfin, S., Makich, A.
(2014). An evaluation of the pilot implementation of Connecticut’s system for educator
evaluation and development. University of Connecticut Center for Education Policy
Analysis. Neag School of Education.

Felch, J., Song, J., Smith, D. (14 August 2010). Who’s teaching LA’s kids? Retrieved from
http://articles.latimes.com/2010/aug/14/local/la-me-teachers-value-20100815

Feng, L. (2010). Hire today, gone tomorrow: New teacher classroom assignments and teacher
mobility. Education Finance and Policy, 5, 3, 278-316.

Feng, L., Figlio, D.N., and Sass, T.R. (2010). School accountability and teacher mobility.
Cambridge, MA: National Bureau of Economic Research. Working Paper. No. 16070.

Firestone, W.A., Blitz, C.L., Gitomer, D.H., Kirova, D., Shcherbakov, A., Nordon, T. (2013).
New Jersey teacher evaluation, RU-GSE external assessment, year 1 report. New
Brunswick, NJ: Rutgers University-Graduate School of Education.

Fleischer, L. (2012 September 9). Teacher grading off to uneven start. The Wall Street Journal.
Retrieved from
http://online.wsj.com/article/SB10000872396390444554704577641950548987704.html?
KEYWORDS=Teacher+Grading

Fletcher, W. (28 September 2013). VAM/AGT Still a Meaningless, Dangerous Number. The United
Teacher. Retrieved from http://4lakidsnews.blogspot.com/2013/09/utla-presidents-perspective-
vamagt.html

Florida Department of Education. (2013). Personnel evaluation data for classroom teachers by
district, 2011-12/Survey 5 Data. Retrieved from
http://www.fldoe.org/profdev/pdf/StatewideResults.pdf

Gentner, D., and Toupin, C. (1986). Systematicity and surface similarity in the development of
analogy. Cognitive Science, 10, 277-300.

Gentner, D., Rattermann, M. J., & Forbus, K. D. (1993).The roles of similarity in transfer:
Separating retrievability from inferential soundness. Cognitive Psychology, 2 5, 524-575.

Garland, S. (2012 December 19). Advice, caution from early adopters of new teacher
evaluations. The Hechinger Report. Retrieved from
127
http://hechingerreport.org/content/advice-caution-from-early-adopters-of-new-teacher-
evaluations_10654/

Glazerman, S. and A. Seifullah (2012). An Evaluation of the Chicago Teacher Advancement
Program (Chicago TAP) After Four Years. Princeton: Mathematica Policy Research.

Goe, L., Bell, C., Little, O. (2008). Approaches to evaluating teacher effectiveness: A research
synthesis. Washington, DC: National Comprehensive Center for Teacher Quality.

Goldhaber, D., Gross, B., Player, D. (2011). Teacher career paths, teacher quality, and
persistence in the classroom: Are public schools keeping their best? Journal of Policy
Analysis and Management, 30, 1, 57-87.

Guin, K. (2004). Chronic teacher turnover in urban elementary schools. Educational Evaluation
and Policy Analysis, 12(42), 1–25.

Halverson, R., Kelley, C., & Kimball, S. (2004). Implementing teacher evaluation systems: How
principals make sense of complex artifacts to shape local instructional practice. In
Educational Administration, Policy, and Reform: Research and Measurement, 153-188.
Information Age Publishing, Inc.

Hanushek, E.A., Kain, J.F., & Rivkin, S.G. (2004) Why public schools lose teachers. The
Journal of Human Resources, 39, 2, 326-354.

Hanushek, E.A., Kain, J.F., O’Brien, D.M., & Rivkin, S.G. (2005) The market for teacher
quality. Cambridge, MA: National Bureau of Economic Research. Working Paper No.
11154.

Heneman, H.G. III, & Milanowski, A.T. (2003). Continuing assessment of teacher reactions to a
standards-based teacher evaluation system. Journal of Personnel Evaluation in
Education, 17(2), 171-195.

Hess, G. A. Jr. (2003). Reconstitution three years later: monitoring the effects of sanctions on
Chicago high schools. Education and Urban Society, 35(4):494-517.

Hill, H. C. (2001). Policy is not enough: Language and the interpretation of state standards.
American Educational Research Journal, 38, 2, 289-318.

Ho, A.D. & Kane, T.J. (2013). The reliability of classroom observations by school personnel.
Measures of Effective Teaching (MET) project, Bill and Melinda Gates Foundation.

Honig, M.I. (2006). New directions in education policy implementation: Confronting
complexity. Albany, NY: The State University of New York Press.

128
Hough, H.J. (2012). Salary incentives and teacher quality: The effect of a district-level salary
increase on teacher retention. Working Paper.

Imbens, G. W., & Wooldridge, J. M. (2009). Recent developments in the econometrics of
program evaluation. Journal of Economic Literature, 47(1), 5-86.

Ingersoll, R.M., and Smith, T.M., (2003). The wrong solution to the teacher shortage.
Educational Leadership, 60, 8, 30-33.

Jacob, B. A., & Levitt, S. D. (2003). Rotten apples: An investigation of the prevalence and
predictors of teacher cheating. Quarterly Journal of Economics, 118, 3, 843-877.

Jennings, N.E. (1996). Interpreting policy in real classrooms: Case studies of state reform and
teacher practice. New York, NY: Teachers College Press.

Kane, T.J. and Staiger, D.O. (2012). Gathering feedback for teaching: Combining high-quality
observations with student surveys and achievement gains. Measures of Effective
Teaching (MET) project, Bill and Melinda Gates Foundation.

Keesler, V.A., Howe, C. (2012). Understanding educator evaluations in Michigan: Results from
year 1 of implementation. Michigan Department of Education. Retrieved from
http://www.michigan.gov/documents/mde/Educator_Effectiveness_Ratings_Policy_Brief
_403184_7.pdf

Kimball, S.M. (2002). Analysis of feedback, enabling conditions and fairness perceptions of
teachers in three school districts with new standards-based evaluation systems. Journal
for Personnel Evaluation in Education, 16, 241-268.

Klem, A.M. & Connell, J.P. (2004). Relationships matter: Linking teacher support to student
engagement and achievement. Journal of School Health, 74, 7, 262-273.

Krieg, J. W. (2006). Teacher quality and attrition. Economics of Education Review, 25(1), 13-27.

Ladd, H.F. (2011). Teachers’ perceptions of their working conditions: How predictive of planned
and actual teacher movement? Educational Evaluation and Policy Analysis, 33, 2, 235-
261.

Lincoln, YS. & Guba, E.G. (1985). Naturalistic inquiry. Newbury Park, CA: Sage Publications.

Lipsky, M. (1980). Street-level Bureaucracy; Dilemmas of the Individual in Public Services.
New York: Russell Sage Foundation.

Llanos, C. (24 August 2011). LAUSD chief Deasy tells educators much work still ahead. Los
Angeles Daily News. Retrieved from http://www.dailynews.com/20110824/lausd-chief-deasy-
tells-educators-much-work-still-ahead
129

Majone, G., & Wildavsky, A. (1978). Implementation as evolution. In Policy Studies Review
Annual 2, 103-117. Beverly Hills, CA: Sage Publications.

Malen, B. & Rice, J. K. (2004). A framework for assessing the impact of education reforms on
school capacity: insights from studies of high-stakes accountability initiatives.
Educational Policy, 18:5:631-660.

Malen, B., Croninger, R., Muncey, D., & Redmond-Jones, D. (2002). Reconstituting schools:
‘testing’ the ‘theory of action’. Educational Evaluation and Policy Analysis, 24:2:113-
132.

Mazmanian, D.A., & Sabatier, P.A. (1989). Implementation and Public Policy. Lanham, MD:
Univeristy Press of America, Inc.

McLaughlin, M.W., (1976). Implementation as mutual adaptation. Teachers College Record, 77,
3, 339-351.

McLaughlin, M.W. (1987). Learning from experience: Lessons from policy implementation.
Educational Evaluation and Policy Analysis, 9, 171-178.

Mertler, Craig A. (1999). Teacher perceptions of students as stakeholders in teacher evaluation.
American Secondary Education, 27, 3, 17-30.

Milanowski, A. (2001). Assessmment of teacher reactions to a standards-based teacher
evaluation system: A pilot study. Journal of Personnel Evaluation in Education, 15, 3,
193-212.

Miles, M.B., Huberman, A.M. (1994). Qualitative data analysis, 2
nd
Ed. Thousand Oaks, CA:
Sage Publications.

Murnane, R.J., & Willet, J.B. (2011). Methods Matter: Improving Casual Inference in
Educational and Social Science Research. New York, NY: Oxford University Press.

National Council on Teacher Quality. (2013). State of the states 2013: Connect the dots: Using
evaluations of teacher effectiveness to inform policy and practice. Washington, DC:
National Council on Teacher Quality.

O’Donnell, P., Lieszkovszky, I. (16 June 2013). Why we’re publishing teachers’ value-added
ratings. State Impact. Retrieved from
https://stateimpact.npr.org/ohio/2013/06/16/grading-the-teachers-why-the-cleveland-
plain-dealer-and-stateimpact-ohio-are-publishing-ohio-teachers-value-added-ratings/

130
Paulson, A. (2012 January 26). Under education reform, school principals swamped by teacher
evaluations. The Christian Science Monitor. Retrieved from
http://www.csmonitor.com/USA/Education/2012/0126/Under-education-reform-school-
principals-swamped-by-teacher-evaluations

Patton, M.Q. (2002). Qualitative research and evaluation methods, 3
rd
edition. Thousand Oaks,
CA: Sage Publications, Inc.

Perlman, C. L. & Redding, S. (2011). Handbook on effective implementation of School
Improvement Grants. Lincoln, IL: Center on Innovation and Improvement. Retrieved
from http://www.centerii.org/handbook/

Peterson, K., Wahlquist, C., & Bone, K. (2000). Student surveys for school teacher evaluation.
Journal of Personnel Evaluation in Education, 14, 2, 135-153.

Pianta, R. C. (2003). Standardized observations from pre-K to 3rd grade: A mechanism for
improving access to high-quality classroom experiences and practices during the p-3
years. New York, NY: Foundation for Child Development. Working paper.

Podgursky, M., R. Monroe, and D. Watson. (2004). The academic quality of public school
teachers: An analysis of entry and exit behavior. Economics of Education Review, 23, 5,
507-518.

Pressman, J.L., & Wildavsky, A.B. (1984). Implementation (3
rd
Ed). Berkeley: University of
California Press.

Rado, D. (17 September 2012). Chicago teachers’ strike: Issues at the center of contract
negotiations. Chicago Tribune. Retrieved from http://articles.chicagotribune.com/2012-
09-17/news/ct-met-chicago-teachers-strike-details-20120917_1_cps-teachers-teacher-
evaluation-chicago-teacher-strike

Rice, J. K., & Croninger, R. C. (2005). Resource generation, reallocation, or depletion: an
analysis of the impact of reconstitution on school capacity. Leadership and Policy in
Schools, 4(2):73-104.

Rivkin, S.G., Hanushek, E.A., & Kain, J.F. (2005). Teachers, schools, and academic
achievement. Econometrica, 73, 2, 417-458.

Rockoff, Jonah E. (2004). The impact of individual teachers on student achievement: Evidence
from panel data. The American Economic Review, 94, 2, 247-252.

Rosenbaum, P., & Rubin, D. (1983). “The Central Role of the Propensity Score in
Observational Studies for Causal Effects,” Biometrika, 70, 41-55.

131
Ross, B. H. (1987). This is like that: The use of earlier problems and the separation of similarity
effects. Journal of Experimental Psychology: Learning, Memory, and Cognition, 13, 629-
639.

Roterham, A.J. (23 September 2010). Rating teachers: The trouble with value-added data. Time
Magazine. Retrieved from
http://content.time.com/time/nation/article/0,8599,2020867,00.html

Sanders, K. (14 September 2011). Teachers union sues state over ‘unconstitutional’ merit pay
law. Tampa Bay Times. Retrieved from http://www.tampabay.com/news/courts/teachers-
union-sues-state-over-unconstitutional-merit-pay-law/1191590

Sanders, W. L., & Rivers, J. C. (1996). Cumulative and residual effects of teachers on future
student academic achievement. Knoxville, TN: University of Tennessee Value-Added
Research and Assessment Center. Retrieved from
http://www.mccsc.edu/~curriculum/cumulative%20and%20residual%20effects%20of%2
0teachers.pdf

Sartain, L., Steinberg, M.P. (2014). Teacher performance evaluation and teacher sorting:
Experimental evidence from Chicago Public Schools. Paper prepared for the 39
th
Annual
Conference of the Association of Education Finance and Policy. Working Paper.

Sartain, L., Stoelinga, S.R., Brown, E. (August 2009). Evaluation of the Excellence in Teaching
Pilot: Year 1 Report to the Joyce Foundation. The Consortium on Chicago School
Research at the University of Chicago.

Sartain, L., Stoelinga, S.R., & Brown, E.R. (2011). Rethinking Teacher Evaluation for Change:
Lessons Learned from Classroom Observations, Principal-Teacher Conferences, and
District Implementation. Chicago: Consortium for Chicago School Research.

Sawchuck, S. (11 June 2013). Race is on to ready teacher evaluations in New York City.
Education Week. Retrieved from
http://www.edweek.org/ew/articles/2013/06/12/35newyork.h32.html

Schmidt, M., Datnow, A. (2005). Teachers’ sensemaking about comprehensive school reform:
The influence of emotions. Teaching and Teacher Education, 21,8, 949-965.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental
designs for generalized causal inference. New York, NY: Houghton Mifflin Company.

Singh, A. (23 September 2011). LAUSD Chief: ‘A tale of two school systems’ (Part II). Eagle
Rock Patch. Retrieved from http://eaglerock.patch.com/groups/schools/p/lausd-chief-a-tale-of-
two-school-systems-part-ii#photo-7874840

132
Spillane, J. P., & Callahan, K.A. (2000). Implementing state standards for science education:
What district policy makers make of the hoopla. Journal of Research in Science
Teaching, 37(5), 401-425.

Spillane, J.P. (1998). A cognitive perspective on the role of the local educational agency in
implementing instructional policy: Accounting for local variability. Educational
Administration Quarterly, 34(1), 31-57.

Spillane, J.P. (2004). Standards deviation: How schools misundersand education policy.
Cambridge, MA: Harvard University Press.

Spillane, J.P., Reiser, B.J., & Reimer, T. (2002). Policy implementation and cognition:
Reframing and refocusing implementation research. Review of Educational Research,
72(3), 387-431.

Stake, R. (1995). The art of case study. Thousand Oaks, CA: Sage.

Strauss, V. (23 December 2013). Errors found in D.C. teacher evaluations (2
nd
update). The
Washington Post: Answer Sheet. Retrieved from
http://www.washingtonpost.com/blogs/answer-sheet/wp/2013/12/23/errors-found-in-d-c-
teacher-evaluations/

Strauss, V. (24 February 2012). NYC releases teachers’ value-added scores –unfortunately. The
Washington Post. Retrieved from http://www.washingtonpost.com/blogs/answer-
sheet/post/nyc-releases-teachers-value-added-scores--
unfortunately/2012/02/24/gIQAtbVXYR_blog.html

Stronge, J. H., & Tucker, P. D. (1999). The politics of teacher evaluation: A case study of new
system design and implementation, Journal of Personnel Education in Education, 13, 4,
218-238.

Strunk, K. O., Weinstein, T.L., & Makkonen, R. (2013). Understanding the Implementation of a
Standards-Based Multiple Measure Teacher Evaluation Reform: Preliminary Evidence
from the Initial Implementation Phase of the Educator Growth and Development Cycle in
Los Angeles. Working Paper.

Taylor, E.S., Tyler, J.H. (2012). The effect of evaluation on teacher performance. American
Economic Review, 102, 7, 3628-51.

Tennessee Department of Education. (2012). Teacher evaluation in Tennessee: A report on Year
1 implementation. Tennessee Department of Education.

The New Teacher Project (2009). Teacher hiring, transfer, and evaluation in Los Angeles
Unified School District. Final Report.
133

Thomas, R.D. (1979). Implementing federal programs at the local level. Political Science
Quarterly, 94, 3, 419-435.

U.S. Department of Education. (2009). Race to the top program: Executive summary. Retrieved
from http://www2.ed.gov/programs/racetothetop/executive-summary.pdf

U.S. Department of Education. (2012a). Teacher Incentive Fund: Program description. Retrieved
from http://www2.ed.gov/programs/teacherincentive/index.html

U.S. Department of Education (2012b). ESEA flexibility policy document. Retrieved from
http://www2.ed.gov/policy/elsec/guid/esea-flexibility/index.html

Weatherly, R., Lipsky, M. (1977). Street-level bureaucrats and institutional innovation:
Implementing special education reform. Harvard Educational Review, 47, 2, 171-197.

Weick, K. E. (1995). Sensemaking in organizations. Thousand Oaks, CA: Sage.

Weisberg, D., Sexton, S., Mulhern, J., Keeling, D. (2009). The Widget Effect: Our National
Failure to Acknowedge and Act on Differences in Teacher Effectiveness. The New
Teacher Project.

Wenglinsky, H. (2002). How schools matter: the link between teacher classroom practices and
student academic performance. Educational Policy Analysis Archives, 10, 12.

Wentzel, K.R. (1997). Student motivation in middle school: The role of perceived pedagogical
caring. Journal of Educational Psychology, 89, 3, 411-419.

West, M. R., Chingos, Matthew M. (2009). Teacher effectiveness, mobility, and attrition in
Florida. In M. G. Springer (Ed.), Performance incentives: Their growing impact on
American K-12 education (pp. 251-271): Brookings Institution Press.

White, B., Cowhy, J., David Stevens, W., and Sporte, S.E. (2012). Designing and Implementing
the Next Generation of Teacher Evaluation Systems: Lessons Learned from Case Studies
in Five Illinois Districts. The University of Chicago Consortium on Chicago School
Research . Research Brief.

Wooldridge, J.M. (2010). Econometric analysis of cross section and panel data, 2
nd
edition.
Cambridge, MA: MIT Press.

Wright, S.P., Horn, S.P., & Sanders, W.L. (1997). Teacher and classroom context effects on
student achievement: Implications for teacher evaluation. Journal of Personnel
Evaluation in Education, 11, 57-67.

134
Yanow, D. (1996). How does a policy mean?: Interpreting policy and organizational actions.
Washington, DC: Georgetown University Press.

Yin, R.K. (2014). Case study research: Design and methods, 5
th
edition. Thousand Oaks, CA:
Sage Publications.

135
Appendix A

Table 1. School Cases - Site and Participant Information Across Selection Criteria
School Level
Number of
Participating
Teachers
Number of
Participating
Administrators
%
Minority
Category
District
Geographic
Location
2010-2011
API
Category
Participating
Teachers Avg.
Years of
Experience
School A Elementary 2 1 Middle East Middle 10
School B Elementary 4 1 Low North High 5
School C Middle 4 1 Middle East Middle 7
School D High 5 3 Middle South Low 8
School E High 3 5 Low West Middle 12
Note. To protect the confidentiality of participating schools we place schools in a “percent minority” and in an API range rather than
identifying their values directly. These ranges were developed by dividing IIP-eligible schools into quartiles and identifying the bottom
quartile as “low minority” or “low API,” and the top quartile as “high” minority or API.

136

Table 2. Teacher Cases - Participant Information
Participant Level Grade/Subject(s)
Years of
Experience
Overall
Years of
Experience
at Site
School A
Teacher 1 Elementary 4th Grade Multiple Subjects 20 15
Teacher 2 Elementary 4th Grade Multiple Subjects 25 5
School B
Teacher 1 Elementary K-5th Grade Special Education 13 4
Teacher 2 Elementary 1st Grade Multiple Subjects 17 4
Teacher 3 Elementary 5th Grade (GATE) Multiple Subjects 12 12
Teacher 4 Elementary 2nd/3rd Grade Multiple Subjects 13 1
School C
Teacher 1 Middle 7th & 8th Grade Algebra 1 7 7
Teacher 2 Middle 6th Grade Math & Science 6 6
Teacher 3 Middle 6th-8th Grade Physical Education 5 2
Teacher 4 Middle 6th-8th Grade Special Education 11 11
School D
Teacher 1 High 11th Grade ELA 11 6
Teacher 2 High 10th & 12th Grade ELA 23 10
Teacher 3 High 9th-12th Grade Math 7 7
Teacher 4 High 10th & 12th Grade AP History 16 16
Teacher 5 High 9th-12th Grade (AP)Chemistry 17 8
School E
Teacher 1 High 9th, 11th, & 12th Grade Art 25 25
Teacher 2 High 9th-12th Grade ELA 22 13
Teacher 3 High 9th-12th Grade Geometry & CAHSEE Prep 18 3
Note. All data was collected during school year 2011-2012 or 2012-2013 during teacher interview and focus groups.

137
Table 3. Administrator Cases - Participant Information
Participant
School
Level
Years of
Experience at
Site
School A
Administrator 1 Elementary 1
School B
Administrator 1 Elementary 9
School C
Administrator 1 Middle 2
School D
Administrator 1 High 10
Administrator 2 High --
Administrator 3 High --
School E
Administrator 1 High 6
Administrator 2 High 6
Administrator 3 High 3
Administrator 4 High 2
Administrator 5 High 4
Note. All data was collected during school year 2011-2012 during administrator interview and focus groups. In
some cases years of experience data was not collected and is left blank in the table.

138
Table 4. Interview and Focus Group Details
Respondents
Sample
Size Time
Length
(Minutes)
Initial Interviews/Focus Groups
Site Administrators 11 March-April 2011/12 60-75
Teachers 18 March-April 2011/12 45-75
Central Office Administrators 5 January-March 2011/12 45-60
District/Labor Leaders 3 March-April 2012/13 45-60
Follow-Up Interviews/Focus Groups
Teachers 4 March-April 2012/13 60
Central Office Administrators 2 March-April 2012/13 60

139
Table 5. Initial Coding Schema Based on a Cognitive Approach to Implementation
1) Individual Cognition- This category includes statements (text/talk) that captures how the implementing agents as individual
sensemakers notice and interprets stimuli and how prior knowledge, beliefs, and experiences influence construction of new
understandings.
a. Prior Knowledge (e.g., aware of Danielson rubrics/framework)
b. Existing Beliefs (e.g., value-laden statements about TQ, data use, eval procedures)
b. Prior Experiences (e.g., STULL evaluation process, using rubrics to evaluate performance)
2) Situated Cognition- This category includes statements (text/talk) that captures how multiple dimensions of a situation influence the
agent’s sensemaking from and about policy.
a. Thought Communities or Worldviews (e.g., teacher or administrator)
b. Social Interactions and Exchanges (e.g., formal and informal interactions)
c. Organizational Arrangements (e.g., norms, chains of communication, siloed)
3) Role of Representations- This category includes statements (text/talk) that captures how policy stimuli and external representations of
ideas about changing practice in policy are used in the sensemaking process.
a. External Representations of Practices/Systems (e.g., policy documents, language)
b. Structure of Learning Opportunities (e.g., trainings, PLCs, admin-teacher interactions)
4) Implementation – This category includes statements (text/talk) that captures how implementing agents actually enact the policy (or a
version of the policy) in the practice.
a. Changes in Practice or Process (e.g., presence or absence of changes in planning/delivering instruction)
b. Understanding (e.g., description of purpose or function of measure, activities, process)
c. Engagement (e.g., indications of participation, depth of engagement with tools)

140
Table 6. District Leader Expectations for Stakeholder Understanding of the Purpose of the EGDC
Purpose of the EGDC
District Leader
Teacher Growth and
Development
Teacher Accountability
1 √ --
2 √ --
4 √ --
5 √ √
6 √ √
Note. This table provides district leaders expectations for stakeholder understanding of the purpose of the EGDC. An “√” indicates
that the individual described the purpose of the EGDC as a function of the goal listed. "--" indicates that the District Leader did not
list the associated goal as a purpose of the EGDC. District Leader 3 was not asked about the purpose of the EGDC given that they
did not play a central role in its design and implementation and is therefore excluded from the table above.

141
Table 7. Teacher and Administrator Understanding of the Purpose of the EGDC
Type of EGDC Understanding

Teacher
Growth and
Development
Teacher
Accountability
Combination
Role Group % N % N % N
Administrators (n=10) 50.0% 5 0.0% 0 50.0% 5
Teachers (n=16) 37.5% 6 37.5% 6 25.0% 4
Note. This table provides the number and proportion of teachers and administrators by the
type of understanding they constructed of the EGDC. Each stakeholder's understanding was
derived from their response to an interview or focus group question asking them to describe
the purpose and goals of the EGDC.

142
Table 8. Examples of Evidence Classification for Factors Shaping Administrator Sensemaking - Previous Evaluation Experiences
Participant Nature/Type Evidence Level
A.a1
Standards, Observation,
Conferencing, Feedback
“I think with what I was doing as an administrator dovetailed
closely to what they were proposing…I do the same thing [as the
EGDC] without the framework...I work off of [the CSTPs]....I
walk in to your classroom, I would script and then I would come
back —and show you the evidence and then we would discuss..."
High
B.a1
Observation,
Conferencing, Feedback
“We did the traditional Stull process [previously]. There was little
pre-observation discussion like what’s required with EGDC but I
did the classroom observations and wrote down my observations."
Mid
C.a1
Standards, Observation,
Conferencing, Feedback
“...prior to last year I was looking at the rubrics and the materials
that are used for BTSA...and I would use the descriptors to help
teachers move from developing to effective. Since probably
October 2010 I started using the work of Charlotte Danielson
and...I shared the rubric with the teachers. I rated them...so I was
already using a very similar process last year here to help me
begin to focus more on an objective way of looking at practice...”
High
D.a1, D.a2, D.a3
Observation,
Conferencing, Feedback
“Usually if we you know we would do the classroom observation
you know we were write down our notes so we would have them
come in and do conference and you know what we saw… what
their reactions to it and then offer whatever assistance that we
could...And it really just was an observation and just kind of
talking…Not consistent though.”
Mid
E.a1, E.a2, E.a4, E.a5 Observation
"Sometimes we do a good job with the teachers. We actually do
this [Stull] until things work...I have my list of eight teachers who
we observe and my discussion is “Who are the people who get
promoted onto the list? Who will get promoted off the list?"
Low
Note. Administrators are identified by their school followed by a small “a” and there associated number, where A.a1 is School A, Administrator 1. In the case of School D and
School E where multiple administrators were interviewed, these pieces of evidence reflect the administrators’ perspectives on average. Administrator 3 in School E is excluded
due to insufficient data.
143
Table 9. Examples of Evidence Classification for Factors Shaping Administrator Sensemaking - Beliefs About Teacher
Capacity/Willingness to Reflect and Grow
Case Site Nature/Type Evidence
A.a1
All Teachers Have the
Capacity to Improve
"...in teaching I think that people develop a comfort level— and I disagree,
because I think it’s ongoing and we’re always changing things. And so, this
[the EGDC] forces you to go back and think about, 'why am I doing this and
what is my expected outcome? And if it didn’t work, why didn’t it work? And,
how am I going to change so that I’m not just doing the same thing…"
B.a1
Some Teachers Have
the Capacity to Improve
"Nobody has brand new teachers on staff anymore, but if we had brand new,
developing teachers, you could counsel them and make them more reflective
teachers. But it is impossible to change the practice among the most
experienced teachers. They tend to put up a wall and don’t want to think
anymore…The ones that reflect, reflect, and the ones that don’t, don’t..."
C.a1
All Teacher Have the
Capacity to Improve
“...I think is a better reflection for the teacher of what they are doing in the
classroom and it’s...easier... for me to then turn around and say, 'This is where
you are at [on the spectrum]...what I envision doing is beginning to focus...on
places where the teacher either needs to really grow to help students achieve
higher or for your teachers that are already effective, beginning to see where
you could begin to push them so that their efficacy in the classroom grows."
D.a1, D.a2, D.a3
All Teacher Have the
Capacity to Improve
“And because the way our [old] system worked...if you’re not doing anything
bad or you just kind of go with the flow you usually just pass...so because
they’ve never have any 'Needs Improvement' [ratings] they had no needs. So
it’s a really different outlook of everybody has needs. Everybody has areas
where they need to grow. You’re not proficient in everything. And let’s look
at some of those areas. So it’s a really different mindset...”
E.a1, E.a2, E.a4, E.a5
Some Teachers Have
the Capacity to Improve
"... I’m talking 10, 15, maybe 20% of the faculty can pull this [the EGDC]
off...the rest of the 80% you’re going to have an attitude problem,
philosophical problem..."
Note. Administrators are identified by their school followed by a small “a” and there associated number, where A.a1 is School A, Administrator 1. In the
case of School D and School E where multiple administrators were interviewed, these pieces of evidence reflect the administrators’ perspective on average.
Also, as mentioned previously, insufficient data was collected on Administrator 3 in School E so this individual is excluded from the analysis.
144
Table 10. Factors Shaping Administrator Sensemaking and Relationship to their Understanding of the Purpose of the EGDC
Participant
Level of Previous
Experience
All Teachers Have the
Capacity/Willingness to Improve
EGDC Understanding
A.a1 High Yes Growth
C.a1 High Yes Growth
D.a1 Medium Yes Growth
D.a2 Medium Yes Growth
D.a3 Medium Yes Growth
B.a1 Medium No Combination
E.a1 Low No Combination
E.a2 Low No Combination
E.a4 Low No Combination
E.a5 Low No Combination
Note. Administrators are identified by their school followed by a small “a” and there associated number, where A.a1 is School A, Administrator 1. Also, as
mentioned previously, insufficient data was collected on Administrator 3 in School E so this individual is excluded from the analysis.

145
Table 11. Examples of Evidence Classification for Factors Shaping Teacher Sensemaking - Previous Evaluation Experiences
Participant Evidence Level
A.t1, A.t2
"In the past..[it's] what did you do for those 30-minute observations three
times a year which isn't necessarily the most effective way to evaluate
yourself, or for someone to evaluate somebody..."
Neutral
B.t1, B.t2, B.t3, B.t4
“I mean I’ve only been here four years, out of my thirteen and she’s unique,
this lady our principal. She’s very thorough [with evaluations] I’d say...and I
thought that [Stull] was really good."
Positive
C.t1, C.t3, C.t4
"...whenever I have been observed [under Stull] it was just for five and
maximum like seven to eight minutes...but with this [the EGDC] it is just
throughout the whole period…"
Neutral
D.t1, D.t3, D.t4, D.t5
“...The Stull was really not an evaluation process at all -- It's basically a
checklist..."
Neutral
E.t1, E.t2, E.t3 "No Stull was just a joke here…” Negative
Note. Teachers are identified by their school followed by a small “t” and there associated number, where A.t1 is School A, Teacher 1. The evidence selected
represents one example of the experience of the teachers listed. Insufficient data was collected on Teacher 2 in School C and Teacher 2 at School D so these
individuals are excluded from the analysis.

146
Table 12. Examples of Evidence Classification for Factors Shaping Teacher Sensemaking - Interactions with Administrators
Participant Frequency Nature Evidence
A.t1, A.t2, B.t1,
C.t4, D.t3, E.t3
Infrequent
Instruction-
Focused
"[During the conference]...we talked about her [my administrators] observations
and the things that she saw positive, things that she was seeing, and not
necessarily saying, “Oh, that’s a bad thing,” but, turning it back on myself to
saying, “How did you feel about it?” and really forcing me to think about that
particular—the instruction that I did and how effective it was and stuff. She
made me more aware of the things that I was doing, and made me think...”
B.t2, C.t3, D.t4,
D.t5, E.t2
Infrequent
Process-
Focused
“...I don’t do bells and whistles. There is my lesson plan. You need to read it.
Come and observe me, tell me what you find, I don’t even think these
administrators understand our standards...We didn’t have a conference. That’s
the big thing. I don’t have a preconference. ”
C.t1, D.t1 Frequent
Process- and
Instruction-
Focused
"...I said, would you – will you be looking for an introductory lesson or a
continuation and they chose introduction...both observed a whole period
and...thankfully right after that period was my conference so they spent a great
deal of time talking with me, discussing [the lesson]."
E.t1 Frequent
Process-
Focused
"...because I have been sort of on my administrators, and I keep them in the
loop...and have had a good experience with their engagement...[the conferences]
happened and they have been genuine and they’ve been supportive, I don’t
know how insightful... not very critical..
Note. Teachers are identified by their school followed by a small “t” and there associated number, where A.t1 is School A, Teacher 1. The evidence selected
represents one example of the experience of the teachers listed. Insufficient data was collected on Teacher 2 in School C and Teacher 2 at School D so these
individuals are excluded from the analysis.

147

Table 13. Factors Shaping Teacher Sensemaking and the Relationship to their Understanding of
the Purpose of the EGDC
Participant
Nature of
Previous
Experience
Teacher-Administrator Interactions
EGDC
Understanding
A.t2 Neutral Infrequent Instruction-Focused Growth
C.t4 Neutral Infrequent Instruction-Focused Growth
D.t1 Neutral Frequent Both Growth
D.t3 Neutral Infrequent Instruction-Focused Growth
D.t4 Neutral Infrequent Process-Focused Growth
D.t5 Neutral Infrequent Process-Focused Growth
B.t1 Positive Infrequent Instruction-Focused Combination
A.t1 Neutral Infrequent Instruction-Focused Combination
C.t1 Neutral Frequent Both Combination
E.t3 Negative Infrequent Instruction-Focused Combination
B.t2 Positive Infrequent Process-Focused Accountability
B.t3 Positive Infrequent Process-Focused Accountability
B.t4 Positive Infrequent Process-Focused Accountability
C.t3 Neutral Infrequent Process-Focused Accountability
E.t1 Negative Frequent Process-Focused Accountability
E.t2 Negative Infrequent Process-Focused Accountability
Note. Teachers are identified by their school followed by a small “t” and there associated number, where A.t1 is School
A, Teacher 1. Insufficient data was collected on Teacher 2 in School C and Teacher 2 at School D so these individuals
are excluded from the analysis.

148
Table 14. The Relationship Between Stakeholder Understanding and Implementation
Teacher-
Administrator(s)
Pair
Administrator
Understanding
Teacher Understanding
Nature of
Implementation
C.a1 – C.t4 Growth Growth Instruction-Focused
D.a1 – D.t3 Growth Growth Instruction-Focused
A.a1 – A.t2 Growth Growth Instruction-Focused
D.a3 – D.t1 Growth Growth Instruction-Focused
A.a1 – A.t1 Growth Combination Instruction-Focused
C.a1 – C.t1 Growth Combination Instruction-Focused
E.a5 – E.t3 Combination Combination Instruction-Focused
B.a1 – B.t1 Combination Combination Process-Focused
B.a1 – B.t2 Combination Accountability Process-Focused
B.a1 – B.t3 Combination Accountability Process-Focused
B.a1 – B.t4 Combination Accountability Process-Focused
D.a1 – D.t5 Growth Growth Process-Focused
D.a2 – D.t4 Growth Growth Process-Focused
E.a2/E.a4 – E.t1 Combination Accountability Process-Focused
C.a1 – C.t3 Growth Accountability Neither
E.a1 – E.t2 Combination Accountability Neither
Note. Teachers and Administrators are identified by their school followed by a small “t” or “a”, respectively, and
there associated number, where A.t1 is School A, Teacher 1 and A.a1 is School A, Administrator 1. Insufficient data
was collected on Administrator 3 in School E, Teacher 2 in School C, and Teacher 2 at School D so these
individuals are excluded from the analysis.

149

Figure 1. The Relationship between Sensemaking and Implementation

Understanding
Previous Experiences
(e.g., Stull Evaluations)
Existing Knowledge
(e.g., NBC, BTSA Provider)
Beliefs/V alues
(e.g., Constructivism)
Organizational Arrangements
(e.g., PLCs, Departments)
Collegial Interactions
(e.g., teacher-administrator)
Structure of Learning
Opportunities
(e.g., workshops, conferences)
Policy Documents
(e.g., training materials,
rubrics/protocols, data reports)
Communications
(e.g., internal memos, media
articles, meeting minutes)
Individual Sensemaking

Situated Cognition

Role of Representations

Implementation

150

Figure 2. Representation of the Multiple-Case (Embedded) Design

Admin1 Admin2 Admin3
Context
School A
Administrator 1
Teacher 1 Teacher 2
Context
School C
Administrator 1
Teacher 4
Teacher 2
Teacher 3
Teacher 1
Context
School D
Teacher 5
Teacher 2
Teacher 3
Teacher 1
Context
School E
Admin1
Teacher 2 Teacher 3
Teacher 1
Admin2 Admin3
Context
School B
Administrator 1
Teacher 4
Teacher 2
Teacher 3
Teacher 1
Teacher 4
Admin5 Admin4
Note. The grayed out boxes indicate participants for which insufficient data on the sensemaking process
and/or implementation of the EGDC was collected. These individuals are excluded from all analyses
presented throughout the paper.

151

Figure 3. List of Documents for Analysis

EGDC Documents
The Educator Growth and Development Cycle Process Guidelines
The Educator Growth and Development Cycle Observation Rubric
The Teaching and Learning Framework
Academic Growth Over Time Reports
Educator Growth and Development Cycle Teacher Self-Assessment
Educator Growth and Development Cycle Teacher Lesson Design
Educator Growth and Development Cycle Training Materials

Other Teacher Evaluation Documents
The STULL Evaluation Process Guidelines
The STULL Evaluation Observation Rubric
The California Standards for the Teaching Profession
Section X. (Evaluation and Discipline) of the LAUSD and UTLA CBA

Internal Communication Documents
Initial Implementation Phase Newsletters
Internal Feedback Reports
Focus Element Memo
Doe vs. Deasy Internal Memo
Teacher and Administrator Recruitment Memo

152

Appendix B

Table 1. Total Panel Sample Size for Elementary and Secondary Teachers

School-Level Analysis Classroom-Level Analysis
School Year Elementary Secondary Elementary Secondary
2007-2008 14,227 12,247 14,184 12,279
2008-2009 12,776 11,432 12,868 11,502
2009-2010 12,234 11,253 12,329 11,313
2010-2011 11,826 10,376 11,779 10,575
2011-2012 11,322 9,669 10,825 9,935
Total 62,385 54,977 61,985 55,604
Note. This table contains sample sizes for all elementary and secondary teachers in non-charter, non-
alternative schools for all relevant samples. The school-level analysis panel includes 16,411 and 15,587
unique elementary and secondary teachers, respectively. The classroom-level analysis panel includes
16,343 unique elementary and 15,566 unique secondary teachers

153

Table 2. Summary Statistics for Elementary Treatment and Comparison Groups

Treated CG1 CG2
Characteristic N Mean N Mean N Mean
Female 209 0.83 1337 0.77+ 11715 0.81
Minority 209 0.64 1333 0.71* 11681 0.66
<=3 Years Exp. 209 0.00 1298 0.04** 11607 0.02*
4-6 Years Exp. 209 0.06 1298 0.10+ 11607 0.08
7-9 Years Exp. 209 0.15 1298 0.13 11607 0.13
10+ Years Exp. 209 0.79 1298 0.73+ 11607 0.78
Permanent 209 0.98 1337 0.89*** 11715 0.95*
Probationary 209 0.01 1337 0.04** 11715 0.02
Temporary 209 0.01 1337 0.06*** 11715 0.03+
Intern 209 0.01 1337 0.01 11715 0.01
Bachelors 209 0.61 1334 0.66 11703 0.67+
Masters 209 0.38 1334 0.33 11703 0.32+
Doctorate 209 0.01 1334 0.01 11703 0.01
# Needs Imp. or Below Stull Ratings 67 0.24 492 0.88+ 4478 0.493
National Board Certified 209 0.10 1337 0.03*** 11715 0.05***
API Score 206 775 1269 769 11434 785*
Note. This table provides summary statistics for the treatment and comparison groups using the full sample of
teachers. Tests of group mean difference between the treatment group and each comparison group are also provided
in each of the comparison group columns with the following significance levels +p<0.10 *p<0.05 **p<0.01
***p<0.001. The treatment group does reduce in sample size slightly when using CG3 as only observations on the
area of common support are used in the final analysis. This smaller sample size matches that of CG3.

154

Table 3. Summary Statistics for Secondary Treatment and Comparison Groups

Treated CG1 CG2
Characteristic N Mean N Mean N Mean
Female 149 0.60 1693 0.50* 10029 0.53+
Minority 146 0.60 1663 0.58 9847 0.55
<=3 Years Exp. 149 0.07 1656 0.09 9850 0.06
4-6 Years Exp. 149 0.19 1656 0.18 9850 0.16
7-9 Years Exp. 149 0.23 1656 0.12*** 9850 0.15**
10+ Years Exp. 149 0.51 1656 0.60* 9850 0.62**
Permanent 149 0.92 1694 0.88 10031 0.91
Probationary 149 0.04 1694 0.05 10031 0.04
Temporary 149 0.01 1694 0.05* 10031 0.04+
Intern 149 0.03 1694 0.02 10031 0.02
Bachelors 149 0.50 1687 0.61** 10001 0.61**
Masters 149 0.47 1687 0.37* 10001 0.38*
Doctorate 149 0.03 1687 0.02 10001 0.02
# Needs Imp. or Below Stull Ratings 50 0.36 701 0.92+ 3527 0.703
National Board Certified 149 0.07 1694 0.02*** 10031 0.03*
API Score 149 685 1656 665*** 9802 687
Note. This table provides summary statistics for the treatment and comparison groups using the full sample of
teachers. Tests of group mean difference between the treatment group and each comparison group are also
provided in each of the comparison group columns with the following significance levels +p<0.10 *p<0.05
**p<0.01 ***p<0.001. The treatment group does reduce in sample size slightly when using CG3 as only
observations on the area of common support are used in the final analysis. This smaller sample size matches
that of CG3.

155

Table 4. Covariates Used in the Propensity Score Selection Model
Variable Definition
Female Gender (1=Female, 0=Male)
Minority Race/Ethnicity (1=Non-White, 0=White)
4-6 Years Exp. Early Career (1=4-6 Years of Experience, 0=All Other)
7-9 Years Exp. Mid-Career (1=7-9 Years of Experience, 0=All Other)
Probationary Employment Status (1=Probationary Teacher, 0=Other)
Intern Employment Status (1=Intern Teacher, 0=Other)
Masters Highest Degree Earned (1=Master’s Degree, 0=Other)
Doctorate Highest Degree Earned (1=Doctorate, 0=Other)
National Board Certified Advanced Certification (1=National Board Certified, 0=Not NBC)
Evaluated in Past 3 Years Evaluation Eligibility (=1 if Evaluated between 2007-2008 and 2009-2010)
Classroom-Level % ELL Classroom ELL Status (=% of ELL Students)
Classroom-Level % FRL Classroom FRL Status (=% of FRL Eligible Students)
Classroom-Level %Proficient or Adv. Classroom Performance (=% of Students Proficient or Advanced in ELA or Math)
IIP School Administrator Participation (=1 if School Participated in the IIP)
Above Avg. IIP Participation @ School Concentration of Participants (=1 if School has Above Average # of IIP participants)
School-Level % ELL School ELL Status (=% of ELL Students)
School-Level %FRL School FRL Status (=% of FRL Students)
School-Level %Minority School Minority Status (=% of Minority Students)
School-Level %Students w/Disabilities School Disabilities Status (=% of Disabilities Students)
API Score School Performance (=API Score)
Note. This table contains all covariates used in the Elementary and Secondary selection models for CG3 identification using nearest neighbor matching. It is
important to note that <=3 Years of Experience and Temporary Employment Status were not included as covariates in the selection model as no participants
were novice or temporary teachers as of 2010-2011 school year when volunteers opted into treatment.

156

Table 5. Results of Elementary Teacher Balancing Tests
Mean T-Test
Variable Sample Treated Control %Bias %Bias Reduced t p>t
Female Unmatched 0.82 0.81 2.30 0.320 0.749
Matched 0.80 0.79 3.00 -30.00 0.260 0.792
Minority Unmatched 0.64 0.65 -3.40 -0.490 0.626
Matched 0.66 0.63 7.20 -109.00 0.670 0.504
4-6 Years Exp. Unmatched 0.06 0.07 -3.60 -0.490 0.623
Matched 0.07 0.03 13.60 -280.90 1.450 0.147
7-9 Yrs. Exp. Unmatched 0.15 0.13 5.80 0.850 0.395
Matched 0.15 0.15 0.00 100.00 0.000 1.000
Probationary Unmatched 0.00 0.01 -5.90 -0.720 0.469
Matched 0.01 0.01 -6.70 -13.10 -0.580 0.563
Intern Unmatched 0.01 0.01 2.80 0.430 0.670
Matched 0.01 0.00 12.40 -346.90 1.420 0.157
Masters Unmatched 0.37 0.33 8.30 1.190 0.234
Matched 0.37 0.39 -4.80 42.10 -0.440 0.660
Doctorate Unmatched 0.01 0.01 5.50 0.940 0.348
Matched 0.01 0.01 0.00 100.00 0.000 1.000
National Board Certified Unmatched 0.10 0.05 20.20 3.480 0.001
Matched 0.05 0.03 4.40 78.00 0.540 0.587
Evaluated in Past 3 Years Unmatched 0.88 0.97 -35.80 -7.670 0.000
Matched 0.95 0.93 6.60 81.60 0.670 0.501
IIP School Unmatched 0.94 0.11 305.60 38.730 0.000
Matched 0.93 0.94 -2.10 99.30 -0.220 0.830
Above Avg. IIP Participation @ School Unmatched 0.90 0.09 281.10 40.930 0.000
Matched 0.89 0.89 -2.00 99.30 -0.170 0.866
School-Level %ELL Unmatched 0.38 0.35 13.80 2.010 0.045
Matched 0.38 0.38 1.40 89.70 0.140 0.888
School-Level %FRL Unmatched 0.87 0.84 10.70 1.420 0.156
Matched 0.88 0.89 -3.60 66.50 -0.390 0.696
School-Level %Minority Unmatched 0.89 0.87 9.90 1.280 0.201
Matched 0.90 0.91 -4.30 57.10 -0.470 0.635
School-Level %Students w/Disabilities Unmatched 0.12 0.12 -0.80 -0.120 0.906
Matched 0.12 0.12 -5.10 -532.30 -0.460 0.642
API score Unmatched 775 785 -15.90 -2.230 0.025
Matched 772 771 1.00 94.00 0.100 0.921

157

Table 6. Results of Secondary Teacher Balancing Tests
Mean T-Test
Variable Sample Treated Control %Bias %Bias Reduced t p>t
Female Unmatched 0.60 0.52 15.40 1.820 0.068
Matched 0.58 0.56 4.80 69.00 0.380 0.705
Minority Unmatched 0.60 0.55 10.30 1.230 0.219
Matched 0.60 0.52 15.90 -53.90 1.260 0.208
4-6 Years Exp. Unmatched 0.19 0.16 6.90 0.850 0.394
Matched 0.20 0.17 8.30 -20.40 0.650 0.516
7-9 Yrs. Exp. Unmatched 0.23 0.15 20.40 2.660 0.008
Matched 0.19 0.20 -2.00 90.20 -0.160 0.874
Probationary Unmatched 0.03 0.03 -2.40 -0.280 0.779
Matched 0.03 0.03 0.00 100.00 0.000 1.000
Intern Unmatched 0.02 0.02 3.90 0.500 0.620
Matched 0.02 0.00 11.80 -205.80 1.420 0.157
Masters Unmatched 0.48 0.38 18.80 2.280 0.023
Matched 0.46 0.50 -9.60 49.00 -0.750 0.453
Doctorate Unmatched 0.03 0.02 8.10 1.130 0.258
Matched 0.02 0.02 -5.40 33.50 -0.450 0.653
National Board Certified Unmatched 0.07 0.03 16.50 2.410 0.016
Matched 0.06 0.06 -3.60 78.30 -0.270 0.791
Evaluated in Past 3 Years Unmatched 0.92 0.94 -9.80 -1.280 0.200
Matched 0.92 0.93 -3.10 68.60 -0.240 0.812
IIP School Unmatched 0.92 0.17 230.70 23.940 0.000
Matched 0.91 0.91 2.40 99.00 0.220 0.828
Above Avg. IIP Participation @ School Unmatched 0.92 0.15 238.50 25.510 0.000
Matched 0.91 0.91 0.00 100.00 0.000 1.000
School-Level %ELL Unmatched 0.21 0.22 -6.90 -0.870 0.387
Matched 0.22 0.23 -0.30 95.40 -0.030 0.979
School-Level %FRL Unmatched 0.79 0.79 1.00 0.110 0.911
Matched 0.79 0.80 -4.40 -348.10 -0.370 0.714
School-Level %Minority Unmatched 0.91 0.89 12.10 1.290 0.197
Matched 0.91 0.92 -2.60 78.40 -0.250 0.806
School-Level %Students w/Disabilities Unmatched 0.11 0.12 -10.40 -1.450 0.147
Matched 0.12 0.12 6.80 34.20 0.690 0.488
API score Unmatched 687 688 -0.80 -0.110 0.914
Matched 675 674 1.40 -72.70 0.120 0.903

158

Table 7. Marginal Effects for Elementary Treated and Non-Treated Teachers School-Level Mobility
CG1 CG2 CG3
Move vs. Stay Margin S.E. P>z Margin S.E. P>z Margin S.E. P>z
Non-Participant Pre-Pilot 0.042 0.003 0.000 0.034 0.001 0.000 0.038 0.008 0.000
Participant Pre-Pilot 0.049 0.008 0.000 0.049 0.008 0.000 0.037 0.007 0.000
Non-Participant Post-Pilot 0.084 0.008 0.000 0.080 0.003 0.000 0.095 0.023 0.000
Participant Post-Pilot 0.088 0.020 0.000 0.088 0.020 0.000 0.073 0.020 0.000
DiD Estimator -0.003 0.023 0.910 -0.007 0.021 0.737 -0.021 0.032 0.515
Note. This table contains the marginal effects for elementary treated and non-treated teachers from model (1) before and after the
EGDC pilot reported as predicted probabilities separately for each comparison group. The table also presents the marginal effect for
the DID estimator calculated using the method outlined above.

Table 8. Marginal Effects for Secondary Treated and Non-Treated Teachers School-Level Mobility
CG1 CG2 CG3
Move vs. Stay Margin S.E. P>z Margin S.E. P>z Margin S.E. P>z
Non-Participant Pre-Pilot 0.067 0.003 0.000 0.067 0.001 0.000 0.093 0.012 0.000
Participant Pre-Pilot 0.060 0.010 0.000 0.060 0.010 0.000 0.052 0.009 0.000
Non-Participant Post-Pilot 0.166 0.010 0.000 0.134 0.004 0.000 0.164 0.034 0.000
Participant Post-Pilot 0.118 0.027 0.000 0.118 0.027 0.000 0.094 0.027 0.000
DiD Estimator -0.041 0.028 0.150 -0.009 0.027 0.739 -0.029 0.044 0.513
Note. This table contains the marginal effects for secondary treated and non-treated teachers from model (1) before and after the EGDC
pilot reported as predicted probabilities separately for each comparison group. The table also presents the marginal effect for the DID
estimator calculated using the method outlined above.

159

Table 9. Difference-in-Difference Estimates for Elementary School-Level Mobility in Pre -and Post-Pilot Periods
CG1 CG2 CG3
Move vs. Stay Margin S.E. P>z Margin S.E. P>z Margin S.E. P>z
Post-Pilot vs. 2008-2009 -0.009 0.026 0.727 -0.002 0.024 0.936 -0.063 0.050 0.212
Post-Pilot vs. 2009-2010 -0.005 0.030 0.878 -0.025 0.029 0.379 -0.035 0.052 0.493
Post-Pilot vs. 2010-2011 -0.013 0.026 0.616 -0.009 0.025 0.706 -0.013 0.051 0.795
Pre-Pilot vs. 2008-2009 0.004 0.021 0.848 0.007 0.021 0.722 -0.050 0.045 0.268
Pre-Pilot vs. 2009-2010 0.008 0.027 0.754 -0.016 0.026 0.532 -0.022 0.046 0.634
Note. This table contains the DID estimates of the marginal effects for elementary treated relative to non-treated teachers from model (2) for post-
pilot (2011-2012), and pre-pilot (2010-2011) time periods reported as predicted probabilities.

Table 10. Difference-in-Difference Estimates for Secondary School-Level Mobility in Pre -and Post-Pilot Periods
CG1 CG2 CG3
Move vs. Stay Margin S.E. P>z Margin S.E. P>z Margin S.E. P>z
Post-Pilot vs. 2008-2009 -0.081 0.032 0.012 -0.043 0.030 0.154 -0.063 0.050 0.212
Post-Pilot vs. 2009-2010 -0.044 0.036 0.220 -0.006 0.034 0.851 -0.035 0.052 0.493
Post-Pilot vs. 2010-2011 -0.042 0.034 0.216 -0.007 0.033 0.833 -0.013 0.051 0.795
Pre-Pilot vs. 2008-2009 -0.038 0.032 0.227 -0.036 0.031 0.244 -0.050 0.045 0.268
Pre-Pilot vs. 2009-2010 -0.001 0.032 0.964 0.001 0.031 0.985 -0.022 0.046 0.634
Note. This table contains the DID estimates of the marginal effects for secondary treated relative to non-treated teachers from model (2) for post-pilot
(2011-2012) and pre-pilot (2010-2011) time periods reported as predicted probabilities.

160

Table 11. Marginal Effects for Elementary Treated and Non-Treated Teachers Classroom-Level Mobility
CG1 CG2 CG3
Move vs. Stay Margin S.E. P>z Margin S.E. P>z Margin S.E. P>z
Non-Participant Pre-Pilot 0.105 0.005 0.000 0.100 0.002 0.000 0.108 0.015 0.000
Participant Pre-Pilot 0.103 0.012 0.000 0.103 0.012 0.000 0.097 0.012 0.000
Non-Participant Post-Pilot 0.125 0.010 0.000 0.117 0.003 0.000 0.113 0.026 0.000
Participant Post-Pilot 0.117 0.023 0.000 0.117 0.023 0.000 0.120 0.025 0.000
DiD Estimator -0.006 0.027 0.821 -0.002 0.025 0.927 0.019 0.038 0.621
Note. This table contains the marginal effects for elementary treated and non-treated teachers from model (1) before and after the EGDC pilot reported as
predicted probabilities separately for each comparison group. The table also presents the marginal effect for the DID estimator calculated using the method
outlined above.

Table 12. Marginal Effects for Secondary Treated and Non-Treated Teachers Classroom-Level Mobility
CG1 CG2 CG3
Move vs. Stay Margin S.E. P>z Margin S.E. P>z Margin S.E.. P>z
Non-Participant Pre-Pilot 0.084 0.005 0.000 0.081 0.002 0.000 0.072 0.014 0.000
Participant Pre-Pilot 0.119 0.018 0.000 0.119 0.018 0.000 0.113 0.018 0.000
Non-Participant Post-Pilot 0.171 0.010 0.000 0.157 0.004 0.000 0.134 0.031 0.000
Participant Post-Pilot 0.161 0.029 0.000 0.161 0.029 0.000 0.151 0.032 0.000
DiD Estimator -0.045 0.034 0.181 -0.033 0.032 0.301 -0.024 0.048 0.626
Note. This table contains the marginal effects for secondary treated and non-treated teachers from model (1) before and after the EGDC pilot reported as
predicted probabilities separately for each comparison group. The table also presents the marginal effect for the DID estimator calculated using the method
outlined above.

161

Table 13. Difference-in-Difference Estimates for Elementary Classroom-Level Mobility in Pre -and Post-Pilot Periods
CG1 CG2 Full Sample
Move vs. Stay Margin S.E. P>z Margin S.E. P>z Margin S.E. P>z
Post-Pilot vs. 2008-2009 -0.006 0.033 0.864 0.011 0.031 0.719 0.036 0.048 0.456
Post-Pilot vs. 2009-2010 -0.010 0.033 0.768 -0.003 0.031 0.912 0.008 0.046 0.869
Post-Pilot vs. 2010-2011
-0.013 0.033 0.696 -0.008 0.031 0.791 0.036 0.046 0.431
Pre-Pilot vs. 2008-2009 0.007 0.033 0.828 0.019 0.031 0.531 0.000 0.044 0.993
Pre-Pilot vs. 2009-2010
0.003 0.032 0.926 0.005 0.030 0.876 -0.029 0.043 0.508
Note. This table contains the DID estimates of the marginal effects for elementary treated relative to non-treated teachers from model (2) for post-pilot (2011-
2012), and pre-pilot (2010-2011) time periods reported as predicted probabilities.

Table 14. Difference-in-Difference Estimates for Secondary Classroom-Level Mobility in Pre -and Post-Pilot Periods
CG1 CG2 CG3
Move vs. Stay Margin S.E. P>z Margin S.E. P>z Margin S.E. P>z
Post-Pilot vs. 2008-2009 -0.058 0.039 0.139 -0.048 0.037 0.196 -0.058 0.054 0.281
Post-Pilot vs. 2009-2010 -0.035 0.037 0.348 -0.025 0.036 0.483 -0.032 0.053 0.541
Post-Pilot vs. 2010-2011 -0.024 0.038 0.523 -0.006 0.037 0.877 -0.021 0.056 0.709
Pre-Pilot vs. 2008-2009 -0.033 0.037 0.364 -0.043 0.036 0.236 -0.037 0.043 0.386
Pre-Pilot vs. 2009-2010
-0.011 0.032 0.740 -0.019 0.031 0.532 -0.012 0.044 0.791
Note. This table contains the DID estimates of the marginal effects for secondary treated relative to non-treated teachers from model (2) for post-pilot (2011-
2012), and pre-pilot (2010-2011) time periods reported as predicted probabilities.

162
Figure 1. Unadjusted School-Level Mobility of Elementary Treated and CG1 Teachers

Figure 2. Unadjusted Classroom-Level Mobility of Elementary Treated and CG1 Teachers

0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Moving Schools
IIP CG1
Pre-SBMMTES Pilot
0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Moving Classrooms
IIP CG1
Pre-SBMMTES Pilot
163
Figure 3. Unadjusted School-Level Mobility of Secondary Treated and CG1 Teachers

Figure 4. Unadjusted Classroom-Level Mobility of Secondary Treated and CG1 Teachers

0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Moving Schools
IIP CG1
Pre-SBMMTES Pilot
0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Moving Classrooms
IIP CG1
Pre-SBMMTES Pilot
164
Figure 5. Unadjusted School-Level Mobility of Elementary Treated and CG2 Teachers

Figure 6. Unadjusted Classroom-Level Mobility of Elementary Treated and CG2 Teachers

0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Moving Schools
IIP CG2
Pre-SBMMTES Pilot
0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Moving Classrooms
IIP CG2
Pre-SBMMTES Pilot
165
Figure 7. Unadjusted School-Level Mobility of Secondary Treated and CG2 Teachers

Figure 8. Unadjusted Classroom-Level Mobility of Secondary Treated and CG2 Teachers

0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Moving Schools
IIP CG2
Pre-SBMMTES Pilot
0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Moving Classrooms
IIP CG2
Pre-SBMMTES Pilot
166
Figure 9. Distribution of Estimated Propensity Scores for Elementary Teachers by
Treatment Condition

0 10 20 30 40 50
0 .1 .2 0 .1 .2
Non-Participant Participant
Frequency
Estimated Propensity Scores for Elementary Teacher Pilot Participation
167
Figure 10. Distribution of Estimated Propensity Scores for Secondary Teachers by
Treatment Condition

0 10 20 30 40 50 60
0 .1 .2 0 .1 .2
Non-Participant Participant
Frequency
Estimated Propensity Scores for Secondary Teacher Pilot Participation
168
Figure 11. Unadjusted School-Level Mobility of Elementary Treated and CG3 Teachers

Figure 12. Unadjusted Classroom-Level Mobility of Elementary Treated and CG3
Teachers

0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Moving Schools
IIP CG3
Pre-SBMMTES Pilot
0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Moving Classrooms
IIP CG3
Pre-SBMMTES Pilot
169
Figure 13. Unadjusted School-Level Mobility of Secondary Treated and CG3 Teachers

Figure 14. Unadjusted Classroom-Level Mobility of Secondary Treated and CG3 Teachers

0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Moving Schools
IIP CG3
Pre-SBMMTES Pilot
0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Moving Classrooms
IIP CG3
Pre-SBMMTES Pilot
170
Appendix C

Table 1. Total Sample Sizes for Elementary and Secondary Teachers Across Years

School-Level Analysis Classroom-Level Analysis
School Year Elementary Secondary Elementary Secondary
2007-2008 14,227 12,247 14,184 12,279
2008-2009 12,776 11,432 12,868 11,502
2009-2010 12,234 11,253 12,329 11,313
2010-2011 11,826 10,376 11,779 10,575
2011-2012 11,322 9,669 10,825 9,935
Total 62,385 54,977 61,985 55,604
Note. This table contains sample sizes for all elementary and secondary teachers in non-charter, non-
alternative schools for all relevant samples. The school-level analysis panel includes 16,411 and 15,587
unique elementary and secondary teachers, respectively. The classroom-level analysis panel includes
16,343 unique elementary and 15,566 unique secondary teachers

171
Table 2. Tests of Mean Difference by Teacher, School, and Classroom Characteristics for Elementary Tested vs. Non-Tested Teachers
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Teacher/School Characteristic
Non-
Tested Tested
Non-
Tested Tested
Non-
Tested Tested
Non-
Tested Tested
Non-
Tested Tested
Female 0.88 0.74*** 0.87 0.74*** 0.87 0.75*** 0.87 0.74*** 0.87 0.74***
Minority 0.66 0.61*** 0.67 0.62*** 0.67 0.61*** 0.67 0.63*** 0.69 0.63***
<= 3 Years Exp. 0.12 0.12 0.10 0.09+ 0.04 0.02*** 0.03 0.02*** 0.02 0.01***
4-6 Years Exp. 0.12 0.14** 0.12 0.14** 0.10 0.11* 0.08 0.08 0.06 0.05***
7-9 Years Exp. 0.17 0.19** 0.16 0.18*** 0.14 0.17*** 0.12 0.14*** 0.11 0.14***
10+ Years Exp. 0.59 0.56*** 0.63 0.60*** 0.72 0.70* 0.78 0.76* 0.81 0.81
Permanent 0.85 0.85 0.87 0.89*** 0.93 0.97*** 0.93 0.95*** 0.94 0.95***
Probationary 0.11 0.12+ 0.08 0.08 0.03 0.01*** 0.02 0.02*** 0.02 0.01***
Temporary 0.02 0.02+ 0.03 0.02* 0.02 0.02 0.03 0.04 0.03 0.04
Intern 0.02 0.01*** 0.02 0.01*** 0.02 0.00*** 0.01 0.00*** 0.01 0.00***
Bachelors 0.73 0.71** 0.71 0.68*** 0.69 0.65*** 0.69 0.64*** 0.68 0.63***
Masters 0.27 0.28* 0.29 0.31** 0.30 0.34*** 0.31 0.35*** 0.31 0.36***
Doctorate 0.01 0.01** 0.01 0.01** 0.00 0.01*** 0.00 0.01*** 0.01 0.01
National Board Certified 0.03 0.03 0.03 0.04+ 0.04 0.04+ 0.04 0.05+ 0.04 0.06***
# Needs or Below Stull Evaluation Ratings 0.63 0.81** 0.69 0.82* 0.51 0.55 0.55 0.63 0.59 0.54
IIP Participant 0.01 0.01 0.01 0.02** 0.01 0.02 0.02 0.02 0.02 0.02*
Classroom % ELL 66.06 69.43*** 64.87 68.23*** 62.98 67.09*** 61.88 66.61*** 60.32 65.75***
Classroom % FRL 71.11 74.58*** 61.24 64.87*** 82.95 83.67+ 83.23 84.05+ 82.28 83.11+
Classroom %ProfAdv 14.14 43.70*** 15.14 47.56*** 14.12 48.88*** 15.55 51.45*** 15.50 52.98***
School-Level % Minority 0.86 0.89*** 0.86 0.88*** 0.85 0.88*** 0.85 0.87*** 0.85 0.87***
School-Level % ELL 0.43 0.43 0.39 0.39 0.37 0.37 0.36 0.36 0.35 0.34
School-Level % FRL 0.83 0.83 0.84 0.84 0.84 0.85 0.84 0.85+ 0.84 0.84
API 744 744 757 756 768 767 786 783* 795 794
Note. This table provides the proportion of teachers in tested and non-tested grades associated with each characteristic using the full school-level sample. Tests of group mean
difference between tested and non-tested teachers are also provided in the tested column with the following significance levels ***p<0.001 **p<0.01 *p<0.05 +p<0.10.

172
Table 3. Tests of Mean Difference by Teacher, School, and Classroom Characteristics for Secondary Tested vs. Non-Tested Teachers
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Teacher/School Characteristic
Non-
Tested Tested
Non-
Tested Tested
Non-
Tested Tested
Non-
Tested Tested
Non-
Tested Tested
Female 0.50 0.56*** 0.50 0.56*** 0.50 0.55*** 0.50 0.56*** 0.50 0.56***
Minority 0.51 0.54** 0.52 0.54** 0.52 0.56*** 0.53 0.57*** 0.53 0.58***
<= 3 Years Exp. 0.17 0.24*** 0.15 0.21*** 0.10 0.12*** 0.05 0.09*** 0.03 0.06***
4-6 Years Exp. 0.14 0.18*** 0.15 0.17*** 0.15 0.18*** 0.15 0.18*** 0.12 0.14***
7-9 Years Exp. 0.15 0.16 0.14 0.16** 0.15 0.17*** 0.14 0.16** 0.15 0.17**
10+ Years Exp. 0.53 0.42*** 0.56 0.46*** 0.61 0.53*** 0.66 0.58*** 0.71 0.63***
Permanent 0.76 0.65*** 0.80 0.71*** 0.88 0.82*** 0.91 0.87*** 0.94 0.90***
Probationary 0.16 0.20*** 0.13 0.17*** 0.07 0.09** 0.03 0.05*** 0.02 0.05***
Temporary 0.04 0.06*** 0.04 0.04 0.03 0.05*** 0.04 0.05** 0.03 0.03
Intern 0.05 0.10*** 0.03 0.08*** 0.02 0.04*** 0.01 0.03*** 0.01 0.02***
Bachelors 0.67 0.69* 0.65 0.66* 0.62 0.62 0.60 0.61 0.59 0.59
Masters 0.31 0.29** 0.33 0.31** 0.37 0.36 0.39 0.38 0.39 0.39
Doctorate 0.02 0.03 0.02 0.03+ 0.01 0.02** 0.01 0.02 0.02 0.03
National Board Certified 0.02 0.02* 0.02 0.03 0.03 0.03 0.03 0.03 0.03 0.04
# Needs or Below Stull Evaluation Ratings 1.16 1.42** 1.28 1.41 0.80 1.04** 0.67 0.98*** 0.88 1.05+
IIP Participant 0.01 0.01+ 0.01 0.01** 0.01 0.01** 0.01 0.02** 0.01 0.02**
Classroom % ELL 75.26 74.08*** 74.65 74.06+ 74.04 73.78 73.26 72.91 71.95 71.62
Classroom % FRL 69.36 71.24*** 65.38 65.91+ 75.60 77.77*** 75.69 78.05*** 76.43 78.34***
Classroom %ProfAdv 17.23 19.46*** 18.84 21.53*** 21.13 23.79*** 22.83 25.13*** 24.98 27.51***
School-Level % Minority 0.89 0.90 0.89 0.90 0.89 0.89 0.90 0.90 0.89 0.89
School-Level % ELL 0.29 0.30** 0.27 0.27** 0.24 0.25** 0.22 0.23*** 0.20 0.21**
School-Level % FRL 0.76 0.77*** 0.76 0.78*** 0.79 0.81*** 0.79 0.80*** 0.81 0.82**
API 639 643** 646 650** 665 669** 683 686* 703 707**
Note. This table provides the proportion of teachers in tested and non-tested grades associated with each characteristic using the full school-level sample. Tests of group
mean difference between tested and non-tested teachers are provided in the tested column with the following significance levels ***p<0.001 **p<0.01 *p<0.05 +p<0.10.

173
Table 4. Wald Tests of School-Level Mobility in Post- vs. Pre-PILOT and Post- vs. Pre-Times Periods for Elementary Teachers
Full Sample Low-Performing Schools
a
High % ELL Schools
b
High % FRL Schools
c
Switch vs. Stay Margin S.E P>z Margin S.E. P>z Margin S.E. P>z Margin S.E. P>z
Post-PILOT vs. 2008-2009 0.001 0.005 0.808 0.003 0.009 0.759 -0.003 0.009 0.742 0.004 0.007 0.592
Post-PILOT vs. 2009-2010 -0.003 0.005 0.559 -0.017 0.011 0.103 -0.011 0.010 0.271 -0.009 0.008 0.281
Post-Times vs. 2008-2009 0.000 0.004 0.923 -0.001 0.009 0.915 -0.009 0.009 0.288 0.001 0.007 0.884
Post-Times vs. 2009-2010 -0.005 0.005 0.352 -0.021 0.010 0.037 -0.018 0.010 0.074 -0.012 0.008 0.130
Leave vs. Stay Margin S.E P>z Margin S.E. P>z Margin S.E. P>z Margin S.E. P>z
Post-PILOT vs. 2008-2009 -0.002 0.005 0.678 0.003 0.009 0.753 0.010 0.008 0.205 0.000 0.007 0.960
Post-PILOT vs. 2009-2010 -0.007 0.005 0.163 -0.009 0.009 0.346 -0.006 0.009 0.532 -0.005 0.007 0.500
Post-Times vs. 2008-2009 0.006 0.004 0.157 0.008 0.008 0.272 0.009 0.007 0.245 0.006 0.006 0.314
Post-Times vs. 2009-2010 0.001 0.004 0.819 -0.003 0.008 0.724 -0.007 0.009 0.380 0.001 0.007 0.915
Note. This table contains the DID estimates of the marginal effects for Elementary tested relative to non-tested teachers from Model 1 for post-PILOT (2011-2012) and pre-
PILOT (2007-2008 to 2009-2010) as well as post-Times (2010-2011) and pre-Times (2007-2008 to 2009-2010) periods calculated using Wald tests and reported as predicted
probabilities.
a
Low-performing schools are schools in the bottom third of API score in LAUSD.
b
High % ELL schools are schools in the top third of proportion ELL students in
LAUSD.
c
High % FRL schools are schools in the top half of proportion of FRL eligible students in LAUSD.

174
Table 5. Wald Tests for Classroom-Level Mobility in Post- vs. Pre-PILOT and Post- vs. Pre-Times Periods for Elementary Teachers
Full Sample
Low-Performing
Classrooms
a

High % ELL
Classrooms
b

High % FRL
Classrooms
c

Switch vs. Stay Margin S.E P>z Margin S.E. P>z Margin S.E. P>z Margin S.E. P>z
Post-PILOT vs. 2008-2009 0.017 0.007 0.021 -0.240 0.056 0.000 0.011 0.014 0.434 0.028 0.016 0.082
Post-PILOT vs. 2009-2010 0.003 0.007 0.720 -0.180 0.052 0.000 0.006 0.014 0.643 0.032 0.019 0.088
Post-Times vs. 2008-2009 0.035 0.008 0.000 -0.002 0.055 0.967 0.034 0.014 0.017 0.025 0.018 0.168
Post-Times vs. 2009-2010 0.020 0.008 0.015 0.058 0.052 0.264 0.030 0.015 0.049 0.029 0.021 0.168
Leave vs. Stay Margin S.E P>z Margin S.E. P>z Margin S.E. P>z Margin S.E. P>z
Post-PILOT vs. 2008-2009 -0.006 0.005 0.232 -0.025 0.041 0.550 -0.012 0.008 0.147 0.004 0.011 0.738
Post-PILOT vs. 2009-2010 -0.010 0.005 0.038 -0.015 0.038 0.698 -0.021 0.009 0.015 -0.004 0.013 0.728
Post-Times vs. 2008-2009 0.006 0.004 0.178 -0.018 0.036 0.620 -0.005 0.008 0.521 0.008 0.010 0.417
Post-Times vs. 2009-2010 0.001 0.004 0.826 -0.008 0.032 0.807 -0.014 0.008 0.084 0.000 0.012 0.995
Note. This table contains the DID estimates of the marginal effects for Elementary tested relative to non-tested teachers from Model 1 for post-PILOT (2011-2012) and pre-PILOT
(2007-2008 to 2010-2011) as well as post-Times (2010-2011) and pre-Times (2007-2008 to 2009-2010) periods calculated using Wald tests and reported as predicted probabilities.
a
Low-performing classrooms are those in the bottom third of proficiency in ELA or Math on average within each school.
b
High % ELL classrooms are those in the top third of
proportion ELL students within each school.
c
High % FRL classrooms are those in the top third of proportion of FRL eligible students within each school.

175
Table 6. Wald Tests for School-Level Mobility in Post- vs. Pre-PILOT and Post- vs. Pre-Times Periods for Secondary Teachers
Full Sample Low-Performing Schools
a
High % ELL Schools
b
High % FRL Schools
c

Switch vs. Stay Margin Std. Err. P>z Margin Std. Err. P>z Margin Std. Err. P>z Margin Std. Err. P>z
Post-PILOT vs. 2008-2009 0.006 0.006 0.325 -0.006 0.013 0.649 -0.015 0.015 0.310 -0.007 0.012 0.548
Post-PILOT vs. 2009-2010 -0.021 0.007 0.005 -0.026 0.016 0.107 -0.052 0.018 0.005 -0.028 0.014 0.044
Post-Times vs. 2008-2009 0.005 0.007 0.478 0.003 0.014 0.796 0.006 0.014 0.650 -0.005 0.012 0.687
Post-Times vs. 2009-2010 -0.022 0.008 0.003 -0.017 0.017 0.313 -0.030 0.018 0.090 -0.026 0.014 0.066
Leave vs. Stay Margin Std. Err. P>z Margin Std. Err. P>z Margin Std. Err. P>z Margin Std. Err. P>z
Post-PILOT vs. 2008-2009 -0.002 0.006 0.710 0.010 0.012 0.428 0.001 0.013 0.923 0.001 0.012 0.919
Post-PILOT vs. 2009-2010 -0.015 0.007 0.024 -0.023 0.013 0.081 -0.020 0.014 0.148 -0.014 0.013 0.296
Post-Times vs. 2008-2009 0.005 0.006 0.375 0.021 0.012 0.070 0.017 0.012 0.169 0.005 0.012 0.649
Post-Times vs. 2009-2010 -0.008 0.006 0.226 -0.011 0.013 0.379 -0.004 0.013 0.734 -0.010 0.012 0.435
Note. This table contains the DID estimates of the marginal effects for Secondary tested relative to non-tested teachers from Model 1 for post-PILOT (2011-2012) and pre-PILOT (2007-
2008 to 2010-2011) as well as post-Times (2010-2011) and pre-Times (2007-2008 to 2009-2010) periods calculated using Wald tests and reported as predicted probabilities.
a
Low-
performing schools are schools in the bottom third of API score in LAUSD.
b
High % ELL schools are schools in the top third of proportion ELL students in LAUSD.
c
High % FRL schools
are schools in the top third of proportion of FRL eligible students in LAUSD.

176
Table 7. Wald Tests for Classroom-Level Mobility in Post- vs. Pre-PILOT and Post- vs. Pre-Times Periods for Secondary Teachers
Full Sample Low-Performing Schools
a
High % ELL Schools
b
High % FRL Schools
c

Move vs. Stay Margin S.E. P>z Margin S.E. P>z Margin S.E. P>z Margin S.E. P>z
Post-PILOT vs. 2008-2009 0.009 0.009 0.333 -0.004 0.019 0.830 -0.015 0.017 0.370 -0.031 0.018 0.082
Post-PILOT vs. 2009-2010 -0.004 0.010 0.705 -0.017 0.020 0.384 -0.011 0.018 0.533 -0.023 0.018 0.208
Post-Times vs. 2008-2009 0.008 0.009 0.376 0.004 0.019 0.843 -0.022 0.016 0.180 -0.021 0.017 0.214
Post-Times vs. 2009-2010 -0.005 0.010 0.618 -0.009 0.020 0.637 -0.018 0.018 0.317 -0.013 0.018 0.471
Note. This table contains the DID estimates of the marginal effects for Secondary tested relative to non-tested teachers from Model 1 for post-PILOT (2011-2012) and pre-PILOT
(2007-2008 to 2010-2011) as well as post-Times (2010-2011) and pre-Times (2007-2008 to 2009-2010) periods calculated using Wald tests and reported as predicted
probabilities.
a
Low-performing classrooms are those in the bottom third of proficiency in ELA or Math on average within each school.
b
High % ELL classrooms are those in the
top third of proportion ELL students within each school.
c
High % FRL classrooms are those in the top third of proportion of FRL eligible students within each school.

177
Table 8. Characteristics of Elementary Tested Teachers who Stay vs. Switch Schools and Stay vs. Leave LAUSD Post-Pilot
Full Low-Performing High % ELL High % FRL
Characteristic Stay Switch Leave Stay Switch Leave Stay Switch Leave Stay Switch Leave
<=3 Years Exp. 0.01 0.04*** 0.04*** 0.01 0.03** 0.01 0.01 0.02 0.02 0.01 0.03*** 0.02+
4-6 Years Exp. 0.02 0.05* 0.04 0.04 0.09** 0.00+ 0.03 0.06 0.03 0.03 0.03 0.03
7-9 Years Exp. 0.12 0.04*** 0.08+ 0.12 0.05+ 0.12 0.14 0.05* 0.08 0.11 0.05* 0.07
10+ Years Exp. 0.85 0.87 0.84 0.84 0.83 0.87 0.82 0.87 0.87 0.86 0.89 0.88
Permanent 0.99 0.82*** 0.80*** 0.98 0.76*** 0.85*** 0.98 0.82*** 0.87*** 0.98 0.84*** 0.86***
Probationary 0.00 0.01 0.06*** 0.01 0.01 0.04*** 0.01 0.00 0.01 0.00 0.01 0.06***
Bachelors 0.64 0.63 0.59 0.63 0.62 0.51* 0.64 0.65 0.54+ 0.63 0.63 0.62
Masters 0.35 0.37 0.41+ 0.36 0.38 0.49* 0.35 0.35 0.46* 0.36 0.37 0.38
Doctorate 0.01 0.00 0.01 0.01 0.00 0.00 0.01 0.00 0.00 0.01 0.00 0.00
National Board Certified 0.06 0.10** 0.05 0.05 0.10* 0.05 0.05 0.12** 0.04 0.06 0.09 0.05
# Needs or Below Stull Ratings 0.45 0.51 3.31*** 0.71 0.50 3.71*** 0.64 0.63 3.77*** 0.46 0.66 3.47***
3rd Grade ELA VAM 3.01 3.12 2.88 2.87 3.06 3.03 2.96 3.32+ 4.23 3.01 3.08 3.27
3rd Grade Math VAM 3.04 3.12 2.91 2.91 3.02 3.19 2.97 3.09 3.72 3.03 3.07 3.46
4th Grade ELA VAM 3.01 2.99 2.42* 2.92 2.92 2.66 3.08 3.06 3.04 3.01 2.93 2.93
4th Grade Math VAM 3.01 3.04 2.58 2.91 2.79 2.44 3.03 3.06 2.85 3.00 2.97 2.86
5th Grade ELA VAM 3.02 2.89 2.85 2.97 2.81 3.12 3.08 2.65* 3.09 3.08 2.73* 2.96
5th Grade Math VAM 3.05 2.75** 2.85 2.99 2.45** 3.35 3.11 2.51** 3.09 3.11 2.59** 2.91
6th Grade ELA VAM 3.05 2.83 3.11 2.95 2.96 3.12 2.82 2.99 2.66 2.98 2.72 3.15
6th Grade Math VAM 3.08 2.64+ 2.42+ 2.89 2.77 2.42 2.85 2.89 2.32 3.09 2.73 2.23+
Note. This table provides the proportion of tested elementary teachers who have the associated characteristic by mobility outcome for the post-pilot period
(SY2011-2012). Tests of group mean difference between tested teachers who stay in their current schools relative to switching schools and for tested teachers
who stay in their current school relative to leaving the district are also provided in the stay and leave columns, respectively. (--) indicates that this outcome could
not be estimated due to insufficient sample size for individuals with the associated characteristic and mobility outcome. ***p<0.001 **p<0.01 *p<0.05 +p<0.10.
178
Table 9. Characteristics of Elementary Tested Teachers who Stay vs. Switch Schools and Stay vs. Leave LAUSD Post-Times
Full Low-Performing High % ELL High % FRL
Characteristic Stay Switch Leave Stay Switch Leave Stay Switch Leave Stay Switch Leave
<=3 Years Exp. 0.01 0.05*** 0.07*** 0.01 0.05*** 0.03** 0.01 0.08*** 0.02 0.01 0.03** 0.06***
4-6 Years Exp. 0.06 0.02* 0.07 0.06 0.01+ 0.10 0.07 0.00* 0.09 0.06 0.00** 0.05
7-9 Years Exp. 0.14 0.08* 0.08* 0.14 0.06* 0.13 0.15 0.08+ 0.16 0.13 0.09 0.10
10+ Years Exp. 0.81 0.86+ 0.78 0.79 0.88+ 0.73 0.78 0.84 0.73 0.81 0.89* 0.79
Permanent 0.98 0.87*** 0.78*** 0.97 0.87*** 0.86*** 0.97 0.84*** 0.80*** 0.97 0.92*** 0.82***
Probationary 0.01 0.01 0.04*** 0.01 0.00 0.02 0.01 0.01 0.02 0.01 0.01 0.01
Bachelors 0.64 0.62 0.64 0.64 0.57 0.59 0.64 0.67 0.65 0.64 0.62 0.66
Masters 0.35 0.38 0.36 0.35 0.44 0.41 0.35 0.33 0.35 0.36 0.38 0.34
Doctorate 0.01 0.01 0.00 0.01 0.00 0.00 0.01 0.00 0.00 0.01 0.00 0.00
National Board Certified 0.05 0.04 0.05 0.05 0.04 0.03 0.05 0.03 0.06 0.05 0.04 0.03
# Needs or Below Stull Ratings 0.53 1.49*** 2.94*** 0.76 1.50 4.22*** 0.58 1.61** 1.61* 0.69 0.70 3.68***
3rd Grade ELA VAM 3.04 2.99 2.53** 2.92 2.78 2.24* 3.06 3.03 2.93 3.02 3.24 2.52+
3rd Grade Math VAM 3.03 3.00 2.55** 2.90 2.59 2.21* 3.05 3.08 2.49 3.02 3.35 2.54
4th Grade ELA VAM 3.04 2.86 2.85 2.93 2.68 2.96 3.17 2.99 2.96 3.06 2.69+ 2.82
4th Grade Math VAM 3.04 2.97 2.71* 2.87 3.01 2.96 3.11 2.99 2.70 3.02 2.82 2.67
5th Grade ELA VAM 3.03 2.91 2.89 2.89 2.82 2.98 3.14 2.94 3.00 3.06 3.03 2.80
5th Grade Math VAM 3.03 2.91 3.03 2.92 2.87 2.99 3.13 2.93 3.21 3.05 3.17 3.11
6th Grade ELA VAM 3.03 2.90 2.95 2.91 2.85 2.62 2.91 3.15 -- 3.01 2.94 3.05
6th Grade Math VAM 3.05 2.80 3.39 2.93 2.89 2.96 2.86 2.78 -- 3.00 2.88 3.36
Third Grade 0.39 0.35 0.33* 0.39 0.22** 0.29+ 0.40 0.33 0.29 0.39 0.28* 0.26*
Fourth Grade 0.31 0.27 0.31 0.31 0.24 0.37 0.32 0.23 0.47* 0.30 0.26 0.37
Fifth Grade 0.29 0.36* 0.36* 0.27 0.39** 0.35 0.28 0.38+ 0.29 0.28 0.35+ 0.30
Sixth Grade 0.05 0.11*** 0.06 0.07 0.19*** 0.08 0.04 0.09* 0.02 0.07 0.14*** 0.12*
Note. This table provides the proportion of tested elementary teachers who have the associated characteristic by mobility outcome for the post-LA Times period
(SY2010-2011). Tests of group mean difference between tested teachers who stay in their current schools relative to switching schools and for tested teachers
who stay in their current school relative to leaving the district are also provided in the stay and leave columns, respectively. (--) indicates that this outcome could
not be estimated due to insufficient sample size for individuals with the associated characteristic and mobility outcome.***p<0.001 **p<0.01 *p<0.05 +p<0.10.

179
Table 10. Characteristics of Elementary Tested Teachers who Stay vs. Switch Placements and Stay vs. Leave LAUSD Post-Pilot
Full Low-Performing High % ELL High % FRL
Characteristic Stay Switch Leave Stay Switch Leave Stay Switch Leave Stay Switch Leave
<=3 Years Exp. 0.00 0.02*** 0.04*** 0.02 0.00 0.00 0.01 0.02** 0.02 0.01 0.05*** 0.07**
4-6 Years Exp. 0.03 0.02 0.04 0.10 0.00 0.07 0.03 0.02 0.02 0.02 0.01 0.00
7-9 Years Exp. 0.13 0.03*** 0.08* 0.17 0.17 0.07 0.14 0.03*** 0.04* 0.14 0.01** 0.13
10+ Years Exp. 0.84 0.93*** 0.84 0.71 0.83 0.86 0.83 0.93*** 0.92+ 0.83 0.92* 0.80
Permanent 0.98 0.92*** 0.80*** 0.90 0.86 0.86 0.99 0.91*** 0.84*** 0.98 0.90*** 0.78***
Probationary 0.01 0.00 0.06*** 0.06 0.00 0.14 0.01 0.00 0.04** 0.00 0.00 0.00
Bachelors 0.63 0.67+ 0.59 0.61 0.57 0.50 0.64 0.69 0.56 0.64 0.57 0.50+
Masters 0.36 0.32 0.41 0.39 0.43 0.50 0.35 0.30 0.44 0.34 0.43 0.50*
Doctorate 0.01 0.01 0.01 0.00 0.00 0.00 0.01 0.01 0.00 0.02 0.00 0.00
National Board Certified 0.06 0.06 0.05 0.01 0.00 0.07 0.06 0.05 0.07 0.07 0.03 0.03
# Needs or Below Stull Ratings 0.40 0.79** 3.31*** 0.53 0.00 9.00*** 0.39 0.19 4.13*** 0.38 0.50 3.40***
3rd Grade ELA VAM 3.05 2.79*** 2.88 2.69 2.71 1.53 3.03 2.74* 2.79 3.01 2.53** 2.59
3rd Grade Math VAM 3.07 2.79*** 2.91 2.49 2.23 1.92 3.09 2.76** 2.85 3.07 2.54** 2.29
4th Grade ELA VAM 3.04 2.71** 2.42* 2.69 2.35 -- 3.10 2.79+ 2.18 3.14 2.63* 1.53
4th Grade Math VAM 3.05 2.68** 2.58 2.49 2.27 -- 3.09 2.64** 2.53 3.12 2.42** 1.92
5th Grade ELA VAM 3.03 2.84+ 2.85 2.86 2.28 -- 2.99 2.77 2.83 2.98 3.04 2.40
5th Grade Math VAM 3.07 2.64*** 2.85 2.61 1.91 -- 3.04 2.71+ 2.93 2.92 3.03 2.64
6th Grade ELA VAM 3.07 2.32** 3.11 -- -- -- 3.19 1.62** 3.69 3.22 2.78 --
6th Grade Math VAM 3.08 2.52+ 2.42+ -- -- -- 3.27 2.29+ 2.31 3.28 2.47 --
Note. This table provides the proportion of tested elementary teachers who have the associated characteristic by mobility outcome for the post-pilot period (SY2011-
2012). Tests of group mean difference between tested teachers who stay in their current placement relative to switching placements and for tested teachers who stay
in their current placement relative to leaving the district are also provided in the stay and leave columns, respectively. (--) indicates that this outcome could not be
estimated due to insufficient sample size for individuals with the associated characteristic and mobility outcome. ***p<0.001 **p<0.01 *p<0.05 +p<0.10.

180
Table 11. Characteristics of Elementary Tested Teachers who Stay vs. Switch Placements and Stay vs. Leave LAUSD Post-Times
Full Low-Performing High % ELL High % FRL
Characteristic Stay Switch Leave Stay Switch Leave Stay Switch Leave Stay Switch Leave
<=3 Years Exp. 0.01 0.03*** 0.07*** 0.02 0.08+ 0.12* 0.01 0.04*** 0.10*** 0.00 0.04*** 0.14***
4-6 Years Exp. 0.06 0.04** 0.07 0.12 0.16 0.24 0.07 0.03* 0.08 0.05 0.06 0.14+
7-9 Years Exp. 0.14 0.07*** 0.08* 0.13 0.13 0.12 0.15 0.05*** 0.05+ 0.14 0.07+ 0.05
10+ Years Exp. 0.80 0.87*** 0.78 0.73 0.64 0.53+ 0.78 0.89*** 0.77 0.81 0.84 0.68
Permanent 0.98 0.91*** 0.78*** 0.85 0.78 0.60** 0.97 0.95+ 0.77*** 0.98 0.88*** 0.78***
Probationary 0.01 0.03*** 0.04*** 0.03 0.13* 0.10 0.01 0.01 0.05*** 0.01 0.04** 0.09***
Bachelors 0.64 0.65 0.64 0.61 0.64 0.65 0.66 0.59+ 0.67 0.67 0.54* 0.74
Masters 0.35 0.34 0.36 0.39 0.35 0.35 0.34 0.40+ 0.33 0.32 0.45* 0.26
Doctorate 0.01 0.01 0.00 0.00 0.01 0.00 0.01 0.01 0.00 0.01 0.01 0.00
National Board Certified 0.06 0.03** 0.05 0.04 0.00+ 0.00 0.04 0.05 0.07 0.06 0.02 0.00
# Needs or Below Stull Ratings 0.48 0.99*** 2.94*** 0.66 0.80 1.80 0.57 0.68 2.15* 0.56 1.51* 5.20***
3rd Grade ELA VAM 3.07 2.82*** 2.53** 2.32 2.10 -- 3.07 2.76** 2.82 3.06 2.68* 2.10
3rd Grade Math VAM 3.07 2.75*** 2.55** 2.11 2.16 -- 3.04 2.76** 2.72 2.98 2.64+ 2.41
4th Grade ELA VAM 3.03 2.96 2.85 2.53 2.34 -- 3.05 2.98 2.98 3.10 3.19 3.56
4th Grade Math VAM 3.05 2.89 2.71* 2.35 1.87 -- 3.08 2.71* 2.79 3.10 3.29 3.16
5th Grade ELA VAM 3.04 2.74** 2.89 2.49 2.52 3.07 2.97 2.85 2.65 2.98 2.74 2.77
5th Grade Math VAM 3.06 2.65*** 3.03 2.40 2.01 3.62 2.99 2.70 2.49 2.96 2.64 2.93
6th Grade ELA VAM 2.99 3.18 2.95 2.85 2.50 2.29 3.07 3.09 -- 3.06 3.35 2.37
6th Grade Math VAM 3.05 2.96 3.39 2.79 2.69 3.31 3.08 2.71 -- 3.25 3.66 3.58
Note. This table provides the proportion of tested elementary teachers who have the associated characteristic by mobility outcome for the post-LA Times period
(SY2010-2011). Tests of group mean difference between tested teachers who stay in their current placement relative to switching placements and for tested teachers
who stay in their current placement relative to leaving the district are also provided in the stay and leave columns, respectively. (--) indicates that this outcome could
not be estimated due to insufficient sample size for individuals with the associated characteristic and mobility outcome.***p<0.001 **p<0.01 *p<0.05 +p<0.10.

181
Table 12. Characteristics of Secondary Tested Teachers who Stay vs. Switch Schools and Stay vs. Leave LAUSD Post-Pilot
Full Low-Performing High % ELL High % FRL
Characteristic Stay Switch Leave Stay Switch Leave Stay Switch Leave Stay Switch Leave
<=3 Years Exp. 0.05 0.09** 0.16*** 0.11 0.10 0.26*** 0.11 0.08 0.22*** 0.08 0.08 0.21***
4-6 Years Exp. 0.13 0.16 0.20*** 0.18 0.17 0.22 0.18 0.18 0.18 0.15 0.19 0.24**
7-9 Years Exp. 0.16 0.19 0.16 0.18 0.21 0.13 0.17 0.26** 0.12 0.16 0.21 0.11+
10+ Years Exp. 0.65 0.56*** 0.49*** 0.54 0.51 0.38*** 0.54 0.48 0.48 0.61 0.52* 0.44***
Permanent 0.92 0.85*** 0.69*** 0.87 0.81* 0.62*** 0.85 0.85 0.64*** 0.89 0.85 0.67***
Probationary 0.05 0.06 0.11*** 0.07 0.06 0.16*** 0.09 0.07 0.14* 0.07 0.07 0.14***
Bachelors 0.58 0.55 0.66** 0.57 0.49* 0.65* 0.58 0.53 0.64 0.61 0.54+ 0.63
Masters 0.39 0.43+ 0.32** 0.41 0.48+ 0.32* 0.40 0.43 0.34 0.37 0.44+ 0.35
Doctorate 0.03 0.02 0.02 0.02 0.03 0.03 0.02 0.04+ 0.02 0.02 0.02 0.02
National Board Certified 0.04 0.03 0.02* 0.03 0.03 0.01* 0.03 0.01+ 0.00* 0.03 0.01 0.02
# Needs or Below Stull Ratings 0.81 1.18+ 3.03*** 0.95 1.00 2.83*** 0.87 1.11 2.46*** 0.75 1.36+ 2.55***
ELA VAM 3.01 2.93 2.84 2.89 2.98 2.26 2.91 2.92 2.28 2.98 2.72+ 3.32
Math VAM 3.05 2.98 2.34* 3.00 3.07 3.11 3.07 3.24 2.35 3.04 2.68+ 2.46
Biology VAM 3.01 3.01 2.24 3.09 2.74 -- 3.15 2.79 -- 3.13 2.60+ --
Chemistry VAM 3.04 3.15 2.21 3.05 3.04 2.21 3.16 2.85 2.96 3.15 3.08 2.96
Integrated Science VAM 2.91 3.02 3.42 3.02 3.13 4.58 3.23 2.78 -- 2.96 2.78 2.63
Physics VAM 2.99 3.03 2.47 2.87 2.74 -- 2.84 2.88 -- 3.04 2.88 --
Science 8 VAM 3.04 2.82 3.23 3.25 2.81 -- 3.08 2.68 -- 2.93 2.64 3.23
US History VAM 3.01 3.00 3.92* 2.99 3.07 3.07 2.96 2.85 -- 3.14 3.13 5.30
World History VAM 3.05 3.10 2.99 3.02 3.03 2.98 3.11 3.10 1.81 3.19 3.48 2.61
Note. This table provides the proportion of tested secondary teachers who have the associated characteristic by mobility outcome for the post-pilot period (SY2011-
2012). Tests of group mean difference between tested teachers who stay in their current schools relative to switching schools and for tested teachers who stay in their
current school relative to leaving the district are also provided in the stay and leave columns, respectively. (--) indicates that this outcome could not be estimated due to
insufficient sample size for individuals with the associated characteristic and mobility outcome. ***p<0.001 **p<0.01 *p<0.05 +p<0.10.

182
Table 13. Characteristics of Secondary Tested Teachers who Stay vs. Switch Schools and Stay vs. Leave LAUSD Post-Times
Full Low-Performing High % ELL High % FRL
Characteristic Stay Switch Leave Stay Switch Leave Stay Switch Leave Stay Switch Leave
<=3 Years Exp. 0.07 0.11** 0.21*** 0.13 0.12 0.23*** 0.11 0.13 0.25*** 0.09 0.09 0.22***
4-6 Years Exp. 0.16 0.19+ 0.22*** 0.20 0.16 0.27* 0.19 0.18 0.24 0.17 0.22+ 0.26**
7-9 Years Exp. 0.16 0.20* 0.16 0.16 0.19 0.13 0.16 0.20 0.14 0.17 0.21 0.14
10+ Years Exp. 0.61 0.51*** 0.41*** 0.52 0.52 0.37*** 0.55 0.49 0.38*** 0.57 0.48** 0.39***
Permanent 0.91 0.84*** 0.64*** 0.86 0.84 0.61*** 0.85 0.83 0.63*** 0.89 0.86 0.61***
Probationary 0.04 0.06+ 0.09*** 0.06 0.05 0.11* 0.06 0.05 0.12* 0.05 0.04 0.09*
Bachelors 0.61 0.60 0.62 0.59 0.61 0.60 0.60 0.62 0.61 0.61 0.59 0.60
Masters 0.38 0.39 0.37 0.40 0.38 0.40 0.39 0.37 0.38 0.38 0.40 0.39
Doctorate 0.02 0.01 0.01* 0.02 0.01 0.01 0.01 0.01 0.01 0.02 0.01 0.01
National Board Certified 0.03 0.03 0.01* 0.03 0.03 0.00* 0.03 0.02 0.00* 0.04 0.02 0.00**
# Needs or Below Stull Ratings 0.74 1.57*** 3.89*** 0.72 1.18+ 3.14*** 0.75 1.69*** 2.85*** 0.68 1.02 3.24***
ELA VAM 3.02 2.97 2.97 2.80 2.87 2.62 3.07 2.90+ 3.11 3.03 3.19 3.00
Math VAM 3.06 3.10 2.86+ 2.85 2.93 2.53+ 3.14 3.28 3.13 3.06 3.23 2.92
Biology VAM 3.03 3.00 3.00 2.99 2.96 -- 3.11 3.00 -- 2.96 3.21 1.90*
Chemistry VAM 3.13 2.91 2.64* 3.12 3.01 2.37* 3.21 2.99 2.22** 3.12 2.79 2.62
Integrated Science VAM 2.96 2.87 2.95 2.97 2.67 2.99 3.19 3.89 -- 2.99 2.33 2.44
Physics VAM 2.91 2.55 3.36 3.04 2.55 -- 3.14 2.70 -- 3.23 1.75+ 3.02
Science 8 VAM 3.05 2.99 3.17 3.04 3.18 -- 3.08 2.84 -- 2.99 2.84 3.35
US History VAM 3.09 2.56** 2.44 3.06 2.44* 2.44 3.27 2.45* -- 3.17 2.84 --
World History VAM 3.12 2.75* 2.92 3.09 2.74 2.84 3.24 2.90 3.04 3.08 3.03 3.19
Note. This table provides the proportion of tested secondary teachers who have the associated characteristic by mobility outcome for the post-Times period (SY2010-2011).
Tests of group mean difference between tested teachers who stay in their current schools relative to switching schools and for tested teachers who stay in their current school
relative to leaving the district are also provided in the stay and leave columns, respectively. (--) indicates that this outcome could not be estimated due to insufficient sample
size for individuals with the associated characteristic and mobility outcome. ***p<0.001 **p<0.01 *p<0.05 +p<0.10.
183
Figure 1. Proportion of Elementary Teachers by School-Level Mobility – Full Sample

0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Switching Schools
Full Sample
Tested Non-Tested
0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Leaving LAUSD
Full Sample
Tested Non-Tested
184
Figure 2. Proportion of Elementary Teachers by School-Level Mobility – Low-Performing
Schools

0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Switching Schools
Low-Performing Sample
Tested Non-Tested
0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Leaving LAUSD
Low-Performing Sample
Tested Non-Tested
185
Figure 3. Proportion of Elementary Teachers by School-Level Mobility – High %ELL
Schools

0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Switching Schools
High % ELL Sample
Tested Non-Tested
0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Leaving LAUSD
High % ELL Sample
Tested Non-Tested
186
Figure 4. Proportion of Elementary Teachers by School-Level Mobility – High %FRL
Schools

0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Switching Schools
High % FRL Sample
Tested Non-Tested
0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Leaving LAUSD
High % FRL Sample
Tested Non-Tested
187

Figure 5. Proportion of Elementary Teachers by Class-Level Mobility– Full Sample

0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Switching Placements
Full Sample
Tested Non-Tested
0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Leaving LAUSD
Full Sample
Tested Non-Tested
188

Figure 6. Proportion of Elementary Teachers by Class-Level Mobility– Low-Performing
Classrooms

0.00
0.20
0.40
0.60
0.80
1.00
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Switching Placements
Low-Performing Sample
Tested Non-Tested
0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Leaving LAUSD
Low-Performing Sample
Tested Non-Tested
189

Figure 7. Proportion of Elementary Teachers by Class-Level Mobility – High %ELL
Classrooms

0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Switching Placments
High % ELL Sample
Tested Non-Tested
0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Leaving LAUSD
High % ELL Sample
Tested Non-Tested
190

Figure 8. Proportion of Elementary Teachers by Class-Level Mobility– High %FRL
Classrooms

0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Staying in the Same School
High % FRL Sample
Tested Non-Tested
0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Leaving LAUSD
High % FRL Sample
Tested Non-Tested
191

Figure 9. Proportion of Secondary Teachers by School-Level Mobility– Full Sample

0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Switching Schools
Full Sample
Tested Non-Tested
0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Leaving LAUSD
Full Sample
Tested Non-Tested
192

Figure 10. Proportion of Secondary Teachers by School-Level Mobility– Low-Performing
Schools

0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Switching Schools
Low-Performing Sample
Tested Non-Tested
0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Leaving LAUSD
Low-Performing Sample
Tested Non-Tested
193

Figure 11. Proportion of Secondary Teachers by School-Level Mobility– High %ELL
Schools

0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Switching Schools
High %ELL Sample
Tested Non-Tested
0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Leaving LAUSD
High %ELL Sample
Tested Non-Tested
194

Figure 12. Proportion of Secondary Teachers by School-Level Mobility– High %FRL
Schools

0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Switching Schools
High %FRL Sample
Tested Non-Tested
0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Leaving LAUSD
High %FRL Sample
Tested Non-Tested
195

Figure 13. Proportion of Secondary Teachers by Classroom-Level Mobility– Full Sample

0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Moving from School
Full Sample
Tested Non-Tested
196

Figure 14. Proportion of Secondary Teachers by Classroom-Level Mobility– Low-
Performing Classrooms

0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Moving from Placement
Low-Performing Sample
Tested Non-Tested
197

Figure 15. Proportion of Secondary Teachers by Classroom-Level Mobility– High %ELL
Classrooms

0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Moving from Placement
High % ELL Sample
Tested Non-Tested
198

Figure 16. Proportion of Secondary Teachers by Classroom-Level Mobility– High %FRL
Classrooms

0.00
0.05
0.10
0.15
0.20
0.25
2007-2008 2008-2009 2009-2010 2010-2011 2011-2012
Proportion of Teachers Moving from Placement
High % FRL Sample
Tested Non-Tested

Abstract (if available)

Abstract Research and practice have now well demonstrated that teachers play a central role in helping students achieve important outcomes (e.g., Baker, 1999

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Levels of interest: the effects of teachers' unions on Congress, statehouses, and schools

PDF

No place like home: a three paper dissertation on K-12 student homelessness & housing affordability

PDF

Are there hidden costs associated with teacher layoffs? The impact of job insecurity on teacher effectiveness in the Los Angeles Unified School District

PDF

School resource allocation in times of economic boom and bust

PDF

The local politics of education governance: power and influence among school boards, superintendents, and teachers' unions

PDF

Building networks for change: how ed-tech coaches broker information to lead instructional reform

PDF

The role of leadership in using data to inform instruction: a case study

PDF

That's not what I asked for: three essays on the (un)intended consequences of California's dual-accountability system

PDF

The teacher evaluation: key components that positively impact teacher effectiveness and student achievement

PDF

Teen parents: outsourcing childcare to keep them connected and engaged in school

PDF

Strategies California superintendents use to implement 21st century skills programs

PDF

More than sanctions: California's use of intensive technical assistance in a high stakes accountability context to close achievement gaps

PDF

Navigating a way out of program improvement: a case study analysis

PDF

Contracting for special education: a case study of a charter school contract for special education

PDF

Reforming developmental education in math: exploring the promise of self-placement and alternative delivery models

PDF

Mandated privatization through program improvement: a case study of the relationship between Action Learning Systems and the Buena Park School District

PDF

Loaded questions: the prevalence, causes, and consequences of teacher salary schedule frontloading

PDF

Contracting for performance: examining the relationship between LAUSD and ALEKS using transaction cost economics

PDF

Student achievement and teacher effectiveness in an era of heightened accountability

PDF

The economic and political impacts of U.S. federal carbon emissions trading policy across households, sectors and states

Asset Metadata

Creator Weinstein, Tracey Lynn (author)

Core Title Teacher evaluation reform in situ: three essays on teacher evaluation policy in practice

School Rossier School of Education

Degree Doctor of Philosophy

Degree Program Education

Publication Date 07/28/2014

Defense Date 05/27/2014

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag OAI-PMH Harvest,sensemaking,teacher evaluation,value‐added

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Strunk, Katharine O. (committee chair), Marsh, Julie A. (committee member), Painter, Gary Dean (committee member)

Creator Email weinstein.tracey@gmail.com

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c3-449444

Unique identifier UC11287796

Identifier etd-WeinsteinT-2748.pdf (filename),usctheses-c3-449444 (legacy record id)

Legacy Identifier etd-WeinsteinT-2748.pdf

Dmrecord 449444

Document Type Dissertation

Format application/pdf (imt)

Rights Weinstein, Tracey Lynn

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA