University of Southern California Dissertations and Theses

Strategies California superintendents use to implement 21st century skills programs

(USC Thesis Other)

Strategies California superintendents use to implement 21st century skills programs

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Running head: CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS

1

Strategies California Superintendents Use to Implement 21st Century Skills Programs

by

Daniel Sweet

A Dissertation Presented to the
FACULTY OF THE USC ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF EDUCATION

May 2014

Copyright 2014 Daniel Sweet

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 2

Dedication
This dissertation is dedicated to my fiancée’ Melinda Dixon and my parents Sandra and
Wallace Sweet. I want to express my love and gratitude for your support throughout the
dissertation process.
To my mother, Sandra Sweet, I would like to express my undying gratitude for the love
and support that you provided throughout the dissertation process. Your support and
encouragement mean the world to me. Thank you for always being there!
To my father, Wallace Sweet, I could not have made it through this process without your
candor and feedback regarding my work. Your attention to details made me strive for
excellence. I truly appreciate the time and effort you gave to support this endeavor.
To my fiancée, Melinda Dixon, your support throughout the process was the fuel to help
me complete this dissertation. I am exceptionally grateful for your love, support, and most of all
patience throughout the process. Your encouragement and support means the world to me!

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 3

Acknowledgements
I would like to acknowledge my Dissertation Committee Chair, Dr. Pedro Garcia. Thank
you very much for your guidance and support throughout the doctoral program at USC. A
special thanks to Dr. Rudy Castruita and Dr. Katharine Strunk for serving as members of my
dissertation committee, I appreciate your guidance, support, and feedback.
I would like to acknowledge my USC sisters, Synee Pearson, Terry Walker, and Shenora
Plenty for sharing this journey with me. You are all exceptional educators that will put this
degree to good use. I look forward to seeing how we progress from here.
I would like to especially acknowledge Lynn Taylor. Throughout my pursuit of this
degree you have always greeted me with encouragement and genuine support. Your warmth
lifted me up during difficult times of this process, for that I am eternally grateful.
To the extended staff and faculty in the Rossier School of Education at the University of
Southern California, I would like to thank you for your time and support in this endeavor.
Specifically, I would like to thank Dr. Ilda Jimenez Y. West for your guidance and feedback
regarding my work. I appreciate your time and effort. To Dr. Rudy Crew, your passion for
education and your wisdom are an inspiration. Thanks to all the Trojan Family who made this
possible.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 4

Table of Contents
Dedication 2
Acknowledgements 3
List of Tables 8
Abstract 9
Chapter One: Overview of the Study 10
Introduction 10
Background of the Problem 12
Global Achievement Gap 16
21st Century Skills 18
Statement of the Problem 19
Purpose of the Study 19
Research Questions 19
Importance of the Study 20
Assumptions 20
Limitations 20
Delimitations 21
Definition of Terms 21
Overview of the Study 24
Chapter Two: Literature Review 25
Introduction 25
Globalization and International Education 26
International Models of Education 28
Finland 28
Asia 30
The Impact of NCLB on School Improvement and the
Implementation of New Reforms 31
Flaws of NCLB 32
Solutions to the Problems Associated with NCLB 34
Current Policy Reponses to NCLB 35
Implementing New Reforms with NCLB 36
21st Century Skills 37
The Case for 21st Century Skills 37
Needs of the Economy Past and Present 38
Principles of Learning and 21st Century Skills 39
Major 21st Century Frameworks 40
Partnership for 21st Century Skills 40
Figure 1: Partnership for 21st Century Skills’ Framework 41
North Central Regional Laboratory 42
Organisation for Economic Cooperation and
Development 43
Three Minds for the Future 43
Implementing 21st Century Skills 44
Assessing 21st Century Skills 45

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 5

Criticisms and Barriers to the Development of
21st Century Skills 48
Leadership Techniques for Implementing Reforms 49
Loose Coupling 49
Distributive Leadership 51
The Role of the Superintendent in Implementing
Change 52
Presentation of Theories 55
Wagner’s Seven Survival Skills 55
Bolman and Deal’s Four Frames of Leadership 56
Summary 57
Chapter Three: Methodology 58
Introduction 58
Purpose of the Study 58
Goals of the Study and Research Questions 59
Design of the Study 60
Methodological Emphasis of the Study 61
Sample and Population 61
Criteria for Selection 62
Process of Selection and Sampling Issues 63
Instrumentation 63
Quantitative Instrumentation 63
Qualitative Instrumentation 64
Data Collection 65
Data Analysis 66
Quantitative Data Analysis 66
Qualitative Data Analysis 67
Triangulation 68
Ethical Considerations 68
Conclusion 69
Chapter Four: Findings 70
Introduction 70
Organization of Data Analysis 71
Descriptive Characteristics 72
Quantitative Demographic Data 72
Qualitative Demographical Data 73
Findings 77
Research Question 1 77
Findings: Responses to Closed-Ended Question 77
Survey Question 1 77
Findings: Interviews 79
Methods for Implementation 79
Professional Development 83

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 6

Technology 85
Allocation of Resources 88
Summary and Discussion: Research Strategies for
Implementing 21st Century Skills’ Programs 90
Research Question 2 90
Findings: Quantitative Survey 91
Findings: Qualitative Interview 92
Formative Indicators 93
Summative Indicators 96
Feedback Loop 97
Readiness Indicators 99
Discussion 100
Research Question 3 100
Findings: Quantitative Survey 101
Findings: Interviews 104
Co-creation 105
Aligning Stakeholders 107
Sustaining 21st Century Skills 111
Discussion 113
Auxiliary Findings 114
Quantitative Survey 114
Discussion 116
Theoretical Framework 116
Conclusion 118
Chapter Five: Summary, Implications, and Recommendations 119
Introduction 119
Statement of the Problem 120
Purpose of the Study 120
Research Questions 120
Methodology 121
Quantitative Data Collection Procedures 121
Qualitative Data Collection Procedures 122
Quantitative Data Analysis 122
Qualitative Data Analysis 122
Summary of Findings 123
Research Question 1 123
Research Question 2 125
Research Question 3 127
Implications for Policy and Practice 129
Future Research 131
Limitations 132
Conclusion 132
References 134
Appendix A 144
Appendix B 145
Appendix C 146

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 7

Appendix D 147
Appendix E 150
Appendix F 151
Appendix G 153

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 8

List of Tables
Table 1: Scale for identifying common logical errors 46
Table 2: Effects of district leadership of reading achievement 53
Table 3: Effects of district leadership on mathematics achievement 54
Table 4: Summary of qualitative data analysis procedures 71
Table 5: Quantitative survey: Superintendent gender 73
Table 6: Quantitative survey: District characteristics 74
Table 7: Qualitative interview: Characteristics for superintendents
and districts 76
Table 8: Strategies to support implementation of 21st century
programs 78
Table 9: Tools to assess students acquisition of 21st century skills 92
Table 10: Roles superintendents are taking to support the implementation
of 21st century skills 102
Table 11: Key stakeholders who superintendents will interact with
during implementation 104
Table 12: 21st century skills’ frameworks 116
Table 13: Superintendents definition of 21st century skills 117

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 9

Abstract
The purpose of this study was to identify strategies used by California superintendents to
implement 21st Century Skills programs in their districts. More specifically, the study
attempted to discover (a) what strategies do superintendents use to develop 21st Century Skills
in their district?, (b) What indicators do superintendents use to evaluate the implementation of
21st Century Skills in their district?, (c) How do superintendents include key stakeholders in
their implementation of 21st Century Skills? This study employed a mixed methods approach
in which 22 superintendents throughout the state of California completed surveys. Among the
superintendents who completed the survey, 10 participated in a semi-structured interview.
Finally, the participating superintendents presented documents for analysis. Triangulation of
the data was used to generate answers to the research questions. The following are some of
the key findings of the study. First, the most common method for implementing 21st Century
Skills is Project Based Learning. Second, superintendents use professional development
methods to build their district’s capacity such as: professional learning communities,
instructional coaching, and train the trainer model. Fourth, superintendents used numerous
formative and summative measures to evaluate their district’s 21st Century Skills
implementation (e.g., graduation rates and portfolio assessments). Fifth, superintendents
evaluated their districts implementation of 21st Century Skills by collecting feedback from key
stakeholders (e.g., principals, teachers, and parents). The results of this study present
strategies that support the development of 21st Century Skills but will require future research to
confirm their effect on student achievement.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 10

CHAPTER ONE: OVERVIEW OF THE STUDY

Introduction
Today we are failing too many of our children. We're sending them out into a 21st
century economy by sending them through the doors of 20th century schools
President Obama (2006, para. 14)
The American education system, in the 20th century, was the preeminent model for the
entire world. America had the most educated workforce in the world, and the highest enrollment
percentage of students in college (National Center on Education and Economics, 2007). A shift
in the world’s economy has occurred, and it has gone from being an industrial-driven economy
to a knowledge-driven economy. During the industrial age, manufactured products were the
United States’ primary tool for the support and growth of its economy. Over the past several
decades there has been a transition to a knowledge-based economy that primarily generates
services (Fadel & Trilling, 2009). The shift in emphasis of the world’s economy has negatively
impacted Americans both individually and collectively (Fadel & Trilling, 2009; Friedman,
2007). A major component of America’s failure to remain competitive is in large part a result of
shortcomings of its educational system. The American education system is failing to: (a) help
individuals obtain gainful employment, (b) sustain economic productivity, and (c) maintain the
democratic processes (Fadel & Trilling, 2009; Organisation for Economic Cooperation and
Development, 2005). The above failings are evidenced in America’s mediocre performance in
the latest international assessments of student achievement. A primary assessment tool, that
compares student achievement worldwide, is the Programme for International Student
Assessment (PISA, 2009). The PISA is designed to measure students’ abilities to demonstrate
competencies in subject areas. Additionally, the PISA analyzes students’ abilities to

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 11

communicate and collaborate effectively as they attempt to solve real world problems
(Organisation for Economic Cooperation and Development, 1999). Overall, America ranks in
the middle of similar economically developed nations in student achievement of language arts,
math, and science based on the results of the 2009 PISA (Organisation for Economic
Cooperation and Development, 2010). Furthermore, in recent years, many United States
employment reports demonstrate that students lack the requisite skills to secure gainful
employment. These employment reports continued to identify that many of the middle class jobs
of the 20th century have disappeared due to automation or exportation of jobs off-shore
(National Center on Education and Economics, 2007). Invariably, the questions that key actors,
the policy makers, the politicians, the educators, specifically superintendents must determine is
how to effectively prepare American students for jobs in the economy of the 21st century.
The latest reauthorization of the Elementary and Secondary Education Act known as No
Child Left Behind Act of 2001 (U.S. Department of Education, 2001) was enacted with the
notion that American students must acquire basic skills to become globally competitive (Baker,
Betebenner, & Linn, 2002). The international student assessment data demonstrates that despite
the efforts to create greater levels of accountability, American schools are still failing to elevate
their educational system (Organisation for Economic Cooperation and Development, 2010). The
PISA was designed to inform governments about ways to create and modify their educational
policies (Organisation for Economic Cooperation and Development, 1999). The performance
issues of the American educational system do not appear to lie with the idea of holding schools
accountable for 100% of the students becoming proficient in basic skills. Instead, the
performance issues lie with what American schools are being forced to address through
accountability measures, i.e. the acquisition of basic skills (Wagner, 2008). As demonstrated by

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 12

the research of Wagner (2008), there is a misalignment between what skills American schools
are attempting to develop and the competencies required to survive in the 21st century.
The idea of 21st Century Skills being incorporated into American schools is not new, but
has taken a more substantial meaning over the past several decades. Businesses, policymakers,
and scholars have all articulated the need for American schools to teach a new set of applied
skills (Fadel & Trilling, 2009; Wagner, 2008) According to Wagner (2008), the “problem,”
simply stated, is that the future of our economy, the strength of our democracy, and perhaps even
the health of the planet’s ecosystems depend on educating future generations in ways very
different from how many of us were schooled.
Background of the Problem
Starting with Lyndon Johnson in the 1960s, the United States government has brought to
bear numerous educational policies based on the premise of equal educational opportunity for all
students (Vinovskis, 2009). The focus of American policies has been to create equal educational
opportunities, to narrow the achievement gaps between minority and economically
disadvantaged students and middle-class White and non-economically disadvantaged Asian
students (Baker et al., 2002; Vinovskis, 2009).
In the 1960s and 1970s, federal education policies were compensatory in nature,
attempting to remedy inequitable educational outcomes (Vinovskis, 2009). President Lyndon
Johnson created the Gardner Commission as a part of his War on Poverty initiative, to explore
the role of federal education aid (Vinovskis, 2009). According to the findings of the committee,
the inequitable funding provided to high-poverty, high-minority schools was the most significant
factor in the unequal outcomes for minority students (Brady & Thomas, 2005). The study’s
finding led to the creation of the Elementary and Secondary Education Act (ESEA) of 1965.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 13

ESEA required that low-income schools receive funding through Title I to receive educational
services comparable to non-Title I funded schools (Podesta & Brown, 2008). The receipt of full
federal funding was contingent upon states and districts creating equivalent levels of financial
backing between Title I and non-Title I schools. The apparent rationale of President Johnson and
Congress was that equal funding would lead to improved outcomes for minority students (Brady
& Thomas, 2005; Podesta & Brown, 2008; McClure, 2008).
Another educational policy that attempted to address the achievement gap was the Head
Start Initiative. The rationale for the development of Head Start was that it would provide
minority students with a pre-Kindergarten education that could prepare them for kindergarten
(Currie & Thomas, 1995). The lack of significant improvement in the outcomes for minority
students as the result of increased federal educational aid led to reforms of a different nature.
The perception that compensatory programs were a sufficient educational policy tool by
itself to narrow the achievement gap gave way to the idea of national education standards. Goals
2000: Educate America Act (Goals 2000, 1994), passed by Congress in 1994, was an attempt to
create national education standards to address issues such as the disparity of educational
outcomes between the wealthy and the poor. There were several objectives associated with
Goals 2000 (1994): (a) all American children will begin school ready to learn, (b) a 90% high
school graduation rate for all Americans, (c) America will be first in the world in mathematics
and science, and (d) the United States will be literate and possess knowledge and skills to be
competitive in the global economy (Paris, 1994). Goals 2000 (1994) also had provisions that
required states to create a comprehensive improvement plan with clear academic standards to be
assessed by statewide testing programs (Goals 2000, 1994). The requirement for clear academic
standards to be assessed across the United States represented the beginning of the accountability

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 14

era. The demand for accountability from states, districts, and schools expanded with the
reauthorization of ESEA (2001).
In 2001, President George W. Bush reauthorized ESEA as the No Child Left Behind Act
(NCLB, U.S. Department of Education, 2001). NCLB (U.S. Department of Education, 2001)
has several goals in mind: (a) by 2014, all Kindergarten-12th grade students in the United States
will be proficient in basic skills (reading, language arts, and mathematics), (b) all states, districts,
and schools would be responsible for ensuring that all significant subgroups developed
proficiency in basic skills (Baker et al., 2002).
Kanter and Lowe (2006) believed that the NCLB’s (U.S. Department of Education, 2001)
provision that requires states, districts, and schools to be accountable for all students represents
progress. One of the perceived advantages of this mandate was that minority students and
parents could no longer be blamed for poor academic performance. However, while the
requirements of NCLB do represent an official effort of the federal government to narrow the
achievement gap between minority students and middle class Whites and Asians, it has largely
been unsuccessful in the achievement of its aims.
Urban school superintendents are now charged with ensuring that all students become
proficient in the identified basic skills. The demand to ensure that all students are proficient in
basic skills has resulted in a narrowing of the curriculum, professional development, and types of
assessments used (Manna, 2006). By narrowing down the elements of schooling, to avoid the
consequences of NCLB (U.S. Department of Education, 2001), many schools have received
program improvement status. The reduction in educational focus to the basic skills by urban
schools has resulted in virtually no attention being paid to higher order thinking and problem
solving skills (Meier & Wood, 2004). In the past several years, there have been several federal

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 15

and local responses (e.g., policy changes) to develop programs to address flaws of NCLB (U.S.
Department of Education, 2001).
In 2009, The National Governors Association (NGA, 2009) approved the Common Core
Standards Initiative (CCSI). The common core standards are national achievement standards for
reading and math. The NGA created these standards to respond to the failings of NCLB (U.S.
Department of Education, 2001), by also creating standards to develop critical thinking. To date,
45 states have adopted Common Core Standards (National Governors Association, 2009). The
implementation process has unofficially begun, in association with the development of the CCSI,
the Smarter Balance assessment has also been designed (Smarter Balanced Assessment
Consortium, 2012). The Smarter Balanced assessment is a multi-format tool that addresses
formative, interim, and summative assessment of the development of skills (Smarter Balanced
Assessment Consortium, 2012). This assessment, which will be implemented in 2014, represents
a shift from the multiple choices to more performance-based assessments. According to Wagner
(2008) this program may be a step in the right direction, because it requires students to justify
their thinking and apply their knowledge to real world situations. However, the impact of the
smarter balanced assessment and common core standards is yet to be determined, and will take
several years to determine their success. Concurrently, with the development of new content
standards, is the changing shape of performance accountability for all relevant actors
(policymakers, central administrators, principals, and teachers).
In the fall of 2011, the United States Department of Education began to offer waivers for
some of the provisions from ESEA. States could apply for waivers if they met a litany of
requirements that force states to maintain some of the accountability requirements outlined by
NCLB (U.S. Department of Education, 2001). Some of the waiver provisions for states

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 16

included: (a) flexibility to create their own guidelines for meeting the criteria associated with
Adequate Yearly Progress (AYP), (b) schools that do not meet AYP requirements are not subject
to corrective action unless determined by the Local Education Agency (LEA), and (c) flexibility
in the use of school funds (United States Department of Education, 2011).
Evidence exists from the results of the PISA and the job report “Are They Really Ready
to Work” (Casner-Lotto & Barrington, 2006) that even the high-performing students in America
are struggling to obtain and develop careers in the competitive global environment. As argued
by Wagner (2008), there are two achievement gaps that exists in America presently: (a) the
achievement gap within the United States between students typically from middle and high
socioeconomic backgrounds (White and Asian students) and students from low socioeconomic
backgrounds (African-American and Latino) and (b) the achievement gap that exists between
American students and children from other economically developed nations (e.g., Finland,
China, and Singapore). This next section defines the global achievement gap and the
ramifications that it has for America.
Global Achievement Gap
“Our nation is at risk. Our once unchallenged preeminence in commerce, industry,
science, and technological innovation is being overtaken by competitors throughout the world”
National Commission on Excellence in Education
(1983, A Nation at Risk, para. 1)
The idea that America has fallen behind the world academically is not a recent
phenomenon, as indicated by this quote to former Secretary of Education Terence Bell in the
1983 The Nation at Risk report. The Nation at Risk report was arguably the first formal
indication that the goals and measurements for success, in the American education system, were

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 17

producing diminishing returns. As previously indicated, recent international student
achievement data from the PISA and Trends in International Math and Science Study (TIMMS,
National Center for Education Statistics, n.d.) further substantiates that America is producing
mediocre academic outcomes in comparison to others in the developed world (OECD, 2009).
The troubling aspect to America’s performance is that the PISA is designed to measure students’
academic competencies in authentic applications. Recent job reports further demonstrate that
even America’s best students lack the requisite skills to be globally competitive. The results of
“Are they Really Ready to Work” (Casner-Lotto & Barrington, 2006) a survey of 400 businesses
indicated that new graduates lack the applied skills that businesses desire for new employees
(Partnership for 21st Century Skills, 2011). It appears that, in the new economic world,
businesses want trained new hires; they do not wish to spend time and money to train new hires.
Moreover, the research of Wagner (2008) and Fadel and Trilling (2009) indicated that the shift in
the basis of the economy to a knowledge economy results in people having numerous careers
over a lifetime. The conclusions made by Wagner (2008) and Fadel and Trilling (2009) that
people will have several careers over a life, in conjunction with America’s poor educational
performance compared to the rest of the world, evidences the need for the development of new
skills. The need of improving the academic achievement of minority students is acute for urban
superintendents. Urban superintendents must support both the development of core
competencies and 21st Century Skills to enable minority students to be globally competitive.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 18

21st Century Skills
There is no universal definition of what 21st Century Skills encompass, but there is
consensus among educators, policymakers, and business people that American students must
develop new competencies. One of the most prevalent definitions of 21st Century Skills comes
from the Partnership for 21st Century Skills (2011). Their definition begins by reiterating the
need for core competencies (English, math, science and social studies); then their definition
further expands to include the need for: (a) interdisciplinary themes, e.g., global awareness,
financial literacy, civic literacy; (b) creativity and innovation; (c) critical thinking and problem
solving; and (d) communication and collaboration (Partnership for 21st Century Skills, 2011).
Another common framework that is utilized by a number of practitioners is the seven survival
skills developed by Wagner (2008).
Wagner (2008) defined the Seven Survival Skills as: (a) critical thinking and problem
solving, (b) collaboration across networks and leading by influence, (c) agility and adaptability,
(d) initiative and entrepreneurialism, (e) effective oral and written communication, (f) accessing
and analyzing information, and (g) curiosity and imagination.
Approximately 13 states have adopted the Partnership for 21st Century initiative to
develop the three R’s (reading, writing, and arithmetic), and integrate them with the four C’s
(critical thinking, communication, collaboration, and creativity) within their standards,
assessments, and professional development (Partnership for 21st Century, 2011). However, the
vast majority of states have not formally recognized that there is a need for the development of
new skills.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 19

Statement of the Problem
Researchers such as Tony Wagner (2008) and Thomas Freidman (2007) believe
American schools are operating an antiquated model of education that does not prepare students
to be global citizens in the 21st century. Friedman and Wagner argued that the current American
accountability requirements exacerbate the problems associated with the global achievement gap.
Current accountability measures, such as No Child Left Behind (U.S. Department of Education,
2001), require that students achieve proficiency in basic content skills. However, legislation like
NCLB (U.S. Department of Education, 2001) is preparing children for a world that no longer
exists (Wagner, 2008). A new knowledge economy exists today and it requires that students
possess a basic mastery of both content specific skills, and 21st Century Skills to sustain and
develop a future America. The question that this study intends to address is how are
superintendents implementing and planning to implement 21st Century Skills’ programs in their
districts.
Purpose of the Study
The purpose of this study is to identify strategies that are being utilized by
superintendents to implement 21st Century Skills in their districts. The examination of practices
utilized by some leading superintendents will help to identify strategies that can be used in other
districts to develop 21st Century Skills for students of their district.
Research Questions
The following research questions were developed to investigate this study’s research.
1. What strategies do superintendents use to develop 21st Century Skills in their district?
2. What indicators do superintendents use to evaluate the implementation of 21st Century
Skills in their district?

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 20

3. How do superintendents include key stakeholders in the implementation of 21st Century
Skills?
Importance of the Study
The goal of this study is to add to the existing body of research regarding implementation
strategies of major reform initiatives, such as 21st Century Skills. Another objective of the study
is to provide current and future superintendents with leadership strategies for the implementation
of critical reforms, like the introduction of 21st Century Skills and meeting of accountability
requirements. The information generated from this study can inform and guide the practices of
school boards. Lastly, the findings of the study can inform other critical district personnel of the
goals and the methods for implementing 21st Century Skills, and the meeting of federal and state
accountability requirements.
Assumptions
This study presupposes that 21st Century Skills, by any current definition, are important
tools to be developed for students by school districts. The strategies, leadership, and vision
employed by superintendents in the implementation of 21st Century Skills are important. For the
purposes of this study, the focus will be superintendents of heterogeneous districts, with a
population of 10,000 students or more. The responses of the superintendents to the interview
protocol are authentic.
Limitations
There are several potential limitations to the findings of this study. First, the sample
population of the study that will be interviewed is limited to the size of the district selected for
the sample. The sample of superintendents to be interviewed is limited to the state of California
and may not reflect the strategies used to implement 21st Century Skills in other states or

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 21

districts. Second, the sample number of superintendents used to gather data for this study is
limited, as is the amount of available time and funds. Three, lack of an agreed universal
definition for 21st Century Skills may generate different interpretations and responses from the
subjects of this study, thereby affecting the studies interpretations and reliability among
interviewees participating in the study. Four, this study presupposes that the responses provided
by superintendents during the interviews are complete and accurate. Finally, the number of
superintendents that met the criteria for an interview were limited to the ones that are planning
to, have, or implementing 21st Century Skills.
Delimitations
The fact that the sample of superintendents is drawn from the state of California may
limit generalizability of the study’s findings to other states and districts within. Based on the fact
that there is no common definition for 21st Century Skills, the findings of this study may not be
generalizable. This study will use Tony Wagner’s (2008) Seven Survival Skills as the
framework for exploring the strategies used to implement 21st Century Skills, while meeting
current accountability requirements.
Definitions of Terms
The following are list of terms and their definitions that will be used throughout the study.
• 21st Century Skills: Tools or skills required to be successful in the current mode of the
economy (Fadel & Trilling, 2009).
• Achievement Gap: The disparity in educational outcomes between minority students and
middle class White and non-economically disadvantaged Asians.
• Adequate Yearly Progress: The federal education legislation, No Child Left Behind (U.S.
Department of Education, 2001), requires that all schools and their significant subgroups

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 22

achieve a pre-set improvement in language arts and mathematics to satisfy the
requirements of the legislation.
• Academic Performance Index (API): This is a state level accountability tool in California
that measures the performance of schools and their significant subgroups of students on a
scale of 200 to 1000. The goal for all schools and their significant subgroups is to
achieve a score of 800 or above (State of California, 1999).
• Common Core Standards Initiative: National content standards for math and language
arts. The common core standards initiative has been adopted by 45 states (Common Core
State Standards, 2012, Standards in Your State).
• Defined Autonomy: An expectation that principals lead within the boundaries of district
defined goals (Marzano & Waters, 2009).
• Elementary and Secondary Education Act: In 1965, President Lyndon B. Johnson,
developed this legislation to attempt to narrow the achievement gap between minority
students and middle class White and non-economically disadvantaged Asian students, by
ensuring the students receive equitable services through equitable Title I funding (Baker
et al., 2002).
• Educate America Act (Goals 2000): The attempt by President Clinton to develop national
education standards (Paris, 1994).
• Global Achievement Gap: The gap between what the highest performing American
schools are teaching and testing versus that all students will need to succeed as learners
and workers in the 21st Century (Wagner, 2008).
• Globalization: Coined in 1985 by economist Theodore Levitt to describe changes in
global economics affecting production, consumption, and investment (Spring, 2009).

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 23

• Local Education Agency: A school district.
• Linked Learning: Career-Themed Academies tailored to the interests of high school
students, e.g., engineering or criminal justice (Lafors & McGlawn, 2013).
• A Nation at Risk: The report developed by former Secretary of Education Terence Bell
that outlined the weak performance of the American school system. The report further
highlighted the need for the development of new skills by schools (National Commission
on Excellence in Education, 1983).
• No Child Left Behind Act: The 2001 reauthorization of the Elementary and Secondary
Act (ESEA) that stated all American students will be proficient in basic skills by the year
2014. All states, districts, and schools will be required to disaggregate the performance
data in all content areas for all major subgroups (Baker et al., 2002).
• Program Improvement: The term used to characterize all schools that have failed to meet
Adequate Yearly Progress goals as outlined by NCLB.
• Project Based Learning: An inquiry-based model of instruction that requires students to
create projects that reflect their learning of content specific knowledge (Bell, 2008).
• Science Technology Engineering Arts and Mathematics Programs (STEAM): Academic
instruction geared towards the sciences, technology, arts, and mathematics.
• Smarter Balanced Assessment: The assessment tool used to coincide with the
implementation of the Common Core Standards Initiative. The assessment is an adaptive
performance based tool that uses formative, interim, and summative data to determine
students’ mastery of the common core standards
(Smarter
Balanced
Assessment

Consortium,
2012).

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 24

• Small Learning Communities: A large school divided into several smaller schools
(Sammon, 2008).
• Status Model: A model for measuring achievement that expects all elements to achieve a
finite score.
• Stretch Goals: Goals created by organizations that require great effort and are difficult to
achieve (Clark & Estes, 2008).
Overview of the Study
This study is composed of five chapters. The first chapter outlined the problem this study
investigated and provided background information about: (a) a brief history of federal
educational policy, (b) an overview of 21st Century Skills, and (c) the achievement gap.
The second chapter is a review of the existing body of literature regarding:
(a) globalization and international education, (b) effects of accountability on education,
(c) 21st Century Skills, and (d) leadership strategies for implementing major reforms. The third
chapter outlines the research methodology to be used for conducting this review. The third
chapter will include: (a) sampling criteria, (b) survey instruments, (c) means for data collection,
and (d) the tools for data analysis.
The fourth chapter of this study will include the findings of the study and the
interpretations made from the results of the study. The fifth, and final chapter of this study, will
present a summary of the findings and their implications for superintendents and the existing
body of research.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 25

CHAPTER TWO: LITERATURE REVIEW
Introduction
Twenty-First Century Skills, as an idea, existed for decades in some shape or form, both
in educational literature and government publications. However, the changing nature of
technology and employment makes 21st Century Skills vital to the economic future of the United
States (Rotherham & Willingham, 2009). In 2004, approximately 60% of U.S. business profits
were earned overseas (Zhao, 2007). A reality of the 21st century is that most people will not
retain a position within the same organization for their entire career (Wagner, 2008).
Corporations like Capitol One are training their employees in different technical areas, preparing
them for a time when their current job will leave, to go off shore. The nature of the knowledge
economy necessitates students possess survival skills, 21st Century Skills (Friedman, 2007). For
the design of this study, the framework used to define 21st Century Skills will be Wagner’s
seven survival skills.
One of the main causes for the stagnation of the United States economy issues is its
educational systems’ inability to prepare students to be globally competitive. The results of the
2009 Programme for International Student Assessment (PISA, 2009) are one of many indications
American students do not possess 21st Century Skills. Furthermore, in the report “Are They
Really Ready to Work” (Casner-Lotto & Barrington, 2006), a survey of over 400 businesses
indicates that new employees lack communication skills, problem solving abilities, and the
ability to collaborate Partnership for 21st Century Skills, 2011). The lack of cohesion among the
purposes of school and its current practices within the United States will lead to poor governing,
ineffective international relations, and poor public understanding of policy issues (Reimers,
2006).

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 26

This review of the literature highlights the impact and ramifications of globalization, and
successful international educational models, on the current and future direction of the U.S.
educational system. It will also review the effect the No Child Left Behind Act (U.S.
Department of Education, 2001) and bureaucratic accountability have on the implementation of
major reform initiatives, such as 21st Century Skills. This literature review will describe many
different aspects of 21st Century Skills including: (a) current frameworks, (b) efforts to
implement 21st Century Skills, (c) the purpose for developing 21st Century Skills, (d) criticisms
and potential barriers to the implementation of 21st Century Skills, and (e) methods for assessing
21st Century Skills. Also, this review of the literature will include leadership strategies for
implementing major reforms. This literature review will conclude with a presentation of the
frameworks used to guide this study. The various subject areas of this literature review are
intended to highlight the problem and research questions that this study will investigate. As
previously indicated, the research questions that this study will address are:
1. What strategies do superintendents use to develop 21st Century Skills in their district?
2. What indicators do superintendents use to evaluate the implementation of 21st Century
Skills in their district?
3. How do superintendents include key stakeholders in the implementation of 21st Century
Skills?
Globalization and International Education
In 2012, Wang conducted a study to learn and describe the changing role of education
throughout the world. According to Wang (2012), the purposes of education have evolved over
time. Some of the current purposes of education include: to prepare students for further
education, immediate employment, updating skills for the future, and for numerous career shifts.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 27

Wang argued further that the advent of new technologies has resulted in the demand by
employers of new skills by employees. According to Wang, over the past half century, as the
demand for new skills have increased, the availability of jobs for low-skilled workers has
diminished significantly.
Moreover, Wang (2012) projected demographic growth trends worldwide. The projected
increasing population will lead to greater demand for highly skilled workers that possess the
ability to respond to developing future technological infrastructure. The changing economic and
infrastructural needs across the globe highlight many challenges for educational systems
worldwide.
Wang (2012) asserted that educational systems must be flexible in meeting these new
demands. A modern educational system must be responsive to the changing demands of the
environment and provide instruction of the soft skills. A worker’s skills must stay updated,
because the idea of a once-in-a-lifetime diploma is no longer sufficient. To survive in the
modern world, all citizens must be lifelong learners (Wang, 2012).
Wang’s (2012) article made two conclusions that have serious implications for the
direction of education and educational policy in the United States. First, there is, and will
continue to be, strong demand for short-term education and worker retraining programs. Second,
the demand for 21st Century skilled workers greatly exceeds the supply (Wang, 2012). The
implication of Wang’s report is that the United States must stop preparing students for jobs that
no longer exist and begin to modify the goals and instruction of their educational system to meet
the demands of the evolving world. The research of Cheng (2007) reinforces the conclusions
and findings made by Wang.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 28

Cheng (2007) listed multiple factors that determine a person’s employability. These
factors include: (a) educational qualifications, (b) work experience, (c) the available information
regarding past employment, (d) guidance regarding employment, (e) the willingness to accept a
job, and (f) the conditions of the job market. The conclusions made Wang (2012) and Cheng
substantiate the need for American schools to work to develop 21st Century Skills in order to
remain globally competitive.
The research of Reimers (2006) also focused on the impact of globalization on education.
Reimers contended that the source of future conflicts in the world will be cultures. Based on this
belief, Reimers argued that the purpose of schooling is to create a quality education that reflects
the needs of society. His conclusion identifies the need to support the development of students’
cultural competency, to enable their ability to participate in a democratic society and
international diplomacy.
International Models of Education
As articulated in Chapter 1, American schools’ performance pales in comparison to that
of many other economically developed nations. This section reviews literature that highlights
specifics regarding how students in other nations have been able to achieve higher levels of
achievement.
Finland. In “What Can We Learn from Finland Successful School Reform,” Darling-
Hammond (2010a) highlighted how Finland, through a systematic shift, reinvented its education
system into one of the best in the world. Finland currently has a 99% completion rate for
primary education and 90% completion rate for secondary education (Darling-Hammond,
2010a). Even with an influx of immigrants and students from low socioeconomic backgrounds,
Finland’s educational system has thrived. These statistics from Finland help demonstrate the

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 29

possibility of succeeding with a culturally and economically diverse population, as in the United
States (Darling- Hammond, 2010a).
The paper identified five core principles that have led to the systematic improvement of
education in Finland (Darling-Hammond, 2010a). These five core principles are: (1) available
resources for the students who need them the most, (2) high standards and supports for special
education students, (3) qualified teachers, (4) a balance between centralization and
decentralization, and (5) evaluation of education (Darling-Hammond, 2010a).
Darling-Hammond (2010a) identified that the most salient changes to education in
Finland have been to the structure of teacher education. A significant change to teacher
education was the shift from three years of preparation to five years (Darling-Hammond, 2010a).
Presently, teacher education focuses on developing teachers’ abilities to teach critical thinking
and higher level thinking skills. Student teachers spend a year planning, developing, and
refining curriculum. Teacher education also focuses on providing instruction for students with
special needs and multicultural education (Darling-Hammond, 2010a).
Darling-Hammond (2010a) further identified other policies that have impacted the
education system in Finland. One of the successful policies developed by the Finnish
government is early intervention for struggling students. Nores (2010) identified that early
childhood interventions improve the educational attainment of students by reducing educational
inefficiencies, such as high school dropouts. Thus, this paper’s findings from Darling-Hammond
(2010a) identified the need for better early-childhood interventions in the US, such as the Head
Start Initiative.
Another policy tool, implemented by the Finnish government to support their educational
system is the equitable distribution of resources. The article discussed additional factors and

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 30

policy tools that have been implemented: (a) small class sizes, (b) access to high quality health
care—free health and dental care, (c) free meals at school, (d) the elimination of the practice of
tracking students by ability, (e) more local autonomy regarding curriculum and pedagogy to meet
light national standards, (d) assessments are used exclusively to gather information about how to
improve curriculum and instruction, and (e) classrooms use small group instruction, student-
centered instruction and project-based learning (Darling-Hammond, 2010a).
A major challenge for the US, towards replicating the success of Finland, requires a
change in values and practices by teachers, parents, school leaders, and policymakers. The idea
that bureaucratic accountability will disappear from U.S. educational policy to allow schools to
use assessment to inform instruction seems unlikely. Also, the findings of the article challenge
the US to redefine and perhaps nationalize the goals of teacher education (Darling-Hammond,
2010a). On the potentially positive side, the United States can shift funding priorities to match
the focuses that have helped lead to the rise of Finland’s educational system.
Asia. Zhao (2007) described how Singapore and several other Asian nations have
adopted the principles of a 21st Century education. Singapore has adopted a new curriculum
called Thinking Schools Learning Nation (Zhao, 2007). The goal of this initiative is to develop
all students into a community of active, creative learners with critical thinking skills. The key
strategies of this initiative include: (a) the explicit teaching of critical and creative thinking skills,
(b) the reduction of subject content, (c) the revision of assessment modes, (d) greater emphasis
on processes rather than outcomes when appraising schools, (e) more varied curriculum, and
(f) more autonomy for schools (Zhao, 2007). The South Korean government has created a
similar policy response to meet the demands of the changing global economy.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 31

In South Korea, the government released the Seventh National Curriculum in response to
current developments relating to globalization, and changes in information and knowledge. In
Japan, the direction of educational policy has taken a similar shift to the ones taking place in
South Korea and Singapore.
The Japanese government is attempting to enhance the emotional education of its
students, to support their development as well-rounded human beings (Zhao, 2007). The second
objective of Japan’s new national curriculum is to develop a school system that provides students
with more choices, by promoting the development of creativity. The third objective of the
Japanese government is to promote a system in which the school’s autonomy is respected
through decentralizing the education administration, enhancing local autonomy, and enabling
independent self-management at the school levels (Zhao, 2007).
Overall, the characteristics’ of these educational systems exemplify many of the ideals of
21st Century education. These models symbolize the need for the development of 21st Century
Skills and the changes related to policy, curriculum, teacher education, and professional
development that can contribute to improving the global standing of the US. The next section
will discuss the effects that bureaucratic accountability and No Child Left Behind (U.S.
Department of Education, 2001) have on the continuous development of the U.S. school system
and its ability to implement 21st Century Skills’ programs.
The Impact of NCLB on School Improvement and the Implementation of New Reforms
The No Child Left Behind Act of 2001 (NCLB, U.S. Department of Education, 2001)
was the reauthorization of the Elementary and Secondary Act (ESEA) that stated that all
American students (K-12) will be proficient in basic skills by the year 2014. NCLB (U.S.
Department of Education, 2001) requires that all states, districts, and schools report the

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 32

disaggregated data of student performance in the major sub-groups. The law requires that all
schools and their major subgroups achieve Adequate Yearly Progress goals in English Language
Arts and mathematics that increase each year, leading to the 2014 deadline (Baker et al., 2002).
NCLB uses a status model to measure the academic growth of schools and students, which
means that all subgroups must meet the adequate yearly progress goals as defined by each state.
These requirements are associated with increasing sanctions that affect all members of the school
(Baker et al., 2002). There are many criticisms associated with the policy goals of NCLB and its
ability to promote continuous improvement. According to Fusarelli and Schoen (2008), federal
officials assert that NCLB has already positively impacted education by reducing the
achievement gap between middle-class White students and other ethnicities. For example, the
United States Department of Education (USDE) reported that since the implementation of
NCLB, the gap between Black 9-year-olds and White 9-year-olds narrowed by 9 points in
reading and 5 points in math (Reimers, 2006). According to Reimer’s (2006) explanations for
why there was growth in English Language Arts (ELA) for students are: (a) students learned
more, (b) students developed better test taking skills, (c) cheating, (d) changes in the test, and
(e) normal fluctuations on grades. Despite purported gains achieved through the legislation,
there are many critics to the goals and provisions of this legislation.
Flaws of NCLB. According to Hawley (2008) and Greenstein (2012), the theory of
action behind NCLB (U.S. Department of Education, 2001) was that continuous improvement
will occur as the result of holding schools accountable for student performance. Furthermore,
this type of accountability would lead to the equitable redistribution of resources by districts and
states, which would improve student achievement and allow states and schools to avoid the
punitive sanctions of the law. According to Hawley, this theory of action was flawed for five

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 33

reasons. First, the idea of universal achievement goals is flawed as the needs of schools are
multiple, changing, and diverse. Moreover, the demands of NCLB (U.S. Department of
Education, 2001) are unreasonable in the short timeframe allotted. According to Elmore (2002),
meaningful improvement for struggling schools takes 3-5 years. Meaningful change is difficult
to accomplish with diffuse goals, high teacher turnover, and difficulty recruiting qualified
teachers. Also, developmental processes are non-linear, which results in fluctuations of test
scores (Elmore, 2002). The requirements for consistent growth in the AYP model does not
account for the way it occurs, ultimately leading to the severe sanctions schools are attempting to
avoid. As highlighted by the findings of Darling-Hammond (2010a) and Zhao (2007),
description of the models of education in Finland and Singapore, using their principles and
philosophies to inform educational practices in the United States practice, is substantiated by
their recent success on the PISA.
A second perceived flaw of NCLB (U.S. Department of Education, 2001) regarded the
difficulty associated with assessing complex learning outcomes with a single measure. The data
generated for high-stakes testing has proven to be statistically unreliable and proven to be
incapable of measuring complex learning tasks (Hawley, 2008).
The core technology of teaching is impacted by the goals of NCLB (Hawley, 2008).
Often based on fear, teachers will alter their practices to defer to the expectations that they are
accountable for meeting (Elmore, 2002; Fusarelli & Schoen, 2008; Hawley, 2008).
A third flaw of NCLB (U.S. Department of Education, 2001) highlighted by Fusarelli and
Schoen (2008) was that NCLB emphasizes attracting and retaining high-quality instructors, but
the policy does not outline a definition for high-quality instruction. The conclusion made by

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 34

Fusarelli and Schoen (2008) alludes to training teachers in scientific methods but does not
provide a definition what new instructional models would entail.
A fourth flaw in the conception of NCLB (U.S. Department of Education, 2001) and its
policy goals according to Hawley, (2008), was the challenge associated in the lack of control of
the inputs students receive before and outside of school (communities and families). The inputs
that Hawley refers to include: cognitive capabilities, prior knowledge, and motivation. This
weakness appears to be most acute for low-income neighborhoods, which was one of the factors
that led to the goals of NCLB (U.S. Department of Education, 2001).
Elmore (2002) found that low-performing schools lack the internal normative structures
necessary to facilitate the growth required by NCLB (U.S. Department of Education, 2001). The
lack of capacity limits schools’ abilities to respond to the goals associated with NCLB (U.S.
Department of Education, 2001). This results often in the narrowing of the curriculum,
professional development, and instruction to avoid the sanctions of NCLB (Elmore, 2002; Meier
& Wood, 2004).
The fifth flaw, as perceived by Hawley (2008), the NCLB (U.S. Department of
Education, 2001) requirements are an impediment to the continuous improvement it was
intended to spur. Hawley (2008) attributed this conclusion to the fact systematic solutions are
required to solve complex issues, but policies such as NCLB are developed piecemeal.
Moreover, the heterogeneity of schools with different characteristics requires complex and
uncertain strategies for change (Hawley, 2008).
Solutions to the problems associated with NCLB. According to Hawley (2008), there
are five possible solutions to the problems associated with the accountability of NCLB (U.S.
Department of Education, 2001). One of the solutions suggested by Hawley was to hold states

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 35

harmless if they raise student achievement standards above the requirements. A second
suggestion, Hawley made was to measure student achievement by using multiple measures. His
third suggestion was that NCLB (U.S. Department of Education, 2001) could be improved by
promoting collaborative problem solving among districts and schools (Hawley, 2008). Hawley’s
fourth suggestion proposed that schools could promote improvement by providing greater
resources to promising practices. Finally, he suggested there must be sufficient time and support
to implement, assess, and modify promising practices (Hawley, 2008). Wagner (2006) asserted
that the improvements in student achievement sought by NCLB (U.S. Department of Education,
2001) can be accomplished with two-way accountability between both the federal and state
governments, and their local education agencies.
Current responses to NCLB. In 2012, the United States Department of Education
began to respond to the flaws highlighted in this review of the literature by offering waivers to
states on select provisions (U.S. Department of Education, 2011, ESEA Waiver Policy). Some
of the provisions in the waivers that directly address the criticisms highlighted in the literature
include: (a) states would have the flexibility to set their AYP goals that have achievable but
ambitious goals, (b) flexibility to transfer funds from Title I to other areas, and (c) LEAs would
no longer be required for compliance with the corrective action required by NCLB (U.S.
Department of Education, 2012). To date, 35 states have received ESEA waivers, but not the
State of California (U.S. Department of Education, 2013). The other recent initiative, that has
responded to aspects that attempts to better address the needs of students, is the Race to the Top
(2009) initiative promoted by U.S. Secretary of Education Arne Duncan and President Obama
(United States Department of Education, 2009

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 36

The Race to the Top (2009) initiative has four critical goals. First, states and districts that
receive funding must adopt standards that prepare students to succeed in college and the
workplace, in order for the U.S. to compete in the global economy (United States Department of
Education, 2009). Second, states and districts must build data systems that measure student
growth and success. Third, states and districts should also inform teachers and principals on how
they can improve student instruction. The third initiative also requires states and districts to
recruit, retain, and reward effective teachers and principals in locations throughout the nation
where low-performing schools need them the most. Lastly, the goal is to improve failing schools
(U.S. Department of Education, 2009). One of the underlying goals espoused in this reform, by
President Obama, is greater creativity and more personalized learning environments (U.S.
Department of Education, 2009). The outcomes of these reforms on student achievement are not
available. While the idea of 21st Century Skills is touted, there is no specific framework
presented. The next section will present ideas within the literature that identify possible
strategies for implementing reforms in conjunction with NCLB (U.S. Department of Education,
2001).
Implementing New Reforms with NCLB
Presently, leaders are trying to be simultaneously responsive to calls for innovation,
critical thinking skills, adaptability, and creativity (21st Century Skills) yet still meet the
demands and adequate yearly progress (AYP) testing targets of NCLB (Brown, 2007). Fusarelli
and Schoen (2008) asserted that NCLB (U.S. Department of Education, 2001) reforms, although
admirable, place competing sets of demands on teachers and schools by simultaneously
implementing standardized, routinized high-stakes testing, while expecting schools to turn out
students who are creative thinkers and real-world problem solvers.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 37

The risks associated with attempting major reforms such as NCLB (U.S. Department of
Education, 2001) for urban superintendents does not provide the potential gains that would prove
worth the risk (Fusarelli & Schoen, 2008). This begets the question, “What conditions in local
contexts are necessary for superintendents to risk implementing reforms with the imperative of
21st Century Skills?” Despite evidence of the need to develop 21st Century Skills there are great
challenges associated with implementing any major reforms that do not relate to the policy goals
of NCLB (U.S. Department of Education, 2001).
21st Century Skills
The idea of 21st Century Skills is not a new one; but the fact is that the U.S. economy
depends on their development for today’s children has made it paramount for U.S. economic
success (Rotherham & Willingham, 2009). Presently, NCLB (U.S. Department of Education,
2001) requires that all children will become proficient in basic skills by 2014. However, as Silva
(2008) indicated it’s not what skills the children have that are important, but what the children
can do with those skills is key. The fact that our nation does not employ assessments similar to
those used by PISA and TIMMS (National Center for Education Statistics, n.d.) indicates the
lack of understanding by politicians and many members of the community at large regarding
how basic and complex skills are developed and what order they are developed. As Silva
asserted, the development of basic skills is only a minimum requirement and policymakers must
understand this idea and develop educational policy related to the needs of the United States.
The Case for 21st Century Skills
As indicated by Obama (2006), we are educating children in 20th century schools and
then sending them out into a 21st century world. Labor reports, international assessments, and
educational literature have indicated for decades the need to develop 21st Century Skills for

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 38

students in our schools. This section will describe how changes in the world demonstrate the
need to develop 21st Century Skills in American schools.
Needs of the economy past and present. During the industrial age, the value chain of
our economy was related to the production of goods (Fadel & Trilling, 2009). The process began
with extracting resources and then extended to the distribution of products and services. Such
jobs required skills like reading, writing, and basic arithmetic (Fadel & Trilling, 2009). The
value chain in the knowledge-based economy centers on the development of the production of
services not goods.
According to Fadel and Trilling (2009), there are four forces that have converged to
demonstrate the need for the development of 21st Century Skills. Fadel and Trilling identified
knowledge work as one of the catalysts for developing 21st Century Skills in students. In
today’s world, knowledge work demands a steady supply of people who use brainpower and
digital tools (technology) to apply their daily work. Fadel and Trilling argued that the need
exists for knowledge workers to create, or innovate, new products and services which solve real
problems and meet the needs of real customers. Today’s digital lifestyles require learning to be
personalized, collaborative, interactive, and creative. Learning research demonstrates the need
for more authentic learning in context that produces more enduring understandings. Fadel and
Trilling extended their argument to indicate that students must build mental models that are
visceral and virtual; ones that make learning visible. Their research highlighted the importance
of internal motivation, or feeling connected to what is being learned, in order to be functional
(Fadel & Trilling, 2009). Another factor that influences the need for 21st Century Skills is the
speed that knowledge develops in the world. The knowledge-doubling curve has changed the
needs for our society, and consequently the skills that students develop. In the 1900s, knowledge

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 39

doubled at a rate of once a century, but today knowledge doubles every 18 months (Greenstein,
2012; Kay & Greenhill, 2013).
Principles of learning and 21st Century Skills. Astleitner’s (2005) principles of
learning support the change required for curriculum and instruction by NCLB (U.S. Department
of Education, 2001). Several of the principles discussed in the article appear to relate to the
development of 21st Century Skills.

The first principle was instruction and should be based on
a design for reflexive learning. Reflexive learning entails an active process of constructing
knowledge in which the contents of memory are meditated by thinking processes. The second
principle of good learning must consider the strengths of the student. This knowledge is
acquired through effective formative assessment. A third principle is that learning should be
acquired in multiple contexts and situations. Astleitner (2005) stated that students must be able
to apply their learning in a variety of contexts. Another principle is that learning must support
both basic knowledge and high-order thinking (Ambrose, Bridges, Dipietro, Lovett, & Norman,
2010; Astleitner, 2005; Silva, 2008). A sixth principle of learning is that instruction should
guide self-regulation. Self-regulated learning was defined as the means that the student uses to
control the learning process related to the given goals (Astleitner, 2005). This principle includes:
collaborative learning, project management, and communication.
A theory of action in the development of NCLB (U.S. Department of Education, 2001) is
that children have to master basic skills first. Anderson and Krathwohl’s (2000) revised Bloom’s
taxonomy and identified that the stages of the learning process do not require a linear sequence
or a specific order of instruction. The U.S. Department of Education report entitled “Measuring
Skills for the 21st Century” regarding the instruction of math further substantiated the idea that
basic skills can be taught in conjunction with other basic skills (Silva, 2008). The research on

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 40

current learning theory further substantiates the need for revisions to curriculum, instruction, and
professional development to meet the economic and policy needs for the United States.
Major 21st Century Frameworks
While there is no consensus about what specifically 21st Century Skills include, there are
several major frameworks being used by districts and schools. The following section will
delineate the different components of the major 21st Century Skills’ frameworks.
Partnership for 21st Century Skills. One of the most prevalent frameworks that
delineate a definition of 21st Century Skills was created by the Partnership for 21st Century
Skills. Figure 1 highlights four different domains of 21st Century Skills and the different areas
of practice that they intersect.
Figure 1 describes the different 21st Century Skills and Outcomes as conceived by the
Partnership for 21st Century Skills (P21, 2009). The first domain in the framework is Core
Subjects and 21st Century Themes, which integrates a mastery of core subjects (English, math,
science and social studies) with the interdisciplinary themes of global awareness, financial
literacy, civic literacy, economic literacy, and health and environmental literacy. The second
domain of P21’s framework is learning and innovation skills. The skills include: (a) critical
thinking and problem solving, (b) communication and collaboration, and (c) creativity and
innovation (Partnership for 21st Century Skills, 2011).

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 41

Figure 1: Partnership for 21st Century Skills’ Framework
Source: Partnership for 21st Century Skills, (2009, p. 1). P21 Framework definitions. Retrieved from
http://www.p21.org/storage/documents/P21_Framework_Definitions.pdf

The third domain of the framework is information, media, and technology skills, which
was developed to prepare students to gain access to information and technology for a myriad of
uses. The skills included in this domain are: (a) information literacy, (b) media literacy, and
(c) information and communications technology (Partnership for 21st Century Skills, 2011).
The final domain developed by P21 is life and career skills. The elements of this domain
include: (a) flexibility and adaptability, (b) imitative and self-direction, (c) social and cross-
cultural skills, (d) productivity and accountability and (e) leadership and responsibility
(Partnership for 21st Century Skills, 2011).
There are agreements among 17 states to implement new standards, assessments, and
professional development programs around the P21 framework. Some of the collaborations
include the development of the Common Core Standards and the Wisconsin Financial Literacy.
The results of these and similar efforts to develop 21st Century Skills are unknown (Partnership
for 21 Century Skills, 2011). The lack of formal policies or implementations to review makes

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 42

conclusions regarding these frameworks difficult at this time. The next section describes a
framework of 21st Century Skills developed by the North Central Regional Educational
Laboratory.
North Central Regional Laboratory. The North Central Regional Educational
Laboratory and the Metiri Group (NCREL, 2003) has developed a framework highlighting
several 21st Century Skills. The first domain as described by this framework is digital-age
literacy. Digital-age literacy encompasses: (a) basic, scientific, economic, and financial literacy,
(b) visual and information literacies and (c) multicultural and global awareness.
The second domain is inventive thinking or creativity. The first sub skill of this domain
is adaptability. The next sub skill is creativity, the act of bringing something new and original
into existence. This domain also includes risk taking and high order thinking (North Central
Regional Educational Laboratory & the Metiri Group, 2003).
The third domain in NCREL and the Metiri Group’s (2003) framework is effective
communication. According to NCREL and the Metiri Group, effective communication includes:
teaming and collaboration, interpersonal skills, social and civic responsibility, and interactive
communication.
The final domain of the NCREL and the Metiri Group’s (2003) framework is high
productivity. The first element of high productivity is the skills of prioritizing, planning, and
managing for results. The next element of this domain is the effective use of real world tools
(e.g., hardware, software, and networking devices). The last skill is the ability to produce
relevant high quality products or services (North Central Regional Educational Laboratory & the
Metiri Group, 2003).

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 43

Organisation for Economic Cooperation and Development. The Organisation for
Economic Cooperation and Development (OECD, 2005), the author of the Programme for
International Student Assessments (PISA, 2009), identified three critical competencies required
for students of the future. These three competencies include: (a) using tools interactively,
(b) interacting with heterogeneous groups, and (c) acting autonomously. While not a direct
reference to 21st Century Skills, it is relevant because the PISA assessments are guided by
applied skills.
Three minds for the future. The renowned Harvard Professor Howard Gardner (2006)
has identified three minds for the future that show a correlation to 21st Century Skills. The first
mind defined by Gardner was the disciplined mind. According to Gardner, a disciplined mind
has mastered one way of thinking. The second mind for the future was the synthesizing mind. A
synthesizing mind gathers, organizes, and synthesizes information from disparate sources in
meaningful ways. The third mind is the creating mind, which generates new ideas that have
applicability to the real world.
The major frameworks of 21st Century Skills have several commonalities. They all
espoused the development of critical thinking and problem-solving skills. They all argued for
the need of communication and collaboration skills. All major frameworks highlighted the need
for personal skills, such as self-direction. Also, the frameworks highlighted the importance of
new literacies like Financial Literacy and Information and Communication Technology (ICT)
literacy.
The perceived limitations of these frameworks are based on the lack of empirical studies
that highlight their effectiveness in practice. The next step to contribute to the literature will be
implementations made by districts and empirical studies to measure their effectiveness.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 44

Implementing 21st Century Skills. Kay and Greenhill (2013) outlined a seven-step
plan for implementing 21st Century Skills. The seven-step plan that the authors developed is
based on the Partnership for 21st Century (2011) skills framework. According to Kay and
Greenhill (2013), the first step in the process of implementation was to adopt a vision. The
vision should identify the definition of 21st Century Skills and specifically what students will
need to be successful. The second step of the implementation process was to create consensus
about the development of 21st Century Skills within the community. They argued the process
for developing the initiative should be co-created with the community to identify the objectives
of the reform. The third step of the implementation process was to align the school system, to
ensure that the various parts of the system are integrated (Kay & Greenhill, 2013). Kay and
Greenhill appeared to indicate that all parts of the system have acquired the capacity for
implementation. This phase requires that all actors (district, school leaders, and teachers) are
prepared for the implementation of 21st Century Skills. The fourth phase of implementation was
to build teacher capacity. Kay and Greenhill argued that this is accomplished by building
teacher’s capacity for the four C’s—communication, collaboration, critical thinking, and
creativity. The fifth phase of implementation was to focus the direction of curriculum and
assessment of the district. The sixth phase was to provide support to teachers through
professional learning communities, personalized learning environments, and providing assistance
through professional development. The last phase of implementation was to improve and
innovate through the existing framework that has been developed (Kay & Greenhill, 2013).
There are two aspects of this implementation guide that limit its saliency. The first aspect is that
it does not provide empirical data that describes what aspects of their plan have been successful
and which elements require refinement. Secondly, this framework did not describe the elements

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 45

necessary for a heterogeneous district to enact a 21st Century Skills’ framework as policy.
However, the framework in “The Leader’s Guide to 21st Century Education: 7 Steps for Schools
and Districts” (Kay & Greenhill, 2013) can serve as a good guide for developing the protocols of
this study and then as a potential tool for the studies’ data analysis.
Assessing 21st Century Skills. As there is no universal definition for 21st Century
frameworks, there is not a universal tool for assessing their development. This section will
describe some of the ways that many 21st Century Skills can be assess42ed. According to
Kyllonen (2012), one method for assessing 21st Century Skills is through the use of self-rating
systems like ones used with the PISA. Self-rating systems can be used to assess: motivation,
engagement, attribution, and use of time. Another self-rating system is called a Behaviorally
Rated Anchor Scale (BARS). The BARS scale assesses the effectiveness that students use to
approach problems (Kyllonen, 2012). In addition to assessing problem solving, this can address
students’ abilities for self-direction. While self-ratings are possible assessment tools, they can
only be one component of comprehensive assessment, as they do not assess applied skills.
Another self-assessment tool is anchoring vignettes, which were used on the 2012 PISA.
Anchoring vignettes demand that students respond to a variety of situations. Kyollnen (2012)
believed that this measurement is more effective because it addresses issues of the BARS scale
and Likert-scale measurements. These assessment tools represent a promising aspect of
assessing some 21st Century Skills, but the study does not address how these types of
assessments can be used on a large scale nor does it comprehensively assess all 21st Century
Skills and content areas, e.g., language arts and mathematics (Kyollnen, 2012).
Marzano and Heflebower (2012) also identified the use of rating scales to measure the
difficulty of a learning objective, and to analyze students’ level of critical thinking on a particular

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 46

assignment. In Table 1, there is an example of a critical thinking rating scale that can be used to
assess this competency across several content areas and through Project Based Learning.

Table 1

Scale for Identifying Common Logical Errors

Scale Logical Errors
Score 4.0 The student applies the strategy in unusual situations or identifies logical
errors that go beyond those that were explicitly taught.
No major errors or omissions regarding the score 4.0

Score 3.0 The student analyzes appropriate information for common logical errors.
No major errors or omissions regarding the score 3.0 content

Score 2.0 The student can describe situations in which it is beneficial to analyze
information for common logical errors.
He or she can describe or recognize the various common logical errors that
have been taught, including those of faulty logic. Attack, weak
reference, and misinformation.
No major errors or omissions regarding the score 2.0 content

Score 1.0 With help, partial success at score 2.0 content and score 3.0 content

Score 0.0 Even with help, no success
Source: Marzano and Heflebower (2012, Appendix B, Analyzing and Utilizing Information: Identifying Common
Logical Errors, locations 4694, 4719).

This scale could be replicated as a model of formative assessment for teachers across
content areas. This rating scale is another component of assessing 21st Century Skills but is also
by no means exhaustive of the methods that need to be used to assess all 21st Century Skills and
content-based skills, e.g., math and science (Marzano & Heflebower, 2012).
Another type of assessment for 21st Century Skills is performance assessments.
However, as previously indicated, there is a much higher cost associated with the use of
performance assessments. Nevertheless, the effectiveness of this assessment can be explored

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 47

when the Smarter Balanced assessment is implemented along with the Common Core Initiative
(Kyollnen, 2012).
Greenstein (2012) suggested an array of formative assessment tools that can be used to
measure 21st Century Skills. Greenstein highlighted the use anecdotal notes, student logs, and
checklists to assess the skills of communication and collaboration. The tools suggested by
Greenstein are good formative measures but she does not indicate what type of assessment could
be used to summative measure these skills.
Another potential tool for measuring 21st Century Skills is through the use of portfolios
and rubrics. The use of portfolios can be an effective tool for demonstrating student progress
towards content standards and their level of growth over a period of time. Additionally, students
can use e-portfolios to track their own progress, schedule, and grades. Greenstein (2012)
indicated that this form of assessment can show evidence of student progress towards the
mastery of specific goals, ability to adhere to a specific schedule, and student reflections about
their learning progress.
Greenstein (2012) noted that portfolio types of formative assessments should only be part
of a larger assessment system of 21st Century Skills. The suggestions of these methods of
assessment made by Greenstein (2012) are very practical, as they are practices already utilized
by many teachers. These suggested tools of measurement will be useful in determining what
indicators superintendents are or are not using to measure the development of 21st Century
Skills.
Darling-Hammond (2010b) conducted a literature review of the policy frameworks used
by several other nations who are attempting to implement 21st Century Skills. Darling-
Hammond’s review included an analysis of the assessment systems used in UK, Singapore,

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 48

Australia, and Finland. These countries used a combination of school-based assessments (open-
ended, inquiry-based papers, and projects), and a series of nationwide, on-demand assessments.
For these nations, students are not assessed in every grade, but at critical points in their academic
career (e.g., only students in grades 3, 5, 7, 9 are assessed in Australia). There are several
findings associated with Darling-Hammond’s study that are potentially useful for the U.S.
educational system. The study of these countries’ policy frameworks indicated that national
curriculum and goals surrounding 21st Century Skills have led to strong performances on
international assessments. The assessments described in Darling- Hammond’s comparative
review of the measure used by other nations (e.g., Finland and Australia) are the types of skills
that employers articulated that they desire in the “Are They Really Ready to Work” report
(Casner-Lotto & Barrington, 2006; Partnership for 21st Century Skills, 2011). Presently, there is
no comprehensive way to conduct a large-scale assessment that will measure all 21st Century
Skills for the US but perhaps the Smarter Balanced or PARCC will represent progress towards
these goals.
Criticisms and barriers to the development of 21st Century Skills. Fadel and Trilling
(2009) highlighted five potential barriers to the implementation of 21st Century Skills in the
United States. The first potential barrier was industrial-age educational policies. The authors
also believed that educational accountability and standardized testing are impediments to
implementation (Fadel & Trilling, 2009). Fadel and Trilling also believed that the educational
publishing industry and values of parents can impede the development of 21st Century Skills.
Lastly, they believed that the teaching profession can impede the development of 21st Century
Skills by resisting the changes that to their practice that would be necessary for effective
implementation (Fadel & Trilling, 2009).

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 49

Rotherham and Willingham (2009) expressed three reservations regarding the
implementation of 21st Century Skills. They believed that educators and policymakers must
ensure the instructional program is complete and that content is not shortchanged for an
ephemeral pursuit of skills (Rotherham & Willingham, 2009). Their article further argued that
states, school districts, and schools need to revamp how they think about human capital;
particularly that teachers must receive effective professional development (Rotherham &
Willingham, 2009). Lastly, assessments must be developed to accurately measure complex
tasks. The criticisms and concerns of these authors represent real challenges that must be
addressed to develop 21st Century Skills but cannot be an impediment to their development.
Leadership Techniques for Implementing Reforms
The ability to implement reforms successfully is dependent on the effectiveness of the
strategies used by leaders. This section of the literature review will discuss barriers to
implementing reforms, general leadership strategies (e.g., distribute leadership), and a meta-
analysis that studied the impact of district leadership on student achievement (Marzano &
Waters, 2009).
Loose coupling. Elmore (2002) described that one of the difficulties of implementing
and sustaining reforms is associated with the loose coupling between schools and the district.
Loose coupling can be described as how U.S. school structures buffer schools from outside
influences and inspection. From Elmore’s point of view, loose coupling creates the appearance
of the management of the technical core of instruction. This idea tends to account for why
promising practices only appear to take root in some classrooms. Elmore believed that the loose
coupling theory explains the tenuous relationship between the district and leaders in schools of
all sizes.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 50

Another barrier to the implementation of new reforms relates to the idea that policy is to
maintain the illusion, between policy makers and the public, that teaching and learning is
actually changing (Elmore, 2002). Elmore (2002) argued that policymakers do not value
individual or collective capacity building; therefore, one of the critical elements of implementing
change, professional development, is ignored when new policies are developed.
Elmore (2002) believed that there are four challenges to leadership and the imperative of
large-scale improvement. The first challenge of implementing large-scale reform related to the
problem of scale because the key units of change activate at very different rates. The second
challenge related to large-scale change is the problem of receiving effective feedback. As
Hentschke and Wohlstetter (2004) described, there is an information asymmetry issue that exists
between the superintendent and the teachers in the classroom. The teacher is the one with the
most information regarding the implementation, and there is not often clear feedback about the
progress of specific reforms. The information asymmetry problem leads to the third challenge
associated with the difficulty of large-scale change. There must be an effective feedback loop
that presents issues with implementation to district leadership (Elmore, 2002). The last
challenge affiliated with large-scale improvement is the problem of standards. Often, solutions
do not exist or are not readily available to implement and must be developed through inquiry
(Elmore, 2002).
Wagner’s (2007) research on the five habits of mind that count extends the challenges
related to reform by arguing that educators are not trained to analyze and solve problems of
practice. His argument continued with the idea that leaders must be able to effectively frame the
problem at hand (Wagner, 2007). An effective framing of the problem must be followed with a
theory of action that fits in with the framing of the problem.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 51

The research of Bolman and Deal (2008) highlighted several organizational and
leadership related challenges to implementing change. One of the challenges with organizations
is to blame people for flaws in policy and performance. A second common challenge for
organizations is that flaws and failures result in the blaming of the bureaucracy. A third flaw of
organization is that people often seek individual power. Collaboration is a common strength of
organizations (Bolman & Deal, 2008).
The problems associated with implementing large-scale change and organizational
impediments are pertinent to the problem of this study. This study is attempting to discover
examples of successful strategies for implementing major reforms like 21st Century Skills. The
next sub-section will describe the potential of distributive leadership as an effective model for
change.
Distributive leadership. According to Marzano, McNulty, and Waters (2005), when
coupled with a singular vision, distributive leadership provides multiple sources of direction and
guidance for the key actors within an organization. Marzano et al. (2005) contended that one of
the purposes of leadership is instructional improvement, and it requires continuous learning. The
environment for continuous learning must also include collaboration and critique. This type of
environment can be established through professional learning communities (PLC). Marzano et
al. (2005) extended their argument with the idea that modeling effective instructional practices is
more effective than providing mandates. Also, they believed a distributive model of leadership
can be established by two-way accountability between the principal (federal and state
government, and the local district) and the agent (principal and teacher) (Hentschke &
Wohlstetter, 2004). Lastly, effective reforms can be established through capacity building. The

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 52

above strategies can be used as a prism for analyzing the responses solicited from
superintendents regarding their strategies for implementing 21st Century Skills.
The role of the superintendent in implementing change. A meta-analysis conducted
by Marzano and Waters (2009) attempted to establish what affect district leadership has on
student achievement. There were two questions that they attempted to address through their
meta-analysis. First, “What is the strength of the relationship between district-level
administrative actions and average student achievement?” The second question of their analysis
attempted to answer is, “What are the specific district leadership behaviors that are associated
with student achievement?” (Marzano & Waters, 2009, Chapter 1, para. 6).
Their meta-analysis included 2,714 districts, 4500 ratings of superintendents and 3.4
million student achievement scores (Marzano & Waters, 2009). The results of their first
question, “What is the strength of the relationship between district-level administrative actions
and average student achievement?” are shown in Table 2.
The effects of reading achievement are best when the quality of the teacher is excellent as
indicated by Table 2. However, the table does indicate that excellent district leadership is
correlated with a significant gain of reading achievement (Marzano & Waters, 2009). These
results demonstrate the need for a high-quality leadership in conjunction with excellent teaching.
Table 3 shows the effects of district leadership on mathematics achievement with consideration
for quality of the key actors (district, school, and teacher).

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 53

Table 2
Effects of District Leadership of Reading Achievement

District

School

Teacher
Predicted Average
Gain for Students at
the 50th percentile
Average (50th
percentile)
Average (50th
percentile)
Average (50th
percentile)

0%
Average (50th
percentile)
Average (50th
percentile)
Superior (84th
percentile)

10%
Average (50th
percentile)
Average (50th
percentile)
Excellent (98th
percentile)

20%
Superior (84th
percentile)
Superior (84th
percentile)
Average (50th
percentile)

7%
Excellent (98th
percentile)
Excellent (98th
percentile)
Average (50th
percentile)
13%
Source: Marzano & Waters, 2009, Chapter 1, para. 6.

The impact of district leadership on mathematical achievement for students is more
powerful than for reading. These results further demonstrate the merit of the focus of the
superintendent as a critical element to implementing 21st Century Skills within schools
(Marzano & Waters, 2009).
Marzano and Waters’ (2009) second question in there meta-analysis was, “What are the
specific district leadership behaviors that are associated with student achievement?” There were
five common behaviors associated with student achievement according the meta-analysis.
First, to ensure collaborative goal setting, Marzano and Waters (2009) found that district
leaders include all relevant stakeholders. Relevant stakeholders include: central office staff,
building-level administrators, and board members.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 54

Table 3

Effects of District Leadership on Mathematics Achievement

District

School

Teacher
Predicted Average
Gain for Students at
the 50th percentile
Average (50th
percentile)
Average (50th
percentile)
Average (50th
percentile)

0%
Average (50th
percentile)
Average (50th
percentile)
Superior (84th
percentile)

14%
Average (50th
percentile)
Average (50th
percentile)
Excellent (98th
percentile)

26%
Superior (84th
percentile)
Superior (84th
percentile)
Average (50th
percentile)

9%
Excellent (98th
percentile)
Excellent (98th
percentile)
Average (50th
percentile)
17%
Source: Marzano & Waters, 2009, Chapter 1, para. 6.

Second, according to the authors, establishing non-negotiable goals for achievement and
instruction was accomplished through collaborative goal setting. Non-negotiable goals are the
results of focus on the areas of student achievement and classroom instruction (Marzano &
Waters, 2009).
The third behavior associated with effective district leadership was board alignment with
and for support for district goals. The results of the meta-analysis indicated that the alignment
and support of the school board in the nonnegotiable goals for achievement and instruction is
essential (Marzano & Waters, 2009).
The fourth behavior associated with effective district leadership was the monitoring of
achievement and instructional goals. Marzano and Waters (2009) believed that effective
superintendents continually monitor district progress toward achievement and instructional goals

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 55

to ensure that these goals remain the driving force behind the district’s actions. They also
contended that effective superintendents ensure that each school regularly examines the extent to
which it is meeting targeted achievement goals (Marzano & Waters, 2009). Any discrepancies
between expected teacher behavior in classrooms as articulated by agreed-upon instructional
models and observed teacher behavior are taken as a call for corrective action.
The fifth, and last common behavior of district superintendents, was the allocation of
resources to support the goals for achievement and instruction. Marzano and Waters (2009)
believed that professional development supported with this funding should be focused on
building the requisite knowledge, skills, and competencies teachers and principals need to
accomplish district goals
There are three positives regarding the impact of their meta-analysis on this study. First,
it demonstrated the role of superintendent leadership is vital. The results substantiate the
research question that addresses how superintendents solicit the support of critical stakeholders
(e.g., the school board). Third, the results of their meta-analysis will guide the development of
the interview and survey protocols.
Presentation of Theories
This research study will use two frameworks to address the problem of implementing
21st Century Skills, while meeting current accountability requirements. One framework that will
be used to analyze and address the problem is Wagner’s (2008) seven survival skills. The
second framework is Bolman and Deal’s (2008) four frames of leadership. The combination of
these frameworks will be used to encompass all of the research questions.
Wagner’s seven survival skills. Wagner (2008) identified seven skills essential for
students to survive in the 21st century. Wagner defined critical thinking and problem solving as

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 56

the ability to apply abstract knowledge to solve a problem and execute a solution. The skill of
collaboration across networks can be defined as working with diverse groups to solve problems.
The idea of leading by influence is motivating groups around the world by a common vision or
goal. The third survival skills of agility and adaptability are defined as the ability to change to
within certain circumstances and to adapt new environments, e.g., workplace and general
(Wagner, 2008). The fourth survival skill of initiative and entrepreneurialism can be defined as
students being self-directed and willing to try new endeavors. The fifth survival skills of written
and oral communication are critical to be effective in business and democratic processes. The
sixth survival skill of accessing and analyzing information relates to know how to find and
identify important information. The seventh survival skills are curiosity and imagination, which
demonstrates one’s ability to detect patterns and opportunities for creating something original
(Wagner, 2008).
Bolman and Deal’s four frames of leadership. Bolman and Deal (2008) defined a
frame as a mental model—a set of ideas and assumptions that you carry in your head to help you
understand and negotiate a particular territory. They identified four frames for effective
leadership. The first frame is structural, which is characterized by a rational world that
emphasizes organizational architecture, including goals, structure, technology, specialized roles,
coordination, and formal relationships. The second frame of leadership is political, which is
characterized by an arena of competing interests for individuals and different groups. The third
frame of leadership is human resources, which is described as a frame that is concerned with
supporting the needs and feelings of the individuals within the organization. The fourth frame is
the symbolic, which is defined as the symbols, their meaning, and the interpretation that people
have regarding specific contexts (Bolman & Deal, 2008).

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 57

Summary
The discussed literature highlighted five specific topics regarding the implementation of
21st Century Skills while meeting federal and state accountability requirements. The review
began by identifying the effects of globalization on the goals of education in the 21st Century.
Then, the review highlighted successful attributes of international models of education that could
be implemented in the United States. The review also highlighted the effects of NCLB (U.S.
Department of Education, 2001) and accountability measures on student achievement and new
educational reforms. The review of literature highlighted critical aspects of 21st Century Skills
including: rationale, current frameworks, and ideas for implementation, assessment, and
criticisms. This review concluded by describing current literature about the role of
superintendents in implementing reforms and the positive behaviors associated with change. The
next chapter will describe the methodology to be employed for this mixed methods study.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 58

CHAPTER THREE: METHODOLOGY
The idea that American children should possess skills, now known as 21st Century Skills,
has existed for decades. As Rotherham and Willingham (2009) argued, the need to develop 21st
Century Skills is now paramount to America’s economic vitality.
Introduction
Purpose of the Study
As indicated in Chapter 1, this study has three primary goals. The first goal of the study
is to enhance the existing body of empirical research on how superintendents implemented 21st
Century Skills’ programs. The second goal of the study is to attempt to identify and address any
gaps that exist in the current body of scholarly research regarding how superintendents
implemented 21st Century Skills. The third objective of this study will be to identify promising
and/or successful practices that could be implemented by other districts possessing similar
characteristics.
The four primary objectives for this chapter are: First, to reiterate the goals of the study,
the associated research questions, and the research methodology that will be used. Second, this
chapter will list various factors of the sampling process, including: (a) the criteria for selecting
the sample, (b) the type of sampling that will be employed by the study, (c) the process for
selecting the sample used in this study, (d) the sample selected, and (e) any anticipated sampling
issues. Third, to describe the instruments used to conduct the study and the conceptual
framework that aided their development. The fourth objective is to articulate the methods
employed for data collection and analysis.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 59

Goals of the Study and Research Questions
The intent of this study is to identify strategies used by superintendents for implementing
21st Century Skills. The aim of NCLB (U.S. Department of Education, 2001) to develop 100%
proficiency of basic skills by 2014 does not effectively address the skills that students will need
to be globally competitive in the 21st century (Fusarelli & Schoen, 2008). In the recent labor
report “Are They Really Ready to Work” (Casner-Lotto & Barrington, 2006), a survey of 400
businesses indicated that employers felt new employees lacked skills, like critical thinking and
collaboration, essentially substantiating the inadequacies of NCLB (Partnership for 21st Century
Skills, 2011). Furthermore, the PISA, an assessment that measures students’ ability to apply
content knowledge, shows that American students are mediocre at best in recent years regarding
performance. As a result of apparent weakness in preparing students for the 21st-century world
economy, this study will attempt to identify strategies used by superintendents in the state of
California to implement 21st Century Skills.
Another aspect of this study will address how superintendents intend to assess their
implementation of 21st Century Skills’ programs. Presently, unlike some states, there is no
formal plan as to how the districts within the state of California will develop and assess the
implementation of 21st Century Skills (Partnership for 21st Century Skills, 2011). Therefore,
this study attempted to find promising practices in the methods for assessing the implementation
of 21st Century Skills.
Within a school district, the highest-ranking administrator is the superintendent. A
superintendent, by definition, possesses the capacity to influence and advance the direction and
primary objectives of the district. However, for success, all reform initiatives require the support
of people at all levels throughout the district (e.g., board of education, assistant superintendents,

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 60

principals, teachers, instructional coaches, and the community at large). This study sought
common leadership strategies, for including key stakeholders, in the implementation of major
reform initiatives such as 21st Century Skills’ programs. The following research questions have
been generated, and were investigated to determine how California superintendents plan to
implement or are implementing 21st Century Skills’ programs while meeting federal and state
accountability requirements.
1. What strategies do superintendents use to develop 21st Century Skills in their district?
2. What indicators do superintendents use to evaluate the implementation of 21st Century
Skills in their district?
3. How do superintendents include key stakeholders in the implementation of 21st Century
Skills?
Design of the Study
This study used a mixed-methods approach in which both qualitative and quantitative
data was collected and analyzed to address the research questions. A mixed-methods approach
was selected to address the objectives of this study because the complexities and many facets to
the issues of implementing major reform initiatives, like the development of 21st Century Skills,
cannot be effectively captured with one method of data collection (Patton, 2002). One benefit to
a mixed-methods approach is that it will support methodological triangulation by verifying the
consistency of the results with multiple approaches, e.g., surveys, interviews, and documents
(Patton, 2002). The value of a mixed-methods study was evident through the triangulation of
different data sources (interviews and document analysis) within the same method (Patton,
2002). Ultimately, the goal of this mixed-methods approach was to create the conditions for
consistency among the existing literature and the data collected in this study.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 61

Methodological Emphasis of the Study
The emphasis among the different methods of data collection and analysis will be
qualitative. The rationale for a qualitatively-based study is that these forms of collection and
analysis (interviews and document analysis) are better suited to understand the complexities of
implementing major reforms like 21st Century Skills; assessing its implementation and the
associated leadership strategies for involving key stakeholders (Maxwell, 2013; Merriam, 2009).
More specifically, a qualitative emphasis elicits greater details from superintendents regarding
situations, actions, and experiences surrounding the different factors for the problems being
investigated (Maxwell, 2013).
The quantitative portion of this study aided the qualitative data collection and analysis in
several critical ways. First, and foremost, the quantitative methods support the identification of
information rich cases that merit a closer examination. Second, quantitative data collection and
analysis provided descriptive information regarding the values, attitudes, and practices of
superintendents, through the results of the survey. The survey can serve to confirm or refute
evidence gathered through qualitative methods of data collection (Creswell, 2009; Maxwell,
2013; Merriam, 2009). Third, the data collected in the quantitative portion of this study
supported the direction of the qualitative aspects of the study, by identifying critical issues that
relate to the research questions of this study
Sample and Population
To best address the research questions, the unit of analysis for the problem in this study is
superintendents of large districts throughout the state of California. Multiple methods of
purposeful sampling will occur to meet the varied needs of the different phases of the study
(Patton, 2002).

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 62

The initial type of sampling that was employed for this study is purposeful criterion-
based sampling. Purposeful sampling is used to select information rich cases regarding the
phenomena or problem being studied (Patton, 2002). As the central administrator of a district,
superintendents are in a prime position to coordinate major reform efforts for their district,
leading to a unique perspective that merits further analysis.
This study also employed intensity sampling as a method for identifying information rich
cases. Intensity sampling is a type of sampling, which identifies units for analysis that display
the characteristics associated with the problem being investigated (Patton, 2002).
The overall goals of these various methods of purposeful sampling are to achieve
representativeness of the population, when drawing conclusions about the findings. A secondary
goal for this type of sampling was to make comparisons to illuminate the similarities and
differences in the data collected in this study (Maxwell, 2013).
Criteria for Selection
As previously stated, the subject unit of analysis for this study is superintendents. The
criteria for selecting superintendents to participate in this study was: (a) superintendents of
districts in the state of California with student enrollments of 10,000 or higher, (b) districts with
a minimum of 33% percent of the population consisting of racial and ethnic minorities (African-
American, Latino, and Asian American), and (c) districts that have at least 15% of the student
population receiving Federal Title I funding. The participants that met these criteria were
identified using demographical data generated from the California Department of Education’s
website www.cde.ca.gov.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 63

Process of Selection and Sampling Issues
Superintendents who met the defined criteria were selected on the basis of their voluntary
decision to participate in all elements of the survey. One of the main issues associated with the
sample identified for this study is generalizability or external validity. The results of this study
will not necessarily be generalizable to different districts, cities, and states. To be applicable to
other settings, the criteria between the sample of this study and the district will need to be similar
(Creswell, 2009).
Instrumentation
The following instruments were used to collect data for this study to ensure a consistent
and reliable approach to data collection: (a) survey, (b) interviews, and (c) document analysis.
Quantitative Instrumentation
The quantitative design can be categorized as a voluntary cross-sectional survey of
superintendents throughout the state of California (Fink, 2013). Three major themes have been
identified in a review of the literature that guided the development of the survey. They are:
(a) 21st Century Skills, (b) responses to accountability measures like NCLB (U.S. Department of
Education, 2001), and (c) leadership strategies for implementing reform. Wagner’s (2008)
Seven Survival Skills and Bolman and Deal’s (2008) four frames were used as models for
developing the proposed protocol.
The survey was created using the Internet based Qualtrics. A letter explaining the goals
and the purpose of the study were included in the initial contact made to the selected sample of
superintendents (see Appendix A). Also available were options for telephone and paper-based
surveys to ensure maximum response rates. Follow up emails were also employed to encourage
responses from all perspective subjects (Fink, 2013).

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 64

The design of the survey used several types of questions to elicit the data required to
answer the research questions. The demographic questions were used to capture data regarding
the general characteristics of the district being surveyed (e.g., what is the student enrollment of
your district). Also, the survey included items designed to obtain biographical information about
the subjects being surveyed (e.g., about the superintendents being sampled). Another type of
item used in the survey was a checklist to ensure that all potentially relevant elements that apply
are present (e.g., which 21st Century Skills’ framework or frameworks is your district using to
guide the implementation of the program). The survey contained numerous Likert-scale based
questions to determined superintendents’ values regarding 21st Century Skills and federal and
state accountability requirements. The Likert questions used a scale from 1-4 (1-strongly
disagree, 2-disagree, 3-agree, and 4-strongly agree) to show superintendents values. The survey
used for this study employed open-ended questions to determine superintendents’ opinions in
depth. The end of the survey included a prompt to solicit interested candidates for interviews in
the qualitative section of the study (Fink, 2013).
Qualitative Instrumentation
The qualitative portion of this study included two instruments to study the problem. The
first instrument used for the collection of data was interviews with superintendents identified
through the intensity sampling associated with analysis of the survey data. Interviews were
necessary for the design of this study because collecting data regarding this problem required
that superintendents provide detailed explanations of their beliefs, attitudes, and practices
regarding implementation of 21st Century Skills (Merriam, 2009).
The interviews associated with this study followed a semi-structured format of several
pre-designed questions, used to gather specific responses, relating to the data needs of the

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 65

research questions. There were also follow-up questions to address issues that required
clarification or to address issues that pertained specifically to unique context of each
superintendent’s district (Merriam, 2009). The protocols were developed using the existing
literature, Bolman and Deal’s (2008) Four Frames of Leadership, Marzano and Waters’ (2009)
meta-analysis on District Leadership That Works, and Wagner’s (2008) Seven Survival Skills.
The final instrument for this study was document analysis. The goal was to obtain
internal and external documents that highlight plans, progress, and efforts to implement 21st
Century Skills while meeting federal and state accountability requirements. The collection of
internal documents provided the opportunity to help analyze the leadership style and practices of
the superintendents surveyed and interviewed (Bilken & Bogden, 2003). Internal documents
provide actual evidence of what the district practices are regarding the problem this study
addresses. They can confirm or refute findings obtained from the other methods of data
collection. Analysis of external documents provided the opportunity to determine how the
district garners the support of the business community and public at large (Bilken & Bogden,
2003).
Data Collection
The data collection for this study was divided into three distinct phases. The initial phase
of the study sent a quantitative survey to superintendents throughout the state of California who
met the sampling criteria. The survey was delivered via Qualtrics with a letter outlining the
purpose and goals of the study. The end of the survey provided interested superintendents with a
prompt to indicate their interest in participating in an interview. The second phase of data
collection was interviews of superintendents who indicated interest in participation (see
Appendix B). Those superintendents that indicated interest were selected based on data

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 66

suggesting the most information rich cases. The final phase of data collection was an analysis of
internal and external documents that detailed aspects of the implementation of 21st Century
Skills’ programs, including frameworks, professional development models, and goals and
leadership strategies for implementation of major reforms. Superintendents surveyed were
requested to supply clarifying or supporting documents.
Data Analysis
To effectively and completely answer the research questions developed for this study, the
researcher analyzed the findings from all three data collection methods (survey, interviews, and
document analysis) to generate theory or theories regarding the subject being investigated.
Quantitative Data Analysis
For this portion of the study, descriptive data about the attitudes, beliefs, and practices
were generated through statistical measures developed by Qualtrics. These results were
imported to the software program Statistical Package for Social Scientists (SSPS). The
descriptive data was used to identify overall trends in the attitudes, beliefs, and leadership
practices surrounding the implementation of 21st Century Skills.
The analysis of the closed-ended questions was conducted using descriptive statistical
measures on Qualtrics (e.g., mean, mode, and standard deviation). The open-ended questions
were coded into categories to find common segments of data that indicate certain trends or
beliefs among the responses given (Fink, 2013). The final goal of data analysis for the
quantitative survey was to identify information-rich cases among those superintendents who
indicated interest in participating in interviews (Patton, 2002).

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 67

Qualitative Data Analysis
As indicated by Merriam (2009) and Maxwell (2013), data analysis begins after the first
collection of data. The initial phase of data collection began with the development of field notes
and memos (Maxwell, 2013). The goal of these two initial data analysis techniques were to
facilitate thinking about the interviews and stimulating ideas regarding potential categories for
the data collected (Bilken & Bogden, 2003; Maxwell, 2013). The field notes and memos were
developed in part by a reconstruction of the interviews conducted and approximating what the
subjects stated during the course of the interviews (Bilken & Bogden, 2003).
The two elements that were the focal point of the field notes for initial analysis included
description of aspects observed during the interviews, e.g., such as body language, demeanor,
and tone of voice. The second aspect of creating field notes was to reflect about what was
observed during the course of the interviews (Bilken & Bogden, 2003). This process occurred
after each interview and the notes were used to modify any inappropriate or ineffective questions
that did not meet the data needs of the research questions. The field notes helped to begin to
segment the data, and to identify patterns and themes that began to emerge. Lastly, the field
notes presented the opportunity to confirm or disconfirm initial patterns identified in the data.
This initial analysis supported the creation of new categories of commonalities throughout data
collected (Bilken & Bogden, 2003). Following the completion of data collection, for this phase,
all interviews were transcribed by professional services and were stored in a password protected
computer.
Following transcription of all interviews, the goal of data analysis was to reduce the data
collected. Data reduction was accomplished by reconfiguring the data and coding the common
segments identified in the transcripts (Miles & Huberman, 1994). This process is known as open

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 68

coding, which is an inductive way to find patterns and themes from the data (Merriam, 2009).
After the initial analysis, the data was displayed graphically to further narrow categories and
subcategories presented by the data. After cleaning and organizing the data, conclusions about
the findings were made.
The last phase of analysis was to examine the internal and external documents collected
from the districts, other states, and organizations. The goal was to confirm, disconfirm, or
enhance the findings from the survey and the interview by examining the articulated practices of
the given district.
Triangulation. The last step to ensure the validity of the findings was to attempt to
triangulate forms of data collected. The findings from the survey were compared to the
interview transcripts. Both the interview transcripts and the survey were compared to the
findings from the document analysis. The purpose of triangulation is to find aspects of the data
that converge, to display common themes, as well as aspects of the data that deviate from the
findings (Patton, 2002).
Ethical considerations. One of the primary concerns with the collection and analysis of
data was to identify, reduce, and eliminate any bias that would limit the validity of the findings
(Merriam, 2009). Prior to completion, the researcher must consider any evidence that
disconfirms the conclusions, to further ensure the validity of the findings (Merriam, 2009).
Lastly, a member check of the superintendents interviewed was conducted to ensure that the
descriptions and quotes were accurate (Merriam, 2009). A member check is a procedure in
which the emerging findings of a study are presented to interviewees to confirm the accuracy of
the findings. The goal of member checks is to ensure the internal validity of the study (Merriam,
2009).

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 69

Conclusion
This chapter included a restatement of the purpose of this study, the research questions,
and the overall study design. A description of the sampling method and criteria as well as the
population that was sampled was included in this chapter. Furthermore, the instrumentation used
for this study was described along with the methods of data collection and analysis. The next
chapter will describe and analyze the findings of the study.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 70

CHAPTER FOUR: FINDINGS
Introduction
This mixed-methods study investigated how California superintendents have
implemented, are implementing, or plan to implement 21st Century Skills within their school
district. There are three distinct objectives that motivated the researcher to conduct this study.
The primary objective for conducting this study was to identify strategies that superintendents
used or will use to implement 21st Century Skills’ programs within their district. The second
objective in conducting this study was to pinpoint the indicators that superintendents used or will
use to measure the progress and effectiveness of their districts’ implementation of 21st Century
Skills. The third objective of this study was to identify ways that superintendents solicited or
will solicit the support of key stakeholders to facilitate implementation of 21st Century Skills’
programs in their districts.
This chapter includes an analysis of data collected for this study and an exploration how a
sample of 10 California superintendents are implementing or planning to implement 21st
Century Skills’ programs within their school districts. This chapter will focus on two objectives.
The first objective of this chapter is to report and summarize the data collected via: (a) an
electronic survey administered by Qualtrics, (b) qualitative interviews of 10 superintendents, and
(c) internal and external documents collected during the course of the study. The second aim of
this chapter is to present and triangulate findings using the quantitative (survey) and qualitative
data (interviews and document analysis) collected along with the existing literature. The
quantitative data (average, median, and mean) was calculated by using the survey program
Qualtrics. Analysis of the qualitative data was conducted on the interviews by utilizing a method

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 71

of data analysis touted by Creswell (2009). Table 4 describes the steps of the method used by
Creswell (2009) to analyze and develop findings from the interviews conducted for this study.

Table 4
Summary of Qualitative Data Analysis Procedures
Procedure Description
1. Organize and prepare the data for analysis Transcribe interviews
Type field notes
2. Read through the data (interview transcripts
and documents)
Reflect on the tone and meaning identified in the
field notes
Develop general ideas about the data
3. Detailed analysis with a coding process Organize the data into categories
Label each category with a term (e.g., individual
research questions)

4. Use the coding process to generate themes Detailed creation of categories
Generate codes for descriptions
5. Advance how themes will be used Use narrative passages from interview transcripts to
convey findings
Include tables and visual illustrations
6. Make interpretation or meaning from the
data
Determine lessons learned from study and compare
findings to literature

Source: Creswell (2009) Research Design: Qualitative, Quantitative, and Mixed Method Approaches, p. 185.

Organization of Data Analysis
The data analysis for this study is organized into six sections. The first section of the
analysis contains: (a) the descriptive characteristics of the survey and interview participants,
(b) response rates for the quantitative survey, and (c) criteria used to select interview
participants. The next three sections contain findings and analysis that pertain to the following
research questions:

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 72

1. What strategies do superintendents use to develop 21st Century Skills in their district?
2. What indicators do superintendents use to evaluate the implementation of 21st Century
Skills in their district?
3. How do superintendents include key stakeholders in the implementation of 21st Century
Skills?
The fifth and sixth section of the analysis will examine the auxiliary findings generated in
this study that do not directly pertain to one of the research questions.
Descriptive Characteristics
A quantitative survey was distributed to 55 superintendents throughout the state of
California. The response rate for the electronic survey was 40% (22 of the 55 selected
participants responded to the questionnaire). The researcher was satisfied with the number of
responses for several reasons: First, the responses met the average return rate of 40% or greater
on surveys distributed by email, as highlighted by Dillman (2000). Second, based on time and
resource constraints, the number of respondents who met the criteria to participate in an
interview (the pool of respondents) was deemed sufficient to meet the data needs for the current
study. Among the 22 respondents to the quantitative survey, 18 met the criteria to participate in
the interview phase of the study. Within the pool of respondents, there were four
superintendents who declined to participate in the interview phase of the study.
Quantitative Demographic Data
Table 5 highlights the gender distribution of the 22 participants of the survey. The
gender distribution of the respondents to the survey was 82% male and 18% female.
The percentage of male and female respondents has been included to help illustrate the
demographics of the study. This table does not present any findings that require further analysis to meet
the data needs of the research questions.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 73

Table 5

Quantitative Survey: Superintendent Gender

Measure Male Female Total
Number of
Superintendents

18 4 22
Percentage of
Superintendents
82 18 100

Table 6 illustrates the demographics of the participants’ districts and the criteria used to
identify the sample student population meeting district selection criteria. As previously
indicated, the criteria used to select participants for the current study was: (a) student enrollment
of 10, 000 students or greater, (b) a minimum of 15% of the students enrolled in the district
receive free or reduced lunch, and (c) the district has a minimum minority enrollment of students
of at least 33%.
The sample population identified for the current study was met based on the data
presented in Table 6. The researcher sought to identify districts that have heterogeneous
characteristics including: (a) race, (b) ethnicity, and (c) Federal Title I standing to help ensure
that the findings of the study could be applied to other districts within California and across the
United States.
Qualitative Demographical Data
The objectives for administering the survey were to: (a) measure superintendents
attitudes regarding the development of 21st Century Skills, (b) determine if districts have or are
planning to implement 21st Century Skills, (c) ascertain the importance that superintendents
place on the implementation of 21st Century

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 74

Table 6

Quantitative Survey: District Characteristics

District

Enrollment
Percentage of Free
or Reduced Lunch
Percentage of
Minority Enrollment
1 24,324 30 71
2 11,456 32 71
3 15,489 54 66
4 28,668 50 82
5 23,771 44 70
6 42,560 65 73
7 25,622 29 57
8 18,852 63 80
9 19,540 67 84
10 26,187 51 45
11 19,126 88 95
12 29,854 48 67
13 53,437 70 46
14 655,494 71 90
15 16,546 42 55
16 13,580 78 46
17 22,848 72 94
18 18,326 48 68
19 56,970 60 79
20 53,785 24 40
21 12,270 63 89
22 10,836 31 70
Source: California Department of Education (2013, retrieved from http://data1.cde.ca.gov/dataquest/)

Skills within their districts, (d) learn the role that superintendents play or intend to play in the
implementation, and (e) identify which stakeholders the superintendents interact with or intend
to interact with during the program implementation.
The researcher believed that superintendents must be: (a) implementing or planning to
implement 21st Century Skills to qualify for participation in the interview phase of the study,
(b) considering implementation of 21st Century Skills to be important or very important,
(c) interacting with or intend to interact with key stakeholders, and (d) planning to play multiple
roles in the implementation of 21st Century Skills within their district, e.g., aligning

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 75

stakeholders, building capacity, and coordinating professional development (Marzano & Waters,
2009).
Ten superintendents were selected to participate in the interview based on their
affirmative responses to the questions. Table 7 highlights the demographics of the interview
participants (gender, ethnicity and education level) and their districts’ demographic information
(enrollment, the percentage of students receiving free or reduced cost lunch, and minority
enrollment).
The superintendents interviewed for these studies are predominantly male (80% of the
participants) and a majority have obtained the highest academic degree (90% of the participants
hold a doctoral degree). The districts have a total combined enrollment of 1,199,541; with an
average of 54,524 students per district, 61% of the students enrolled in the districts of study
receive free or reduced lunch (common measurement tool for socioeconomic status based on
Federal Title I status), and the districts list an average, 78%, of their enrollment as minority
students. The demographics indicate that 21st Century Skills’ programs are being implemented
in heterogeneous districts that meet the criteria for selecting a sample for the current study
(districts of 10,000 students or more, districts with 33% minority enrollment, and districts with
15% or more receiving free or reduced lunch).

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 76

Table 7

Qualitative Interview: Characteristics for Superintendents and Districts

Superintendent Profile District
A Gender: Female
Ethnicity: White
Education: Doctoral Degree

Enrollment: 11,456
Free and Reduced Lunch: 32%
Minority: 71%
B Gender: Male
Ethnicity: Latino
Education: Doctoral Degree

Enrollment: 28,688
Free and Reduced Lunch: 50%
Minority: 82%
C Gender: Male
Ethnicity: White
Education: Doctoral Degree

Enrollment: 18,852
Free and Reduced Lunch: 63%
Minority: 82%
D Gender: Male
Ethnicity: White
Education: Master’s Degree

Enrollment: 19,540
Free and Reduced Lunch: 67%
Minority: 84%
E Gender: Female
Ethnicity: White
Education: Doctoral Degree

Enrollment: 19,126
Free and Reduced Lunch: 88%
Minority: 95%
F
Gender: Male
Ethnicity: White
Education: Doctoral Degree

Enrollment: 655,494
Free and Reduced Lunch: 71%
Minority: 90%
G White Gender: Male
Ethnicity: White
Education: Doctoral Degree
Enrollment: 13,830
Free and Reduced Lunch: 78%
Minority: 46%
H Gender: Male
Ethnicity: Latino
Education: Doctoral Degree

Enrollment: 22,848
Free and Reduced Lunch: 72%
Minority: 94%
I Gender: Male
Ethnicity: White
Education: Doctoral Degree

Enrollment: 18,326
Free and Reduced Lunch: 48%
Minority: 68%
J Gender: Male
Ethnicity: White
Education: Doctoral Degree
Enrollment: 23,771
Free and Reduced Lunch: 44%
Minority: 70%

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 77

Findings
Research Question 1
What strategies do superintendents use to implement 21st Century Skills’ programs in
their districts?
This first research question sought to identify the various strategies employed by
superintendents in the state of California to implement 21st Century Skills within their district.
The strategies for implementation addressed in the interview protocol (see Appendix C) for this
research question included: (a) methods for implementing 21st Century Skills’ programs, (b) the
role of professional development in the implementation process, (c) how resources were
allocated, (d) role of technology in the implementation, and (e) how does the superintendent
align the different components of the system (e.g., stakeholders and resources).
The methods used to capture the data needs for these questions were: (a) an electronic
survey link from Qualtrics emailed to superintendents, (b) qualitative interviews, and
(c) document analysis of internal and external documents from the districts studied. Samples of
each protocol can be found in Appendices C-E.
Findings: Responses to Closed-Ended Question
Table 8 highlights superintendents’ responses made to survey question 1: “Which of the
following strategies will you or did you use to build your district’s capacity to implement 21st
Century Skills?
Survey Question 1

Which of the following strategies will you or do you use to build your districts
capacity to implement 21st Century Skills?
Twenty of 22 superintendents responded to the first question. One hundred percent of the
superintendents who answered research question one used, or intend to use, professional

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 78

Table 8
Strategies to Support Implementation of 21st Century Programs

Strategy

Number of Superintendents
Percentage of
Superintendents
Professional Development 20 100
Curriculum Writing 15 75
Instructional Coaching 18 90
Other 4 20

development as a tool for developing their district’s capacity to implement 21st Century Skills.
The research of Marzano and Waters (2009) highlighted how professional development supports
the growth of district initiatives. Ninety percent of the superintendents responding indicated that
they are, or will use, instructional coaching as a tool for developing their district’s capacity to
implement 21st Century Skills, and 75% of the responding superintendents indicated that they
used, or will use, curriculum development as a tool for building their district’s capacity (Sweet,
2014). The literature demonstrated that all three of the selected strategies (professional
development, curriculum writing, and instructional coaching) have been highlighted as tools for
developing capacity for districts to implement a reform (Kay & Greenhill, 2013; Marzano &
Waters, 2009).
Question one also presented the option for respondents to provide other strategies by
allowing responses in an open-ended format. Four respondents (18%) selected this option,
listing the following strategies to build their district’s capacity: (a) community support,
(b) business leader support, (c) learning walks, (d) board policies, (e) hiring practices, (f) the
implementation of Science Technology Engineering Art and Mathematics Programs (STEAM),

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 79

and (g) sample lessons for teachers. The four superintendents who imputed other strategies for
supporting the implementation do not have statistical significance to support their use as
strategies for other districts. However, these responses do require further probing to clarify their
potential in supporting future implementation of 21st Century Skills’ programs.
Findings: Interviews. The coding and analysis of interview transcripts led to the
development of several categories for the findings of this research question. The categories that
developed as a result of the coding and analysis of the interview transcripts include: (a) strategies
for incorporating 21st Century Skills’ programs in the districts’ curriculum, (b) professional
development strategies, (c) technology as a tool for implementation, and (d) allocation of
resources. The following section will describe the key findings developed from the categories.
Methods for implementation. The researcher’s review of the literature highlighted that
there is no universal methodology or format used by districts throughout the country to develop
21st Century Skills. The analysis of the interview transcripts revealed three methods that are
being used to implement 21st Century Skills including: (a) project-based learning, (b) curriculum
writing, and (c) linked learning academies.
The interviews revealed that there are three methods that districts are using to implement
21st Century Skills’ programs into their districts. The first method identified in the interviews to
implement 21st Century Skills’ programs is curriculum writing or creating units of study.
Curriculum writing, or units of study, entails districts creating subject-specific content around the
districts definition of 21st Century Skills, e.g., communication, collaboration, critical thinking,
and problem solving (Sologuk, Stammen, & Vetter, 2001). The study revealed that 40% (4 out
of 10) of the superintendents’ districts are using curriculum writing, or units of study, as a
primary strategy for the implementation of 21st Century Skills’ programs. Under the format of

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 80

curriculum writing, the method teachers use for instruction does not necessarily change while the
goals and objectives of the lesson change to include 21st Century Skills. There are limited, to
no, empirical-based studies that show the impact of this approach; however, the writings of Kay
and Greenhill (2013) contended that curriculum writing around the four C’s is an essential step
to the implementation of 21st Century Skills. The logic of their argument would appear to be
that a district-level rewriting of the curriculum to include 21st Century Skills is essentially
creating a coherent implementation versus a piecemeal approach made by individual teachers.
The second method which 30% (3 out of 10) of the districts studied are using in their
high schools is a program known as Link Learning. According to Lafors and McGlawn (2013),
Linked Learning is a high school based initiative designed to create multiple pathways to college
and careers for low income Latino and African American students through the creation of career-
and subject-themed academies or small learning communities. In Superintendent C’s district,
there are several high schools that use such career-based academies including one for students
wanting to study engineering. The following quote from Superintendent C illustrates how his
district is using Link Learning to implement 21st Century Skills:
The freshman engineering class is working with two Caltrans bridge engineers. And their
job is go up and down the state, test bridges, safety component design, you know
redesign them for a retrofitting. And so this is a bridge that would've been designed. If
the kids had to design it, it had to meet certain engineering criteria. And to have that
relationship with the Caltrans engineer and the students, that level of relevancy was
essential. And teams of students work together again, you know, in bringing you know
that collaboration. That works together, developing the communication skills to design

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 81

the bridge and then communicate with adults, professionals, and industry professionals.
And they create the critical thinking because they had to design it. (Superintendent C)
A document provided by Superintendent C demonstrates that there are some positive
indicators for increases in student’s academic achievement and their educational attainment, as
results of attending Linked Learning academies. According to a presentation that Superintendent
C made to his board of education, the Linked Learning academies have an average API of 895
compared to the state average of 762. Furthermore, students in the district who attend Linked
Learning academies have on average 13% higher passage rate on the California High School Exit
Examination (CAHSEE) taken by students in the 10th grade than students who attend traditional
high schools. The research of LaFors and McGlawn (2013) in a study that examined the
academic achievement of four schools implementing the program found that Link Learning
increased the graduation rates of students, but provided mixed results regarding other indicators
of achievement. The findings from the current study indicate that Linked Learning academies
within Superintendent C’s district had a significantly higher API than traditional high schools in
the state of California contradicting the findings of Lafors and McGlawn (2013) regarding
similar programs.
The State of California has supported the Linked Learning initiative with Assembly Bill
790 (AB 790). AB 790 is a bill designed to improve the graduation rates of students in the state
through career academies that support the development of college- and career-ready skills.
AB 790 authorized superintendents’ use of funds from the federal and state budget to support the
development of Linked Learning academies (State of California, 2012). The results of the study
conducted by Lafors and McGlawn (2013) showed that the Linked Learning Initiative does
expand access for low-income students of color. The Linked Learning academies appear to

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 82

represent a viable method for implementing 21st Century Skills. The limitation of Linked
Learning academies appears to be an inability to implement at the middle and elementary levels.
The third method identified through data analysis is Project Based Learning (PBL); it is
the most common method of implementation found in the current study for developing 21st
Century Skills (see Appendix F). Project Based Learning is a method of instruction in which
students receive mini-lessons in a specific content, and then create projects applying the
knowledge concepts generated in the mini-lesson (Kay & Greenhill, 2013). Six out of 10
interviewees (60%) identified this as the method that their district is using to implement 21st
Century Skills. The superintendents who identified PBL, as their districts main tool for
implementing, described the integration of PBL as a process that requires an incremental
approach. According to Superintendent D, their district’s approach to using PBL is:
They're not going whole hog or redoing everything for Project Based Learning, but there
will be an expectation that each teacher produce, or each team produce, two integrated
projects a year when they use PBL averages, one each semester. And gradually that will
increase, but we’re going to ease into that approach because it isn't the be-all, end-all.
But if it's done well, the PBL can be an effective strategy for incorporating the 21st
Century Skills. (Superintendent D)
According to Kay and Greenhill (2013), PBL is a valuable tool for implementing 21st
Century Skills because: (a) it has an inquiry-based method of learning that leads to authentic
application of content specific skills, (b) allows for student voice and choice, (c) creates open-
ended questions, (d) creates essential knowledge and skills. The positive attributes espoused by
Kay and Greenhill for PBL present a strategy for narrowing the global and national achievement
gaps. These gains would likely be achieved on the PISA examinations that require students to

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 83

apply their content knowledge to authentic situations in a manner similar to PBL. A large-scale
empirical study would be required to confirm that PBL causes improved academic achievement;
its potential does signify the need for greater consideration as a reform strategy.
Overall, the findings from the interviews indicated that there are multiple methods for
implementation with Project Based Learning being the most commonly used strategy among the
superintendents interviewed. All of the methods (Curriculum Writing, Linked Learning, and
PBL) present viable options for superintendents of other districts interested in implementing 21st
Century Skills.
Professional development. The method and mode for professional development is an
essential component to the success of any reform. Marzano and Waters (2009) identified a
positive correlation between professional development and an increase in student achievement.
Therefore, one of the goals of the interview portion of this study was to gather in-depth
information regarding if and how superintendents do or intended to use professional
development in building their districts’ capacity to implement 21st Century Skills. The two most
prevalent answers describing how superintendents build capacity within their districts are on-
going coaching and professional learning communities. The following section will describe how
superintendents use these as tools to support the implementation of 21st Century Skills in their
districts.
Ninety percent of (9 of 10) respondents to the survey and subsequent interviews
identified on-going coaching as a strategy for supporting the implementation of 21st Century
Skills. More specifically, districts are using a “train-the-trainer” model to facilitate the
implementation. A train-the-trainer model to professional development can be defined as a small
portion of the district receives professional development regarding a particular methodology or

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 84

strategy (Birman, Desimone, Porter & Garet, 2000). The small group that received the initial
training then trains the rest of the district. Superintendent G described his district’s strategy for
the use of ongoing coaching with the following quotation:
Our Ed Services department, our assistant superintendent who has really spent a lot of
time and effort into doing that, and so one of the things that we've done, um, and using
Ed Service's resources, that includes Director of Curriculum, and Assistant Super of Ed
Services, the Gate Coordinator, is they've created a professional development plan, um,
and structured, actually, ah, the actual professional development for a Train the Trainers
model, and so we call them "iCoaches." So this year, we rolled out the iCoaches, um, or
rolled out the plan to, to bring in the iCoaches. So we've already had three professional
development days with our iCoaches specifically in their 21st Century Skills, and then
those teachers. (Superintendent G)
The strategy of training the trainers appears to be an efficient strategy for districts to use
in supporting the implementation of 21st Century Skills. As cited by professional development
experts, this strategy is an effective tool for facilitating changes in practice (Kay & Greenhill,
2013; Marzano & Waters, 2009). The iCoaches can provide the needed on-going professional
development required to ensure that the implementation of 21st Century Skills is maintained.
Another strategy that was identified by 70% (7 out of 10) of superintendents was the use
of professional learning communities (PLC) as a tool for building capacity. PLCs are tools that
are designed to ensure that the assessment, instruction, and the curriculum used match the needs
of students to ensure their academic achievement (Dufour, Dufour, Eaker, & Many, 2010). In
the instance of the current study, professional learning communities were designed to help
teachers, administrators, and districts develop 21st Century Skills within their districts. The

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 85

respondents highlighted several different types of PLCs used to support implementation
including: (a) within schools, (b) within districts, (c) principals within a district. Superintendent
A described the function and value of PLCs to districts implementing 21st Century Skills with
this statement:
So we have national district conversation but we’ll have northern California, southern
California, district conversations which is very helpful when one district like Grape
Unified is ahead of us by about a year. And so we’ve met with them and they’re coming
here actually for the next meeting in September. And so what, who’s doing what, and
how can we help each other not. So a different than between schools or even between
classrooms, you know, same idea. What can we . . . How are we gonna . . . How are you
making this work? We are very, very committed to this. Let’s say we had a piece of it, I
will tell you real fast is that last year, ‘cause you know, once a month we have a
leadership meetings and they are about, they aren’t about, uh, uh, the nuts and bolts of
things. (Superintendent A)
As Fullan (2011) indicated in “Six Secrets of Change,” organizations attempting to
change a facet of the organization most commonly learn from peers that are at a similar stage of
the implementation process, versus organizations that have already accomplished the desired
changes. As demonstrated by the research of Dufour et al. (2010) and Marzano and Waters
(2009), PLCs represent a powerful tool for building capacity. PLCs promote continuous
improvement by enhancing the pedagogical skills of participants and making the best practices
for implementing 21st Century Skills explicit to participants.
Technology. The ability to use technology in modern society has become paramount to
success of all citizens (Fadel & Trilling, 2009). The use of technology is often linked to the

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 86

development of 21st Century Skills. One of the goals of the interviews was to identify how
technology is being used to support the implementation of 21st Century Skills in the participants’
districts.
The following section highlights various ways that technology is being used, by the
superintendents’ districts, to implement 21st Century Skills. Superintendent I’s district is using a
“bring-your-own-device program” that allows students to supply their own technology. The
“bring-your-own-device program” is a strategy designed to bring different forms of technology
to support the districts use of Project Based Learning. Superintendent G and J’s districts are
using different devices for different grade levels, e.g., mini IPADs for grade levels K-2, IPads for
grades 3-8, and laptops for high school students in districts F, G, I and J. For students and
certain schools, who cannot afford to purchase the needed technology, two districts are
subsidizing the cost through local bond measures or private organizations.
The interviews of the superintendents revealed different purposes/functions for
technology: (a) provide individualized learning environments with access to experts in the field,
used as primary sources of information, and the ability to collaborate with peers across the
world, (b) document analysis of Superintendent G’s district indicated that the district uses
computer applications from Soliloquy learning to provide enrichment and intervention across all
curricular areas, (c) in Superintendent I’s district, technology is used to provide students with
rubrics for the projects they are completing, (d) Superintendent I’s district uses an on-line system
to allow parents and students to track grades throughout their high school career, and the on-line
system allows teachers to provide feedback that students can use to view their grades, and (e) the
following quote from Superintendent J highlights how the districts envision the use of
technology as supporting the goals of 21st Century Skills Learning:

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 87

The second thing is it gives you access to communicate with anybody around the world.
So, one of our goals is that we're using that for kids to access experts, access peers from
around the world about, you know, engaging in certain kinds of problem discussions and
what not, and then the third thing it does, and the iPad may be not quite as good at this as
the computer, is producing something that demonstrates what you've learned. So that's
what we hope the kids are using it for, is to get to the web and, you know, get powerful
access to information, communicate, and then demonstrate what they know.
(Superintendent J)
The above quote echoes the sentiment expressed by three additional participants
(Superintendents F, G, and I) regarding the need for students to access experts to develop
curricular and 21st Century Skills, as it is not always practical or cost-efficient to bring experts
or peers physically into the classrooms. The use of technology makes experts or professionals in
the field accessible to students who are experiencing link-learning or project-based learning
models, for developing 21st Century Skills. Technology in conjunction with PBL or Link
Learning can make education more meaningful and relevant to students than traditional forms of
instruction. According to Delawsky and Johnson (2013), PBL has a positive impact on the
emotional engagement of students while being inconclusive regarding cognitive engagement.
The results showed promise for PBL because emotional engagement is an important component
to student achievement. Nevertheless, further research is required to conclusively determine the
impact of PBL on student achievement. Another theme that appeared relevant in the strategies
chosen and employed by superintendents, in their implementation of 21st Century Skills, is how
resources are allocated in their districts.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 88

Allocation of resources. The use of people and material resources has direct relevance
to the success of any educational reform. Marzano and Waters (2009) identified that the
allocation of resources is one of five essential components, used by high performing districts, to
support their instructional goals and achievement. The findings of Marzano and Waters help
focus on how superintendents can best allocate resources, by identifying the contextual factors
that can facilitate the success of 21st Century Skills’ district programs. Therefore, this interview
question attempted to identify how superintendents allocate financial, material, and human
resources to support the implementation of 21st Century Skills.
The interviews revealed several strategies utilized by superintendents to allocate
resources in support of the implementation of 21st Century Skills. Four (40%) of the
superintendents identified building up their districts’ technological infrastructure as a starting
point for the implementation process. The participants’ districts have used the following tools to
secure the resources for building the technological resources: (a) donations from private
foundations, (b) state funds (e.g., California Assembly Bill 790), and (c) local bond measures.
The four districts are using these sources of funding to: (a) build or expand their schools
wireless capacity, (b) purchase digital devices for students, (c) provide professional development
for administrators, district personnel, and teachers, and (d) make changes to the physical plant of
schools. The analysis of the interview transcripts highlighted two key findings regarding the
allocation of resources. First, Superintendent H described difference in the state’s funding
strategy compared to previous initiatives.
This year was an, was absolutely critical. Uh, this is the only time in my 20 plus years in
education that I can remember the state looking at an initiative and saying, "We're, we're
not, you know, we're not just gonna throw you money after the fact or we’re not gonna

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 89

throw you after your scores or you know, if you’re seen as deficient or something and
give you money to do it. We're actually gonna be proactive and we're gonna help you
and give you resources for proper implementation of something. And so that having
those common core, core funds for the materials for the technology, for the training
related to that is absolutely key. So those resources are very, very helpful.
(Superintendent H)
The idea of properly funding the initiative without stipulations seems to have surface
appeal to interested parties. However, the question becomes: What will be the model for
properly funding 21st Century Skills’ programs? Five respondents identified the use and
distribution of technology as integral to the implementation of 21st Century Skills’ programs.
These five respondents (Superintendents F-J) revealed that their districts’ use of technology
including tablets and laptops.
While initial funding may be sufficient to fit the current technology and professional
development needs for the district, it does not guarantee its long-term funding. Document
analysis of Superintendent C and Superintendent I’s districts’ indicated that the funding they are
receiving from private organizations is gradually being phased out. The research of Marzano
and Waters (2009) highlighted that high performing districts need to maintain efficient allocation
of resources, to ensure continued student achievement. It is necessary for a collection of districts
to demonstrate the positive effects on student achievement and educational attainment to solicit
the likely required funding for public institutions and private benefactors.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 90

Summary and Discussion: Research Strategies for Implementing 21st Century
Skills’ Programs
As observed through interviews, the most prevalent method for implementing 21st
Century Skills in the current study is Project Based Learning. The most commonly reported
methods for developing capacity to implement 21st Century Skills are professional learning
communities and a train-the-trainer model. The use of technology to support the implementation
has been through expanding access and use of technology, by making digital devices available to
students. The districts, of the superintendents interviewed, used a combination of public and
private funding, which includes both donations from organizations supporting the cause and
funds provided to the district for the Common Core Standards Initiative. This research seems to
indicate that there are several effective models for implementing 21st Century Skills’ programs.
A key to sustaining efforts is to maintain the flow of resources that includes on-going
professional development. The research of Marzano and Waters (2009) validated the use of on-
going professional development as a tool for sustaining the goals of reforms like 21st Century
Skills.
Research Question 2
What indicators do superintendents use to evaluate the implementation of 21st Century
Skills?
One essential practice that effective superintendents followed to ensure the success of
their districts was to monitor student achievement and instructional goals (Marzano & Waters,
2009). The conclusion of Marzano and Waters (2009) highlighted the importance of identifying
the indictors that superintendents use to evaluate the implementation of 21st Century Skills.
Thus, the second research question sought to explore the achievement and instructional

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 91

indicators used by California superintendents to evaluate the implementation of 21st Century
Skills.
Data analysis of the transcripts revealed several categories of indicators used by the
participating superintendents, they include: (a) assessed instructional goals, (b) monitored
achievement goals/educational attainment, and (c) measured stakeholder satisfaction and
concerns.
The methods used to capture the data needs for these measures were: (a) an electronic
survey link from Qualtrics was emailed to potential research participants, (b) qualitative
interviews, and (c) analysis of internal and external documents from the studied districts.
Samples of each protocol can be referenced in appendices C-E.
Findings: Quantitative survey. The quantitative survey measured how the responding
22 superintendents were currently or intend to measure students’ acquisition of 21st Century
Skills in their districts. Among the respondents, 85% of superintendents indicated that they used
or intend to use performance assessments to measure students’ acquisition of 21st Century Skills.
Also, 65% of the respondents selected content- and performance-based rubrics as a tool for
measuring students’ acquisition of 21st Century Skills. This result further indicated that 30% of
the respondents intend to use self-rating scales. As discussed by Kyllonen (2012), self-rating
scales are assessment implemented in conjunction with the PISA to measure students’
perceptions of their acquisition of 21st Century Skills. Additionally, 20% of the respondents
indicated they would use some other assessment to measure students’ acquisition of 21st Century
Skills. The open-ended responses included: (a) the College and Work Readiness Assessment
(CWRA), (b) learning walks, and (c) Northwest Evaluation Assessment’s (NWEA) Computer

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 92

Adaptive Assessments. Table 9 displays the results of the item identifying how superintendents
are measuring students’ acquisition of 21st Century Skills.

Table 9

Tools to Assess Students Acquisition of 21st Century Skills

Assessment Tool
Number of Superintendents
Selecting Assessment
Percentage of Superintendents
Selecting Assessment
Self-Rating Scale 6 30
Performance Assessment 17

85
Content and Performance
Based Rubrics

13 65
Other 4 20
Not Sure 1 5

The responses to this question indicated that superintendents are, or will be, using
multiple measures to determine how well students are acquiring 21st Century Skills in their
district. In general, there are a few uncertainties regarding how the acquisition of 21st Century
Skills is being measured by districts in the state of California. The research of Kay and Greenhill
(2013) validated the use of multiple measures as a tool for assessing the development of 21st
Century Skills. The qualitative interviews served as an opportunity to gather more information
regarding how these multiple measures serve as indicators as to the development of 21st Century
Skills within the districts.
Findings: Qualitative interview. The coding and analysis of interview transcripts
resulted in several themes or categories highlighting what indicators superintendents are using to

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 93

measure their students’ acquisition of 21st Century Skills. The following categories were
developed as the result of themes identified in data analysis of interview transcripts:
(a) formative indicators, (b) summative indicators, (c) feedback loop, and (d) readiness
indicators.
Formative indicators. The strategy of measuring on-going progress of the initiative is
fundamental to vitality of a reform. As highlighted by numerous researchers including Marzano
and Waters (2009), the effective use of formative assessment is associated with improved
academic achievement. Therefore, the interview questions for the current study were developed
to identify the different formative measures used by superintendents regarding the development
of 21st Century Skills.
The interviews revealed that superintendents were, or will be, using the following
formative indicators to measure their district’s implementation: (a) learning walks with
observational checklists, (b) content specific rubrics, (c) teacher reflections, and (d) one district
will use an integrated on-line measurement system that presents parents and students with
rubrics, current and past, of student performance on projects. The most common formative
measure identified in the interviews was rubrics. The superintendents identified two purposes
for the use of rubrics. First, to measure students’ acquisition of 21st Century Skills; and the
second purpose for using rubrics was to identify how well the goals developed for the
implementation were being achieved in schools throughout the district of study. The final
purpose identified by data analysis for the use of rubrics was to assess districts’ readiness to
implement 21st Century Skills.
All 10 of the superintendents indicated that their districts were or will use content specific
rubrics to measure students’ acquisition of 21st Century Skills. Among the superintendents who

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 94

indicated that they were or will use rubrics, 40% indicated that they will use the rubrics created
in district PLCs as offered by EdLeader 21. EdLeader 21 is a private organization that provides
professional development, curricular materials including rubrics, and facilitates PLCs
nationwide. Superintendent A briefly highlighted her rationale for using EdLeader’s rubrics with
the following quote, “So you know them too and they are very progressive . . . Um, they’re,
they’ve already created the Rubrics for how do you measure 21st Century Skills.”
(Superintendent A, Personnel Communication, July 26, 2013).
The selection of this assessment tool appears to be a reasonable decision based on the fact
the rubrics have already been implemented by districts in states outside of California as a part of
their 21st Century Skills’ initiatives. Measuring 21st Century Skills is different than measuring
the acquisition of basic content skills. With basic content skills there are often finite ways of
analyzing student work, because either its accurate or inaccurate. The development of 21st
Century Skills requires rubrics because they contextualize the development of skills in ways that
other methods cannot measure. Therefore, the use of rubrics appears to be an appropriate
indicator of student performance, and the degree of implementation accomplished by the district,
schools, and individual teachers. Appendix G presents an observation checklist based on rubrics
that highlight how this measurement tool is and can be implemented. A quote from
Superintendent D highlights how the rubrics can be used to measure a district’s readiness to
implement 21st Century Skills.
There were some old Rubrics that we tried to use last year and we didn’t, uh, it, it didn’t
work. We tried to use them as a self-assessment tool and everything and all the teachers
evaluated themselves in the high end of the rubric. It was totally useless because what it
told me is that they didn’t know what they were, they're talking about. Because they,

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 95

when I would go around to classrooms specifically looking for people who are, uh, you
know, who understand what the 21st Century Skills are and actually doing something
different in their classroom, uh, you know to, to incorporate those skills into their
teaching and student learning, I see very little of it. I do see some, but it's from teachers
who are getting it, but it's very sporadic. (Superintendent D)
Measuring the readiness of the entire district or individual schools is a necessary step to
support the successful implementation of 21st Century Skills (Kay & Greenhill, 2013). One
finding from the data analysis does not constitute a theme for the overall study, but it merits
discussion, is Superintendent I’s district’s use of Reverberation, an on-line management system
for tracking the development of 21st Century Skills. Overall, Reverberation is a learning
management system that supports the use of Project Based Learning within Superintendent I’s
district (On-Line Platform), provides PBL on a daily basis. Teachers, students, and parents use
Reverberation to access course resources, project plans, assignments, a multi-dimensional grade
book, and its online resources for teachers’ course agendas, project management, a peer feedback
tool, and a student discipline tool. Other features of Reverberation include school and
instructional resources, for example: sample rubrics, library of sample PBL projects, student
recruitment samples, counseling and college planning materials, technology infrastructure, and
on-line software tutorials. The purpose for mentioning this finding is that Reverberation has the
potential to be replicated by other districts in support of their implementation of 21st Century
Skills. The evidence of the system was presented in the district’s website and by Superintendent
I during the course of the interview. In addition to the formative indicators, there are many
summative measures used by superintendents to indicate the success of their implementation of
21st Century Skills.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 96

Summative indicators. The most commonly identified goal for implementing 21st
Century Skills’ Programs is to ensure that students are college and career ready. The use of
summative indicators enables superintendents to measure the implementation effectiveness over
an extended period (Marzano & Waters, 2009).
Analysis of the interview transcripts revealed several common summative measures of
academic achievement used to evaluate the effectiveness of the program implementation.
Among the superintendents, there are three whose districts have begun implementation in at least
a portion of their districts. The three districts are using several achievement indicators to
measure the effectiveness of their 21st Century Skills’ initiatives. These indicators include:
(a) report card system with criteria developed by district personnel, (b) examining the number of
students enrolled in Advanced Placement courses, (c) student attendances rates, (d) student
discipline statistics, (e) completion rates of the A-G requirements for high school students in the
state of California, (f) student scores on the College and Work Readiness Assessment,
(g) passage rates for the California High School Exit Exam (CAHSEE), and (h) evaluation of
students’ progress towards the four C’s in 5th, 8th, and 12th grade. These various summative
measures serve as an indicator for the overall implantation progress of 21st Century Skills.
Superintendent B described the impact of Linked Learning (a 21st Century Skills’ initiative) on
the achievement of his district with the following quote:
The indicators of success, I think I've already talked about our students in terms of higher
GPAs. Um . . . higher levels of post-secondary experience um . . . and I would say less
discipline issues, higher passage rates on um the high school exit exam. These are all
indicators to me that the effort that we're involved in with Link Learning is far superior to
any reform effort that I've been involved. (Superintendent B)

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 97

The combination of these indicators would seem to demonstrate the promise of 21st
Century Skills’ programs (e.g., Link Learning) have on student engagement and student
achievement. Superintendent B presented a document to the researcher demonstrating that
participating Linked Learning high schools have an average API score of 895 compared to the
state average of 752. Superintendent B and I also reported a reduction in suspensions and
increase in the passage of California’s A-G requirements.
The interview with Superintendent I presented similar results of summative measures.
According to Superintendent I (personnel communication, September 4, 2013), his district has seen
an increase of 11% in students’ completion of A-G requirements and an increase in the
graduation rate from 80 to 87%.
An analysis of the documents presented by Superintendent B validated his assertions in
the quote previously shown. These findings merit further research to validate the results before a
broad conclusion can be made as there are a limited number of surveyed districts that have
formally implemented 21st Century programs. The documents and interviews for
Superintendent B and I demonstrated the potential of 21st Century Skills to improve the
academic achievement of students, but require larger studies to confirm program effectiveness.
Feedback loop. One common practice among effective superintendents is to monitor
student achievement (Marzano & Waters, 2009). One method for monitoring the progress of
implementation is to get feedback from key stakeholders. Among the superintendents
interviewed, several commonalities emerged regarding the methods used to collect feedback
from key stakeholders in their district. The methods used to collect feedback from stakeholders
(e.g., parents, teachers, principals, and students) included: (a) interviews, including one-on-one
meetings, (b) focus groups, and (c) surveys. Superintendent H described the importance of

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 98

receiving feedback from students following their graduation as being paramount to sustaining
and improving the 21st Century Skills’ program within his district. His quote describes the point
of view of servicing students’ needs with the implementation of 21st Century Skills’ program.
My ultimate goal is, honestly, student and parent surveys, client surveys, right? Just go
to the people who we’re serving and that’s really kids and parents, uh, mostly kids to me,
quite honest with you. It's a little bit more challenging obviously at the elementary level.
Um, but parents can help you know there too. Um, but certainly at the secondary level
when we went to high school, when we're talking to high school kids about you know,
what is going well, what's not going well, we feel, and then tracking our graduates and
serving our graduates even after they leave us.
And as they get out into the world one, two, three, four, or five years removed
from our school system, asking them how they felt about their preparation. What could
have been better, what could have been done differently, those kinds of things?
(Superintendent H)
This quote highlights the type of qualitative data necessary to implement, sustain, and
ensure the success of a reform like 21st Century Skills. The perspective of students, regarding
implementing, is necessary but seemingly overlooked by some; for most effective
implementation, superintendents need to evaluate the different components of their district’s 21st
Century Skills’ program. Superintendents must gather information from those closest to the
implementation to ensure that the goals of the initiative are being met. One problem highlighted
by Hentschke and Wohlstetter (2004) was the information asymmetry problem in which those
farthest from the reform lack information regarding its progress. Therefore, as highlighted by

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 99

Marzano and Waters (2009), effective superintendents gather feedback from key stakeholders to
ensure that the predetermined achievement and instructional goals are being met.
Readiness indicators. The participants in the current study have indicated that their
districts are at various levels of planning or implementing 21st Century Skills. Therefore, a
commonality among superintendents in the planning or preparation stage has been to use
indicators that demonstrate their district’s readiness to implement 21st Century Skills. The
indictors during the course of the interviews included: (a) self-evaluation rubrics for principals
and teachers, (b) focus groups, and (c) learning walks. In the following quote, Superintendent D
highlights how a survey presented to the faculty of the schools in his district demonstrated to him
a lack of understanding regarding the definition of 21st Century Skills.
There were some old Rubrics that we got from Ken’s previous organization we tried to
use last year and we didn’t, it, it didn’t work. We tried to use them as a self-assessment
tool and everything and all the teachers evaluated themselves in the high end of the
Rubric. It was totally useless because what it told me is that they didn’t know what they
were they're talking about. Because they, when I would go around to classrooms
specifically looking for people who are, uh, you know, who understand what the 21st
Century Skills are and actually doing something different in their classroom, you know
to, to incorporate those skills into their teaching and student learning I see very little of it.
I do see some, but it's from teachers who are getting it, but it's very sporadic.
(Superintendent D)
This quote from Superintendent D supported the notion that districts must measure
various elements of their personnel’s readiness prior to proceeding with the implementation. A

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 100

lack of awareness or capacity can negate any positive effects intended with any reform. The
need to educate teachers is most salient because they are the implementers of the initiative.
Discussion. The implementation of major reforms requires multiple measures of
progress throughout the implementation process. The interviews revealed a multitude of
measures to assess academic achievement, the satisfaction of stakeholder and stakeholders’
knowledge, and capacity to implement. As the research of Marzano and Waters (2009)
demonstrated, high performing districts have superintendents who monitor the progress that
schools make towards goals of particular reforms. Therefore, the identification of multiple
indictors supports the growth spotted by districts for Superintendent B and I.
One caveat from the current study that requires discussion is that all the districts are at
various stages of the implementation process. There are three districts that are in the goal-setting
phase of the process for implementing 21st Century Skills. There are three districts that are in
the process of developing plans around their mission and vision statements to be readied for
implementation. Four of the interviewees’ districts are implementing district wide, or in a
significant number of schools. The varying degrees of implementation do not allow for
definitive conclusions regarding what indicators are being successfully utilized to measure the
progress of 21st Century Skills. Nevertheless, the findings can provide a road map for other
districts that decide to implement 21st Century Skills. The next section presents the findings
from research question three.
Research Question 3
How do superintendents include key stakeholders in the implementation of 21st Century
Skills?

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 101

To promote continuous improvement, superintendents must include all stakeholders in
setting goals for an initiative (Marzano & Waters, 2009). This finding was the impetus for
developing research question three. This research question sought to identify the various
methods for including stakeholders in the implementation of 21st Century Skills. Another
objective for this research question was to identify specific strategies used by the superintendents
to solicit support of key stakeholders in 21st Century Skills’ initiatives.
The methods used to capture the data needs for this question were: (a) an electronic
survey link from Qualtrics was emailed to potential research responders, (b) qualitative
interviews of 10 superintendents, and (c) document analysis of internal and external documents
from the districts studied. Samples of protocols can be referenced in Appendices C-E.
Findings: Quantitative survey. The electronic survey had several options for
respondents to illustrate their role in their districts implementation of 21st Century Skills. The
primary objective was to identify superintendents that demonstrated a level of involvement with
stakeholders and the project implementation in order to meet the data needs of the research
study. To identify potential direct-interview participants, one of the survey questions asked what
role did or will the superintendent take to support the implementation of 21st Century Skills.
The results of this question, found 95% of respondents overseeing or intending to oversee the
implementation of the project. Among the respondents, 10% identified themselves as taking a
role in the process as a researcher, and 25% of the respondents identified professional
development as one of the roles that they had, or will have, during the implementation process.
Half of the respondents to the survey selected aligning the stakeholders as one of their roles they
took, or will take, during the implementation process. Moreover, 25% of the respondents to the
quantitative survey identified that they have, or will have, a role in leading focus group

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 102

discussions in the implementation process. Additionally, 20% of the respondents indicated
taking roles in the open-ended “other” response. The other roles identified by superintendents in
response to this item included: (a) coordinating leadership training for principals, teachers, and
other district staff, (b) communication of the vision, (c) aligning resources, (d) goal setting with
principals and teachers, (e) supporting with time, money, and people, (f) modeling 21st Century
Skills’ behaviors, (g) communicating to parents and community, and (h) aligning policies and
contracts with teachers. Table 10 displays the roles that superintendents indicated they are, or
will take, in the implementation process (20 responded).

Table 10
Roles Superintendents are taking to Support the Implementation of 21st Century Skills

Role
Number of Superintendents
Taking the Role
Percentage of Superintendents
Taking the Role
Overseer 19 95
Researcher 2 10
Professional Development 5 25
Aligning Stakeholders 10 50
Leading Focus Group
Conversations

5 25
Other 4 20

Two primary functions that respondents have, or will take, to support the implementation
are as the project overseer and the aligner of stakeholders. The superintendents who responded
that they are aligning stakeholders were valued because they can be regarded as information rich

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 103

cases that should demonstrate how superintendents solicit support for the implementation of 21st
Century Skills.
Another goal of the survey was identification of the stakeholders that the superintendents
interacted, or intend to interact, with during the implementation process. The results of the
survey question indicate that all superintendents interact with their board of education,
principals, and cabinet members. Among the respondents, 95% interacted or will interact with
the assistant superintendents of their districts. All respondents indicated that they have, or will,
interact with: (a) board of education, (b) cabinet members, and (c) principals. Table 11
represents the different stakeholders that superintendents will interact with during the
implantation process.
As indicated with Table 11, all superintendents will interact with cabinet members, board
of education, and cabinet members. Virtually, all of the superintendents will interact with their
assistant superintendents, and most superintendents will also interact with: (a) curriculum
specialists, (b) teachers, and (c) vendors. The results of this response must be analyzed in
conjunction with the response from the survey question that asks superintendents what their role
is, or will be, in the implementation (Table 10) to determine if they possessed valuable
information to obtain regarding the data needs of the current study. All respondents to the
survey indicated that one of the roles they will take is an overseer. As previously shown in Table
10, half of the superintendents indicated that one of the roles was to align stakeholders. The role
of overseer would seem to require significant interaction with key stakeholders, such a role
requires political aligning to ensure the effectiveness of the initiative. The qualitative interviews
attempted to gather specific strategies that the superintendents used in the various roles they took
to support the implementation of 21st Century Skills.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 104

Findings: Interviews. The qualitative interviews addressed the role and function of the
superintendent in greater detail to identify common themes used or to be used in future
implementations. Data analysis of the interview transcripts revealed several categories/themes
for the strategies that superintendents used, or will use, to include stakeholders in the
implementation. The strategies that emerged from data analysis included: (a) stakeholders by
co-creating various elements of their districts 21st Century Skills’ programs, (b) aligning key
stakeholders, (c) creating a risk-free environment among stakeholders to support the initiative,
(d) educating stakeholders about the initiative, and (e) celebrating successful achievement of
milestones in the implementation process.

Table 11

Key Stakeholders who Superintendents will Interact with during Implementation

Stakeholder
Number of
Superintendents
Percentage of
Superintendents
Principals 20 100
Board of Education 20 100
Cabinet Members 20 100
Assistant Superintendents 19 95
Curriculum Specialists 14 70
Assistant Principals 13 65
Teachers 13 65
Vendors 9 45
Other 5 25

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 105

Co-creation. One theme that the data analysis revealed regarding how superintendents
included key stakeholders in the implementation process is through co-creation. The process of
co-creation is one in which key stakeholders are included setting goals for the initiative, lead
professional development, and have a voice regarding the implementation (Marzano & Waters,
2009). Data analysis revealed that co-creation in the participants’ districts manifests itself in
several ways: (a) developing a mission, (b) creating vision around 21st Century Skills, and
(c) writing curriculum for the development of 21st Century Skills.
For three of the superintendents whose districts are in the initial stages of the
implementation process, co-creation is utilized through the development of mission and vision
statements for the districts 21st Century Skills’ initiatives. Superintendent A described the
process of creating the mission and vision statements by including the whole community
(personal communication, July 26, 2013). In Superintendent A’s district, the process began with
discussion and letters that articulate the goals which parents and stakeholders have for their
children. According to Superintendent A, the next step in her district’s process of developing
goals for the district was a meeting for cabinet members to identify common words and phrases
in the letters submitted by parents. The process used by Superintendent A translated into
meetings of key stakeholders who developed goals with statements that began: “students will.”
This description of the process used by Superintendent A does not present itself as a pattern used
by the other participants in the interview. However, Superintendent A’s description of the
process does provide a potential model for districts that are just beginning the process of
planning the implementation of 21st Century Skills.
Another method that the participating superintendents used to facilitate co-creation of the
different components of their districts’ 21st

Century Skills were: (a) write curriculum,

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 106

(b) develop assessment tools to measure students development of 21st Century Skills, (c) to lead
professional development. Several of the participants revealed that there was some consternation
from teachers regarding the purpose and goals for their districts 21st Century Skills’
implementation. In the instance of Superintendent D, he wanted to create an assessment tool for
teachers around the instruction of 21st Century Skills. The strategy that he employed to develop
an assessment tool was to use in-district teachers and curriculum specialists to develop it. The
following quote highlights his goals and the purpose for this strategy:
But, I wanted them to write the tool so it wasn’t a bargaining issue and, uh, and they
seem to, they're saying at this point that they trust my motives, but yet they're still not
coming to the table one-to-one now. The first time they showed up they brought their
bargaining team. I said no, no, no, no. I said we’re not bargaining. I said these are not
your curriculum experts. I want your curriculum experts. I said I can't tell you whom to
send to the table, but I'd like you to send to the table teachers who wrote the curriculum,
because there were a lot of them. (Superintendent D)
This quote highlights an important issue regarding the presentation of 21st Century Skills
into district: the need to include key stakeholders’ voice into all aspects of the implementation
process, especially staff teachers. The success of a new teacher evaluation surrounding the use
of 21st Century Skills requires the support and understanding of teachers. The resistance as
described by Superintendents B and D can be reduced or eliminated by allowing the voices of
key stakeholders (i.e. teachers) to participate in the planning stages of the process (Marzano &
Waters, 2009). Superintendent D’s awareness of their resistance led him to use the human
resources approach to ensure their participation and support (Bolman & Deal, 2008).

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 107

Superintendent D argued that if the teachers were included in the planning process they would
believe their goals and concerns would be best addressed.
Overall, the strategy of co-creating goals with key stakeholders appears to be an essential
strategy in obtaining support for implementing 21st Century Skills within a district. The
following quote from Superintendent B summarizes the importance of using co-creation as a tool
for soliciting support from key stakeholders (personal communication, July 29, 2013).
So, it, it's, you know, pretty sophisticated, where, um, you know, it's a lot of . . . how
could I say . . . communication between the district side and the school side, because if
you don't have co-creation, you know, between classroom and district mandates, you
know, things are implemented typically superficially. (Superintendent B)
This quote highlights the importance of co-creation in the development of educational
reforms by all key stakeholders, to ensure success. As supported by the research of Marzano and
Waters (2009), collaborative goal setting is one practice of effective implementation by
superintendents. Another theme that developed through analysis of the transcripts was the need
to align key stakeholders during the implementation process.
Aligning stakeholders. The need of continuity among key stakeholders regarding all
aspects of the 21st Century Skills’ programs was consistently identified in the analysis of the
interview transcripts. The research of Marzano and Waters (2009) concluded that effective
district leadership practices include aligning stakeholders. Based on analysis of the interview
transcripts there are several key elements to aligning stakeholders, which include educating
stakeholders about 21st Century Skills and presenting a clear message about the goals of the
program.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 108

One of the commonalities among the responses of the participating superintendents was
the need to educate stakeholders about the initiative. The consensus among the participants was
that key stakeholders must be: (a) aware of the definition of 21st Century Skills and (b) the role
21st Century Skills play in preparing students to be college and career ready.
The interviews revealed several common strategies used by superintendents to educate
stakeholders within districts about 21st Century Skills. These strategies included: (a) presenting
showcase lessons to stakeholders, (b) providing literature about 21st Century Skills to key
stakeholders, (c) presenting exemplars of other districts who have already implemented 21st
Century Skills, (d) one-on-one conversations with key stakeholders regarding the definition and
value of developing 21st Century Skills, and (e) workshops/professional development for key
stakeholders (e.g., parents, district personnel, and teachers). The one commonly identified
purpose for using the strategies by superintendents, for educating stakeholders, was to make
stakeholders aware of how schooling will change with the implementation of 21st Century Skills.
Superintendent B highlights the importance of beginning the change process by describing how
he has attempted to educate top leaders in his district.
I've enrolled some of my cabinet members as well as some of my lead principals to this
course called Innovator Accelerator through the University of Phoenix. And basically,
it's a course on how do management leaders promote change innovation in, an
organization so try . . . educate ourselves differently. And, and this course is an online
hybrid course that includes many implements . . . many instructional shifts that we expect
to see in the classroom. So, I'm having my top leaders experience those shifts so they can
understand how learning should be different. Um, and, and, and it's hard to manifest

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 109

change if you don't have a clarity to how does a change should . . . like. (Superintendent
B)
To align stakeholders, requires that they understand and value the proposed change, to
the way that schools in their district educate children. Marzano and Waters (2009) highlighted
educating stakeholders as a tool used by effective district leaders. As indicated by
Superintendent B (personal communication, July 29, 2013), for 21st Century Skills to become a
lasting change to the way children are educated, the stakeholders must understand and have the
capacity to implement the changes that have been mandated.
Data analysis revealed that another tool for aligning stakeholders is the setting of goals
using a common language and phrases regarding the initiative among all district stakeholders and
personnel. More than half of the superintendents interviewed believe that developing goals
among all stakeholders with common language is a way to facilitate alignment in the district.
The superintendents interviewed for the current study believed that the process begins
with the development of instructional and achievement goals by key stakeholders (e.g.,
principals, teachers, and assistant superintendents). Analysis of study transcripts revealed that
the district begins with overarching goals for 21st Century Skills that should be customized into
goals for individual school sites including principals and teachers. Superintendent I (personal
communication, September 4, 2013) made several comments that espouse the value of this
strategy. The first comment he made speaks to the idea of beginning the process of alignment by
creating common language about the goals of the 21st Century Skills’ initiative among key
stakeholders.
We simplify the message; we repeat the message, so that everybody’s clear on what the
goals are. We have three goals: college and career, which I’ve talked about; closing the

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 110

achievement gap (I think our board calls it equitable opportunities, but we just basically
say closing the achievement gap); and then the third one is to instill 21st Century Skills.
This instills 21st Century Skills. (Superintendent I)
This statement establishes that it is imperative that stakeholders understand the goals of
and for the development of 21st Century Skills. A mandate for developing 21st Century Skills
without clear goals can lead to the superficial implementation, as described in comments
previously made by Superintendent B. Aligning key stakeholders is a common practice of
effective district leaders, and developing continuity through common language about 21st
Century Skills would seem to facilitate those goals (Marzano & Waters, 2009).
Another technique of creating alignment is allowing schools to have some form of
defined autonomy, to achieve district goals (Marzano & Waters, 2009). The districts of the
superintendents studied are heterogeneous and, therefore, require a degree of autonomy in the
pursuit of the common goal of developing 21st Century Skills. Therefore, a mandate that forces
all schools to implement 21st Century Skills in an identical fashion can present a threat to the
overall success of the initiative. Superintendent G (personal communication, September 11,
2013) highlighted that during his implementation process, he realized from feedback provided by
his principals, that to meet the needs of the school community there was the need to allow
schools to customize the strategies used to pursue goals in developing 21st Century Skills.
What we try to create is just an understanding of what 21st Century Skills are. But we do
believe in this district that each site has a unique ecosystem that can . . . you need to
know the needs of your community and implement these skills to ensure that those . . .
they're connected to those needs. So, how a lesson may be delivered may change, may be

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 111

different from site to site but they need to focus on those, those specific skills.
(Superintendent G)
The conclusion drawn by Superintendent G (personal communication, September 11,
2013) about allowing schools and teachers to customize lessons and goals seems to be a
necessity for schools, especially in large districts that have very heterogeneous populations. This
conclusion is supported by the positive correlation found by Marzano and Waters (2009)
regarding schools being given defined autonomy. The last objective of this question was to
ascertain strategies used by superintendents to sustain the implementation of 21st Century Skills.
Sustaining 21st Century Skills. To implement 21st Century Skills requires maintaining
the commitment and support of key stakeholders. The interviews for the current study revealed
several strategies for maintaining the support of key stakeholders in the implementation of
process for 21st Century Skills. The strategies identified by superintendents for maintaining
support for the implementation of 21st Century Skills include: (a) having a clearly defined plan
with achievable goals, (b) creating a forgiving environment, and (c) creating teacher leaders.
According to three of the 10 superintendents interviewed, one of the key elements to
maintaining and building upon the implementation of 21st Century Skills is to establish
achievable milestones in the implementation process. In the work of Clark and Estes (2008),
they discussed the importance of creating clear and concrete goals as paramount to the success of
change within an organization. The notion of clear and concrete goals is highlighted in the
analysis by Superintendent J’s (personal communication, July 30, 2013) belief that developing
milestones that can be achieved creates momentum towards the larger objectives of the imitative.
In the following comments, he expresses how the achievement of underlying goals creates a
confidence in stakeholders that maintains the momentum of the goals for 21st Century Skills.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 112

But I think, unless you have sort of a very clear cut, we want you to do this, this is your
first step, and then they can have some success doing that, and then that success leads to
some confidence and that confidence leads to, you know, other instructional initiatives or
other instructional activities that the teacher might take on to support it, you know, just
because they're developing in their own practice, I think that's the way the change
happens successfully. You gotta pick something that they can do successfully that's
definable, measurable, and, and that can be achieved quickly, and then you can start
building successes upon successes. (Superintendent J)
The notion of having a clearly defined order for the implementation process, with
achievable goals, is a necessary step towards meeting the overall objectives. In the era of
bureaucratic accountability that has led to unachievable stretch goals this pragmatic approach
appears to have a better chance for success. A stretch goal is a very difficult objective to achieve
(Clark & Estes, 2008) As argued by Clark and Estes (2008), when stretch goals are forced upon
employees, and they fail to achieve them, it discourages further attempts to persist towards
meeting the goals of a mandate. Therefore, the use of achievable milestones towards the larger
objective of developing 21st Century Skills is a necessity, due to the significant changes in
practice that are required. Creating a forgiving environment is another strategy for maintaining
support for an initiative like 21st Century Skills. An environment in which the predominant
method for motivating people to achieve a goal is punitive consequences can have a deleterious
effect on the people and the goals of an organization. Therefore, an environment that permits
mistakes to support individual and collective growth is more likely to achieve its long-term
objectives.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 113

Creating lasting changes in behavior of administrators and teachers is a challenge to truly
accomplish the ultimate goals of any reform. Superintendents who participated in this study
seemed to grasp that reality and believed that it is necessary to create a forgiving environment for
administrators. Five of the 10 superintendents verbalized support for this notion. Superintendent
F (personal communication, August 7, 2013) in this statement highlights the necessity for
allowing principals and teachers to make mistakes in the implementation of 21st Century Skills.
Principals have to first and foremost provide a culture and a space for teachers to make
mistakes. No one's going to, um, believe this, who thinks this is going to go perfect,
period. I don't care if it's a district our size or a little, tiny district., adults who are
learning to implement something new, just like kids do, will make mistakes. If we do
make a mistake six times, um, then we'll learn from that. (Superintendent F)
Superintendent F’s (personal communication, August 7, 2013) comments highlight a
human resources approach to leadership that is necessary to supporting the initiatives that require
changes in pedagogy and educational philosophy. The national landscape for educational policy
over the past decade has been tied to punitive consequences associated with a failure to narrow
the achievement gap, as demonstrated by reforms such as No Child Left Behind (Hawley, 2008).
It remains to be seen how the policy landscape will change if 21st Century Skills become a state
or federal mandate. District leaders’ use of the human resources frame represents the greatest
likelihood of creating lasting changes, such as the development of 21st Century Skills (Bolman
& Deal, 2008).
Discussion. The purpose of the third research question was to identify strategies that
superintendents used for including stakeholders in the implementation of 21st Century Skills.
The researcher believed that identifying common strategies for soliciting the support of

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 114

stakeholders could provide a road map for other superintendents and researchers in the support of
21st Century Skills’ initiatives. Data analysis revealed that supporting the implementation of
major mandates involves the inclusion of stakeholders in a process of co-creating: goals, lesson
plans, and professional development. As forwarded by Superintendent B (personal
communication, July 29, 2013), meaningful implementation by administrators and teachers
requires that they believe in the mandate that is being implemented. The interviews also
revealed that stakeholders can be aligned through the development of common language and
goals, teacher directed professional development, and writing curriculum.
These strategies require further study to examine both the short- and long-term impact on
supporting the implementation of 21st Century Skills. The superintendents interviewed for the
current study further detailed that key stakeholders need a plan of achievable goals, with a
clearly identified sequence, creating a forgiving environment for administrators, and creating
teacher leaders. Overall, the strategies identified by the participants of these studies can be
universally applied to other major reforms that occur within an organization. The
superintendents surveyed used primarily the human resource and political frames (Bolman &
Deal, 2008) to garner support of key stakeholders. The variance such as where the districts are
in the process of implementing 21st Century Skills restricts the ability to find commonalities in
approaches for soliciting support of key stakeholders.
Auxiliary Findings
Quantitative survey. The quantitative survey also sought to measure items that
provided background information regarding how districts are, or plan to, implementing 21st
Century Skills. One of the quantitative survey questions asked superintendents to select a 21st
Century Skills’ framework that guided their district’s implementation of 21st Century Skills.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 115

Sixty-seven percent of superintendents selected Partnership for 21st Century Skills. Also among
the respondents, 17% of the superintendents indicated that North Central Regional Educational
Laboratory and the Metiri Group’s 21st Century Skills’ framework as influencing their district’s
21st Century Skills’ implementation. Twenty-eight percent of the superintendents selected Tony
Wagner’s Seven Survival Skills as the 21st Century Skills’ framework that influenced their
district’s implementation. The remaining third of the respondents selected another 21st Century
Skills’ framework as influencing their districts implementation. Among the other frameworks
selected in the open-ended option of this item included: (a) EdLeader 21, (b) Linked Learning,
(c) Lagan Strategies, (d) New Tech Network, (e) Buck Institute for Education, and (f) Digital
Promise/League of Innovative Schools. Among the open-ended responses, five respondents
selected EdLeader 21. Table 12 highlights the different frameworks used by 18 of the surveyed
superintendents to support their district’s implementation of 21st Century Skills.
The results from Table 12 demonstrate the most commonly used framework is the one
developed by Partnership for 21st Century Skills. This observation simply provides background
information, but did not merit further inquiry during the interviews or document analysis portion
of the study due to the goals of the study. The quantitative survey also sought to see what
competencies superintendents believe are included in a 21st Century Skills’ framework. One of
the quantitative survey items asked respondents to select competencies that they define as 21st
Century Skills, including skills described by: (a) Partnership for 21st Century Skills, (b) NCREL
and the Metiri Group, (c) Wagner’s (2008) Seven Survival Skills, (d) the research of Fidel and
Trilling (2009). The subject allowed superintendents to check all skills that they thought were
included in the definition of 21st Century Skills (see Table 13).

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 116

Table 12

21st Century Skills’ Frameworks

21st Century Skills’
Frameworks
Number of
Superintendents
Percentage of
Superintendents
Partnership for 21st Century
Skills

12 67
North Central Regional
Educational Lab (NCREL-
enGauge)

3 17
Tony Wagner’s Seven
Survival Skills
5 28
Other 6 33

The competencies presented by this subject were selected by a majority of the
respondents. The most popular skills included in the superintendent’s definitions were what are
commonly referred to as the four C’s (communication, collaboration, problem solving, and
critical thinking) (Partnership for 21st Century Skills, 2011). An in-depth analysis of the
responses to this item is beyond the data needs for the current study.
Discussion. The interviews/documents and surveys revealed that Partnership for 21st
Century Skills and EdLeader are the most commonly used frameworks among the participants of
the current study. The auxiliary findings have been presented to provide greater clarity and
depth to the findings from the research questions associated with the current study.
Theoretical framework. To better understand the findings, the theoretical framework of
Bolman and Deal (2008) was used to support data analysis. As previously indicated, Bolman
and Deal developed a leadership framework that is based on mental models used for different
areas. The findings from the study demonstrate that superintendents supported implementation

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 117

Table 13

Superintendents Definition of 21st Century Skills
21st Century Skill Number of
Superintendents
Percentage of
Superintendents
Critical Thinking 22 100
Problem Solving 21 95
Collaboration 21 95
Taking Initiative 17 77
Effective Oral Communication 20 91
Accessing Information 18 82
Curiosity 12 55
Imagination 18 82
Entrepreneurialism
16 73
Analyzing Information 19 86
Effective Written
Communication

18 82
Ability to Use Current
Technology

18 82

using the human resources. Evidence of the human resources frame was evident in several
aspects of research question one and three. These findings include: (a) seeking feedback of key
stakeholders, (b) celebrating the achievement of milestones, (c) including teachers in the
development of curriculum, (d) using teachers to lead professional development, and (e)
including all stakeholders in the development of goals. Symbolic leadership was evident when
several superintendents discussed the need to market the initiative (Bolman & Deal, 2008). The

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 118

attempt to align all stakeholders by educating them about the initiative and using common
language to support the initiative is reflective of the structural framework. The creation of a
forgiving environment that develops achievable goals is reflective of the human resources frame
of leadership.
Conclusion
In summary, Chapter 4 included the results of a quantitative survey developed to measure
attitudes, behaviors, and practices regarding the implementation of 21st Century Skills. The
chapter included the results of 10 interviews with superintendents regarding their districts’
implementation of 21st Century Skills. Multiple sources of information were used to conduct the
analysis of the data including the results of the survey, interviews, and documents.
For the quantitative data, basic statistical measures were used including mean, mode, and
median. Questions were used to generalize about the attitudes, opinions, roles, and practices of
superintendents who are implementing, or planning to implement, 21st Century Skills within
their districts. The qualitative data was included in the form of coded categories and quotes used
to illustrate themes about strategies for implementing 21st Century Skills. Indicators were used
to measure the effectiveness of implementation and how superintendents included key
stakeholders in the implementation.
A synopsis of the current study’s goals, the research questions, the methodology used for
the study, and recommendations for current practice will be included in Chapter 5. Then a
summary of the most salient findings and conclusions will be presented. Finally,
recommendations for future research will be made.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 119

CHAPTER FIVE: SUMMARY, IMPLICATIONS AND RECOMMENDATIONS
Introduction
“Today we are failing too many of our children. We're sending them out into a 21st
century economy by sending them through the doors of 20th century schools.”
President Obama (2006, para. 14)
The goals of the American educational system seem to be incongruent with the needs of
today’s world economy. In the 20th century, students were being prepared to participate in an
American economy that was primarily based on the manufacturing of material goods. The above
quote from President Obama alludes to the fact that the mode of the American and advanced-
world economies have changed. The primary driver for today’s American- and advanced-world
economies has shifted to the service industries (Fadel & Trilling, 2009). This new mode of
economy requires that children develop skills to serve in this new arena. The competencies that
are required beyond the development of basic content knowledge are 21st Century Skills. The
current study attempts to address this reality by examining how California superintendents are
implementing or planning to implement 21st Century Skills in their districts.
Summaries of the key findings, of the current study, as reported in Chapter 4 will be
included in Chapter 5. To provide context for the key findings, the following elements of the
study will be reviewed: (a) the background information regarding the topic of the study, (b) the
problem this study intended to address, and (c) the methodologies used to collect and analyze the
data. This chapter will also include implications of key findings for administrators and future
researchers to build upon. The chapter concludes by identifying study limitations and suggested
areas for future research.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 120

Statement of the Problem
The principles and goals of the American educational system have become antiquated as
the needs of the world’s economy have changed. The current American accountability
requirements for K-12 schools largely mandate the development of basic skills only. However,
the mode of the new world economy is knowledge based, one that requires K-12 education be
predicated on the development of basic content knowledge and 21st Century Skills. The
problem that the current study addressed was to identify strategies used by California
superintendents to implement 21st Century Skills’ programs in their districts.
Purpose of the Study
The purpose of this study was to add to the existing body of research regarding how
superintendents implemented or plan to implement 21st Century Skills. Moreover, the study
intended to identify strategies for implementing 21st Century Skills that could be replicated by
the administrators of other school districts in the state of California and other states.
Research Questions
To guide this research study, the following questions were used as the foundation in
conducting the research.
1. What strategies do superintendents use to develop 21st Century Skills in their district?
2. What indicators do superintendents use to evaluate the implementation of 21st Century
Skills in their district?
3. How do superintendents include key stakeholders in the implementation of 21st Century
Skills?

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 121

Methodology
The current study employed mixed methods using both qualitative and quantitative
methods for data collection. The unit of analysis for the study was superintendents in the state of
California. The quantitative aspect of the study involved an electronic survey sent to 55
superintendents, throughout the state of California, via Qualtrics. Twenty-two superintendents
completed the survey. The survey consisted of both closed-ended and open-ended items. The
qualitative portion of the study included 10 interviews of current superintendents, and an analysis
of internal and external documents obtained from the interviewees’ districts. The mixed-
methods approach to data collection (survey, interview, and document analysis) enabled the
researcher to use triangulation to cross check and validate the findings.
Quantitative data collection procedures. The data collection procedure for the study
involved a voluntary cross-sectional survey. The researcher’s design of the survey was
influenced by the research of Marzano and Waters (2009), on effective district leadership and
common 21st Century Skills’ frameworks, including: NCREL and the Metiri Group, Partnership
for 21st Century Skills, and Tony Wagner’s (2008) Seven Survival Skills. The superintendents
were invited to participate by way of a letter of recruitment that included a link to the survey.
The survey used for the study was composed of 12 open- and closed-ended questions with
responses posted by a link to the Qualtrics website. The survey instrument is shown in Appendix
D.
Prior to data collection, the survey was field tested by administering to three doctor-of-
education candidates at the University of Southern California, and three former district
superintendents. The survey was designed to gather quick responses that measured the attitudes
and behaviors of California superintendents regarding the implementation of 21st Century Skills

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 122

in their districts. The survey instrument was designed to identify superintendents who presented
information rich cases regarding their district’s implementation of 21st Century Skills.
Qualitative data collection procedures. The primary tool for the collection of
qualitative data was interviews. The interview protocol for the study was designed with a semi-
structured approach that blended pre-written questions with follow-up questions, to clarify or
gather more in-depth information. Prior to the interviews, the study’s protocol was developed
using the findings of: (a) Marzano and Waters’ (2009) research on effective district leadership,
(b) Kay and Greenhill’s (2013) recommendations for implementing 21st Century Skills, (c) the
four frames of leadership (see Chapter 2) as espoused by Bolman and Deal (2008), and
(d) Senge’s (2012) work on structures for Schools that Learn. The interview questions for the
study were grouped by the studies three primary research questions. The interviews began with a
brief explanation of the goals of the research project; the initial questions were designed to help
develop rapport with the participants. The interview protocol is located in Appendix E.
Quantitative data analysis. The Qualtrics software employed for this study calculated
the basic measures of central tendency for each question. The data analysis began with a
discussion of the response rates to the survey. Descriptive data was included to illustrate the
demographics of the respondents to the survey, and the criteria used to select superintendents as
participants. Then, responses to the items were analyzed by research questions in conjunction
with the qualitative data that was collected for the current study.
Qualitative data analysis. The process followed for analyzing the interview transcripts
was based on the six-step method designed by Creswell (2009). First, the researcher captured
field notes following each interview. Second, the interviews were transcribed via the Internet
using the Rev service at www.rev.com. Third, the field notes and transcripts were examined to

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 123

note the researcher’s initial impressions of the data. Fourth, the data from the transcripts were
grouped into categories (the three primary research questions for the study), and then each
category was labeled. Fifth, the coded data was organized into specific groups within the
research questions. Sixth, the researcher identified narrative passages that supported the
findings. To conclude, the researcher generated findings from the categories in relation to the
research questions that they sought to answer.
Summary of Findings
Research Question 1
What strategies do superintendents use to develop 21st Century Skills in their district?
An implementation of 21st Century Skills requires a variety of strategies based on the
numerous and varied contextual elements involved in the initiative. Data analysis revealed
strategies for four different aspects of the implementation of 21st Century Skills. The first
category is the method that districts used to implement 21st Century Skills.
Data analysis revealed three methods for implementing 21st Century Skills including:
(a) Project Based Learning, (b) Linked Learning, (c) curriculum writing or the creation of units
of study. The literature reviewed in Chapter 2 highlights the benefits of PBL/Linked Learning to
acquiring 21st Century Skills. According to Astleitner (2005), the principles of learning
demonstrate that learners must be able to apply their knowledge in a variety of contexts.
Authentic learning must support basic knowledge and high order thinking. Project Based
Learning is an inquiry based method in which students select a topic of interest, and then have
the opportunity to collaborate and problem solve to complete the project (Kay and Greenhill,
2013). Bell (2008) highlighted, that as the result of using PBL, standardized test scores have
been improved in sections of Dubuque, Iowa and Boston, Massachusetts.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 124

The second major theme, for the first research question, is about how professional
development is used to build a district’s capacity to implement 21st Century Skills. The two
most commonly identified approaches for professional development were professional learning
communities and instructional coaching.
During the interviews, superintendents identified the use of professional learning
communities as a tool for building their district’s capacity to implement 21st Century Skills.
Data analysis revealed several different PLCs being used to build district capacity for
implementing 21st Century Skills. The types of PLCs that were identified during the interviews
include: (a) inter-district PLCs, (b) intra-district PLCs, (c) principal PLCs, (d) regional PLCs,
and (e) school-site based PLCs. A review of literature highlights the benefits of PLCs in relation
to building individual and collective capacity. The research of Hattie (2012) demonstrated that
allowing teachers to collaborate on the development of curriculum, lessons, and assessment is
one of the most significant factors to supporting student learning. PLCs are a strong tool for
developing 21st Century Skills as it allows teachers to learn about students’ prior achievements,
their modes of thinking, and their level of knowledge in a given content area (Hattie, 2012;
Marzano & Waters, 2009). PLCs can enable principals and teachers to identify the learning
targets associated with 21st Century Skills, to select the approach for instruction, and to critique
the approach. The collaboration associated with PLCs can ensure that learning targets are
optimized around the achievement and instructional goals for 21st Century Skills, as developed
for the needs of a given district (Hattie, 2012).
The second identified strategy used by the participating 10 superintendents was on-going
instructional coaching. The literature reviewed in Chapter 2 supports the use of ongoing

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 125

professional development as a tool of effective district leadership in supporting continuous
improvement of academic achievement (Marzano & Waters, 2009).
The third major strategy that superintendents use to implement 21st Century Skills is
their methods/strategies used to obtain funds and select the best allocation of funds to support the
initiative. An analysis of the interview transcripts revealed that there are three methods that
superintendents use to acquire funding for 21st Century Skills’ initiatives. The three significant
funding strategies are: (a) private donors or organizations, (b) state funding, e.g., Assembly Bill
790 (State of California, 2012), and (c) local bond measures.
The interview transcripts further revealed that superintendents allocated resources to:
(a) expand their district’s technological infrastructure, e.g., schools’ Wi-Fi capabilities,
(b) purchase technology devices for students and staff that included: Apple IPADs, Google
Chromebooks and Laptops, (c) provide professional development opportunities for
administrators and teachers, and (d) changes to the physical plants of the schools to reflect 21st
Century learning environments. The findings of the current study coincide closely with the
literature reviewed in chapter 2 regarding effective leadership. The research conducted by
Marzano and Waters (2009) identified that effective district leadership successfully allocates
resources to support the academic achievement of its students. Research question 2 presents key
findings regarding indicators that superintendents use to evaluate their district’s implementation
of 21st Century Skills.
Research Question 2
What indicators do superintendents use to evaluate the implementation of 21st Century
Skills in their district?

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 126

The analysis of data regarding this research question yielded four categories of indicators
used by superintendents to evaluate the implementation of 21st Century Skills in their district as:
(a) formative measures, (b) summative measures, (c) readiness indicators, and (d) feedback
loops.
The formative indicators that superintendents identified as being used to measure the
effectiveness of their district’s implementation of 21st Century Skills include: (a) content
specific rubrics, (b) learning walks, (c) electronic portfolios of student work. As
highlighted
in
the review of literature in Chapter 2, Marzano and Heflebower (2012) and Greenstein (2012)
championed these formative assessment measures as effective tools for measuring students’
acquisition of 21st Century Skills. The interviews further revealed several summative indicators
that are used to measure the implementation of 21st Century Skills in the participants’ districts.
The summative measures identified as being used by participating superintendents
include: (a) graduation rates, (b) CAHSEE passage rates, (c) A-G completion rates for high
school students, (d) suspension rates, and (e) CWRA passage rates. The literature in Chapter 2
highlighted that effective district leaders continually monitor achievement data to ensure that the
district goals are being met (Marzano & Waters, 2009). Moreover, effective superintendents
measure the extent their schools are meeting their instructional targets. The monitoring of the
formative and summative measures indicates that the superintendents have effective plans for
monitoring student achievement. Data analysis also revealed that several superintendents are
using readiness indicators to measure their districts’ implementation of 21st Century Skills.
The readiness indicators that were used by districts and schools within districts included:
observational rubrics that measured school’s readiness, e.g., teachers’ capacity and knowledge of
21st Century Skills. One superintendent’s district required schools to have evidence of fully

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 127

developing the districts other major initiatives (e.g., ELD, AP courses) prior to receiving
permission to implement 21st Century Skills. The literature regarding the use of indicators to
measure a district’s or school’s readiness to implement 21 Century Skills is lacking and is open
for further study. The interview transcripts revealed that districts are also collecting feedback
from stakeholders to assess the success of the implementation.
The feedback loop is the fourth measure that superintendents are using to evaluate their
districts’ progress towards implementing 21st Century Skills. Data analysis revealed three
methods that superintendents use to gather feedback from key stakeholders. The methods
employed by superintendents to collect feedback include: (a) interviews (one-to-one meetings),
(b) stakeholder surveys, and (c) focus groups. The literature reviewed in Chapter 2 highlighted
the importance of monitoring progress towards the goals of the initiative (Marzano & Waters,
2009). Hentschke and Wohlstetter’s (2004) research highlighted initiators (e.g., superintendents)
of new policies lack information regarding the progress of their new reform because they are
removed from the implementation. Hentschke and Wohlstetter dubbed this phenomenon as the
information asymmetry problem. Thus, the use of the feedback loop by the superintendents in
this study appears to be a promising and a necessary practice for evaluating the implementation
of 21st Century Skills. The next section will include a description of the key findings, from the
study, regarding how superintendents solicit the support of key stakeholders in the
implementation of 21st Century Skills.
Research Question 3
How do superintendents include key stakeholders in the implementation of 21st Century
Skills?

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 128

Analysis of the interview transcripts revealed three major themes regarding how
superintendents include key stakeholders in the implementation of 21st Century Skills. The three
themes identified from data analysis are: (a) co-creation, (b) aligning stakeholders, and
(c) sustaining the initiative.
One major theme is to co-create different aspects of the implementation of 21st Century
Skills. Co-creation was evidenced in three particular aspects of the implementation of 21st
Century Skills for the current study. The idea of co-creation for the superintendents in the study
was evident in the development of the mission, vision, and goals for their 21st Century Skills’
program. The function of co-creating included the development of curriculum and assessment
for 21st Century Skills between administrators, district personnel, and teachers. The findings of
Kay and Greenhill (2013) and Marzano and Waters (2009) demonstrated effective leaders use
co-creation to include key stakeholders as a strategy for soliciting their support for the initiative.
Data analysis affirmed that aligning key stakeholders is a tool used by superintendents to support
the implementation of 21st Century Skills.
A second theme identified by data analysis is that superintendents align key stakeholders
to support the goals for their 21st Century Skills’ initiative. The first step for aligning key
stakeholders, by the study’s participants, was to educate stakeholders about 21st Century Skills.
The second step, identified by superintendents, was that the board must understand the value and
reasons for developing 21st Century Skills.
To educate stakeholders regarding 21st Century Skills, superintendents conducted
literature groups, provided showcase lessons, exemplars from other districts, and held one-on-
one meetings. Marzano and Waters (2009) highlighted aligning stakeholders to the goals of a
reform as one of the practices of effective superintendents. The last theme is strategies used by

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 129

superintendents to maintain support of key stakeholders in their districts’ 21st Century Skills’
programs.
A third theme identified how superintendents maintained the support of key stakeholders
for their 21st Century Skills’ programs. Through data analysis, one strategy identified was the
superintendent had a clearly defined plan for developing 21st Century Skills, with achievable
milestones throughout the process. As put forth by Clark and Estes (2008), maintaining
stakeholder morale is supported by clear, concrete goals. Another instrument used by
superintendents for maintaining stakeholder support is the creation of a supportive environment.
A supportive environment was described as one that does not impose immediate punitive
consequences for the failure to achieve the goals of the initiative. As highlighted by the
comments of several superintendents, a full implementation takes five years, and that requires
patience and commitment of the staff (e.g., principals and teachers) tasked with implementing
the reform. A final strategy identified by superintendents, as a tool for maintaining support, was
the use of teacher leaders to coordinate various elements of their 21st Century Skills’ programs
(e.g., curriculum and assessment).
Implications for Policy and Practice
This study has implications for superintendents implementing or planning to implement
21st Century Skills, both within the state of California and the rest of the United States. This
study adds to the existing body of knowledge on: strategies for implementing 21st Century
Skills, evaluating the implementation of 21st Century Skills, and strategies for including key
stakeholders in the program implementation. The following are the researcher’s
recommendations for administrative staff and other researchers based on recommendations of
other research. However, future research will need to be conducted to confirm the effects that

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 130

the suggestions have on student achievement in relation to the development of 21st Century
Skills.
1. Create a long-term plan (3-5 years) for implementation that: (a) highlights the mission of
the districts 21st Century Skills’ program, (b) describes amount and how the resources for
the 21st Century will be allocated, (c) aligns the allocation of resources to the goals of the
initiative, (d) includes a professional development plan to build capacity in an on-going
nature for principals and teachers, (e) identifies measurable achievement and instructional
standards, and (f) clearly determines the phases of the implementation and the
corresponding actions and goals of each step.
2. Curriculum, instruction, and assessment of 21st Century Skills must be linked together;
and then aligned to CCCS to ensure compatibility of district, state, and federal goals.
3. Market and brand 21st Century Skills’ programs for the students, parents, community.
4. To implement 21st Century Skills, use a combination of Project Based Learning or
Linked Learning along with the continued instruction of core academic subjects.
5. Use professional learning communities that include key stakeholders to build capacity
and support the development of 21st Century Skills (Marzano & Waters, 2009).
6. Professional development must be continuous and aligned to the goals of the program
(Kay & Greenhill, 2013; Marzano & Waters, 2009).
7. Use instructional coaches to facilitate local site implementation of 21st Century Skills.
8. Use multiple measures to evaluate the implementation of 21st Century Skills including:
(a) electronic portfolios that include content rubrics, current and past student
performance, and curricular materials, (b) graduation rates, (c) suspension rates,
(d) completion of A-G requirements, (e) CAHSEE passage, and (f) learning walks.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 131

9. Co-create the mission, vision, and goals for 21st Century Skills’ program with all key
stakeholders (Marzano & Waters, 2009).
10. Align key stakeholders around: (a) a vision, (b) a mission, and (c) instructional goals.
This alignment of key stakeholders for all three aspects of the implementation should
include the use of a common language about 21st Century Skills’ development (Kay &
Greenhill, 2013; Marzano & Waters, 2009).
11. Sustain the initiative (Create 2nd Order Change—Marzano, McNulty, & Waters, 2005)
by: (a) soliciting the support of all key stakeholders (e.g., administrators, parents,
teachers, community, and board members); (b) by creating consensus about mission,
vision, and instructional objectives; and (c) create achievable milestones for the initiative.
12. Educate key stakeholders on the definition of 21st Century Skills. Provide exemplars of
other schools and districts that are implementing 21st Century Skills. Give concrete
examples of the type of instruction that are associated with 21st Century Skills.
Future Research
Implementing 21st Century Skills into California school districts is a relatively new and
developing phenomenon. Administrators (i.e. superintendents) must continue to identify
effective implementation strategies for 21st Century Skills, establish indicators to evaluate the
implementation, and find ways to involve key stakeholders. Following are suggestions for future
research regarding ways superintendents can implement 21st Century Skills in their individual
districts.
1. This study should be replicated in other California districts and states to determine the
accuracy of this study’s findings. The replication of this study in other California

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 132

districts and states could also identify other strategies for implementation, indictors to
evaluate the implementation, and strategies for including key stakeholders.
2. Another study should examine how districts integrated the goals of 21st Century Skills’
initiatives with other major district, state, or federal goals (e.g., the Common Core
Standards Initiative).
3. A study should be conducted to identify strategies for implementing 21st Century Skills
in elementary schools, as the findings for the current study show the origins and
implementation for programs primarily in secondary schools.
4. A study or studies should be employed to examine the effectiveness of the findings for
each of the three primary research questions.
Limitations
There were several limitations associated with the current study. Central limitations of
the study, as indicated in Chapter 1, were time and distance. The availability of superintendents
to participate in qualitative data collection was limited to last summer due to their availability
during the school year. Second, the findings were also limited to the criteria used to identify the
sample population for the study, and may not reflect strategies used by smaller districts or other
states. Third, the accuracy of the findings was based on the assumption that the superintendents’
responses to survey and interviews were complete and accurate. Fourth, the findings may not
have included all relevant information to the research questions due to information not sought.
Fifth, the findings may have been limited by the researcher’s personal biases regarding the topic.
Conclusion
The current study attempted to identify California superintendents’ strategies for
implementing 21st Century Skills. The literature reviewed for the current study highlighted the

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 133

need for schools to support the development of 21st Century Skills, alongside basic content
knowledge. As Wagner (2008) argued, the “problem,” simply stated, is that the future of our
economy, the strength of our democracy, and perhaps even the health of the planet’s ecosystems
depend on educating future generations in ways very different from how many of us were
schooled. The findings for the study are potentially a road map for other California districts and
states to follow suit regarding the development of 21st Century Skills by the students of their
districts. The newness of 21st Century Skills’ programs in California requires that further
research be conducted to confirm the effect that 21st Century Skills have on student
achievement.
The development of 21st Century Skills is not another educational fad,
but a reality that the American educational system must address.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 134

References
Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman, M. K. (2010). How
learning works: Seven research-based principles for smart teaching. San Francisco, CA:
Jossey-Bass.
Anderson, L., & Krathwohl, D. (2000). A taxonomy for teaching, learning and assessing: A
revision of Bloom’s taxonomy of educational objectives. Saddle River, NJ: Pearson
Education.
Astleitner, H. (2005). Principals of effective instruction: General standards for teachers and
instructional designers. Journal of Instructional Psychology, 32(1), 3-7.
Baker, E. L., Betebenner, D. W., Linn, R. L. (2002). Accountability systems: Implications of
requirements of the No Child Left Behind Act of 2001. Educational Researcher, 31(6),
3-16.
Bell, S. (2008). Project-based learning for the 21st Century: Skills for the future. A Journal of
Educational Strategies, Issues and Ideas, 83(2), 39-43.
Bilken, S., & Bodgan, R. (2003). Qualitative research for education: An introduction to theory
and methods. Saddle River, NJ: Pearson Education.
Birman, B. F., Desimone, L., Porter, A. C., & Garet, M. S. (2000). Designing professional
development that works. Educational Leadership, 57(8), 28-33.
Bolman, L.G., & Deal, T.E. (2008). Reframing organizations: Artistry, choice and leadership.
San Francisco, CA: Jossey-Bass.
Brady, K. P., & Thomas, J. Y. (2005). The Elementary and Secondary Act at 40: Equity,
accountability, and the evolving federal role in public education. Review of Research in
Education, 29(51), 51-68.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 135

Brown, C. P. (2007). Unpacking standards in early childhood education, Teachers College
Record, 109(3), 635-665
Brown, J. S. (2006, October). New learning environments for the 21st Century: Exploring the
edge. Change, 38(5), 18-24.
Casner-Lotto, J., & Barrington, L. (2006). Are they really ready to work? Employers'
perspectives on the basic knowledge and applied skills of new entrants to the 21st century
U.S. workforce. Washington, DC: Partnership for 21st Century Skills.
Cheng, Kai-Ming. (2007, May). Education, but for what? Challenges from the workplace. Paper
presented at the World Bank Institute course, “Education for Development and
Competitiveness: Challenges and Opportunities for Post-Basic Education,” Washington,
DC.
Clark, R. E., & Estes, F. (2008). Turning research into results: A guide to selecting the right
performance solutions. Charlotte, NC: Information Age.
Common Core State Standards. (2012). Standards in your state. Retrieved from
http://www.corestandards.org/standards-in-your-state/
Creswell, J. W. (2009). Research design: Qualitative, quantitative, and mixed methods
approaches. Los Angeles, CA: Sage Publications.
Currie, J., & Thomas, D. (1995). Does Head Start make a difference? American Economic
Review, 85(3), 341.
Darling-Hammond, L. (2010a). What can we learn from Finland successful school reform.
National Education Association. Retrieved from http://www.nea.org/home/40991.htm
Darling-Hammond, L. (2010b). The flat world and education: How America’s commitment to
equity will determine our future. New York, NY: Teachers College Press.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 136

Delawsky, S., & Johnson, C. (2013). Project-based learning and student engagement. Academic
Research International, 4(4), 599.
Dillman, D. A. (2000). Mail and internet surveys: The total design method (2nd ed.). New York,
NY: Wiley.
Dufour, R., Dufour, R., Eaker, R., & Many, T. (2010). Learning by doing: A handbook for
professional learning communities at work. Bloomington, IN: Solution Tree.
Elmore, R. F. (2002). Bridging the gap between standards and achievement. Retrieved from:
http://www.gtlcenter.org/sites/default/files/docs/pa/3_PDPartnershipsandStandards/TheI
mperativeforPD.pdf
Fadel, C., & Trilling, B. (2009). 21st century skills: Learning for life in our times. San Francisco,
CA: Jossey-Bass.
Fink, A. (2013). How to conduct surveys. Thousand Oaks, CA: Sage Publications.

Friedman, T. L. (2007). The world is flat: A brief history of the twenty-first century. New York,
NY: Picador, Farrar, Straus and Giroux.
Fullan, M. (2011). The six secrets of change: What the best leaders do to help their
organizations thrive and survive. San Francisco, CA: Jossey-Bass
Fusarelli, L., & Schoen, L. (2008). Innovation, NCLB, and the fear factor: The challenge of
leading 21st century schools in an era of accountability. Educational Policy, 22(1), 181-
203.
Gardner, H. (2006). Five minds for the future. Boston, MA: Harvard Business School Press.
Goals 2000: Educate America Act. (1994). Pub. L. 103-227. Retrieved from
http://lamar.colostate.edu/~hillger/laws/goals-2000.html

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 137

Greenstein, L. (2012). Assessing 21st century skills: A guide to evaluating mastery and authentic
learning. Thousand Oaks, CA: Corwin Press.
Hattie, J. (2012). Visible learning for teachers: Maximize impact on learning. New York, NY:
Routledge.
Hawley, W. D. (2008). NCLB and continuous school improvement. In G. Sunderman (Ed.),
Holding NCLB accountable: Achieving accountability, equity and school reform
(pp. 173-190). Thousand Oaks, CA: Corwin Press.
Heflebower, T., & Marzano, R. (2012). Teaching and assessing 21st century skills. Bloomington,
IN: Solution Tree Press.
Hentschke, G. C., & Wohlstetter, P. (2004). Cracking the code of accountability. University of
Southern California Urban Ed., Los Angeles, CA.
Kanter, H., & Lowe, R. (2006). From new deal to no deal: No child left behind and the
devolution of responsibility for equal opportunity. Harvard Education Review, 76(4).
Kay, K., & Greenhill, V. (2013). The leader’s guide to 21st century education: 7 steps for
schools and districts. Upper Saddle River, NJ: Pearson.
Kyllonen, P. C. (2012, May 8). Measurement of 21st century skills within the Common Core
State Standards. Paper presented at the K-12 Center at ETS Invitational Research
Symposium on Technology Enhanced Assessments, Washington, DC. Retrieved from
http://www.k12center.org/rsc/pdf/session5-kyllonen-paper-tea2012.pdf

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 138

Lafors, J., & McGlawn, T. (2013). Expanding access, creating access: How linked learning
pathways can mitigate barriers to college and career access in schools and districts.
Retrieved from:
http://www.edtrust.org/sites/edtrust.org/files/Expanding%20Access%20Creating%20Opti
ons%20Report.pdf
Manna, P. (2006). Control, persuasion and educational accountability: Implementing the No
Child Left Behind Act. Educational Policy, 20(3), 471-494.
Marzano, R., McNulty, B., & Waters, T. (2005). School leadership that works: From research to
results. Alexandria, VA: ASCD.
Marzano, R., & Waters, T. (2009). District leadership that works: Striking the right balance.
Bloomington, IN: Solution Tree Press.
Maxwell, J. A. (2013). Qualitative research design: An interactive approach. Los Angeles, CA:
Sage Publications.
McClure, P. (2008). The history of educational comparability in Title I of the Elementary and
Secondary Act of 1965. Retrieved from
http://www.americanprogress.org/issues/education/report/2008/06/10/4567/ensuring-
equal-opportunity-in-public-education/
Meier, D., & Wood, G. (2004). Many children left behind: How the No Child Left Behind Act is
damaging our children and our schools. Boston, MA: Beacon Press.
Merriam, S. B. (2009). Qualitative research: A guide to design and implementation. San
Francisco, CA: Jossey-Bass.
Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook.
Thousand Oaks, CA: Sage.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 139

National Center for Education Statistics. (n.d.) Trends in international mathematics and science
study (TIMSS). Retrieved from https://nces.ed.gov/TIMSS/
National Center on Education and Economics. (2007). Tough choices, tough times: The report of
the new commission on the skills of the American workforce. San Francisco, CA: Jossey-
Bass.
National Commission on Excellence in Education. (1983). A nation at risk: The imperative for
educational reform. Retrieved from
http://datacenter.spps.org/uploads/SOTW_A_Nation_at_Risk_1983.pdf
National Governors Association. (2009). Forty-Nine states and territories join Common Coe
Standards Initiative. Retrieved from http://www.nga.org/cms/home/news-room/news-
releases/page_2009/col2-content/main-content-list/title_forty-nine-states-and-territories-
join-common-core-standards-initiative.html
Nores, M. (2010). The economics of early childhood intervention. In D. J. Brewer & P. J.
McEwan (Eds.), Economics of education (pp. 191-195). San Diego, CA: Academic Press.
North Central Regional Educational Laboratory, & the Metiri Group. (2003). enGauge
®
21st
Century Skills: Literacy in the Digital Age.

Retrieved from
http://pict.sdsu.edu/engauge21st.pdf
Obama, B., President. (2005). 21st century schools for a 21st century economy. Remarks of
Senator Barack Obama, Chicago, IL. Retrieved from http://obamaspeeches.com/057-
21st-Century-Schools-for-a-21st-Century-Economy-Obama-Speech.htm

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 140

Organisation for Economic Cooperation and Development. (1999). Measuring student
knowledge and skills. Retrieved from:
http://www.oecd.org/edu/school/programmeforinternationalstudentassessmentpisa/336
93997.pdf
Organisation for Economic Cooperation and Development. (2005). The definition and
selection of key competencies. Retrieved from: http://
http://www.oecd.org/pisa/35070367.pdf
Organisation for Economic Co-operation and Development. (2009). OECD Annual Report:
2009. Retrieved from http://www.oecd.org/newsroom/43125523.pdf
Organisation for Economic Cooperation and Development. (2010). What students know and
can do: Student performance in reading, mathematics and science. Retrieved from:
http://www.oecd.org/pisa/46643496.pdf
Paris, K. (1994). A leadership model for planning and implementing change for school to work
transition. University of Wisconsin-Madison, Madison, WI: Center on Education &
Work.
Partnership for 21st Century Skills. (2011). Framework for 21st century learning. Retrieved
from: http://www.p21.org/storage/documents/1.__p21_framework_2-pager.pdf
Partnership for 21st Century Skills. (2009). P21 Framework definitions. Retrieved from
http://www.p21.org/storage/documents/P21_Framework_Definitions.pdf
Patton, M. Q. (2002). Qualitative research and evaluation methods. Thousand Oaks, CA: Sage.
Podesta, J., & Brown, C. (2008). Ensuring equal opportunity in public education: How local
school district funding hurt disadvantaged students and what federal policy can do about
it. Center for American Progress, 1-110.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 141

Programme for International Student Assessment (PISA). (2009). PISA 2009 Results. Retrieved
from https://nces.ed.gov/surveys/pisa/pisa2009highlights.asp
Race to the Top. (2009), Fact sheet: The Race to the top: Promoting innovation, reform, and
excellence in America’s public schools. Retrieved from http://www.whitehouse.gov/the-
press-office/fact-sheet-race-top
Reimers, F. (2006). Citizenship, identity and education: Examining the public purposes of
schools in an age of globalization. Prospects, 36(3), 275-295.
Rotherham, A. J., & Willingham, D. (2009). 21st Century: The challenges ahead. Educational
Leadership, 67(1), 16-21.
Sammon, G. (2008). Creating and sustaining small learning communities: Strategies and tools
for transforming high schools. Thousand Oaks, CA: Corwin Press.
Senge, P. (2012). Schools that learn: A fifth discipline fieldbook for educators, parents and
everyone who cares about education. New York, NY: Crown Business.
Silva, E. (2008). Measuring skills for the 21st century. Washington, DC: Education Sector.
Retrieved from
http://www.educationsector.org/sites/default/files/publications/MeasuringSkills.pdf
Smarter Balanced Assessment Consortium. (2012). Smarter balanced assessments. Retrieved
from http://www.smarterbalanced.org/smarter-balanced-assessments/
Sologuk, S., Stammen, R., & Vetter, R. (2001). A collaborative approach for creating curriculum
and instructional materials. Journal of Technology and Teacher Education, 9(2), 199-
210.
Spring, J. (2009). Globalization of education: An introduction. New York, NY: Routledge.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 142

State of California. (1999). Public Schools Accountability Act of 1999. Retrieved from
http://www.leginfo.ca.gov/pub/99-00/bill/sen/sb_0001-
0050/sbx1_1_bill_19990405_chaptered.html
State of California (2012). Career technical academies: Linked learning pilot program. Retrieved
from:
http://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=201120120AB790
Sweet, D. (2014). Strategies California superintendents use to implement 21st Century Skills.
Unpublished raw data.
U.S. Department of Education. (2013). Elementary & Secondary Education: ESEA Flexibility.
Retrieved from http://www2.ed.gov/policy/elsec/guid/esea-flexibility/index.html
U.S. Department of Education. (2011). ESEA Waiver Policy Document. Retrieved from:
http://www2.ed.gov/policy/elsec/guid/esea-flexibility/index.html
U.S. Department of Education. (2001). No Child Left Behind Act of 2001. Washington, DC:
United States Department of Education.
U.S. Department of Education. (2009). Race to the Top: Executive summary. Retrieved from:
http://www2.ed.gov/programs/racetothetop/executive-summary.pdf
Vinovskis, M.A. (2009). From a nation at risk to No Child Left Behind: National education
goals and the creation of federal education policy. New York, NY: Teachers College
Press.
Wagner, T. (2006). Five habits of mind. Retrieved from:
http://www.lwsd.org/school/ics/Library/Pages/Five-Habits-of-Mind.aspx
Wagner, T. (2007). Leading for change: Five ‘habits of mind’ that count. Retrieved from
http://www.edweek.org/ew/articles/2007/08/15/45wagner.h26.html?print=1

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 143

Wagner, T. (2008). The global achievement gap: Why even our best schools don’t teach the new
survival skills our children need – and what we can do about it. New York, NY: Basic
Books.
Wang, Y. (2012). Education in a changing world: Flexibility, skills, and employability. Retrieved
from http://www-
wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2012/05/23/00035616
1_20120523022400/Rendered/PDF/691040WP00PUBL0ability0WEB050110120.pdf
Zhao, Y. (2007). Education in a flat world: Implications of globalization on education. Edge,
2(4), 3-19.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 144

Appendix A
E-mail Recruitment Letter

August 26, 2013

Dear [Name],
My name is Daniel Sweet. I am a doctoral student in the Rossier School of Education at the
University of Southern California and am conducting a research study as part of my dissertation
process under the direction of Dr. Pedro Garcia. This study focuses on identifying how
superintendents of large districts in the state of California implement or plan to implement 21st
Century Skills. You have been selected as the superintendent of a large diverse district in the
state of California. The completion of this survey will take up to 5 minutes.

Participation in the study is entirely voluntary. Your identity will remain confidential at all
times. Your relationship with USC will not be affected whether or not you participate in this
study.

If you have any questions, please contact me via email or phone: Daniel Sweet,
danielbs@usc.edu or 310-925-8231.

Thank you for your participation
Daniel Sweet
Ed.D. Candidate
University of Southern California

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 145

Appendix B
Interview Cover Letter/Email

June 2013

Dear [Name],

My name is Daniel Sweet. I am a doctoral student in the Rossier School of Education at the
University of Southern California and am conducting a research project as part of my dissertation
process under the direction of Dr. Pedro Garcia. My study focuses on the strategies that
superintendents of large districts in the state of California are using to implement or planning to
implement 21st Century Skills. Thank you for your willingness to participate in a 30 minute
interview.

Participation in this study is entirely voluntary. Your identity will remain confidential at all
times. Your relationship with USC will not be affected whether or not you participate in this
study.

Our interview has been scheduled to take place at your office on ____________ at ___________.
(date) (time)

Attached you will find a list of topics and questions we may cover during the interview. If you
have any questions, please contact me via email or phone: [Daniel Sweet danielbs@usc.edu or
(310) 925-8231.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 146

Appendix C
Interview Protocol

1. How do you define 21st Century Skills?

2. Where is your district in the process of implementing 21st Century Skills?

3. What goals did you or your district set, or will set, for achieving the development of
21st Century Skills within your school district? (RQ #2)

4. How did you or will you establish goals for the implementation of 21st Century Skills
within your school district? (RQ#2) Describe the process, strategies used, and people
interacted with; how did you include stakeholders?

5. How did or will your district allocate resources to support the implementation of 21st
Century Skills? — e.g., technical, material, financial, and human

6. How will you monitor/assess the goals that you established for 21st Century Skills?
(RQ#2) — What are your indicators of success?

7. What role does or will professional development play in the implementation of 21st
Century Skills in your district?

8. How does or will your district collect feedback regarding the implementation of 21st
Century Skills, and do you feel feedback regarding implementation is or has been
useful? (RQ#2) — What indicates success?

9. Upon monitoring the results of the implementation of 21st Century Skills, how will you
or have you worked to sustain progress of the elements that you’ve found to be
successful in implementation of 21st Century Skills?

10. What will you do, or have you done, to make modifications to aspects of the
implementation that are not meeting the goals previously defined?

11. How do you or will ensure that all parts of the system in your district are aligned to
implement 21st Century Skills? (RQ#1 and RQ#3) — e.g., people, resources,
infrastructure

12. How did you or will you solicit the support of key stakeholders for the implementation
of 21st Century Skills? (RQ#3)-

13. How do federal and state accountability requirements impact your plans to implement
or your actual implementation of 21st Century Skills?

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 147

Appendix D
Survey Protocol

Implementing 21st Century Skills in California

Q1 Which of the following best describes your professional background?
m Elementary Education
m Secondary Education
m Business
m Other ____________________

Q2 Do you believe that the full development of basic skills is sufficient for American
students to be globally competitive?
m Yes
m No

Q3 In consideration of current Federal (Adequate Yearly Progress) and State (Academic
Performance Index) accountability requirements, how important do you consider the
implementation of 21st Century Skills?

Provide your opinion below
Choose One Answer
Not at all Important m
Very Unimportant m
Very Important m
Extremely Important m

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 148

Q4 Which abilities do you consider part of 21st Century Skills? (Check all that apply)
m Critical Thinking
m Problem Solving
m Collaborating with others
m Taking Initiative
m Effective Oral Communication
m Accessing Information
m Curiosity
m Imagination
m Entrepreneurialism
m Analyzing Information
m Effective Written Communication
m Ability to use current technology

Q5 Is your district considering or already implementing 21st Century Skills?
m Yes
m No

Q6 What is or what will be your role in the implementation of your school district’s 21st
Century Skills’ Program? (Check all that apply)
m Overseer
m Researcher
m Professional Development
m Aligning Stakeholders
m Leading Focus Group Conversations
m Other roles (please specify) ____________________

Q7 Who do you or will you interact with during the implementation process of 21st Century
Skills? (Check all that apply)
m Assistant Superintendents
m Board of Education
m Cabinet Members
m Principals
m Assistant Principals
m Curriculum Specialists
m Teachers
m Vendors
m Other (please specify) ____________________

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 149

Q8 Which of the following 21st Century Skills’ frameworks influences the implementation
of 21st Century Skills in your school district? (Check all that apply)
m Partnership for 21st Century Skills
m North Central Regional Educational Laboratory (NCREL–enGauge 21st Century
Skills)
m Tony Wagner's Seven Survival Skills
m Other (please specify) ____________________
m None of the above

Q9 Which of the following strategies will or do you use to build your districts capacity to
implement 21st Century Skills? (Check all that apply)
m Professional Development
m Curriculum Development
m Instructional Coaching
m Other (please specify) ____________________

Q10 What types of assessments does the district use or intend to use to measure students’
acquisition of the instructional goals associated with 21st Century Skills? (Check all that
apply)
m Self-Rating Scales
m Performance Assessments
m Content and Performance Skills Based Rubrics
m other ____________________
m not sure

Q11 Thank you for taking the time to complete this survey. Please feel free to add any
additional comments.

Q12 Can I contact you for a follow-up interview related to this topic?
m Yes
m No

Q13 Again, thank you time and interest. If you have consented to a follow-up interview,
please provide your information below
m Name: ____________________
m District and City: ____________________
m Email Address: ____________________

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 150

Appendix E
Interview Protocol Connection to Framework
Question Connection to
Marzano and Waters
(2009)
Connection to Bolman
and Deal (2008)
1 How do you define 21st Century
Skills?

none none
2 Where is your district in the process
of implementing 21st Century Skills?

none none
3 What goals did you or your district
set, or will set, for achieving the
development of 21st Century Skills
within your school district? (RQ #2)

Setting non-
negotiable goals for
instruction
Structural
4 How did you or will you establish
goals for the implementation of 21st
Century Skills within your school
district?

Collaborative goal
setting
Structural
5 How will you monitor/assess the
goals that you established for 21st
Century Skills?

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 151

Appendix F
Sample Project Based Learning Lesson for Elementary School

Grape Unified School District

Lesson Format for PBL
1. Enduring Understanding
2. Driving Question
3. Problem Statement

Sample Topic
1. Enduring Understanding – Understand the origins, accomplishments, and geographic
diffusion of the renaissance, reformation, and scientific revolution; i.e. Leading figures
who made advances in literature, the arts, science, mathematics, and human anatomy.
2. Driving Question: How did the evolution of Western Europe and the Church influence
World History?
3. Problem Statement: How can we as European art and history students create and design
an annotated mural to illustrate the evolution of Western European society and how the
church influenced world history so that anyone anywhere can visit the ACMS Art Project
to see and learn from the art.

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 152

1/2nd Grade
1. Driving Question: How can we as community planners improve the Grape Junction
campus?
2. Today’s Activity: In an effort to celebrate Grape Junction’s 60th birthday, 1st and 2nd
grade students are taking on a variety of beautification projects around campus. This
class has to paint the garbage cans. Today students will work in collaborative teams to
plan their designs

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 153

Appendix G
Observational Checklist

Grape Unified School District
1. Evidence of best practices and strategies being used schoolwide
a. driving question posted
b. agenda that outlines student work versus teacher work
c. evidence of need to knows posted
d. standards of behavior posted
e. organized protocols for technology use
f. respectful interactions between teachers and students
g. evidence of student voice and choice (in selection of products, work (roles), how
they use their time
h. learning target is posted
i. student and team improvement goals guide learning

2. Evidence of the 4 C’s (Critical Thinking, Creativity, Communication, Collaboration)
a. majority of students conduct themselves in their own roles in the group (leaders,
recorder, time keeper)
b. majority of students can explain thinking to others and support it with evidence
c. majority of students follow polite rules for speaking and listening
d. majority of students can use like technology to facilitate their work
e. majority of students gather relevant information from reliable sources
f. majority of students demonstrate multiple original ideas for responding to a problem,
investigation, or challenge
g. majority of students demonstrate willingness to share ideas and ask for feedback from
others

3. Regardless of instructional strategy (direct instruction, group work, small groups with
teacher, and independent work), about what percentage of students were actively engaged
in the learning process, and if appropriate, can you tell what they are working on and why
it is important
a. >50% student engagement
b. 50% student engagement
c. <50% student engagement

CALIFORNIA SUPERINTENDENTS EFFECT 21st CENTURY SKILLS 154

4. What evidence is there that students are working in collaborative teams on complex tasks
(student feedback may be necessary to evaluate these questions)? Please select all that
apply.
a. student seated together focused on common project-based tasks
b. students discussing project work and/or supporting one another with project-based
tasks
c. students are able to articulate what they are learning and why they are learning it
d. teacher is facilitating around the room as needed or working with small groups
e. teacher is checking for understanding and offering formative feedback, as students
respond
f. other

5. What is the approximation of students using digital devices for learning?
a. 100%
b. more than half
c. <50%
d. 0%

6. Use of Technology — What kinds of tools (posted or available) are being used to support
group work? (Observers may have to ask students to show these tools as they may be
housed digitally) Please check all that apply.
a. Digital Devices (tablet, laptop, notebook, etc.)
b. Agendas
c. Project pacing charts or task lists
d. Group contracts
e. Small group workshops with the teacher

Abstract (if available)

Abstract The purpose of this study was to identify strategies used by California superintendents to implement 21st Century Skills programs in their districts. More specifically, the study attempted to discover (a) what strategies do superintendents use to develop 21st Century Skills in their district?, (b) What indicators do superintendents use to evaluate the implementation of 21st Century Skills in their district?, (c) How do superintendents include key stakeholders in their implementation of 21st Century Skills? This study employed a mixed methods approach in which 22 superintendents throughout the state of California completed surveys. Among the superintendents who completed the survey, 10 participated in a semi‐structured interview. Finally, the participating superintendents presented documents for analysis. Triangulation of the data was used to generate answers to the research questions. The following are some of the key findings of the study. First, the most common method for implementing 21st Century Skills is Project Based Learning. Second, superintendents use professional development methods to build their district’s capacity such as: professional learning communities, instructional coaching, and train the trainer model. Fourth, superintendents used numerous formative and summative measures to evaluate their district’s 21st Century Skills implementation (e.g., graduation rates and portfolio assessments). Fifth, superintendents evaluated their districts implementation of 21st Century Skills by collecting feedback from key stakeholders (e.g., principals, teachers, and parents). The results of this study present strategies that support the development of 21st Century Skills but will require future research to confirm their effect on student achievement.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

A study of California public school district superintendents and their implementation of 21st century skills

PDF

A study of California public school district superintendents and their implementation of 21st century skills

PDF

A study of California public school district superintendents and their implementation of 21st-century skills

PDF

The process secondary administrators use to implement twenty‐first century learning skills in secondary schools

PDF

What strategies do urban superintendents utilize to address global challenges in the implementation of 21st century skills

PDF

A study of California public school district superintendents and their implementation of 21st century skills

PDF

A study of California public school district superintendents and their implementation of 21st -century skills

PDF

The role of district leadership and the implementation of 21st century skills through professional development

PDF

21st century skills as equity pedagogy to teach academic content standards: a socioeconomically diverse district's implementation

PDF

Evaluating the implementation of 21st century skills and learning

PDF

The development of 21st century skills among incarcerated youth: a needs assessment of teachers

PDF

21st century superintendents: the dynamics related to the decision-making process for the selection of high school principals

PDF

Impact of leadership on professional development programs on districts' and schools' implementation of 21st century skills

PDF

How the University of Costa Rica is preparing educational leaders in secondary schools to help students develop 21st-century skills to meet the demands of multinational corporations for increased...

PDF

The influence of leadership on the development of 21st century skills throughout Irish schools

PDF

An examination of traditional versus non-traditional superintendents and the strategies they employ to improve student achievement

PDF

Leadership strategies employed by K–12 urban superintendents to improve the academic achievement of English language learners

PDF

Strategies used by superintendents in developing leadership teams

PDF

Preparing students for the future - 21st century skills

PDF

The critical aspects of oversight that suburban superintendents, as instructional leaders, must employ to improve instruction

Asset Metadata

Creator Sweet, Daniel (author)

Core Title Strategies California superintendents use to implement 21st century skills programs

School Rossier School of Education

Degree Doctor of Education

Degree Program Education (Leadership)

Publication Date 04/30/2014

Defense Date 02/03/2014

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag 21st century skills,implementing 21st century skills,OAI-PMH Harvest,superintendents

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor García, Pedro Enrique (committee chair), Castruita, Rudy Max (committee member), Strunk, Katharine O. (committee member)

Creator Email danielsweet1422@gmail.com

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c3-408457

Unique identifier UC11295416

Identifier etd-SweetDanie-2473.pdf (filename),usctheses-c3-408457 (legacy record id)

Legacy Identifier etd-SweetDanie-2473.pdf

Dmrecord 408457

Document Type Dissertation

Format application/pdf (imt)

Rights Sweet, Daniel

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA