University of Southern California Dissertations and Theses

A needs assessment and evaluation of an elementary afterschool STEM program in a small urban school district

(USC Thesis Other)

A needs assessment and evaluation of an elementary afterschool STEM program in a small urban school district

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

1

A NEEDS ASSESSMENT AND EVALUATION OF AN ELEMENTARY AFTERSCHOOL
STEM PROGRAM IN A SMALL URBAN SCHOOL DISTRICT

by

Joseph Malcolm Calmer

A Dissertation Presented to the
FACULTY OF THE USC ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF EDUCATION

May 2015

Copyright 2015 Joseph Malcolm Calmer
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

2
Acknowledgements
I would like to acknowledge my wife, son, family, and friends for their continuous
support. My colleague was also instrumental in the completion of this process. Additionally, my
committee was essential in assisting me throughout this endeavor. I would like to thank Dr.
Freking for giving me guidance; Dr. Rueda for helping me organize my thoughts; in addition to,
Dr. Sinatra for your feedback that enabled me to develop as a professional.
An acknowledgement is also given to Cohort 2012, especially to the Saturday group who
has helped me learn more about education, leadership, accountability, and diversity via
curriculum and experience during my pursuit of advanced learning. I would also like to thank
the entire Rossier faculty for their guidance and motivation. A special thank you for the district
is well deserved for allowing us to use their sites and staff for data collection and scrutiny.
This project has facilitated my learning in regards to research and how to report the
research. My hope now is that this project will help this specific site, in addition to others like it
to become a better program, which will subsequently enable students to learn STEM through the
practice of science and discovery. Education has always been about the advancement of
learning.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

3
STATEMENT OF CO-SUTHORSHIP
This document reports the results of a capstone project that was completed as part of the Ed.D
culminating program requirements. It was designed as a dissertation of practice that targets
authentic problems of practice and that provides an opportunity for students to demonstrate skills
and competencies that will be required in future career activities. The present project was a needs
assessment/evaluation of a specific program, and was designed to address concerns of this
specific site rather than being designed as a generalizable research project. In line with the goal
of reflecting real world practice, this project was carried out collaboratively between Joseph
Calmer and Rebecca Acosta. To accurately reflect the division of work on this project, some of
this document is co-authored, and the specific chapters are labeled to reflect the collaborative
authorship where appropriate.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

4
Table of Contents
List of Tables 8
List of Figures 9
Abstract 10
Chapter 1: Overview of the study 11
Background of the Problem 11
Statement of the Problem 13
Purpose of the Project 13
Limitations and Delimitations 14
Definition of Terms 15
Chapter 2: Literature Review 17
Introduction 17
USC and Elementary Afterschool STEM Program Collaboration 17
Importance of STEM 18
Definition of STEM 18
History of STEM in Education 20
Transition to an integrated approach 21
Increase attention in STEM education 22
Why STEM Education is Important 23
Literacy in STEM education 23
Changing technology 23
Changing standards 23
The Role of NGSS 24
Elements of STEM Education 25
Literacy 25
Relevance 26
Areas of Focus for the Needs Assessment and Evaluation of SUSD 27
Needs Assessment 27
Evaluation 27
Logic Models in Organizations 28
Four Domains of Analysis 28
Instructional Practices 28
Criteria for STEM Instruction 29
Inquiry 29
Relevancy 29
Literacy 30
Assessment 31
Summary of Key Points 31
Staff Knowledge of Content and Pedagogy 32
Pedagogical content knowledge 32
Summary of Key Points 34
Curriculum 34
Summary of Key Points 36
Organizational Elements 36
Organizational culture 37
Organizational goals 38
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

5
Organizational barriers 38
Resources 38
Summary of Key Points 38
Issues of Concern at SUSD 39
Elementary After School STEM Program’s Stated Goals 39
Year one 39
staff goal 39
youth goal 40
Year two 40
staff goal 40
youth goal 40
Instructional Practices 40
Staff 40
Curriculum 40
Organizational 41
Measuring Outcomes in STEM Programs 41
Chapter 3: Methodology 42
Introduction 42
Site Description 42
Small Urban School District 42
Connection to Elementary Afterschool STEM Program 42
SUSD Mission Statement 43
Project Context 44
Location and number of sites 44
Student Demographics 44
Site Staff 45
Instrumentation 46
Student Interest in STEM Survey 46
Student Career Awareness in STEM Survey 46
Staff Knowledge of STEM Survey 47
Questionnaire 47
Document Analysis 48
Observations 48
Data Collection 51
Sequence of work for the study 51
Initial meeting 51
Site Visit 51
Scope of work 52
Conference call 52
8/13/14 meeting 52
IRB 53
Survey Administration 53
Questionnaire 53
Document analysis 53
Observations 54
Data Analysis 54
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

6
Survey 54
Questionnaire 54
Document analysis 55
Another University’s Initial Needs Assessment Results 55
Observations 55
Chapter 4: Findings (Coauthored by Joseph Calmer and Rebecca Acosta) 56
Introduction 56
Participant Demographics 56
Study Participants 57
Tutors 57
Site leads 60
Site supervisors 61
Students 61
Framing Question Analysis 62
Afterschool STEM Program question 1: Staff attitude
towards STEM 62
Tutor survey 62
Site lead survey 66
Afterschool STEM Program question 2: Amount of
STEM implemented 67
Tutor survey 68
Site lead survey 69
Supervisor questionnaire 70
Afterschool STEM Program question 3: Student voice
and choice in Afterschool STEM Program 72
Student survey 72
Student observation 73
Relationships 74
Relevance 74
Youth voice 75
Summary 75
Question 1: Staff perception of STEM and LIAS principles 75
Question 2: Implementation of STEM for Afterschool STEM
Program goals 76
Question 3: Student voice and choice in activities 77
Chapter 5: Discussion (Coauthored by Joseph Calmer and Rebecca Acosta) 78
Summary 78
Discussion of Findings 79
Unexpected outcomes 79
Recommendations 80
Developing real organizational goals 80
Action item 80
Measuring student outcomes 81
Action item 81
Increase training 82
Action item 82
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

7
Created Tiered Training 83
Action item 83
Implementation of STEM 83
Action item 84
Student voice and choice in learning 84
Action item 85
Enhance their current curriculum 85
Action item 86
Increase staff members with STEM interests 87
Stimulating students’ affective domain 87
Action item 87
Limitations 87
Appendices: 90
Appendix A: Afterschool STEM Program/ USC: Scope of Work 90
Appendix B: Afterschool STEM Program: Student Survey 94
Appendix C: Afterschool STEM Program: Tutor Survey 99
Appendix D: Afterschool STEM Program: Site Lead Survey 107
Appendix E: Afterschool STEM Program: Administrative Questionnaire 111
Appendix F: SUSD Strategic Action Plan 112
Appendix G: Afterschool STEM Program Staff Job Descriptions 113
Appendix H: STEM career interest survey 115
Appendix I: Secondary teacher’s pedagogical content knowledge instrument 117
Appendix J: Dimensions of Success Rating Sheet for Roller Coasters 121
Appendix K: Child Assent Form 122
Appendix L: Information Sheet for Parents 123
Appendix M: Information Sheet for Adults 125
Appendix N: Student survey results 127
References 130

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

8
List of Tables
Table 1: Practices and Crosscutting Concepts in the NGSS 25
Table 2: Combined Demographic Data of the Students from the Schools that
Participated in Afterschool STEM Program 45
Table 3: Dimensions of Success: Domain 1: Features of the
Learning Environment 49
Table 4: Dimensions of Success: Domain 2: Activity Engagement 49
Table 5: Dimensions of Success: Domain 3: STEM Knowledge and Practices 50
Table 6: Dimensions of Success: Domain 4: Youth Development in STEM 50
Table 7: Summary of Survey Responses Sent, Received, and Analyzed 57
Table 8: Afterschool STEM Program’s Tutor Percentage and Participation in
this Project from Each of Their Sites 58
Table 9: Descriptive Statistics for Afterschool STEM Program’s Tutor Survey
Items about the Environment of Afterschool STEM Program 63
Table 10 Results from the Site Leads’ Survey that identifies the Needs in
Afterschool STEM Program 67
Table 11. Descriptive Statistics for Afterschool STEM Program Student Surveys 72

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

9
List of Figures
Figure 1: Tutor Age Distribution at Afterschool STEM Program 59
Figure 2: Tutor Current Undergraduate Degree Matriculation Plan 60
Figure 3. Students’ School Attended for Afterschool STEM Program 62
Figure 4. Statistics From Tutors Perspective on Afterschool STEM Program’s Resources and
Supplies, Training, and Instructional Space 65
Figure 5. Percent of Tutors and Their Perspective on Their Training in STEM, Science and
Engineering Practices and Afterschool STEM Program’s Mission Statement
66
Figure 6. Average Results from Tutors Responses about their Daily Activities Completed in
Classrooms at Afterschool STEM Program 69
Figure 7. Site Lead Responses to the Planning of Daily Activities 70
Figure 8. Results of Youth Development in STEM Domain 74

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

10
Abstract
This project was a needs assessment and evaluation of an elementary afterschool STEM program
in a small urban school district. The organization we studied will be called Elementary
Afterschool STEM Program to protect the identity of the organization that resides is a small
urban school district. Three framing questions were created to organize the data: 1) Does the
staff at Elementary Afterschool STEM Program have a positive attitude toward STEM and their
LIAS principles? 2) To what extent is Elementary Afterschool STEM Program implementing the
four components of STEM in their daily plans to achieve their program’s goals?
and 3) To what extent does Elementary Afterschool STEM Program provide opportunities for
student voice and choice in their learning?. To answer these questions, data from the afterschool
program’s students, tutors, site leads, and administration was sought. To support
recommendations, literature was reviewed on the history of STEM education, STEM instruction,
STEM programs practices, and effective STEM pedagogy. Also, literature about the literacy, the
CCSS, and the NGSS were investigated as possible solutions as enhancements to their
afterschool STEM program. Data was collected and analyzed to produce recommendations for
program improvement. Data from student, tutor, and site leads along with an administrative
questionnaire, classroom observations and document analysis was conducted to support the
recommendations. The analysis included quantitative analysis on the surveys and qualitative
analysis on the observations and document analysis. These methods were used to triangulate data
and generate recommendations. Recommendations included creating SMART goals, increase
staff training, implementing a full STEM curriculum, increasing students’ affective domain,
measuring student outcomes, increase student voice and choice, and enhancing their STEM
curriculum.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

11
CHAPTER 1: OVERVIEW OF THE STUDY
Background of the Problem
Herbert Smith long ago described why science education has not been successful.
He explains that most science curricula are written in a way that treats science as
discipline (Bybee, 2013a; Smith, 1969). Science is more than a discipline; it is a process
of gaining knowledge. Unfortunately, science education in the past decade has simply
been treated as a set of facts to be learned (Brown, Brown, Reardon, & Merrill, 2011).
This approach does not lead to a deep understanding of the processes of science by
students. Currently, there are a new set of science standards, called the Next Generation
Science Standards (NGSS) that approach science from a more process-orientated,
inquiry-based, integrated approach (NGSS Lead States, 2013). One goal of the new
standards’ is to ameliorate the lackluster understanding of science by students and
increase students’ knowledge and interest in STEM related fields (Organisation For
Economic Co-Operation Development, 2007).
There is a mode of thought that integrates different fields of study for the purpose
of increasing student understanding of science and scaffold interlocking concepts. The
term ‘STEM’ refers to an acronym that combines the independent content areas of
science, technology, engineering, and mathematics into a single subject. The notion of
STEM education is an eponymous description for an instructional approach that may help
students understand the integrated nature of science with technology, engineering, and
mathematics. The use of STEM reflects an impending curricular shift in science
educational practice that integrates different content areas (Committee on Integrated
STEM Education, National Academy of Engineering, & National Research Council,
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

12
2014; National Governors Association Center for Best Practices, 2010; NGSS Lead
States, 2013). To achieve real results in science education will require a paradigm shift
almost tantamount to a revolution (Bybee, 2010a, 2010c; Kuhn, 1970). The shift needs to
be from the traditional science instructional approach to an integrated STEM approach
(Bybee, 2013a; Committee on Integrated STEM Education et al., 2014). The new NGSS
reflect this new approach to science instruction and increase the ubiquity of the nature of
science in schools (Flammer, 2006).
The progress of science learning and education in the United States has been slower than
other developed countries (OECD, 2010). To ameliorate this lackluster performance, a different
approach to science instruction is needed. One emergent thought about science instruction has
been to integrate it with other fields, namely technology, engineering, and mathematics. This
new approach to science instruction falls into the common understanding of STEM education.
The problems of science instruction have long been identified. Herbert Smith (1969)
identified the problems of science instruction as being rooted in literacy and the lack of attention
to the affective domain in science instruction. He stated that “Although the cognitive aspects of
learning are not neglected, the affective aspect of learning is highly significant to most
educators” to reach specific learning goals (Smith, 1969). Smith recognized early that science
instruction needed to be rooted in psychology. Later others have proven his essential tenet
(Donovan & Bransford, 2004). One method of placing science instruction within the affective
domain of students is via an afterschool program.
Afterschool programs allow for content to be delivered in a variety of ways. Most
afterschool programs have structure, partnerships, and missions that promote instruction in a way
that satiates the interests, beliefs, and motivations of children. Afterschool programs have the
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

13
ability to connect content instruction to the affective domain of students (Huk & Ludwigs, 2009).
While there is much more research on STEM programs being conducted than in previous times,
STEM programs continue to struggle with effective implementation (Schiavelli, 2010). This
concern also existed in the STEM program of this project. This report is on the development,
collection, and review of relevant data to help this afterschool program meet their students’
STEM instructional needs.
Statement of the Problem
Small Urban School District has an after school program called Elementary Afterschool
STEM Program. Elementary Afterschool STEM Program has completed their first year of
implementation and would like to evaluate the implementation of their year one goals. In
addition to a year 1 evaluation, they are starting their year two implementation and would like to
complete a needs assessment of their year two goals.
Elementary Afterschool STEM Program’s ultimate STEM objective is to improve SUSD
students' interests in STEM-based careers. To begin the attainment of Elementary Afterschool
STEM Program’s ultimate program goal, a needs assessment and an evaluation of their training
and instructional programs were conducted. Based on the needs assessment and evaluation of
this project a series of recommendations were made.
Purpose of the Project
The purpose of this study was to collect data on Small Urban School District’s (SUSD)
afterschool program. The data was collected and analyzed to provide formative information on
the current status of Elementary Afterschool STEM Program with respect to their goals. The
analysis of data collected were used as evidence to inform the recommended strategies for
Elementary Afterschool STEM Program’s goal actualization and identify real organizational
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

14
outcomes (Altschuld & Witkin, 2000). From that, SMART goals can be generated and used for
program improvement. SMART goals are goals that are specific, measurable, authentic, realistic,
and time-dependent (Anderson, Krathwohl, & Bloom, 2001; Rueda, 2011).
This purpose of this project was to evaluate, build a framework of resources, and to
generate a series of recommendations, using the needs assessment process. The outcomes from
this project will help inform Elementary Afterschool STEM Program create the after school
program they envision. This project was collaboration between SUSD, USC, and the school site
for the program improvement of Elementary Afterschool STEM Program.
This project provided information on the perceptions of selected aspects of the
functioning of an afterschool STEM program. Along with data from survey of students, tutors,
and staff, the effectiveness of instructional practices of STEM were measured and observed.
This project was completed to help Elementary Afterschool STEM Program improve and
develop their STEM instruction. Ultimately, the goal of using STEM instructional practices,
especially in the affective domain of afterschool programs, is to produce graduates in the STEM
career fields.
Limitations and Delimitations
This project used a variety of data collection methods, including surveys, questionnaires,
and observations. While the intent of this project is not to produce generalizable data, these
forms of data collection are subject to limitations in the generalizability of their conclusions.
Self-report surveys are limited because they rely on the subjects’ perspective, which may be
skewed and biased. Questionnaires are limitations because participants sometimes respond in
ways that they perceive will make the interviewer satisfied. Also, observations are limitations
because the observer may have a bias (Creswell, 2008). Lastly, there was a limited amount of
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

15
time and resources to collect data for this project, which is a project limitation. Similarly, that
data collection window may not have allowed some respondents to participate in the data
collection process. This project was designed to help SUSD evaluate their afterschool program,
so data collected and recommendations made were designed to be site specific and limited in
their generalizability.
Definition of Terms
STEM
STEM is the integration of four distinct content areas: science, technology, engineering,
and mathematics, with an emphasis on real-world applicability.
STEM Education
STEM education is an interdisciplinary approach to student learning that integrates the
four distinct disciplines of STEM and emphasizes the relevancy of these four disciplines
(Vasquez, 2013).
STEM Teaching
STEM teaching is the instructional practices of STEM education. This can occur in a
traditional setting or an out-of-school setting, like an afterschool program or camp. Teaching
STEM requires a unique set of skills and pedagogy. The instruction needs to be based more on
process learning rather than content learning (Bybee, 2013b).
STEM pipeline
The term ‘STEM pipeline’ is commonly used to refer to the desire to fill STEM related
careers via students how study STEM related fields in college, high school, and primary school
(Committee on Highly Successful Schools or Programs in K-12 STEM Education, 2011). It is
often thought that if students are interested in STEM early, the STEM pipeline will remain full.
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

16
Integrated STEM
Integrated STEM is the name given to an instructional practice of STEM teachers for the
pedagogy of STEM principles (Committee on Integrated STEM Education et al., 2014).
Pedagogical Content Knowledge
Pedagogical Content Knowledge (PCK) describes the knowledge of a teacher that
combines content knowledge and pedagogical knowledge. Teachers have a unique skill set that
requires them to have content knowledge and the ability to regenerate that knowledge into others
(Shulman, 1986).
Nature of Science
The Nature of Science is an understanding of how society and science interacts. It is a
metacognitive approach to the field of science that explore the philosophy of science and
combines the field of psychology in its description of ‘science’ (McComas, Clough, &
Almazroa, 1998).

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

17
CHAPTER 2: LITERATURE REVIEW
Introduction
USC and Elementary Afterschool STEM Program Collaboration
A collaboration between USC and Elementary Afterschool STEM Program was
established to identify the scope of work of this project. Elementary Afterschool STEM Program
works inside of a larger entity called the California Afterschool Network. The California
Afterschool Network has various STEM afterschool programs in place. At the behest of the
California Afterschool Network via the Los Angeles County Office of Education (LACOE),
various STEM afterschool programs requests to have needs assessments and evaluations
completed to measure and ensure their effectiveness. For this project, the Small Urban School
District’s afterschool program, Elementary Afterschool STEM Program was linked to USC to
complete the program needs assessment and evaluation. USC has the research background and
structure to complete those types of assessments. Therefore, a collaboration between USC and
Elementary Afterschool STEM Program has emerged to identify and recommend a future course
of action to improve their STEM program. This, this chapter will provide research background
and context, which informed the data collection, analysis, and interpretation for this project.
The purpose of this project was to conduct a needs assessment and evaluation of
the STEM program at Small Urban School District’s (SUSD) called Elementary
Afterschool STEM Program. This literature review chapter presents a theoretical and
practical framework for Elementary Afterschool STEM Program’s approach of STEM
instruction. In this literature review, the importance of STEM was reviewed, including
defining STEM and describing the role of the NGSS in education. It also reviews the
needs assessment and evaluation process. This section also reviews the topic and
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

18
structure of Logic Models. Literature that was pertinent to Elementary Afterschool
STEM Program specifically was reviewed. This includes describing and delineating the
four domains that informed the recommendations of Elementary Afterschool STEM
Program’s STEM program. The four domains that were used are 1. Instructional
Practices, 2. Staff Knowledge and Requirements, 3. STEM Curriculum, and 4. The
Organizational Structure. Lastly, the issues of concern at SUSD will be address
specifically.
Importance of STEM
Definition of STEM
McComas (2014) describes STEM as the acronym that refers to the commonly linked
disciplines of Science, Technology, Engineering, and Mathematics. He continues to describe
STEM as a term used to amalgamate the vital skills that the next generation of learners will have
to develop during their formal education. He essentially uses ‘STEM’ as an eponym for the
important skills that the next generation of learners will have to develop (McComas, 2014).
STEM education is often implicitly described as connecting real-world situations to the
classroom content, i.e. have curriculum that is relevant. STEM is a very loaded term that is used
to describe many things, both explicitly and tacitly.
Despite STEM education being ubiquitous, the definition of it is ambiguous. STEM
started out as an idea from the National Science Foundation (NSF) as a means to apply the
concepts of science and math in a real-world context for students (Sanders, 2009). This lead to
the concept that teaching science in conjunction with other subjects, namely technology,
engineering, and mathematics will lead to increased inquiry, relevance, and interest in STEM
fields in the classroom (NGSS Lead States, 2013).
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

19
Sanders (2009) describes the practice of STEM instruction in the classroom as different
than its connotation as an integrated subject. The typical STEM classroom currently teaches
science in isolation of the other subjects (Sanders, 2009). This can be linked to the fact that there
are many definitions of STEM that are used in practice. Bybee describes the many facets that
use STEM in their curricular activities. He notes that STEM is used to describe “centers”,
“programs”, or “committees” (Bybee, 2013a). A disconnect exists between the idea of STEM
education and actualization of STEM instruction. STEM can also be used to refer to “science
literacy”.
Fang, Lamme, and Pringle (2010) define ‘Science Literacy’ as the capacity to use science
content knowledge in a functional way. They go on to illuminate the increased need of science
literacy because of increased use of technology in our society (Fang, Lamme, & Pringle, 2010).
Their identification of science literacy codifies the integrated nature of STEM with the necessity
of technology to create 21
st
century learners that will solve societal problems. This outlines the
importance of science literacy for STEM teaching.
From this notion of STEM, two concepts emerge: STEM education and STEM
integration. Bybee describes STEM education as the purpose, policy, programs, and practice of
schools and districts to instruct students (Bybee, 2013a). STEM integration has been defined as
the method for making STEM subjects more connected to the “real-world” and relevant to
students (Committee on Integrated STEM Education et al., 2014).
There is another working definition of STEM education. Hoachlander and Yanofsky
(2011) describe STEM education as an interdisciplinary approach that combines the subjects of
science, technology, engineering, and mathematics, versus separating them, for the goal of
creating meaningful learning experiences for students (Hoachlander & Yanofsky, 2011). Their
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

20
expansion of the definition of STEM education is insightful because it incorporates the outcomes
of students’ learning as part of the definition. Despite the term ‘STEM’ being used in a variety
of ways, the impact on students should be consistent and a common goal for educators. A STEM
education should improve a student’s understanding of how things work and their ability to use
technology (Bybee, 2010b). Science literacy is described as the knowledge and skills students
should attain during their time in the K-12 educational system (American Association for the
Advancement of Science, 1993). By using the concepts of STEM education and Integrated
STEM, they can provide a foundation for schools to promote science literacy.
History of STEM in Education
The problems of science instruction have long been identified (Smith, 1969). Science
instruction has historically focused on the cognitive development of students at the expense of
developing interests and fostering inquiry. Osborne, Simon, and Collins (2003) state the lack of
instructional emphasis on the affective domain of students has caused a lackluster interest in
science (Osborne, Simon, & Collins, 2003). After school programs can serve the role of
providing affective stimulation for students in science beyond content only (Mayer, 2011;
McComas, 2006). McComas’ description of science instruction elucidates how science can be
taught in an environment that activate students’ affective domain and cause an interest in STEM
fields.
Benchmarks for Science Literacy first described and illuminated the importance of
science standards in education. The Benchmarks for Science Literacy established the foundation
for the Integrated STEM approach for science education by stating: “The basic point is that the
ideas and practice of science, mathematics, and technology are so closely intertwined that we do
not see how education in any one of them can be undertaken well in isolation from the others ”
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

21
(American Association for the Advancement of Science, 1993). The Benchmarks articulate the
integrated nature of science, mathematics, and technology, but the subsequent content standards
did not reflect that integrated relationship of the disciplines (California Department of Education,
2000). Hence, the Next Generation Science Standards (NGSS) were created to make the tacit
explicit.
The NGSS were created in response to the need to have explicit knowledge of science
and engineering integrated in K-12 schools. The Committee on a Conceptual Framework for
New K-12 Science Education Standards illuminate the new vision of science education as
incorporating the nature of science, science and engineering practices, crosscutting concepts, and
the technology of the 21
st
century (Committee on Conceptual Framework for the New K-12
Science Education Standards, 2011). This transition phase, from old science standards to new, is
unique and allows for creativity of implementation (Spiegel, Quan, & Shimojyo, 2014).
Transition to an integrated approach. For the history of science standards, the science
content standards have been isolated, as have all subjects’ standards. The creation of the
integrated STEM approach was to provide students with the curricular framework of integration
of previously, isolated subject matter. This integration is more closely aligned with the reality of
science (Committee on Integrated STEM Education et al., 2014). The integrated approach
explicitly describes engineering and technology as applications of science and math.
The integrated approach parallels the current effort of the Common Core State Standards
(CCSS). The CCSS are designed to prepare students for college and career by identifying the
knowledge and skills needed for successful matriculation into post-secondary life (National
Governors Association Center for Best Practices, 2010). Along with the standards, there is an
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

22
identification of 21
st
century skills that complement the standards. STEM education and the
NGSS can support the 21
st
century skills needed by students (Hilton, 2010).
Increase attention in STEM education. STEM has received increased attention in the
past decade because of the desire to improve science education’s curriculum and instruction
(National Academy of Engineering and National Research Council, 2014). STEM has increased
in importance because of the perception of China and India’s increased capacity for science and
math aptitude (Sanders, 2009). Also, it has been noted that science and math instruction has
lacked real-world and contextual connections in its methodology (Committee on Highly
Successful Schools or Programs in K-12 STEM Education, Board on Science Education, Board
on Testing and Assessment, Division of Behavioral and Social Sciences and Education, &
Council, 2010). An effect of this perception has caused more money to be awarded to STEM-
based educational programs, which increases the interest of districts to adopt STEM incentives
(Zuger, 2012).
McComas et al. (1998) described a National Science Foundation survey that concluded
that 40% of Americans were interested in science. But, despite that reported interest in science,
the level of understandings of how science operates remains near nil. The report found that 64%
of respondents had level IV science understanding, the lowest possible level of the survey
(McComas et al., 1998). These findings illuminate the need for an approach to science education
that is more effective.
In summary, STEM has an amorphous definition that is used to mean many different
pedagogical concepts. Vasquez (2013) has created a working definition of STEM. Her
definition of STEM acknowledges the four titular subjects, but also describes the need for the
content to be infused in a rigorous and relevant learning environment (Vasquez, 2013). This
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

23
definition goes into the realm of becoming a foundation or a philosophy of teaching, which
elucidates how STEM teaching is a paradigm shift from the typical science instruction to
becoming a process-orientated and skills-based pedagogy for the 21
st
century.
Why STEM Education is Important
Literacy in STEM education. Along with being literate in a general sense, there is a
growing need to have schools to teach for the goal of science literacy (American Association for
the Advancement of Science, 1993; Bybee, 1997; John, 2007; McComas, 2004). Science
literacy and an understanding of how science operates within our society is needed by each
citizen (Committee on Conceptual Framework for the New K-12 Science Education Standards,
2011). Science literacy is connected to the 21
st
century skills and needed to be job-attractive in a
global economy (Brooks & Normore, 2009; Hilton, 2010; Spring, 2008). Science literacy skills
include adaptability, complex communication skills, and the ability to solve nonroutine problems
(Hilton, 2010). The listed skills are perennial and transferable to post secondary careers.
Changing technology. As the 21
st
century has started and progressed, technology has
changed rapidly. Technology in the field of education has also seen an increase via the use of
laptops, tablets, and mobile devices. Ball and Forzani (2009) describe the innovative use of
technology as a necessity for the improvement of schools and the professional teacher (Ball &
Forzani, 2009). The Committee on Highly Successful Schools or Programs in K-12 STEM
Education (2011) describes the use of technology in classroom as an essential part of effective
instruction (Committee on Highly Successful Schools or Programs in K-12 STEM Education,
2011).
Changing standards. From The Benchmarks for Science Literacy, the National and
California Content Standards were created (Washington, 1996). These standards focused
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

24
primarily on the set of facts and concepts that are needed for science literacy, but did not address
how to teach the standards or developing skills in science practice. The NGSS addresses the
later concerns and frames the former (Committee on Conceptual Framework for the New K-12
Science Education Standards, 2011).
The Role of NGSS
Along with the CCSS, a new set of science standards were written and are currently in the
process of being adopted by various states. The new science standards are called the Next
Generation Science Standards (NGSS) (NGSS Lead States, 2013). These new standards are
meant to address the current lack of science understanding that exists among Americans (Bybee,
1997; Committee on Conceptual Framework for the New K-12 Science Education Standards,
2011). There is an identified need for improved science education (Committee on Integrated
STEM Education et al., 2014). One process for improving science education is by improving its
standards. Once the standards are improved, then improved instruction can subsequently occur.
The NGSS are designed to not only frame science content, but also inform science instruction
(Bybee, 2013b; Committee on Integrated STEM Education et al., 2014). Learning science
through proper science education is the goal of the new science standards (Committee on
Conceptual Framework for the New K-12 Science Education Standards, 2011).
The NGSS attempts to expand science instruction by organizing the standards in a way
that illuminates the crosscutting nature of the concepts and STEM disciplines (Bybee, 2013b).
Table 1 describes the practices and crosscutting concepts in the NGSS, which is a conceptual
shift from the previous California Content Standards. The practices in the NGSS are intended to
help students develop habits of mind and skills that professional scientists and engineers use.
The cross cutting concepts are intended to provide an organizational structure that makes sense
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

25
to students and bridge individual content areas for maximum concept integration (NGSS Lead
States, 2013).
Table 1.
Practices and Crosscutting Concepts in the NGSS.
Practices of the NGSS Crosscutting Concepts in the NGSS
Asking and defining problems Patterns
Developing and using models Cause and effect: mechanism and
explanation
Planning and carrying out investigations Scale, Proportion, and quantity
Analyzing and interpreting data System and system models
Using mathematics and computational
thinking
Energy and matter: flows, cycles, and
conservation
Constructing explanations and designing
solutions
Structure and function
Engaging in argument from evidence Stability and change
Obtaining, evaluating, and communicating
information

Note. The practices and crosscutting concepts in the NGSS (NGSS Lead States, 2013).
Elements of STEM Education
Literacy. Literacy is an essential part of STEM education because some of the practices
outlined in the NGSS infer literacy skills (NGSS Lead States, 2013). For example, the practices
of “developing and using models”, “constructing explanations, and designing solutions”, and
“obtaining, evaluating, and communicating information” requires students to utilize their literacy
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

26
skills by transferring their ideas through a medium, either speech or written. Fang et al.
described using the literacy skills of observing, identifying, describing, and experimentally
investigating natural phenomena within the expression of language and the written form for
demonstration of knowledge (Fang et al., 2010). There is a necessity for incorporating literacy in
STEM education.
To expand on literacy instruction and how afterschool programs can supplement literacy
instruction, Moje, Ciechanowski, Kramer, and Ellis (2004) argued that science literacy ought to
be taught in a “third space”. They define the “third space” as a hybrid of the learner’s home and
community (first space) and the structured realm of traditional education (second space). The
“third space” is a place where education and culture meet to enable cultural relevancy into a
learner’s educational experience (Moje, Ciechanowski, Kramer, & Ellis, 2004). Afterschool
programs can be built as a “third space” that can serve students’ needs and foster learning
experiences because they can create an environment that is a blend of traditional instruction, but
create a more community-based environment for learning (Jensen, Dabney, Markowitz, &
Selsor, 2003).
Relevance. A key element of content standards and their acceptance by educational
leaders is their relevance and usability. Students learn more when the subject matter is relevant
and connected to the real-world (Fensham, 2009). An essential feature of STEM teaching is the
nature of connecting content to students’ lives. Afterschool programs have a duty to demonstrate
the relevance of the subjects of their instruction. Moreover, afterschool programs have the
adaptability and structure to provide that relevance through its instruction and facilitation.
In sum, the above section suggests that an increased attention to STEM, STEM education,
and STEM instruction are based on changing social and economic needs of our culture, low
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

27
achievement of sub-groups of students, and the new developments in educational theory and
research. There is a need for programs focusing on STEM to evaluate the effectiveness and
collect data on their performance.
Areas of Focus for the Needs Assessment and Evaluation of SUSD
This project will combine a need assessment with some elements of program evaluation.
It will be informed by a logic model, which captures the ideal program inputs and outputs as a
way to guide data collection. The process of needs assessment and evaluation are described
here.
Needs Assessment
The purpose of a needs assessment is to accomplish two tasks, first, they identify what
should be and what currently is (Altschuld & Kumar, 2010). This step allows the actual causes
of the problems to be identified. When the proposed causes are identified, then, secondly,
recommendations to the organization can be made (Altschuld & Kumar, 2010; Gupta, 2007).
For this project, SUSD has identified goals for their staff and their students. Using a
needs assessment model at this site allows for the measurement of their identified objectives for
A) increasing staff competence, confidence, and motivation in STEM program implementation
and B) increasing student interest, engagement, and knowledge of STEM processes and concepts
(California Afterschool Network, 2013). Once the proper needs are identified and accurately
measured, then the next phase of an action plan can be implemented. The needs assessment and
evaluation approach will allow the measurement and development of their proposed objectives.
Evaluation
Evaluations are conducted in organizations to gather data so that decisions can be made
(Alkin, 2011). SUSD has a number of existing instructional programs operating in their
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

28
Elementary Afterschool STEM Program afterschool program. The evaluation of the exiting
Elementary Afterschool STEM Program was conducted to determine the alignment of their
stated goals and actual praxis.
Program in Education Afterschool and Resiliency (PEAR) has designed a method of
measuring and evaluating afterschool STEM programs. Their method relies on four elements of
data collection. Each element has three dimensions to add context to data collection. The four
elements and dimensions are described in tables 2-6:
Through observation each dimension of each element can be measured and analyzed for
its impact on STEM learning for the participants (Program in Education Afterschool and
Resiliency, 2014a).
Logic Models in Organizations
Logic models are used as organizational tools to demonstrate relationships among
variables. The creation of a logic model allows variables to be displayed pictorially. Also,
Logic models identify the variables to be measured over time and illuminate the organization’s
plan for the future. A logic model allows the stakeholders to see the designed outputs, expected
outcomes, and long-term goals and ensure that a structured plan for goal attainment is manifest.
Essentially, logic models allow for organizations to plan for success (W.K. Kellogg Foundation,
2001).
Four Domains of Analysis
To organize and focus the areas of data collection and analysis, four areas of program
functioning were examined. These domains provided the foundation for the recommendations
for Elementary Afterschool STEM Program. The four areas are: A) Instructional Practices, B)
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

29
Staff knowledge, C) Curriculum Structure and Design, and D) Organizational Factors. The
following section describes each of the four domains and their connection to STEM education.
Instructional Practices
Instructional practices are commonly referred to the science of instruction (Rueda, 2011).
Instructional practices vary in their notion of strategies, from increasing literacy, making science
relevant, connecting to one’s culture, and/or using an inquiry-based approach.
Criteria for STEM Instruction
Inquiry. Lee (2002) describes the role that inquiry plays in science learning. She also
notes that most inquiry methods do not address the needs of diverse learners (Lee, 2002).
Developing a process if inquiry instruction for diverse learners will be essential for the
population at Elementary Afterschool STEM Program. Although Lee (2002) identifies the need
for inquiry in science instruction, she articulates areas of need for diverse learners in an inquiry-
based science classroom. One of Lee’s guidelines for meeting the needs of diverse learners is
adding more discourse in science instruction and less rigidity in pedagogy. She also contends
that teachers need to improve and add students’ cultural values in instruction. Lastly, she
contends that children’s cognition needs to be addressed for constructing an inquiry-based
classroom (Lee, 2002).
Relevancy. Tomlinson (2009) argues that teaching needs to incorporate students’
readiness, interest, and learning profile. She describes readiness as a student’s preparedness for
the instructional goals and interest as the attention and investment of learning by the learner.
The learning profile is the student’s culture, intelligence preference, and the interaction of each
of those items (Tomlinson, 2009). An afterschool program needs to organize and structure its
program to meet those qualifications.
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

30
Hand, Noton-Meier, Staker, and Bintz (2009) have developed a pedagogical tool to bring
relevancy into science instruction called the Science Writing Heuristic (SWH). The SWH
involves a process of group work that has students: ask guiding questions, identify procedures
that will be safe and test those questions, collect and display observations from those tests,
evaluate the evidence and dialogue about their claims, then reflect on the entire process. This is
a process that encourages discussion and argumentation to validate results and claims in science
(Hand, Noton-Meier, Staker, & Bintz, 2009). The SWH can be used as a way to stimulate the
affective domain of students.
Literacy. Au (2003) argues that having a balanced literacy approach, that is built on the
foundations of constructivist theory, diverse learners are more successful in becoming literate
(Au, 2003; DeBoer, 2000). It is the need of an afterschool program to incorporate literacy
instructional practices to supplement the needs of diverse learners (Lee, 2005). STEM
instruction benefits when strategies for literacy are utilized (Fang et al., 2010). The notion of
literacy is a tacit feature in science instruction; there is a need to make it more prominent in
STEM education especially in light of the adoption of the new standards. The expectation for
science classrooms is that literacy will be the foundation of instruction (Lee, Quinn, & Valdés,
2013; National Governors Association Center for Best Practices, 2010; Valencia & Wixson,
2013).
Fang et al. (2010) identify an instructional approach that uses literacy development to
foster inquiry in the science classroom. They describe various approaches that increase literacy.
For example, they describe how to use trade books in a literature circle fashion that can help
students relate to science and get more in depth to a particular topic than a textbook (Fang et al.,
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

31
2010). Using literacy as an approach to inquiry can help students foster their interests in STEM
subjects.
Assessment. Noam et al. (2004) describes the social and academic needs for afterschool
programs or out-of-school space as a place for youth development. They also argue that
afterschool programs or out-of-school programs can be used as a transitional space for learning
and academic growth. From that, they recommend that the case study approach to evaluating
these programs as the only method for creating an accurate evaluation. This ensures that the
program is measured in their context and has an accurate and meaningful evaluation (Noam et
al., 2004).
A measurement tool for afterschool programs was created to measure the diverse nature of
afterschool programs. The tool, Dimensions of Success (DoS) was developed as an
observational tool specifically for afterschool programs or out-of-school learning environments.
It is designed to measure various dimensions of afterschool STEM programs in their specific
context, since there is a large variability within afterschool programs. It was designed by the
Program in Education, Afterschool, and Resiliency (PEAR) to evaluate STEM Programs on 12
dimensions (Program in Education Afterschool and Resiliency, 2014b). This tool will facilitate
the data analysis of the classroom observations at SUSD.
Summary of Key Points
Instructional practices of an afterschool STEM program should have inquiry, relevancy,
and engagement in their structure. Developing and integrating literacy in STEM instruction can
supplement and enhance STEM program effectiveness. To ensure program effectiveness,
programs need to be assessed and the Dimensions of Success tool can be used as that assessment
component.
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

32
Staff Knowledge of Content and Pedagogy
This concept of knowledge as a domain of assessment is important. This domain
measures the level of knowledge that is needed for individuals and groups to accomplish their
goals (Rueda, 2011). This domain addresses the notion of Pedagogical Content Knowledge
(PCK) and the specific content knowledge needed to work in a STEM afterschool program.
Lastly, motivational factors were reviewed. For programs to be successful, a certain level of
competency is required of its staff. The literature has various definitions and description as to
the meaning of a competent teacher. The National Council on the Acceptation of Teacher
Education (2006) describes the preparation and education of teachers has a positive effect on
student learning, therefore developing teachers is essential, if student knowledge outcomes are
desired (National Council for Accreditation of Teacher Education, 2006).
Pedagogical content knowledge. Ball et al. (2008) describe the impact that teachers’
knowledge of content play in student learning, but content knowledge is only context to the
pedagogy that is needed for instruction (Ball, Thames, & Phelps, 2008). Their work is based on
Shulman’s concept of pedagogical content knowledge (PCK) (Shulman, 1986). They expand on
his original ideas by looking at math instruction and identifying the problems that arose in a
mathematics classroom. They argue that there is a need to develop Shulman’s notion of PCK.
They created categories of PCK that are essential to teaching content, since knowledge of
content is not sufficient in being able to teach that content. The first category is ‘common
content knowledge’, which means teachers must have knowledge on content that can be used
beyond the subject. The second category is ‘specialized PCK’, which means that teachers need
to understand the most minute of details of content, so they can help all students understand a
concept. Lastly, the third category is ‘knowledge of content and student’, which means that a
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

33
teacher should have knowledge about where students’ misunderstandings could arise during
instruction (Ball et al., 2008). The ability to understand content and pedagogy in these varied
ways is essential an essential skill in effective teaching (Carney & Indrisano, 2013).
The Power of Discovery states that collaboration and partnerships between out-of-school
programs, traditional classroom instruction, business & industry, and higher education are the
foundation of their model (California Afterschool Network, 2013). This type of horizontal
collaboration reflects an understanding of the need to have well trained professionals working in
their programs. Furthermore, these collaborations allow for resources to be combined and
readily available to staff to use as curricular elements.
Teachers and instructional staff, in this case, are the most important instructional factors
in the classroom learning formula (McComas et al., 1998). That being said, it is imperative that
the teachers and staff at SUSD have the appropriate knowledge for STEM instruction. Various
authors describe the level and type of knowledge needed to be effective at STEM instruction
(Anderson et al., 2001; Hess, 2013; Read, 2013).
McComas et al. (1998) define ‘Nature of Science (NOS)” as a fertile hybrid arena which
blends aspects of social studies combined with cognitive science to describe what science is, how
it works, how scientists operate as a social group, and how society itself interacts with science
(McComas et al., 1998). From this description, it becomes clear that the subject of NOS is not
about the facts of science, but the interaction of science knowledge with society, the
environment, and the learner. The concept of NOS is explicitly compatible with STEM teaching.
A deep understanding of NOS is a requirement of STEM teaching (Herman, 2010).
Matthews (1998) explains that teaching the nature of science to students is necessary so
they can distinguish between good science from parodies and pseudoscience (Matthews, 1998).
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

34
The ability of a teacher to be able to instruct science comes from their knowledge of the subject.
The teacher’s knowledge of the subject impacts the learning environment of the students.
Evans (2011) describes the role that teacher qualifications make on a mathematics
classroom. His research identified the difference that exists in qualified teachers and the impact
of this disparity on student learning (Evans, 2011). Highly Successful Schools or Programs for
K-12 STEM Education (2011) found that the preparation of a teacher has an impact on students’
test scores in Math and Science. The report also identifies that although expertise in content and
pedagogy are needed in teaching STEM content, it is often the case that there is severe lack of
qualified teachers in the classroom (Highly Successful Schools or Programs for K-12 STEM
Education, 2011).
Summary of Key Points
Having teachers develop an understanding of the nature of science improves
implementation of science concepts in teacher’s instruction (McComas et al., 1998). It is
imperative that SUSD’s staff has a precise and functional understanding of NOS, so their pupils
can have qualified models of science educators to learn from. Using the research notion of NOS
allows for teacher development of PCK. Currently STEM is used to refer to the essential tenets
of the NOS, which has been described through research and can be used to help train STEM
teachers for their duty as classroom teachers. Organizations that are in the business of education
must have staff that are knowledgeable themselves of the content that will be delivered.
Curriculum
For STEM education, there is a need for a redesign of the standard mode of curriculum
development. Drake and Burns (2003) describe an approach to curriculum design that unifies
subjects and learning experiences (Drake & Burns, 2003). Curriculum in this sense is the
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

35
structure, the materials, and the purpose of the STEM content that is contained in this afterschool
program. The STEM approach requires that curriculum be transdiciplinary in nature.
Fullan, Hill, and Crévola (2006) describe that learning in students can only occur when
instruction is organized and focused (Fullan, Hill, & Crévola, 2006). This means that the
curriculum of an afterschool program needs to be organized and focused on a singular purpose.
Drake and Burns (2003) outline a curriculum development plan that makes explicit the
multidisciplinary nature of knowledge. They start with a “backward design” approach of the
curriculum plan. Then they identify the evidence to support the student learning and finish with
describing the types of learning experiences that should be incorporated to support the plan.
Along with these elements, they outline the fundamental questions that should be answered in
the curriculum design: “what should students know”, “what should students do?”, and “who
should students be?”. They describe this framework as the bridge (Drake & Burns, 2003). The
bridge can be used to design curriculum that integrates many subjects. Their questions are not
content specific and can be used to frame any curriculum design plan.
STEM as a process of teaching has many forms. The “right” method of STEM teaching is
not found in the literature. There are certain criteria of effective and meaningful learning in
students. Gee (2001) describes a need for a curriculum to address the cultural needs of the
student to be effective and foster increased literacy (Gee, 2001). Bybee (2013) describes STEM
curriculum as needing to fulfill the “4 Ps”: purpose, policy, programs, and practices. By
purpose, he means that the activity has an articulated goal. He explains the policies as needing to
support the purpose by creating concrete translations of the purpose, so the goals can be
accomplished. The program outlines the materials, resources, and technology that will be used
to realize the curricular means. Lastly, the practices are the specific actions and pedagogy that
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

36
the staff will bring to support the goals of the STEM curriculum (Bybee, 2013b). An effective
STEM curriculum can look differently across sites, but need to have fundamental criteria and
structure.
The framework of the NGSS allows for connections to other subjects, in fact, it is
encouraged. The NGSS standards even have a section where they connect the CCSS to a
Disciplinary Core Idea (Bybee, 2013b). Table 2.1 above shows the practices and crosscutting
ideas of the NGSS. Those elements illuminate how the knowledge and concepts in science are
ubiquitous in nature and need to be taught that way. STEM instruction in an afterschool program
needs to capitalize on the integrated nature of the NGSS and STEM practices.
Summary of Key Points
Curriculum is the integral plan of instructional design of an afterschool program. The
curriculum of an afterschool program has the ability to be flexible, so student interests and needs
can be incorporated. An essential feature of curriculum design is its dedication to integration.
The curriculum needs to be designed from a “backward design” approach, but be framed by
standards. The new NGSS is design to facilitate curriculum design that is multidisciplinary in
nature.
Organizational Elements
Another domain that will be examined for this needs assessment is the Organizational
domain. This domain will look at the additional needs required after the instructional practices,
knowledge, and curriculum domains have been satisfied. Once it occurs that the staff is
knowledgeable, the instructional practices are sound, and the curriculum is sufficient and a
problem still exists, then the organizational structure needs to be evaluated. In this case the
organization could be lacking in allowing goals to be achieved (Rueda, 2011). This section will
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

37
look at the training procedures, the process of accomplishing work, their resources, and the
facilities.
Organizational culture. Schein (2010) defines organizational culture as the language,
meaning, and ultimately the “social order” of the organization. He calls it the “here and now” of
the organization. He describes it as the cohesive and background context that determines how an
organization functions and operates. The culture of an organization is impacted by the leader’s
vision and structure of that organization. The mission and vision of an organization are products
of a leader’s beliefs and structure of that organization, which influences the culture and “social
order” (Schein, 2010). He describes the notion of ‘culture’ as an abstraction, but its affects are
pivotal in organizational change.
Peterson and Deal (2002) have created a widely accepted definition for “school culture”.
They say that organizational culture is the “norms, values and beliefs, rituals and ceremonies,
and the symbols and stories” that make up a school (Peterson & Deal, 2002). The identity of a
school is determined by its culture. The ability for an organization, like a school or afterschool
program, to accomplish its goals, a positive and cohesive academic culture needs to be
established.
Muhammad (2009) describes the importance of schools developing a specific culture that
either allows them to accomplish their mission or impedes their ability to foster student learning.
He found that schools have four types of persons in an educational organization; believers,
tweeners, survivors, and fundamentalists. The believers and the fundamentalists are on opposite
ends of the desire for school change; the believers want to change policies and programs for
school change, while the fundamentalists do not want to have change. Each view has a valid and
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

38
impactful stance, but it is the duty of the organizational leader to ameliorate their views and do
what is best for the students (Muhammad, 2009).
Organizational goals. Organizational goals are needed to reach the desired level of
performance by an organization. Rueda (2011) describes goals as occurring in three states:
global, intermediate, and performance. He states that global goals are long term and are
tantamount to an organization’s mission statement. They are a declaration of the purpose of an
organization. Intermediate goals are designed to deconstruct the mission statement into its
component parts for manageable tasks. Lastly, the performance goals are the measureable
objectives that need to be accomplished in the short term (Rueda, 2011).
Organizational barriers. There are a variety of things that can be considered
organizational barriers. Ultimately, the organization’s culture is created through the processes
and materials of the organization (Clark & Estes, 2008). The processes and materials that will be
looked at for Elementary Afterschool STEM Program will be staff training, available planning
time for teaching staff, access to resources, and the space utilization.
Resources. Jensen et al. (2003) describe the environments needed for learning to occur.
They outline the physical requirements that are conducive to learning. In their work, they
articulate the right amount of size, light, color, temperature, and noise that have been measure to
support learning in classrooms (Jensen et al., 2003). Information of this sort can be used to
design the resources needed for Elementary Afterschool STEM Program.
Summary of Key Points
Organizational culture affects all aspects of the organization. The structure, processes,
and materials must be understood in order for the organization to achieve their mission and
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

39
objectives. Organizational culture is created through training, policies, resources, and they
utilize their space.
Issues of Concern at SUSD
The issues of concern of SUSD are taken from the Scope of Work that was established by
LACOE. This document outlines the processes and duties of this project. It also identifies the
responsibilities of the collaboration that exists between Elementary Afterschool STEM Program
and SUSD.
Elementary Afterschool STEM Program’s Stated Goals
The goals stated for years one and two are aligned to the research on STEM practices.
Elementary Afterschool STEM Program set goals that combined student learning goals, staff
knowledge goals, and their organization. Elementary Afterschool STEM Program’s goals were
identified from the “Scope of Work” that was created from the agreement between USC’s
Rossier School of Education and SUSD. Elementary Afterschool STEM Program also created a
Strategic Action Plan (SAP) for 2014-2015, which was used to generate their organizational
goals. The progressions of their goals are listed below.
For year one. There is a focus on “attitude: about STEM; embed LIAS principles
(learning is active, collaborative, meaningful, support mastery, and broadens horizons) in STEM
programming.
Staff goal. Staff can lead STEM learning opportunities, “I understand various STEM
processes, I know how to ask good questions that lead learning; and I know how to focus on
STEM learning goals.”
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

40
Youth goal. Youth can choose to be a STEM professional is that is something I desire; I
understand the tools that STEM professionals use in their work; and I am confident that I can
participate in STEM learning opportunities successfully.
For year two. There is a focus on Science and Engineering content in physical, earth,
and life sciences, build on attitude toward STEM learning and STEM processes, have program-
wide Science and Engineering Fair (no individual project “judging”)
Staff goal. Staff can deliver STEM content by building my own understanding of the
information and developing my facilitation skills to support student learning; I can apply key
STEM processes to the implementation of STEM learning consistently; and I understand the
steps in strong STEM programming.
Youth goal. Youth have learned key STEM content through active hands-on and minds-
on experiences; I can apply the scientific method and /or engineering process to discovering the
answer to my own STEM-related question; and I am confident I can participate successfully in a
Science Fair as part of a team.
Instructional Practices
SUSD would like to ensure this project gives them a list of effective STEM programs.
Staff
SUSD would like to have feedback on the Professional Development they deliver to their
staff.
Curriculum
SUSD would like feedback on the STEM program they use for their curriculum.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

41
Organizational
SUSD would like suggestions and recommendations, based on the findings, for a future
direction of Elementary Afterschool STEM Program’s STEM program. They would also like to
create evaluate measures for their program.
Measuring Outcomes in STEM Programs
There is a program tool that measures the effectiveness of out-of-school (afterschool)
STEM programs. It is called the Dimensions of Success (DoS) tool. This observational tool was
developed by the Program in Education, Afterschool, and Resiliency (PEAR) center in Harvard
University to measure the impact of STEM programs. The DoS tool measures twelve indicators
via rubrics (see tables 3-6) (Program in Education Afterschool and Resiliency, 2014a). This tool
was used to measure the outcomes of Elementary Afterschool STEM Program’s STEM program
during the observations of classrooms.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

42
CHAPTER 3: METHODOLOGY
Introduction
The purpose of this study was to collect data on Small Urban School District (SUSD) and
their afterschool program, Elementary Afterschool STEM Program, to determine the current
needs of their STEM implementation. This data will be analyzed to measure their progress at
achieving their goals of implementation. The analysis of this data was completed to create
recommendations for SUSD in order to realize their organizational goals (Altschuld & Witkin,
2000).
Site Description
Small Urban School District
Small Urban Schools District (SUSD) has an after school program called Elementary
Afterschool STEM Program. Their ultimate aim is to improve students' interests in STEM fields
through instruction, projects, and engaging activities (see Appendix F). SUSD is school district
that is in southern California and has nine elementary schools, three high schools, and two
learning centers. Like most schools, they have an afterschool program that supplements daily
instruction and provides a healthy place for their students to congregate during the hours after
school. Ultimately, they want to create an interest in STEM fields so that the STEM pipeline
(i.e. STEM careers) can be partially filled by their students in the long term.
Connection to Elementary Afterschool STEM Program
SUSD is a school district that has fourteen schools, nine of which are elementary schools.
Elementary Afterschool STEM Program is currently working with seven of SUSD’s elementary
schools. Each site has approximately 100 students participating in the Elementary Afterschool
STEM Program (Bellflower Unified School District, 2014b). Elementary Afterschool STEM
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

43
Program is the afterschool program that delivers STEM instruction to SUSD. They are
contracted from Los Angeles County of Education (LACOE) and work in conjunction of SUSD
to deliver educational services in the form of afterschool instruction to the participating sites.
SUSD Mission Statement
This is the mission statement that is on their website. “The mission of the Small Urban
School District is to provide the pathway for all students to attain the expertise and develop skills
of academic excellence that will empower them to:
• Become lifelong active learners
• Demonstrate respect for themselves and others in a dynamic, diverse and global society
• Become responsible, informed, productive, independent and contributing citizens
• Perform successfully in their chosen field and in society
We Believe That
• Every student deserves to learn every day
• Positive relationships and a strong sense of community connect students to learning
• Teachers who challenge and care for students make a significant impact on students’
lives
• Standard of Excellence, Nothing Less will be achieved from every individual in our
learning community
We Commit To
• Providing each student with an appropriate and challenging educational experience
• Maintaining a respectful environment that fosters learning through positive relationships
among students, adults and our diverse community
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

44
• Hiring and retaining only the best educators and paraprofessionals while investing in their
success
• Providing quality education based on high standards, effective practices, continuous
improvement, and innovation” (Bellflower Unified School District, 2014a)
Elementary Afterschool STEM Program Project Context
Location and number of sites. The location of data collection occurred at different
sites, depending on the data collected. For the surveys (student, tutors, and site leads), all sites
for Elementary Afterschool STEM Program were asked to participate and subsequently
surveyed. Elementary Afterschool STEM Program operates within seven sites in SUSD. Only
the seven participating elementary schools were surveyed for this project. The students, tutors,
and site leads were surveyed from those sites. The administration was surveyed at the district
office and participating contracted employers that the district office selected. The contracted
employer trains and hires the tutor staff at Elementary Afterschool STEM Program.
Observations were conducted at a single site.
The observations occurred at a single site. The site that was observed was Zeta
Elementary. Elementary Afterschool STEM Program selected this site for observations because
they indicated that their goal is to create a model that can be replicated and implemented in
SUSD’s other elementary sites. The intention was that the observations and recommendations
from this site could be used as a model of STEM education implementation through out the
remaining participating schools.
Student Demographics. Each of the seven sites has similar student demographics. Each
elementary school has K-6
th
grade. Each school has a similar make up of student ethnicity data.
The schools have a large Hispanic/ Latino population (65-85%), then a similar Black population
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

45
(20-30%), and a small White population (5-7%). Other groups do not represent more than seven
percent of the demographic make up of the school. Table 2 describes the ethnicity data for the
Elementary Afterschool STEM Program’s schools in SUSD. Along with similar ethic data, each
school has similar socioeconomic demographics. The percent of students on free/ or reduced
lunch is 84.9% (California Department of Education, 2014).
Table 2.
Combined Demographic data for the schools that participate in Elementary Afterschool STEM
Program

Enrollment Percent of Total
American Indian or Alaska Native 22 0.47%
Asian 130 2.68%
Native Hawaiian or Pacific Islander 48 0.99%
Filipino 119 2.45%
Hispanic or Latino 3589 73.98%
Black or African American 623 12.84%
White 233 4.80%
Two or More Races 86 1.77%
None Reported 1 0.02%
Total 4851 100.00%

Site Staff. Elementary Afterschool STEM Program has various staff members that form
its organizational structure within SUSD. The Elementary Afterschool STEM Program
coordinator and director worked with Elementary Afterschool STEM Program and SUSD during
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

46
the time of data collection and analysis. Along with the district staff, each site has a site
coordinator that oversees instruction and the tutors that lead instruction and classroom activities.
Elementary Afterschool STEM Program works with various entities for staffing needs, training
needs, and curricular needs.
Instrumentation
Student Interest in STEM Survey
Krapp and Prenzel (2011) describe interest in science as being content specific in nature
and not solely based on enjoyment. They used the items from the PISA science survey and
analyzed the pertinent items for their results. The items from Krapp and Prenzel (2011) and PISA
(2007) influenced our survey items relating to “interest in learning science”, “enjoyment of
science”, and “participation in science related activities” (Krapp & Prenzel, 2011; Organisation
For Economic Co-Operation Development, 2007). The Elementary Afterschool STEM Program
student survey had items that were created based on their recommendations. Our survey was
designed to measure how much students enjoy doing science-orientated tasks during activities.
Student Career Awareness in STEM Survey
Kier et al. (2014) developed a STEM career interest survey for middle school students
(Kier, Blanchard, Osborne, & Albert, 2014). This survey measured the awareness of STEM
careers for middle school students. The goal of STEM career awareness is to create interest and
innovations, while having students recognize their academic strengths. Kier et al. (2014) argues
that interest in STEM begin at middle school, but this survey will attempt to make primary
students aware of possible careers in STEM (Kier et al., 2014). Their survey was designed to
assess the interests in middle school students, so the survey for Elementary Afterschool STEM
Program will be adapted to fit the needs of their serviced grades (grades 3-6). This survey
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

47
influenced certain items on the Elementary Afterschool STEM Program student survey. A copy
of the STEM Careers Survey can be found in appendix H.
Staff Knowledge of STEM Survey
To begin an analysis of STEM program effectiveness, the staff’s knowledge of STEM
must be assessed. Aydeniz and Kirbulut (2014) identify the problems of measuring PCK in
teachers in light of reform efforts. They designed an instrument called Secondary Teachers’
Science Pedagogical Content Knowledge for measuring teacher PCK. Their tool categorizes
responses into three categories: naive, developing, and sophisticated (Aydeniz & Kirbulut,
2014). This tool was used and adapted to influence items for the Afterschool STEM Program
tutor survey. Furthermore, this tool was designed for preservice teachers, which have limited
teaching experience, so the tool can be used to identify areas of need in terms of PCK for an
afterschool teaching staff who are not necessarily certified teachers. A copy of the Secondary
Teacher’s Scientific Pedagogical Content Knowledge instrument can be found in appendix I.
The question items from the student and tutor surveys were influenced by extant surveys,
but were modified to fit the needs of Elementary Afterschool STEM Program. The surveys need
to reflect the needs and interests of Elementary Afterschool STEM Program, so specific surveys
were constructed to collect the pertinent data. The focus of the questions centered on the areas of
needs addressed by Elementary Afterschool STEM Program. These instruments were used
because research has demonstrated their reliability and validity. T he outcomes of these survey
results will be analyzed quantitatively.
Questionnaire
A questionnaire was distributed to the Elementary Afterschool STEM Program
Coordinator. This questionnaire was designed using a semi-structured format, that had two
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

48
multiple choice items and thirteen open response items (Alkin, 2011). This format allowed for
fairly consistent responses from the administration. The questionnaire was emailed to the
coordinator to disseminate to pertinent individuals. The questionnaire was anonymous and
voluntary. The purpose was to get specific data on items regarding Elementary Afterschool
STEM Program that was not available in the surveys.
Document Analysis
The analysis of certain documents from SUSD is a form of data collection of artifacts
from Elementary Afterschool STEM Program. The uses of their documents demonstrate the
alignment of goals via curriculum, agendas, and calendars. These resources are a reflection of
the work and pedagogy of Elementary Afterschool STEM Program. The purpose of analysis is
to measure the programs goals with the tools and resources they are implementing. There is not
a standard, accepted method of analyzing documents. The purpose of the collection of pertinent
documents was to measure the alignments of goals to resources used. The use of this form of
data was used to support the data already collected from individuals and allow for a larger
context.
Observations
Observations are used to see the implementation of Elementary Afterschool STEM
Program’s STEM program in action. The observation is a way to collect data on environments
that allow for detailed and specific data. In education, observational data is a way to measure the
use and impact of classrooms, pedagogy, and facilities.
The observations of classroom instruction were conducted using the Dimensions of
Success (DoS) observation tool. The observational data provided feedback on the Elementary
Afterschool STEM Program’s implementation of STEM education. The DoS observation tool
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

49
has three dimensions in four domains, making a total of twelve dimensions that will be rated
according to the DoS rubric. The tables below show the dimensions along with a brief
description for context.
Table 3.
Dimensions of Success: Domain 1: Features of the Learning Environment
Organization Measures the preparation of material, if
time is allocated well, and if there is a back
up plan
Materials Measures the appropriateness and appeal of
the instructional materials
Space Utilization Measure the types of space, quantity, and if
there are distractions

Table 4.
Dimensions of Success: Domain 2: Activity Engagement
Participation Measure student’s access to activates, how
much prompting id needed, an equity of
student involvement
Purposeful Activities Measures meaningfulness and time spent
guiding students
Engagement with STEM Measures hands-on and minds-on nature of
the activities

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

50
Table 5.
Dimensions of Success: Domain 3: STEM Knowledge and Practices
STEM Content Learning Measures the accuracy of the facilitator, the
connections between concepts, and student
understanding
Inquiry Measures how much watching vs. using
STEM practices
Reflection Measures facilitator’s prompts for
reflection, involvement of students in
reflection, and depth of reflection

Table 6.
Dimensions of Success: Domain 4: Youth Development in STEM
Relationship Measures atmosphere, negative
interactions, and type of intersections
Relevance Measures facilitators determination of
connections of science content and student
involvement in discussions
Youth Voice Measures student voice and ownership

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

51
Data Collection
Sequence of work for this study
Initial Meeting. There was initial meeting on April 23, 2014 that was conducted over
the phone to interview and introduce the project goals and expectations to the participating
members (SUSD, LACOE, and USC). During the meeting, SUSD described their three-year
plan for their current STEM program that described its development and implementation. They
also described a few measurable goals that will allow them to monitor their progress, i.e.
surveys, student learning outcomes, utilization of Bloom’s & the Depth of Knowledge (DOK)
during instruction, and staff confidence. But it was identified that there is a lack of content
knowledge among their staff and confidence about STEM lesson implementation. SUSD
mentioned that each site has limited resources available for its participants, i.e. using only paper,
etc. It was identified that their program’s participants are first through sixth grade students at
seven sites within the SUSD; with approximately 100 students at each site.
Among their goals, SUSD wanted to: A) to increase staff and students positive “attitudes”
towards STEM fields and B) SUSD wanted to participate in the science fair. C) SUSD would
like to teach metacognitive skills to their participants. D) SUSD would like to train their staff in
the use of the scientific method and its application to engineering and the curriculum’s activities.
Also, E) SUSD mentioned they desire to incorporate more technology in their programs along
with F) connecting the STEM activities to real-world, “STEM career clusters”. Lastly, G) SUSD
wants their staff to gain confidence in using STEM lessons from their curriculum, Sci-gineering.
Site Visit. In May 2014, there was a site visit at Zeta Elementary to establish the
guidelines for the project and SUSD’s expectations of the USC research group. During this site
visit, it was determined that an initial needs assessment was already completed for their year one
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

52
implementation of Elementary Afterschool STEM Program. This initial needs assessment survey
was administered by another university. Another university’s data from this survey was included
in our analysis of documents.
From these initial meetings, it was established that further data would still be needed to
be collected in regards the initial needs of the program. The data will determine future strategies
and recommendations for Elementary Afterschool STEM Program. The types of data collected
will be in the form of surveys, observations, a questionnaire, and document analysis.
Scope of Work. A Scope of Work (SOW) was created by LACOE for Elementary
Afterschool STEM Program and USC to identify the scope and sequence of the project. The text
of the actual document is located in appendix A:
Conference Call. There was a conference call on 7/24/2014 to calibrate the parties and
update the project progress. There was a discussion about next steps and a timeline created for
those next steps. During this call it was established that the preliminary another university needs
assessment data would be collected on 7/30/14 and follow up meeting will occur on 8/13/14 at
SUSD.
8/13/14 Meeting. This meeting took place at the SUSD annex office. In the meeting, we
discussed the topics and plans for the project. In this meeting, it was determined that the scope
of the project would look at the goals of Elementary Afterschool STEM Program. Although the
actual goals and data specifics were not identified, the general topics of STEM attitudes,
Elementary Afterschool STEM Program’s alignment, and modes of data collection were
discussed. Also, an overview of the Scope of Work was addressed to illuminate any questions
and discrepancies.
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

53
IRB. This project was designed to collected observational data from human subjects, so
university research approval was needed for data collection. The approval from the Institutional
Review Board (IRB) was achieved on 10/15/2014.
Survey Administration. There were three surveys created. The students had a survey
regarding their experience and perspective of Elementary Afterschool STEM Program. The full
version of the student survey is located in appendix B. The tutors had a survey that asked
questions about their training and experience while working for Elementary Afterschool STEM
Program. Information about the education and demographic data was also collected for analysis.
The full version of the tutor survey is located in appendix C. Lastly, the site leads’ survey asked
questions about the organization and implementation of STEM within Elementary Afterschool
STEM Program. The full version of the site lead survey is located in appendix D. The survey
was launched on 10/21/2014 to begin data collection and was closed on 11/19/2014.
Questionnaire. A questionnaire was designed to collect open-ended data about
Elementary Afterschool STEM Program from the administration. This data is purposive in
nature, because a specific group of participants were desired for input. These results were
analyzed qualitatively to determine the emergent themes from the respondents.
Document Analysis. Curriculum, meetings, and calendar documents from SUSD and
Elementary Afterschool STEM Program were collected for analysis. These documents were
analyzed for STEM content, alignment towards their identified goals, and commensurability with
the SAP to create recommendations of improvement.
During our initial meetings, it was mentioned that another university did and initial Needs
Assessment. That data was collected and evaluated the items to determine if it will be useful for
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

54
our subsequent evaluation. The items that are pertinent and yield results will be reported in the
analysis chapter.
Observations. Classroom observations occurred two times during the data collection
window of 10/21/2014 and 11/19/2014. These times were agreed upon by SUSD and USC. The
researcher conducted the observations of two classrooms during a single visit. The observation
was organized and evaluated using the DoS observation tool.
During the observational session, the researcher collected data on what the teacher was
doing, what the students were doing, what the teacher said, how the students responded, and the
time spent on certain activities. The researcher also listened for student-to-student conversations
to determine the degree of content understanding by the students. Another dimension of the DoS
tool is analysis of the learning environment, so the research also collected data on the abiotic
factors of the learning environment. This data was used for the qualitative analysis portion of
data collection. The data collect will be analyzed using the methods described by DoS (Program
in Education Afterschool and Resiliency, 2014a).
Data Analysis
Survey
The surveys were analyzed and basic descriptive statistics were completed through the
Qualtrics website. The percentages, means, and standard deviations were calculated and
displayed in chapter 4. The data produced from the surveys of the students, tutors, and site leads
are used as the evidence for the recommendations that are in chapter 5.
Questionnaire
The questionnaire was administered through Google Forms. The link for the
questionnaire was given to the Program Coordinator for dissemination to the participants of their
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

55
choice. Data from the questionnaire of Elementary Afterschool STEM Program administrators
were analyzed via qualitative methods. The questionnaire was coded for themes. The themes
are reported in chapter 4. The themes’ categories will be instructional practices, staff
knowledge, curriculum, and organizational elements.
Document Analysis
Various documents were analyzed for alignment with Elementary Afterschool STEM
Program’s goals and SAP. The documents analyzed were agendas, calendars, and staff training
materials. These documents were coded for the emergent themes of instructional practices, staff
knowledge, curriculum, and organizational elements.
Another University’s Initial Needs Assessment Results
Elementary Afterschool STEM Program has already had one year of implementation. In
that initial year, the University of California, Irvine for Afterschool STEM Program, created a
needs assessment survey. That data was collected and analyzed for correspondence to
Elementary Afterschool STEM Program’s current needs and goals and the purpose of this
project. The pertinent results are reported in table 13 and analyzed in chapter 4. The results
from this needs assessment were reviewed for themes to improve implementation for year two.
Observation
Classroom observations were completed on November 5, 2014 for the purpose of data
collection and subsequent analysis. The co-researcher, Rebecca Acosta, collected the
observation data. The data collected was analyzed using the Dimensions of Success (DoS) tool.
The DoS tool is commonly used as a for of evaluation of STEM afterschool programs (Program
in Education Afterschool and Resiliency, 2014a). Results are reported and displayed graphically
in the analysis chapter.
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

56
CHAPTER 4: FINDINGS
Joseph Calmer and Rebecca Acosta coauthored this chapter
Introduction
This chapter presents the findings obtained from the evaluation of Small Urban School
District’s Elementary Afterschool STEM Program. Specifically, this project sought to examine
three areas of Elementary Afterschool STEM Program: 1) evaluating feedback on current STEM
programming during STEM implementation; including staff attitudes toward STEM
implementation and 2) feedback on professional development for STEM professionals;
specifically, those that are delivering the instruction to the students who participate in
Elementary Afterschool STEM Program. This chapter is divided into several sections including
the demographics of both the survey respondents and questionnaire participants, as well as a
discussion of the research findings from the quantitative and qualitative data collected. The
analysis of our findings of this needs assessment and evaluation were centered around the
following framing questions:
1. Does the staff at Elementary Afterschool STEM Program have a positive attitude
toward STEM and their LIAS principles?
2. To what extent is Elementary Afterschool STEM Program implementing the four
components of STEM in their daily plans to achieve their program’s goals?
3. To what extent does Elementary Afterschool STEM Program provide opportunities
for student voice and choice in their learning?
Participant Demographics
The participants for this needs assessment were Small Urban School District Elementary
Afterschool STEM Program tutors, site leads, site supervisors, and students (grades 4-6).
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

57
According to Elementary Afterschool STEM Program’s documents, Elementary Afterschool
STEM Program employed 39 tutors, 7 site leads, and 3 coaches with approximately 700 students
total throughout the program. Table 2 displays the number of surveys that were distributed,
returned, and used for the project analysis. Only surveys that were 100% complete were
considered valid and used for analysis in this project.
Table 7.
Summary of survey responses.

Sent Returned Analyzed Response rate
Site Leads N=15 N=15 N=15
100%
Site Supervisors N=8 N=8 N=8
100%
Students
N=293 N=293 N=266 91%
Tutors
N=24 N=24 N=20 86%

Study Participants
Tutors. Twenty-four tutors responded and completed the tutor survey, which
corresponds to an 86% completion rate. The participants who responded to the tutor survey were
from five of the six Elementary Afterschool STEM Program sites. Delta Elementary and Epsilon
Elementary tutors did not participate in this survey (see Table 3). Sixty-seven percent of tutor
survey respondents indicated that they have been at Elementary Afterschool STEM Program for
fewer than two years. Only two participants indicated that they have been employed with
Elementary Afterschool STEM Program for more than five years.
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

58
Table 8.
Elementary Afterschool STEM Program’s Tutor percentage and participation in this project
from each of their sites
Tutor school site Responses (N=20) Percent of total
Alpha Elementary 2 10
Beta Elementary 7 35
Gamma Elementary 2 10
Delta Elementary 0 0
Epsilon Elementary 0 0
Zeta Elementary 4 20
Epsilon Elementary 5 25
Note. Only 20 tutors responded to this question in the survey. Delta and Epsilon Elementary did
not have any respondents.
Forty-four percent of tutors indicated that their age was between 18-21 years old and 44%
indicated that there age was between 22-25 years. Also, the age groups of 26-29 and above 30
are represented at 8% and 4%, respectively (see Figure 2).
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

59

Figure 1.
Tutor age distribution at Elementary Afterschool STEM Program.
All the tutors in Elementary Afterschool STEM Program were enrolled in a college and
identified their undergraduate matriculation plans. The tutors had a range of degree aspirations.
Although 48% of the participants had an educational focus, only five respondents indicated their
degree focus as a science-related. Similarly, 28% indicated that their future career goal was
STEM-related. Figure 3 displays tutors’ responses about their future degree. According to the
tutor survey responses, 71% of Elementary Afterschool STEM Program tutors are in non-STEM
related fields.
18-‐21
years
old

22-‐25
years
old

26-‐29

years
old

>30
years
old

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

60

Figure 2.
Tutor current undergraduate degree matriculation plan
Site leads. Fifteen site leads responded to the site lead survey and nine completed the
entire survey for a 60% completion rate. The site leads are responsible for managing their tutors,
communicating site effectiveness to supervisors, and ensuring that Elementary Afterschool
STEM Program is implemented properly at their site. Full descriptions of site leads’ duties are
in Appendix G. The site leads’ survey responses indicated that the respondents were 63%
Hispanic and 38% Black or African American, which is a similar representation of the
demographic data of the tutors and students in Elementary Afterschool STEM Program. Seventy-
five percent of the respondents reported their highest degree attained was a high school diploma
while 25% reported that they earned a Masters degree. One hundred percent of the respondents
indicated their highest degree was not STEM related. The degree areas reported were
exclusively in Social Science and Education.
17%

21%

4%

3%

41%

14%

Science
(Biology,
Chemistry,
etc.)

Social
Science
(sociology,
history,

cultural
studies)

Arts

Engineering

Education

Undecided

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

61
Site supervisors. Site supervisors were identified as persons who work in the
administrative offices at Small Urban School District or among one of Elementary Afterschool
STEM Program’s contracts. A questionnaire was sent to Small Urban School District’s
Elementary Afterschool STEM Program coordinator, who then sent the questionnaire to any
person fitting the data collection criteria, as determined by Elementary Afterschool STEM
Program. There were a total of five respondents. Demographic information was not collected on
this group in order to protect the anonymity of the respondents and to maintain confidentiality of
the results.
Students. Elementary Afterschool STEM Program students were identified as those
students who participated in Small Urban School District’s Elementary Afterschool STEM
Program. The program is comprised of seven elementary schools within Small Urban School
District. For the purpose of this project, we analyzed the survey data from students at six schools
who were in fourth, fifth, and sixth grades. Additionally, fourth-grade classrooms at Zeta
Elementary were observed. Delta Elementary was not included in this analysis because none of
the respondents indicated Delta Elementary as their current school site. The distribution of
respondents by grade level can be seen in Figure 3 below.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

62

Figure 3.
Students’ School Attended for Elementary Afterschool STEM Program
Framing Question Analysis
Elementary Afterschool STEM Program Question 1: Staff Attitude towards STEM.
The first framing question was: Does the staff at Elementary Afterschool STEM Program have a
positive attitude toward STEM and their LIAS principles? These survey questions were
designed to determine and assess the staffs’ attitude towards STEM, their understanding of
STEM, and their commitment to the LIAS principles (Learning in Afterschool and Summer) of
Elementary Afterschool STEM Program. This question is based on the following Elementary
Afterschool STEM Program goal to: “Focus on attitude about STEM; embed LIAS principles in
STEM programming” (see Appendix A).
Tutor survey. The tutors’ survey asked questions about tutors’ experiences working at
Elementary Afterschool STEM Program and the amount of training the tutors received on STEM
topics. According to the survey, 95% of the tutors responded that Elementary Afterschool
15%

23%

20%

0%

20%

22%

Alpha
Elementary

Beta
Elementary

Gamma
Elementary

Delta
Elementary

Epsilon
Elementary

Zeta
Elementary

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

63
STEM Program has “always” provided a safe environment, improved students’ academic
performance, helped develop relationship with peers, and helped students develop new skills (see
Table 9 for statistics). According to their responses, tutors’ attitudes are committed to
Elementary Afterschool STEM Program’s LIAS principles. From the data, tutors responses
demonstrated a positive feeling towards the impact of Elementary Afterschool STEM Program
and its purpose. Tutors consistently felt positive about the educational environment at
Elementary Afterschool STEM Program (See Table 9). However, an area that was not consistent
with Elementary Afterschool STEM Program’s LIAS principles was the tutor’s perception of
their students’ desire to be a part of the Elementary Afterschool STEM Program. The mean
response for the tutors was 1.48 (SD=0.59). This indicates that tutors perceived few students had
a desire to be apart of Elementary Afterschool STEM Program.
Table 9
Descriptive Statistics for Elementary Afterschool STEM Program’s tutor survey items about the
environment of Elementary Afterschool STEM Program
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

64
Tutors M SD
Our program provides a physically safe learning
environment for students.
1.04 0.21
Our program provides an emotionally safe learning
environment for students.
1.04 0.21
Our program supports improvement in student academic
performance.
1.00 0.00
The curriculum includes activities and approaches aimed at
improving the leadership and character development of
students.
1.13 0.34
I believe students like coming to the program. 1.48 0.59
Our program helps students to develop self-confidence. 1.09 0.29
Our program provides opportunities for students to build
friendships with peers.
1.04 0.21
Our program helps students develop new skills. 1.09 0.29

Note. Tutors’ responses to questions about Elementary Afterschool STEM Program’s
environment from the tutor survey. Not all tutors responded to this question (N=23).
The majority of tutors (82%) responded “always” to the questions about their preparation
and training for success in Elementary Afterschool STEM Program (M=1.17, SD=0.38). Despite
tutors reporting that they felt that their preparation and training was adequate, their responses
about activity space and program resources were varied. The results indicated that a significant
number of tutors (56%) “always” felt that space and resources were an impediment to their
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

65
assigned duties. Figure 5 presents the tutors’ responses to items concerning training, resources,
and program space.

Figure 4.
Statistics from tutors perspective on Elementary Afterschool STEM Program’s Resources and
Supplies, Training, and Instructional Space.
Tutors were asked if they received enough training on various topics, including STEM
content knowledge (SCK), science and engineering practices (SEP), and Elementary Afterschool
STEM Program’s mission. The tutor surveys indicated that 82% and 91% of the respondents felt
that they received enough training on Science and Engineering Practices (SEP) and Elementary
Afterschool STEM Program’s mission, respectively (see Figure 5). However, the tutors
perception of the amount of training of STEM Content Knowledge (SCK) and lab safety
responses were more varied, (82% and 65%, respectively). This means that tutors recognize and
0

10

20

30

40

50

60

70

80

90

Resources
and
supplies
Received

proper
training
Adequate
space

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

66
report that their level of content knowledge varied across Elementary Afterschool STEM
Program.

Figure 5.
Percent of tutors and their perspective on their training in STEM, science and engineering
practices and Elementary Afterschool STEM Program’s mission statement.
Site lead survey. The site lead survey covered topics related to the logistics and attitudes
toward Elementary Afterschool STEM Program. There were only 9 completed surveys.
Therefore, the sample size was smaller and more varied than the tutors’ surveys.
One question asked the site leads about their needs and priorities. The results showed a
variance among respondents (see Table 10). Training was reported 50% of the time as the
highest area of need, M= 1.75, SD=0.89. Conversely, site visits were reported 62% of the time
as the least area of need, M= 4.5, SD=0.76. The notion of coaching and consultation was only
76
78
80
82
84
86
88
90
92
94
96
STEM content knowledge Science and Engineering Processes Program Mission/ Vision
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

67
reported 12% of the time as an area of need. These results indicated a desire to improve STEM
pedagogy through more training.
Table 10
Results from the site leads’ survey that identified the needs in Elementary Afterschool STEM
Program.
Statistic Percent Identifying
a ‘Pressing Need
M SD
Training 50 1.75 0.89
Information and
Resources
25 3 1.6
Coaching 12 2.13 0.64
Consultation 12 3.63 1.19
Site Visit 0 4.5 0.76

Note. Site Leads’ response about their sites’ needs and priorities from the site lead survey (N=8).
Elementary Afterschool STEM Program Question 2: Amount of STEM
Implemented. The framing question for this section was: To what extent is Elementary
Afterschool STEM Program implementing the four components of STEM in their daily plans to
achieve their program goals? This question was designed to assess the staff’s practice and
implementation of science and engineering content during the week, it is based on the
Elementary Afterschool STEM Program goal of building a strong science and engineering
focused curriculum (see Appendix A).
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

68
Tutor survey. Figure 6 shows the responses of the tutors regarding daily activities in
Elementary Afterschool STEM Program. The questions from this instrument had the options
“1= never, 2=less than once a month, 3= once a month, 4= 2-3 times a week, 5=once a month,
6=2-3 times a week, and 7= daily”. A mean of 6 or greater indicated that activities were
completed daily. The tutor survey indicated that tutors did not implement technology and
science in the same frequency across Elementary Afterschool STEM Program sites. The use of
their “Sci-gineering” curriculum once a week was consistent with Elementary Afterschool
STEM Program’s goals of STEM lessons and tutor responses. These responses were consistent
with the Elementary Afterschool STEM Program’s curriculum/ activities planning calendar. The
tutor survey indicated that the promotion of healthy behaviors activities “always” (82%)
occurred and the use of Microsoft products “never” occurred (43%), M= 6.38, SD= 1.53 and
M=2.46, SD 1.77. Similarly, Elementary Afterschool STEM Program’s weekly plan indicated
that 2 days a week were spent with the curriculum “Kids Lit” and “Kids Math”, which are ELA
and math curriculums. The results indicated that sites are using these later forms of curricula 2-3
times a week, as verified by classroom observations.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

69

Figure 6.
Average results from tutors responses about their daily activities completed in classrooms at
Elementary Afterschool STEM Program
Site lead survey. Along with asking the tutors how much time they spent on the
classroom-based learning activities, site leads were asked about the amount of time spent
planning and supporting tutors with their classroom activities. The results of site leads responses
to STEM lesson planning are represented in Figure 7. The results show that 25% of site leads
spend 2-3 times per week planning a science activity (M= 2.63, SD= 0.52). Meanwhile, 12%
reported that math activities are planned daily (M=1.88, SD=0.35). Interesting 12% of site leads
indicated that engineering activities were not planned. The results reflected a varied
understanding of the curriculum and the nature of STEM programs.
0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Never
Less

than

Once
a

Month

Once
a

Month

2-‐3

Times
a

Month

Once
a

Week

2-‐3

Times
a

Week

Daily

Use
of
Microsoft
products

Kidz
Math

Kidz
Lit

Scigineering
curriculum

Science
based
activities

Technology
based
activities

Activities
promoting
healthy

behaviors

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

70

Figure 7.
Site Lead Responses to the Planning of Daily Activities
The site leads were asked about the amount of time supporting the components of STEM
(science, technology, engineering, and mathematics). According to the site lead survey, 87% of
the site leads supported math activities 2-3 times a week at their sites. Also, 62% of site leads
supported science and technology activities weekly. In contrast, site leads had conflicting reports
about engineering activities. Twelve percent reported that engineering activities did not occur,
while 87% state that they occurred monthly (M=3.25, SD= 0.71). This result indicated that the
understanding of the activities is not explicitly identified as STEM activities by the site leads.
Supervisor questionnaire. From the questionnaire responses, the site leads described
the development of Elementary Afterschool STEM Program’s STEM program as an evolving
process that relied on feedback from the previous year’s implementation. For example, a theme
0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Everyday
2-‐3
times/

week

Once
a
week
This
activity
is

not
offered

A
MATH
related
activity

An
ENGINEERING
related
activity

A
TECHNOLOGY
related
activity

A
SCIENCE
related
activity

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

71
emerged that building on the program’s strength is a goal of Elementary Afterschool STEM
Program’s improvement plan. One respondent stated, “Throughout this process we are aiming to
be mindful of youth choice and development as we identify and expand upon our partnerships in
education.” This suggests that Elementary Afterschool STEM Program site leads understand
that growth areas exist in the program. To build on their needs, the respondents consistently
reported that they primarily collected data on tutors and staff in the form of observations by
outside consultants and teachers on special assignment.
Elementary Afterschool STEM Program used curricular materials that are created by
venders and consultants. Respondents indicated that their role as a lead is not to generate the
STEM curriculum, but to focus on researching, facilitating, and discovering partnerships that
will enhance Elementary Afterschool STEM Program’s afterschool program. The facilitators
brought materials and activities from existing curriculum to their sites. The respondents all
indicated that the currently used materials and curriculum were “highly effective” and “very
beneficial” in student learning about STEM.
Two of the five respondents indicated that pre and post surveys were used to evaluate
Elementary Afterschool STEM Program students. One respondent stated that the another
university online toolbox (survey) was used to measure student and tutor growth. From the data
collected and site administrator’s responses, the measurement of student growth was not done in
an explicit and systematic way. Eighty percent of respondents indicated that observations were
used to measure accountability in students’ understanding of the curriculum. From that, one
respondent said that collected observational data were used to inform subsequent meetings with
staff.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

72
Elementary Afterschool STEM Program Question 3: Student voice and choice.
Student survey. Table 11 displays the results of the analysis of the student survey.
Students were asked to respond to a series of statements and identify which best described them
using a 4-point Likert Scale (1= This describes me, 2= This mostly describes me, 3= This sort of
describes me, 4= This does not describe me). Most students (64%) responded positively to being
able to ask their own questions about science (M=2.04). 44% of students answered, “This
describes me” while 21% of students answered, “This mostly describes me”. Forty percent of
students answered, “This describes me” (M=2.03), when stating that the activities that they
completed during Elementary Afterschool STEM Program allowed them to think about what
they were doing rather than simply following directions. Fifty-six percent of the students
(M=1.7) stated that the activities allowed them to use their hands and mind.
Table 11
Descriptive Statistics for Elementary Afterschool STEM Program student surveys.
Students N M SD
I am able to ask my own questions about science. 281 2.04 1.09
The activities I do allow me to think about what I am doing,
rather than simply follow directions.
278 2.03 1.04
The activities in the program allow me to use my hands and my
mind.
279 1.7 0.94
The data that was collected and analyzed on the student surveys contradicted the
observational data from the classroom. Possible explanations for this discrepancy will be
discussed in Chapter 5.
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

73
Student observations. In an effort to gain a better understanding of students’ role in
Elementary Afterschool STEM Program, the Dimensions of Success (DoS) observation tool was
used to pinpoint strengths and weaknesses as well as opportunities for students to interact with
STEM practices (PEAR, 2010). This instrument was used to collect qualitative data and assign a
rating in twelve different dimensions: organization, materials, space utilization, participation,
purposeful activities, engagement with STEM, STEM content learning, inquiry, reflection,
relationships, relevance and youth voice. For this project, students were observed in their
classroom on November 4, 2014. The lesson was observed for an hour. During the observation
tutor and student behaviors were recorded. The notes were then used as evidence to assign a
rating based on DoS’s 4-level rubric for increasing quality.
The lesson observed was part of Elementary Afterschool STEM Program’s Sci-Gineering
curriculum; Module #4, Build and Test, lesson 8. The purpose of the lesson was to continue
student learning about acceleration and how it affected the travel of a marble on a roller coaster.
Sixteen fourth grade students participated in the lesson: eight boys and eight girls.
Students were inside a general education classroom. Desks were all faced forward in a
traditional setting where the direct-instruction took place.
The results of this lesson focused on the fourth domain of the DoS Observational Tool:
Youth Development in STEM. This domain was selected because it mirrors Elementary
Afterschool STEM Program’s third essential question for this project: “To what extent does
Elementary Afterschool STEM Program provide opportunities for student voice and choice in
their learning?” (See Figure 8). It rated the facilitator-student interactions as well as student-
student interactions. More specifically, it magnified the way interactions were encouraged or
discouraged during participation in STEM activities for the duration of the observation, whether
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

74
or not the activities made STEM relevant and meaningful to students' everyday lives, and how
the interactions allowed youth to make decisions and have a voice in the learning environment
and community (www.pearweb.org/tools/dos.html, 2015). The DoS Observer Rating Sheet can
be found in its entirety in Appendix J.

Figure 8.
Results of Youth Development in STEM Domain
Relationships. Weak evidence existed to show that the relationships among students,
student-facilitator, and facilitator-facilitator were positive. Several examples of facilitator
comments included: “Demetrius!” “Get your hands off that!” “That sounds like an argument,
not a compliment.” “Boys. Boys. We’re not outside.” “Around! Around! That is over, not
around!” A few positive comments included: “Yours looks good,” while walking away. “You
did this right the first time.”
Relevance. Limited evidence of relevance was observed during the activity.
Additionally, limited effort was observed to make the lesson relevant to students’ lives. The
0
1
2
3
4

Youth
Voice

Relationships

Relevance

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

75
facilitator made an isolated reference to acceleration while peddling a bike when students were
copying the definition onto their paper but not during the hands-on part of the activity. The
conversation did not continue beyond the teacher’s reference to the bicycle.
Youth Voice. Little evidence of activities encouraging student voice occurred while the
observation took place. The first tutor told a group of boys, “Don’t share with anyone. We’ll
share at the end,” while the second facilitator said to the students, “Remember what I said.” The
students immediately stopped talking and little conversation between students took place after
that.
Summary
Question 1: Staff perception of STEM and LIAS principles.
The results from the tutor survey and staff lead survey indicated that the staff had a
positive attitude towards Elementary Afterschool STEM Program’s LIAS principles at the time
of this report, however their attitude towards STEM was not directly measured. Tutors felt that
Elementary Afterschool STEM Program helped students gain content knowledge and skills,
indicating program effectiveness. However tutors’ attitudes about STEM could only be inferred
from their positive outlook about Elementary Afterschool STEM Program’s mission and
principles. Elementary Afterschool STEM Program was able to increase their staffs’ attitudes
from their first year of implementation, based on the evidence. This shows that participants
perceived that Elementary Afterschool STEM Program and its consultants had increased their
effectiveness at training their tutors. The survey did not allow for the respondents to elaborate.
Therefore, an additional survey should be conducted to identify current and future needs.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

76
Question 2. Implementation of STEM for Elementary Afterschool STEM Program's
program goals.
According to Elementary Afterschool STEM Program tutors, the data collected supported
the practices of Elementary Afterschool STEM Program completing STEM activities once a
week at the participating sites. Conversely, the site leads indicated that they spend 2-3 days a
week on planning a science and technology activity. Elementary Afterschool STEM Program’s
current calendar has STEM- based activities once a week, which does not support the research-
supported practices for an effective STEM program. Elementary Afterschool STEM Program
had a discrepancy with what activities were planned and what was actually delivered to the
students. The results showed that activities were not categorized equally or consistently across
the sites.
The respondents identified the activities on which they spent their afterschool academic
time. Elementary Afterschool STEM Program’s program calendar outlines STEM activities
occurring once a week. From the tutor surveys, site lead surveys, and questionnaire, STEM
activities occur once a week. Conversely, not all respondents agreed on which specific activities
are done or how they are categorized. The data support the notion that sites do differ in the types
of activities they perform in the classroom. From the data, it is clear that the nature of the
activities supported in the classrooms were not all identified as STEM. According to the results,
each site does engage in simultaneous activities and instruction, but the activities are minimal in
STEM or were not identified as such. Respondents of the tutor survey consistently reported that
they were doing certain activities, such as math, ELA, and “healthy activities,” but varied on the
how much they engaged the students in STEM activities. The student outcomes of Elementary
Afterschool STEM Program were not measured in student knowledge growth and their resources
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

77
were not assessed for effectiveness. The results supported the notion that all four components of
STEM were not practiced equally across the sites.
Question 3. Student Voice and Choice in Activities
The data collected on the student surveys indicated that the majority of the students had
positive self-efficacy related to science. Most students felt that they could ask questions about
science while in class. Additionally, students felt that they were able to use their hands and
minds during their Elementary Afterschool STEM Program time. While students responded
positively on the survey, the classroom observations provided some conflicting data. Students
were not given the opportunity to ask questions and have them answered. Most students were
actively engaged on the project but received minimal guidance and reference to the lesson’s
objective. The discussion of these questions and recommendations from these findings are
presented in the next chapter.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

78
CHAPTER 5: DISCUSSION
Joseph Calmer and Rebecca Acosta coauthored this chapter.
Summary
Elementary Afterschool STEM Program is an afterschool program that provides STEM
education to elementary students in the Small Urban School District. This project sought to
evaluate Elementary Afterschool STEM Program’s first year of implementation as well as a
needs assessment for year two and beyond. Surveys were distributed and completed by
Elementary Afterschool STEM Program tutors, site leads and students (grades 4-6 only).
Additionally, site supervisors completed a questionnaire. The data collected from these surveys
and questionnaires were combined with site observations and document analysis for
triangulation.
Our findings indicated that the tutors agreed with the mission and principles of
Elementary Afterschool STEM Program while simultaneously having ambiguous feelings
toward the actual structure of the academic activities within the program. The tutors felt they
received enough training to be effective. Moreover, the site leads identified training as a need.
Site leads also stated that STEM instruction occurred once a week. Additionally, site leads
stated that ELA and math activities occurred twice a week. The frequency of these subjects
being taught aligned with the program structure, but not Elementary Afterschool STEM
Program’s STEM learning goals. Elementary Afterschool STEM Program’s site leads also
reported, via questionnaire, that the only tutor evaluation and program evaluation that occurred
was through observations with a protocol for the next evaluative step.
All data that was collected helped provide evidence to support our recommendations for
Elementary Afterschool STEM Program’s effectiveness of STEM instruction and
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

79
implementation. Unexpected outcomes that were discovered throughout the project are also
discussed below as well as limitations that could have impacted the data collected.
Discussion of Findings
Unexpected outcomes
During the analysis of results, some unexpected outcomes were identified. All of the site
leads indicated that their professional education and university degrees were not STEM related.
However, research supports the notion that content knowledge supports one’s pedagogical
knowledge (Carney & Indrisano, 2013; DeBoer, 2000). Therefore, Elementary Afterschool
STEM Program would benefit by having staff members that have formal knowledge of education
and policy, as well as content knowledge in the STEM fields. Currently, California is in the
process of transitioning science standards (Spiegel et al., 2014). The NGSS are structured in a
way to utilize modes of scaffolding Crosscutting Concepts and Disciplinary Core Ideas
throughout a student’s educational career (NGSS Lead States, 2013). Elementary Afterschool
STEM Program can use these standards to frame their program and foster STEM learning in
their students.
In Chapter 4, the discrepancy between the results from student surveys and the student
observation were not expected. We would have expected the observation to validate student
responses on the survey. Students responded positively when asked about opportunities to add
voice to their learning, whereas in the observation, students were not given the opportunity to
express the learning that had taken place and make connections to their everyday life.
Furthermore, during the data collection phase of the project, it became clear that the
perception of respondents suggested that the curriculum being used was not meeting the
students’ needs or the program’s instructional needs. Although the students were being exposed
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

80
to many concepts in science, the perception was that retention of those lessons was not occurring
with the existing curriculum. However, the cause could be curricular, pedagogical, or
instructional. It is not known at this time.
Recommendations
After carefully analyzing the data collected from surveys, questionnaires, classroom
observations and existing data for this project, several recommendations have been made. These
recommendations are focused around the three framing questions: (1) Does the staffs at
Elementary Afterschool STEM Program have a positive attitude toward STEM and their LIAS
principles? (2) To what extent is Elementary Afterschool STEM Program implementing the four
components of STEM in their daily plans to achieve their program’s goals? (3) To what extent
does Elementary Afterschool STEM Program provide opportunities for student voice and choice
in their learning?
Develop or revise organizational goals. Research has identified the characteristics of
effective organizational goals. At present, the goals in Elementary Afterschool STEM Program’s
strategic action plan do not fit the measures of effective goal development. Therefore,
Elementary Afterschool STEM Program should consider creating performance goals, short-term
objectives that are used for specific tasks, that support their intermediate and global goals
(Rueda, 2011). Once the performance goals are created, the gaps can be identified and the path
towards goal attainment can begin. These goals can be constructed through the creation of a
logic model, which would aid in measuring outcomes over time (Regional Educational
Laboratory, 2014; W.K. Kellogg Foundation, 2001).
Action item. The development of SMART goals needs to occur before the remaining
recommendations. A smart goal is specific, measurable, action-oriented, realistic and timely
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

81
(Smith, 1994). This can be accomplished with meetings and professional development for the
various stakeholders associated with Elementary Afterschool STEM Program. The meetings
should be focused on specific outcomes for developing a long-term vision and/or mission for
Elementary Afterschool STEM Program.
A logic model can also be developed to organize the vision and goals for Elementary
Afterschool STEM Program. The OC STEM Initiative has a long history of STEM program
implementation. Therefore, meetings with their leadership should be made to aid in the
development of Elementary Afterschool STEM Program’s vision and goals to ensure alignment
and effectiveness.
Measure student outcomes directly. The analysis of the first framing question
illuminated the need for Elementary Afterschool STEM Program to measure student knowledge
objectively, rather than by observation only. It was stated that teacher observations and other
anecdotal methods were used for student measurement, but students’ knowledge was not
measured directly. It is recommended that Elementary Afterschool STEM Program develop an
instrument to measure students’ content knowledge throughout the program. Collection of this
data could also be used to measure the effectiveness of curriculum and therefore aid in making
more informed program decisions in the future.
Action Item. Forty-four percent of students felt that science makes them feel “lost and in a
jumble”. Elementary Afterschool STEM Program needs to decrease this number. One way to
help is to analyze the way in which instructional time is being used. Tutors can collect data
quickly through the use of exit cards, increased time for student reflection, and the use of
technology applications on tablets and iPads. A suggested app is called Near Pod
(www.nearpod.com). On this app, tutors would generate quizzes and all students would answer
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

82
simultaneously through their own device. Another suggestion would be using a Senteo
Interactive Response System. With this, the tutor would pose a question and students would
answer by using a clicker. Tutors would have immediate feedback on student understanding and
be able to adjust future instruction.
Furthermore, the DoS observational tool should be used more frequently at all
Elementary Afterschool STEM Program school sites, and across all grade levels. This would
ensure continuity with measuring identified student outcomes. Increasing the use of the DoS
observational tool would give Elementary Afterschool STEM Program a better indication of the
quality of their program over time (www.pearweb.org, 2015). With an increase in observational
frequency, Elementary Afterschool STEM Program would build a database of lessons taught.
Therefore, they could receive and develop customized reports on trends and student outcomes
across their sites.
Increase training. The analysis supports the need for more training in their identified
areas. Training needs to be targeted and aligned with the organizational outcomes identified by
Elementary Afterschool STEM Program. Site leads have indicated that training was needed.
Therefore, a survey should be constructed to identify the specific needs of these tutors. This
recommendation would satisfy the action item on Elementary Afterschool STEM Program’s
current SAP.
Action item. It is recommended to use the National Academies Press as a resource to
gather research and training materials on the specific items of interest to Elementary Afterschool
STEM Program. Professional development should be created using the materials of existing
programs with effective professional development at this time. One such resource is the
Association of Supervision and Curriculum (ASCD). The ASCD offers professional
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

83
development in STEM, literacy, student engagement, and many others via recognized
professionals in educational research such as Douglas Fisher, Nancy Frey, Mike Schmoker, and
Robert Marzano.
Create tiered staff training. All staff could benefit from receiving regular training and
professional development. It should be noted that training in content and STEM-related fields
would be beneficial as well. Tiered staff training would consist of sessions that build upon prior
content knowledge as previous concepts had been mastered. It would enable Elementary
Afterschool STEM Program staff members to learn new strategies and concepts that are
unknown to them. Research has shown that proper content knowledge will increase staffs’ self-
efficacy and motivation in their organization (Schein, 2010). Despite the tutors indicating that
they have received enough training and the site leads indicating that training is a need, the cogent
thing to do for Elementary Afterschool STEM Program is to identify the training topics that need
to be expanded and progress from there.
Action item. Elementary Afterschool STEM Program needs to identify specific areas of
training based on current tutor content knowledge and pedagogies. Elementary Afterschool
STEM Program supervisors could create a questionnaire or survey with topics that meet their
program’s goals and current Strategic Action Plan. Based on the responses received, training
could be scheduled and implemented to increase tutor competency.
Implement STEM instruction. The full implementation of STEM instruction in
Elementary Afterschool STEM Program is recommended. While completing this needs
assessment, Elementary Afterschool STEM Program was delivering STEM lessons once a week
for one hour. The full actualization of STEM pedagogical practices would incorporate CCSS
Math and ELA. Therefore, all students’ learning would be positively affected. Based on the
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

84
surveys and observations, STEM instruction was not practiced in accordance with the
recommended research-based perspective at Elementary Afterschool STEM Program. However,
according to research, increasing STEM instruction should help ensure students learn science
and essential practices at a faster rate (Schiavelli, 2010).
Increased training and intentional Professional Development is also recommended.
Elementary Afterschool STEM Program’s weekly schedule should incorporate more STEM
activities that would compliment Common Core State Standards Math and ELA instruction. The
discrepancy between tutors and site leads understanding of STEM could potentially be
eliminated by increased training. Literature supports and recommends that STEM-based
activities occur daily in order to be effective (Gomez & Albrecht, 2013; Program in Education
Afterschool and Resiliency, 2014a; Schiavelli, 2010).
Action item. Build upon the intended practices of the Next Generation Science Standards.
Practice the strategies from other partnership programs in the Orange County Science,
Technology, Engineering and Mathematics (OC STEM Initiative) and reevaluate their academic
calendars. According to the National Science Teachers Association, in order to achieve full
STEM Implementation, a program should include instruction, curriculum, assessment, teacher
preparation and professional development (www.nsta.org, 2015). Additionally, we suggest that
Elementary Afterschool STEM Program build relationships with its stakeholders to help generate
financial resources and increased public support to enhance student learning.
Student voice and choice in learning. While the results of the student survey were
encouraging, the observational data from the DoS observational tool did not confirm the student
survey results. In fact, the results contradicted one another. Therefore, more research is
necessary to validate the results. It is recommended that Elementary Afterschool STEM
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

85
Program have a staff member trained in DoS so that they can maintain ongoing observations to
compare the delivery of instruction and quality of student learning over time
(www.pearweb.com, 2015). Another option for Elementary Afterschool STEM Program is to
take advantage of the pre-established relationship with the OC STEM Initiative in which regular
observations can be made from an OC STEM

Initiative member and the quality of instruction
and student learning can be compared not only within Elementary Afterschool STEM Program,
but also across the participating schools in Orange County, California.
Furthermore, results from the student survey and what was noted during the classroom
observations indicated a need for an increase in student engagement. It is recommended that
Elementary Afterschool STEM Program build in an increased amount of time for students to ask
questions and reflect on their learning at the conclusion of each lesson.
Action item. Students need to be given the opportunity to explain their learning
experience and share their ideas and opinions about structuring the activities to complete the
project at hand (www.pearweb.org, 2015). Elementary Afterschool STEM Program students
should also be given multiple opportunities to make decisions and choices with tutor-selected
constraints placed upon them. While completing the daily STEM activity, students must be
provided the environment where they feel they can take personal and group responsibility for
important aspects of their learning and participation in activities. Additionally, giving students
the opportunity to share their ideas outside of Elementary Afterschool STEM Program to school
and community members such as at school board meetings and community gatherings would
increase their voice.
Enhance Current Curriculum. Increased STEM pedagogy can be developed by the
implementation of various STEM resources. Currently, Elementary Afterschool STEM Program
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

86
uses Sci-Gineering; a curriculum that was designed for regular classroom teachers. Curriculum
is only effective when delivered properly by trained facilitators. It is recommended that
Elementary Afterschool STEM Program increase their pedagogical training for its staff, so that
the curriculum can be utilized more fully. The new NGSS standards have many identified and
vetted resources for STEM program development that can be used to support their current
curriculum (Committee on Integrated STEM Education et al., 2014; NGSS Lead States, 2013).
Also, the NGSS standards are congruent with the CCSS, so Elementary Afterschool STEM
Program can increase their Math and English curricula by strongly adopting the practices
identified by the NGSS.
Action item. Elementary Afterschool STEM Program should consider an alternative
STEM curriculum. An OST STEM curriculum needs to be student centered. It should be
focused on science and engineering practices rather than content alone. Students need
opportunities to be active and thinking about the activity they are engaged in. They should also
be given opportunities to collaborate and engage in habits of mind that enhances their
understanding and meaning-making of real-world phenomena. It is recommended that
Elementary Afterschool STEM Program look at the Consumers Guide to Afterschool Science
Resources for direction on choosing a curriculum that fits their program’s goals. The Science
After School (SAS) Consumer’s Guide was developed by the Coalition for Science After School
to help afterschool programs find high-quality materials to enhance their existing program. This
guide will provide the opportunity to read expert reviews of products, curriculum, activity kits,
instructor guides and more. You can find this website at:
http://www.sedl.org/afterschool/guide/science/.
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

87
Increase staff members with STEM interests. Along with the implementation of
NGSS and aligning NGSS tenets to their organizational goals, Elementary Afterschool STEM
Program should develop a recruitment plan to hire more STEM-degree focused undergraduates.
Having staff that has the content background and interest will enhance their STEM program by
building interest in staff and students.
Stimulate students’ affective domain. It is recommended that more emphasis on
inquiry and constructive practices will increase engagement and increase students’ affective
domain. For example, adopting the Science Writing Heuristic would help students build content
knowledge while engaging them in the practices of science and engineering (Hand et al., 2009).
The development and integration of NGSS practices will explicitly make the activities engaging
and attractive to the students. The reestablishment of the partnership with OC STEM Initiative
should be used as a source for modeling the effective STEM program. The identified
stakeholders should visit and observe OC STEM Initiative programs for ideas and networking.
Another way to stimulate students’ affective domain in STEM is through the use of trade
books. The use of trade books can cover a wide range of topics. Most importantly, they can be
used for science, engineering, literature, and any non-fiction-based topic. The National Science
Teacher’s Association produced a list of the year’s best trade book, organized by topic and
grade, each year. It can be found on their website
Limitations
This project collected data from students at six of seven elementary schools that
participated in Small Urban School District’s Elementary Afterschool STEM Program. This
could have impacted the outcome of data with over 100 student responses missing as well as
several tutors.
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

88
The site observations only took place at one participating school, Zeta Elementary.
While this was Small Urban School District and Elementary Afterschool STEM Program’s
decision, it is difficult to assume that the results from the DoS observation tool were reflective of
all schools that participate in Elementary Afterschool STEM Program. Therefore, the results
cannot be generalizable.
Additionally, the tutors, those individuals who teach the lessons to Elementary
Afterschool STEM Program students, may have been nervous about being observed for fear of
being evaluated, among other reasons. This may have impacted the delivery of their lessons.
Finally, it was difficult to measure whether the participants of the surveys and questionnaires
answered honestly. The tutors may have felt pressure to answer the questionnaires but feared
that their job was on the line had they answered truthfully.
The wording of the questions on the surveys and questionnaires may have also affected
the results. For example, when most tutors’ responses varied on the situation of space and
resources, it may have been better to have had a follow-up, open-ended response section to hear
tutors’ feedback in the matter rather than only a selection of answers to choose from.
Furthermore, the questions were worded in a way that seemed to convey the investigator’s
intentions. However, during the project implementation and analysis, that intention was not
realized. This became apparent while analyzing the questionnaire. The responses seemed to
reflect the convoluted nature of the questions; therefore, the results may not be as precise and
reliable as would be hoped.
Elementary Afterschool STEM Program was very interested in ascertaining data about
the improvement of their staff’s attitudes towards STEM from year one. While analyzing the
tutor survey, it was realized that most tutors had only been employed with Elementary
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

89
Afterschool STEM Program for less than one year. Therefore, a measurement of growth could
not be reliable or valid. The results can only be applicable to this current year of
implementation.
Lastly, the survey and questionnaire items were generated by the authors and were
influenced by the Elementary Afterschool STEM Program’s organizational goals and documents.
The data collected for this project was from Elementary Afterschool STEM Program’s site leads,
tutors, and students. Therefore, the recommendations are specific to Elementary Afterschool
STEM Program in Small Urban School District. The recommendations have limited
generalizability because of limited and targeted data collection. The data collected was on
Elementary Afterschool STEM Program’s specific goals and measures.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

90
APPENDICES
Appendix A. Afterschool STEM Program/ USC: Scope of Work
09/18/14

Afterschool
STEM
Program/USC
Rossier
School
of
Education

Scope
of
Work

This document describes the scope of work for:
Afterschool
STEM
Program,
Small
Urban
School
District

to be conducted by:

Rebecca Acosta, Joseph Calmer, under the supervision of Drs. Sinatra and Rueda
University of Southern California, Rossier School of Education
3470
Trousdale
Parkway

Los
Angeles,
CA
90089

The purpose of this document is to outline the scope of work for the project. Through the
project, USC Doctoral students will provide consultative support to Afterschool STEM Program
Afterschool programs around the topic of Science Technology Engineering and Math (STEM) as
part of their dissertation program.
Afterschool STEM Program afterschool STEM needs

The Afterschool STEM Program team identified several areas of need that they would like
assistance with:
• List of effective STEM programs
• Evaluative feedback on current STEM programming
• Feedback on PD for STEM professionals
• Suggestions and recommendations for future directions
• A final report that includes an executive summary of findings
• Access to the complete findings in dissertation form

USC capacity

USC agrees to provide the following supports to Afterschool STEM Program:

Support options
• Help analyze pre-existing needs assessment
• Develop an evaluation of 2014-1015 STEM programming

In addition to those supports, the graduate students have also offered additional added value
based upon their own professional expertise.
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

91

• Provide recommendations on professional development opportunities and impact
(Joseph)

Timeline for rollout and implementation

May-June 2014

USC team will provide Afterschool STEM Program with:
• Biographies for the students involved in the project
• A sample letter for requesting approval for research to be distributed to appropriate
district representatives.
• A list of data or information they will need to complete their recommendations.

Afterschool STEM Program will provide USC team with:
• Copies of your needs assessment (pre and post data)
• Information about professional development efforts up until this point (providers and
content)
• Afterschool STEM Program STEM inventory (products, curriculum, kits, etc.)

Both USC Team and Afterschool STEM Program
• Develop a Memorandum of Understanding defining your collaboration
USC Students at this time will also start to develop their dissertation proposals and prepare to
defend them.
Afterschool STEM Program team should identify any administrative barriers to USC students
collecting data and developing a needs assessment. Please secure a letter of approval from the
appropriate source (Superintendent, Board, etc.) to continue with the project that addresses and
removes those barriers.
July-August 2014
USC
Team

• Applying
for
IRB
approval

• Defending
dissertation
proposals

• Provide
feedback
on
needs
assessment

• Developing
evaluation
plan
and
tools
(interviews,
surveys,
focus
groups,
etc.)

• Provide
recommendations
on
blended
learning

• Provide
recommendations
on
professional
development

Please
keep
in
mind
that
the
USC
students
will
be
providing
those
recommendations
based
upon

data
that
Afterschool
STEM
Program
has
already
collected.

The
more
robust
comprehensive

evaluation
report
will
be
delivered
at
the
end
of
the
year.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

92

Afterschool
STEM
Program
Team

• Planning
for
programs
opening

• Securing
curriculum

• Development
of
professional
development
plan

• Solidifying
partnerships/aligning
resources
based
on
feedback

•
Preparing
stakeholders
for
implementation
of
STEM
initiative

September-‐
October
2014

USC/Afterschool
STEM
Program
Team

• Implementation
of
2
nd

year
of
Afterschool
STEM
Program
STEM
strategic
initiative

• Survey
distribution

• Focus
groups

• Interviews

November
–December
2014

USC/Afterschool
STEM
Program
Team

• Collection
of
surveys

• Analyzing
data

January-‐April
2015

USC
team

• Creation
of
Evaluation
report

• Defense
of
work

May
2015

USC
Team

• Delivers
final
evaluation
report
to
Afterschool
STEM
Program.

Afterschool
STEM
Program
Team

• Hosts
meeting
to
review
results
and
recommendations.

Afterschool
Technical
Assistance
Unit
Recommendations

1.

We
recommend
that
the
USC
team
clearly
define
the
evaluation.
The
"evaluation"
tool
(or
suite

of
tools)
and
process
should
be
operationalized
explicitly
so
that
all
parties
are
clear
about
what

will
occur
when
and
with
whom.

For
example,
some
questions
that
should
be
addressed

include:

(1)
Is
this
evaluation
process
going
to
involve
surveys,
focus
groups,
and
interviews?,
(2)

Who
will
be
the
target
audience
(e.g.,
Program
Director,
Site
Coordinator,
Instructional
day
staff)

for
each
of
the
data
collection
instruments?,
(3)
which
school
sites
will
participate
in
the
study
as
a

whole
(e.g.,
elementary,
middle,
high),
and
(4)
how
will
the
results
be
aggregated
and
reported?

2.

We
recommend
you
pre-‐schedule
conference
calls
for
the
entire
year
to
occur
at
least
once
a

month
to
maintain
momentum
and
alignment.

3.

We
also
recommend
you
plan
and
schedule
at
least
2
in
person
meetings
that
include
all
parties

sometime
between
now
and
the
end
of
the
2015
school
year.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

93
4.

Each
team
should
begin
to
gather
the
documents
as
stated
in
the
timeline
and
prepare

deliverables.

This project will end after the final meeting to review the results of the evaluation and to discuss
recommendations. That meeting should occur in May of 2015.
Onward:
• List of effective STEM programs
• Direction of STEM education and integration

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

94
Appendix B. Afterschool STEM Program: Student Survey
Q1 Please indicate the school you attend for Afterschool STEM Program
! Alpha Elementary (1)
! Beta Elementary (2)
! Gamma Elementary (3)
! Ramona Elementary (4)
! Epsilon Elementary (5)
! Zeta Elementary (6)
! Eta Elementary (7)

Q2 Please indicate your grade level
! 4th grade (1)
! 5th grade (2)
! 6th grade (3)

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

95
Q3 Please indicate which statement describes you

This
describes

me
(1)

This
mostly

describes
me
(2)

This
sort
of

describes
me
(3)

This
does
not

describe
me
(4)

“Science
makes

me
feel
like
I
am

lost
in
a
jumble

of
numbers
and

words”.
(1)

!
!
!
!

I
am
able
to

complete
my

science

homework

independently.

(2)

!
!
!
!

I
earn
good

grades
in

Science

activities.
(3)

!
!
!
!

Do
you
agree

with
the

statement,

“Science
is

fun.”?
(4)

!
!
!
!

My
afterschool

program
has

made
me

interested
in

Science
(5)

!
!
!
!

I
am
interested

in
Science

activities.
(6)

!
!
!
!

I
like
my

Science
class.

(7)

!
!
!
!

If
I
perform
well

in
Science,
it

will
help
me
get

a
job
in
Science

in
the
future.

(8)

!
!
!
!

Do
you
agree

with
the

!
!
!
!

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

96
statement:
“I

would
enjoy

being
a

scientist.”?
(9)

I
am

comfortable

using

technology.
(10)

!
!
!
!

I
know
some

tools
that

scientists
use.

(11)

!
!
!
!

I
know
some

tools
that

engineers
use.

(12)

!
!
!
!

My
after-‐school

program
has

helped
me
in

my
daily

classroom.
(13)

!
!
!
!

My
after-‐school

program
has

helped
me
in

my
daily

classroom,

especially
in

science.
(14)

!
!
!
!

I
understand

that
science

uses
a
process

of
observing,

collecting
data,

and
explaining

evidence.
(15)

!
!
!
!

I
am
able
to
ask

my
own

questions
about

science.
(16)

!
!
!
!

The
activities
I

do
allow
me
to

think
about

!
!
!
!

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

97
what
I
am

doing,
rather

than
simply

follow

directions.
(17)

The
activities
in

the
program

allow
me
to
use

my
hands
and

my
mind.
(18)

!
!
!
!

I
get
to
use
a
lot

of
tools
and

materials

during
STEM

instruction.
(19)

!
!
!
!

I
know
what
the

letters
STEM

stand
for.
(20)

!
!
!
!

Completing
the

activities
in

STEM
allows
me

to
think
like
a

scientist.
(21)

!
!
!
!

I
am
interested

in
participating

in
a
Science
Fair

now
that
I
have

participated
in

Afterschool

STEM
Program.

(22)

!
!
!
!

I
have
been
able

to
earn
higher

grades
on
my

homework
as
a

result
of

participating
in

Afterschool

STEM
Program.

(23)

!
!
!
!

I
am
interested

in
pursuing

answers
to

!
!
!
!

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

98
scientific
or

engineering

questions.
(24)

I
feel

comfortable

with
science

topics.
(25)

!
!
!
!

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

99
Appendix C. Afterschool STEM Program: Tutor Survey
Q1 Please answer the following questions in an effort to gather accurate data about Small Urban
School District’s Afterschool STEM Program program.
! Alpha Elementary (1)
! Beta Elementary (2)
! Gamma Elementary (3)
! Delta Elementary (4)
! Zeta Elementary (5)
! Eta Elementary (6)

Q2 Please indicate how long you have worked at Afterschool STEM Program in your current
position.
! Less than 1 year (1)
! 1-2 years (2)
! 3-4 years (3)
! More than 5 years (4)

Q3 Please respond to the following statements, as you have experienced while working at
Afterschool STEM Program.
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

100

Always
(1)
Sometimes
(2)
Never
(3)
Not
sure
(4)

Our
program

provides
a

physically
safe

learning

environment
for

students
(1)

!
!
!
!

Our
program

provides
an

emotionally
safe

learning

environment
for

students
(2)

!
!
!
!

Our
program

provides

exciting
and

engaging

enrichment

opportunities

for
students
(3)

!
!
!
!

Our
program

supports

improvement
in

student

academic

performance

(4)

!
!
!
!

The
curriculum

includes

activities
and

approaches

aimed
at

improving
the

leadership
and

character

development
of

students
(5)

!
!
!
!

I
involve

students
in

decision
making

about
program

activities
(6)

!
!
!
!

I
believe
!
!
!
!

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

101
students
like

coming
to
the

program
(7)

Our
program

helps
students

to
develop
self-‐
confidence
(8)

!
!
!
!

Our
program

provides

opportunities

for
students
to

build

friendships
with

peers
(9)

!
!
!
!

I
have
close

relationships

with
students
in

the
program

(10)

!
!
!
!

Our
program

helps
students

develop
new

skills
(11)

!
!
!
!

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

102
Q4 Please respond to the following statements, as you have experienced while working at
Afterschool STEM Program.

Always
(1)
Sometimes
(2)
Never
(3)
Not
sure
(4)

I
keep
parents

informed
about

their
child's

participation
in

the
program
(1)

!
!
!
!

I
receive
the

support
I
need

from
the
Site

Supervisor
(2)

!
!
!
!

I
receive
the

support
I
need

from
the

teachers
(3)

!
!
!
!

I
respond

quickly
and

appropriately
to

any
concerns

that
the
Site

Supervisor

might
have
(4)

!
!
!
!

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

103
Q5 Please indicate how often each of the following activities occur at your site:

Never

(1)

Less
than

Once
a

Month

(2)

Once
a

Month

(3)

2-‐3

Times
a

Month

(4)

Once
a

Week
(5)

2-‐3

Times
a

Week
(6)

Daily
(7)

Activities

promoting

healthy

behaviors

(e.g.,
physical

activity,

gardening)
(1)

!
!
!
!
!
!
!

Technology

based

activities
(e.g.,

using

computers,

multimedia

projects)
(2)

!
!
!
!
!
!
!

Science
based

activities
(e.g.,

robotics,

environmental

science

projects)
(3)

!
!
!
!
!
!
!

Sci-‐gineering

curriculum
(4)

!
!
!
!
!
!
!

Kidz
Lit
(5)
!
!
!
!
!
!
!

Kidz
Math
(6)
!
!
!
!
!
!
!

Use
of

Microsoft

products;

Office,

PowerPoint,

etc.
(7)

!
!
!
!
!
!
!

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

104
Q6 Please answer the following questions regarding your training for Afterschool STEM
Program.

Always
(1)
Sometimes
(2)
Never
(3)
Not
sure
(4)

I
have
the

resources
and

supplies
needed

to
do
my
job

well
(1)

!
!
!
!

I
receive
the

training
that
I

need
to
be

successful
(2)

!
!
!
!

I
receive
the

coaching
that
I

need
to
be

successful
(3)

!
!
!
!

I
have
the

confidence
to

do
my
job
well

(4)

!
!
!
!

I
have
the
space

to
do
my
job

well
(5)

!
!
!
!

I
have
close

relationships

with
my

colleagues
(6)

!
!
!
!

I
get
the

support
that
I

need
from
the

Team
Lead
(7)

!
!
!
!

I
get
the

support
that
I

need
from
the

leadership
team

(8)

!
!
!
!

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

105
Q7 Please indicate your level of knowledge for the following items:

I
received
enough
training
(1)
I
need
more
training/

information
(2)

Classroom
management
(1)
!
!

STEM
content
(2)
!
!

Science
and
Engineering

Processes
(3)

!
!

Program
Mission/
Vision
(4)
!
!

Lesson
design
(5)
!
!

Assessing
student

knowledge
(6)

!
!

Behavior

Agreements/Discipline

Procedures
(7)

!
!

Lab
Safety
(8)
!
!

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

106
Q8 Demographic Information: What is your age?
! 18-21 years old (1)
! 22-25 years old (2)
! 26-29 years old (3)
! >30 years old (4)

Q9 What level of school have you completed?
! High School Graduate (1)
! Some college (2)
! AA (3)
! BA (4)
! Advanced College Degree (5)

Q10 Is your (future) degree STEM-related?
! STEM related (1)
! Not STEM related (2)

Q11 What is your future degree(s) in? (Check at that Apply)
" Science (Biology, Chemistry, etc.) (1)
" Social Science (sociology, history, cultural studies) (2)
" Arts (3)
" Engineering (4)
" Education (5)
" Undecided (6)

Q12 What is your Ethnicity? (Optional)
! White/ Caucasian (1)
! Hispanic or Latino (2)
! Black or African American (3)
! Native American or American Indian (4)
! Asian / Pacific Islander (5)
! Other (6)

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

107
Appendix D. Afterschool STEM Program: Site Lead Survey
Q1 Please indicate the level of your role in Afterschool STEM Program:
• District Level Employee (1)
• Other (2)

Q2 Please identify the amount of time spent on planning or supporting the following activities:

Everyday
(1)
2-‐3
times/

week
(2)

Once
a
week

(3)

1-‐3
times
a

month
(4)

This
activity

is
not
offered

(5)

A
SCIENCE

related

activity
(1)

•
•
•
•
•

A

TECHNOLOGY

related

activity
(2)

•
•
•
•
•

An

ENGINEERING

related

activity
(3)

•
•
•
•
•

A
MATH

related

activity
(4)

•
•
•
•
•

Q3 In the past year, how many of the following STEM related support services have you
received as a Site Supervisor
• Training on how to integrate elements of STEM into my program
activities (1)
• Coaching on how to implement STEM training content into my
program activities (2)

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

108
Q4 Please rate how the following factors have impacted the implementation of STEM related
activities within Program APPLE?

Very
challenging

(1)

Somewhat
of
a

challenge
(2)

A
small
challenge

(3)

Not
a
challenge

(4)

Adequacy
of

space/facilities

(1)

•
•
•
•

Availability
of

trained
staff
(2)

•
•
•
•

Access
to

resources,

curricula,

and/or

materials
(3)

•
•
•
•

Time
(e.g.,

competition

from
other

activities)
(4)

•
•
•
•

Access
to

Internet

connected

computers
(5)

•
•
•
•

Student
interest

(6)

•
•
•
•

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

109
Q5 Please rate the following:

Completely
agree

(1)

Mostly
agree
(2)
Somewhat
agree

(3)

Do
not
agree/

Not
aligned
(4)

The
goals
of

SUSD
are

aligned
with

Afterschool

STEM
Program.

(1)

•
•
•
•

The

Supplemental

Tutor
Agency

supports
the

educational

goals
of

Afterschool

STEM
Program

(i.e.
Rigorous

instruction,

being
culturally

relevant,
and

engaging)
(2)

•
•
•
•

Q6 What are your most pressing needs related to implementing STEM activities within
Afterschool STEM Program? Please drag your responses into the order of your choice, from
greatest to least pressing needs.
______ Training (e.g., integrating STEM into program activities, evaluating STEM activity
outcomes) (1)
______ Information and resources (e.g., existing STEM curricula, developing partnerships with
outside agencies, funding opportunities, evaluation tools ) that I may be able to use in
Afterschool STEM Program (2)
______ Coaching (e.g., how to integrate and/or improve STEM components in program
activities) (3)
______ Consultation (e.g., how to integrate and/or improve STEM components in program
activities) (4)
______ Site Visits (to those who have already implemented effective STEM activities) (5)

Q7 What are your goals for students participating in STEM activities in Afterschool STEM
Program? Check all that apply.
• Increased knowledge of STEM disciplines (1)
• Increased interest in and positive attitude toward STEM disciplines (2)
• Increased awareness of and interest in pursuing STEM careers (3)

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

110
Q8 Are there other goals that you feel are needed for Afterschool STEM Program? If so, please
write them below

Q9 What is your Ethnicity?
• White/ Caucasian (1)
• Hispanic or Latino (2)
• Black or African American (3)
• Native American or American Indian (4)
• Asian / Pacific Islander (5)
• Other (6)

Q10 What is your highest degree attained?
• High School Diploma (1)
• Bachelors of Science or Arts (2)
• Master of Science or Arts (3)
• Doctorate (4)

Q11 Is your degree STEM related?
• STEM related (1)
• not STEM related (2)

Q12 What was your degree(s) in? (Check at that Apply)
• Science (Biology, Chemistry, etc.) (1)
• Social Science (sociology, history, cultural studies) (2)
• Arts (3)
• Engineering (4)
• Education (5)
• Other (6)

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

111
Appendix E. Administrative Questionnaire
Administration Questionnaire:

1. What do you think was the most beneficial meetings, trainings, or other support materials
that assisted you in supporting your staff?
2. How effective do you find partnerships in your work?
3. From your perspective, how effective at creating STEM understanding in the students, do
you think “Sci-gineering” was in the past year? Give specific examples.
4. How is the student knowledge gained from Afterschool STEM Program measured?
5. What are the most productive systems or products that you use?
6. What do you see as your role in the creation of science, technology, engineering, or math
activities?
7. What is being done to implement the Common Core State Standards in Afterschool
STEM Program? (to support the daily instruction in SUSD)
8. What is the biggest lesson you learned about STEM program implementation this year?
Give a specific time and what you learned from it.
9. What will you do differently for year 2 or year 3?
10. What barriers do you foresee about year 2 implementation?
11. How is student achievement measured in Afterschool STEM Program?
12. How is the curriculum measured for accountability?
13. How are you evaluating any STEM activities offered in your program?
14. How are families invited and involved with the Afterschool program?

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

112
Appendix F. SUSD Strategic Action Plan
Strategic Action Plan 2014-2015
Instructional
Strategic Action – Support the District’s Plan to Measurably Improve Student
Achievement
A Analyze Data from 2013-2014 Surveys for Program Improvement
B Improve Kidz/Lit, Kidz/Math, and STEM Practices
C Implement Basic Skills Practice 3X weekly (1
st
-3
rd
HFW, and 4
th
-6
th
Math Facts)
D Identify and Implement Common Core Practices in Afterschool
Strategic Action – Support the District’s Plan to Improve the Use of Technology
A Connect “Type to Learn” to School Assignments
B Implement Project-Based Learning Assignments through Technology Clubs with Student
Leaders
C Purchase Big Universe On-Line Leveled Reading Program

Human
Strategic Action – Support Improved Student Guidance Services (Youth Development
in APPLE)
A Increase Opportunities for Student Voice and Choice
B Implement Student Led Project-Based Learning and Clubs
C Continue with Student Behavior Agreements
D Pilot Girl Scout Project for Anti-Bullying and Self Esteem
E Continue Healthy Living Practices Through AmeriCorps Coaches and NEOP Partnership
Strategic Action – Increase Public and Teacher Confidence in APPLE
A Provide Opportunities for Tutors to Connect Routinely with and Learn from Teachers of
Their Students
B Provide Tutor Training for Communicating Effectively with Families and Teachers
C Include APPLE Activities in School Calendar
D Increase Parent and Teacher Participation at APPLE Showcases
E Increase Publication of APPLE Newsletter
F Identify Additional Partnerships
Strategic Action - Highly Skilled Staff
A Implement “Tiered” Staff Development Based on Tutor Evaluations
B Introduce Tutors to DII, Thinking Maps, and Nancy Fetzer Practices (Awareness of Best
Practices)
C Provide Program-Specific Leadership and Coaching Training to Site Supervisors
D Provide Site Supervisors with Opportunities to Participate in District Staff Development
and/or Committees

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

113
Appendix G. Afterschool STEM Program Tutor and Site Leads Job Description
Tutors (Afterschool STEM Program):
Complete all Team Member responsibilities above with increased responsibilities as assigned
including, but not limited to:
• Create and maintain a room environment that best facilitates instruction and class
ownership;
• Enthusiastically participate with students, managing behavior throughout program
duration and communicating with team to effectively handle situations in an appropriate
manner;
• Actively engage students throughout the classroom and program remembering to make
meaningful connections with students equally;
• Provide guidance and participate with students during academic instruction; which
includes but not limited to STEM, Kidz Lit, Kidz Math, PBL, Game Day, Type to Learn
4, and or district driven initiatives.
• Make a clear objective to check homework if homework is complete and correct.
• Be a resource for students to discover what they know and need to know, helping
students to understand processes of solving and not just give answers;
• Direct volunteers to group students effectively, utilizing all resources provided to keep
students engaged and interested;
• Be accountable for student attendance procedures, being mindful of the number of
students in their group;
• Assist with rotation process of students from one activity to another;
• Assist with attendance procedures, including sign-in and sign-out processes at the
beginning and the ending of the day;
• Assist with distribution & cleanup of snacks to students;
• Evaluate all students fairly and consistently; praising for students for their efforts and
guiding them through their challenges.
• Communicate absences and call-ins as detailed in Staff Attendance Policy and
Procedures; committing to evaluation expectations and areas for improvement as
opportunities to grow and learn.

Lead Tutors (Afterschool STEM Program):
Complete all respective Team Member, Tutor, and Instructor responsibilities above with
increased responsibilities as assigned including, but not limited to:
• Be a beacon of communication for team members, supervisors, volunteers, managers and
other concerning parties;
• Attend all Lead Trainings and Meetings with the intent to contribute and benefit from
discussion and information presented;
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

114
• Facilitate discussion, training reminders, and announcements with team; taking initiative
to find solutions rather than excuses,
• Respond to emails consistently, relaying information to site staff, and hold them
accountable for given information;
• Work with site supervisors to ensure program effectiveness and communicate needs
immediately;
• Ensure deadlines are met team-wide;
• Email weekly reports to management, noting site accomplishments and challenges;
keeping log of students who need extra attention;
• Acknowledge team members successes and share in their challenges; talk to team
members about how you would handle decisions;
• Arrive early and remain on site to triple-check that all opening and closing duties are
completed;
• Communicate supply orders and arrange pick-up/delivery

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

115
Appendix H: STEM Career Interest Survey
STEM Career Interest Survey (STEM-CIS)
Optional Demographic Questions
1. Date

2. First name

3. Last name

4. Grade

5. Gender

6. Teacher

7. Race

8. Period

9.School

Directions: Students will complete the STEM-CIS online via iPod Touches or computers. Each question is a Likert
scale with the following choices:
Strongly Disagree (1), Disagree (2), Neither Agree nor Disagree (3), Agree (4), Strongly Agree (5)
Science
S1 I am able to get a good grade in my science class.

S2 I am able to complete my science homework.

S3 I plan to use science in my future career.

S4 I will work hard in my science classes.

S5 If I do well in science classes, it will help me in my future career.

S6 My parents would like it if I choose a science career.

S7 I am interested in careers that use science.

S8 I like my science class.

S9 I have a role model in a science career.

S10 I would feel comfortable talking to people who work in science careers.

S11 I know of someone in my family who uses science in their career.

Mathematics
M1 I am able to get a good grade in my math class.

M2 I am able to complete my math homework.

M3 I plan to use mathematics in my future career.

M4 I will work hard in my mathematics classes.

M5 If I do well in mathematics classes, it will help me in my future career.

M6 My parents would like it if I choose a mathematics career.

M7 I am interested in careers that use mathematics.

M8 I like my mathematics class.

M9 I have a role model in a mathematics career.

M10 I would feel comfortable talking to people who work in mathematics careers.

M11 I know someone in my family who uses mathematics in their career.

Technology
T1 I am able to do well in activities that involve technology.

T2 I am able to learn new technologies.

T3 I plan to use technology in my future career.

T4 I will learn about new technologies that will help me with school.

T5 If I learn a lot about technology, I will be able to do lots of different types of careers.

T6 My parents would like it if I choose a technology career.

T7 I like to use technology for class work.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

116
T8 I am interested in careers that use technology.

T9 I have a role model who uses technology in their career.

T10 I would feel comfortable talking to people who work in technology careers.

T11 I know of someone in my family who uses technology in their career.

Engineering
E1 I am able to do well in activities that involve engineering.

E2 I am able to complete activities that involve engineering.

E3 I plan to use engineering in my future career.

E4 I will work hard on activities at school that involve engineering.

E5 If I learn a lot about engineering, I will be able to do lots of different types of careers.

E6 My parents would like it if I choose an engineering career.

E7 I am interested in careers that involve engineering.

E8 I like activities that involve engineering.

E9 I have a role model in an engineering career.

E10 I would feel comfortable talking to people who are engineers.

E11 I know of someone in my family who is an engineer.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

117
Appendix I: Secondary teacher’s pedagogical content knowledge
Secondary teachers’ scientific pedagogical content knowledge (STSPCK)
Instructions
The purpose of this study is to measure pre-service secondary school teachers’ PCK.
This questionnaire consists of three parts and 30 statements in total.
Part A. Measures your knowledge of curriculum (n = 7).
Part B. Measures your knowledge of instructional strategies (n = 9).
Part C. Measures your knowledge of assessment strategies (n = 14).
There are multiple statements in each of these three categories. Please provide an example of
how you would address each statement as a science teacher.
Your honest and detailed responses can potentially help teacher educators to develop targeted
professional development opportunities in the area of physical science to improve science
teachers’ PCK.

PART A. Curriculum
Statement Provide examples and comments for
each statement.
1. The teacher uses an inquiry-based science
curriculum.

2. The teacher uses problem-based science
curriculum.

3. The teacher pays attention to the relevancy of
curriculum to the students’ everyday lives.

4. The teacher pays attention to the relevance of
curriculum to his/her students’ cultural backgrounds.

5. The teacher uses a curriculum that emphasizes the
nature of science.

6. The teacher uses a curriculum that emphasizes the
history of science.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

118
PART A. Curriculum
Statement Provide examples and comments for
each statement.
7. The teacher uses a curriculum that is not gender-
biased. (It provides examples from and biographies
of both male and female scientists.)

PART B. Instruction
Statement Provide examples and
comments for each statement.
1. The teacher starts the lesson by helping children to
discover what they already know about the concepts
taught in the lesson.

2. The teacher identifies the limitations and strengths of
his/her students’ prior conceptions.

3. The teacher uses probing questions to help students
retrieve relevant information and experiences.

4. The teacher uses guiding questions to help students
integrate relevant information and experiences into what
they are currently learning

5. The teacher provides experiences for the students to
understand the limitations of their initial ideas.

6. The teacher engages students in sharing and discussion
to help them consider their individual ideas in relation to
the ideas of others.

7. The teacher provides experiences (supports lecture
notes with static or interactive visuals, models) for the
students to understand the plausibility/intelligibility of
scientifically accurate conceptions.

8. The teacher supports her explanation of scientific
concepts by using analogies or metaphors.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

119
PART B. Instruction
Statement Provide examples and
comments for each statement.
9. The teacher supports her explanations with
demonstrations to make the concepts intelligible to her/his
students.

PART C. Assessment
Statement Provide examples and
comments for each
statement.
1. The teacher poses open-ended questions.
2. The teacher uses problems that require students to engage in
methods of inquiry.

3. The teacher asks problems that require students to explain a
concept.

4. The teacher asks problems that require students to justify their
understanding of the concept.

5. The teacher uses problems that require the students to
communicate their understanding of the concept through
multiple means.

6. The teacher uses problems requiring students to develop their
own methods of investigation.

7. The teacher uses problems that emphasize students’
understanding of the relationships among ideas under study.

8. The teacher challenges his/her students to interpret graphical
representations of scientific data.

9. The teacher requires his/her students to explain their
reasoning when giving an answer to a question.

10. The teacher challenges his/her students to evaluate each
other’s ideas and answers.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

120
PART C. Assessment
Statement Provide examples and
comments for each
statement.
11. The teacher challenges his/her students to consider
alternative methods for solutions to the problems.

12. The teacher asks his/her students to explain concepts to one
another.

13. The teacher asks questions that require students to explain a
scientific phenomenon through a model.

14. When a student asks a question, which of the following
action does the teacher take:

1. ____She/he just tells the answer.
2. ____She/he wants the student to find the answer on his/her
own by asking leading questions.

3. ____She/he wants her students to find the answer by asking
his/her peers.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD

121
Appendix J: DoS Rating Sheet for Roller Coasters

OBSERVER RATING SHEET

Date of Observation:
Site/Location Name:
Observer Name:
Co-Observer Name: N.A.

FIELD NOTES
Appendix K: Child Assent Form
University of Southern California
Rossier School of Education
Waite Phillips Hall, 3470 Trousdale Parkway, Los Angeles, CA 90089

CHILD ASSENT FORM

Title: Needs Assessment and Program Evaluation for SUSD’s Elementary Afterschool
STEM Program

Rebecca Acosta and Joseph Calmer are graduate students at USC and want to learn about
students’ interest in science, technology, engineering and mathematics (STEM). We
hope to find out what interests students in these STEM subjects. One way to learn about
it is to conduct a research study; the people conducting the study are called researchers.

Your mom/dad/Legal Guardian have told us we can have you answer questions about the
study. You can also talk this over with your mom or dad. It is up to you if you want to
take part. You can say “yes” or “no.” No one will be upset if you don’t want to take
part.

This study will be a regular part of your Elementary Afterschool STEM Program
activities.

You will receive your regular STEM lessons. You may be asked about what you learned
in your lesson or whether you liked the activity. You may also be asked questions about
what you learned from the activity.

Researchers do not always know what will happen to people in a research study. We
don’t expect anything to happen to you, but you might not like to answer the questions.

Your teacher or parents will not have access to your answers. You will not be graded on
answers you provide in the study.

If you have any questions, you can as one of the researchers.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD
123
Appendix L: Information Sheet for Parents
University of Southern California
Rossier School of Education
Waite Phillips Hall, 3470 Trousdale Parkway, Los Angeles, CA 90089

INFORMATION/FACTS SHEET FOR NON-MEDICAL
RESEARCH—Information Sheet for Parents

Title: Needs Assessment and Program Evaluation for SUSD’s Elementary Afterschool
STEM Program

PURPOSE OF THE STUDY
This study explores your child’s learning in Small Urban School District’s Elementary
Afterschool STEM Program after-school Science, Technology, Engineering, and
Mathematics or STEM program. We hope to find out more about your child’s motivation
and efficacy for STEM. Your child is invited to be in this study because he or she is a
student in the after-school program where the evaluation is being conducted.

This project will be a regular part of your child’s regular instruction. In other words, the
instruction will take place during the after-school instruction as part of the planned
instruction. There will be no added requirements outside of the classroom. You child’s
participation in the study is completely voluntary.

PARTICIPANT INVOLVEMENT
Your child has received lessons about STEM. Your child may be asked about what he or
she has learned in the after school STEM program, if he or she is interested in science,
and if he or she knows what careers in Science are available to him or her. The study will
take place one day and will only take 10-15 minutes to complete a survey.

ALTERNATIVES TO PARTICIPATION
You and your child’s relationship with the school will not be affected, whether or not
your child participates in this study. Your child will be asked to continue with his or her
regular class activities, and will not be asked to complete the survey or participate in the
focus group, if s/he doesn’t want to participate.

CONFIDENTIALITY
All data will be coded by assigning a numerical code. No names will be attached to the
information collected. We will not use the name of your student in anything that we
produce.

Your child will not be graded on participation in the study. Your child’s teachers will not
know whether or not your child was one of the participants.

Data will be stored in a computer which is password protected.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD
124
An executive report will be created for SUSD and Elementary Afterschool STEM
Program at the conclusion of this project. Information obtained from this project will be
included in the report. However, no identifiable information will be included.

The members of the research team and the University of Southern California’s Human
Subjects Protection Program (HSPP) may access the data. The HSPP reviews and
monitors research studies to protect the rights and welfare of research subjects.

The data will be stored for future use. When the results of the research are published or
discussed in conferences, no identifiable information will be used.

INVESTIGATOR CONTACT INFORMATION
Rebecca Acosta at rmacosta@usc.edu and Joseph Calmer calmer@usc.edu
Ed.D. Students
Rossier School of Education
University of Southern California
3470 Trousdale Parkway
Waite Phillips Hall
Los Angeles, CA 90089-4036

IRB CONTACT INFORMATION
University Park Institutional Review Board (UPIRB), 3720 South Flower Street #301,
Los Angeles, CA 90089-0702, (213) 821-5272 or upirb@usc.edu

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD
125
Appendix M: Information Sheet for Adults
IRB-Information Sheet for Adults
University of Southern California
Rossier School of Education
Waite Phillips Hall, 3470 Trousdale Parkway, Los Angeles, CA 90089

INFORMATION/FACTS SHEET FOR NON-MEDICAL
RESEARCH—Adult Permission Form

Title: Needs Assessment and Program Evaluation for SUSD’s Elementary Afterschool
STEM Program

You are invited to participate in a research study conducted by Rebecca Acosta and
Joseph Calmer, Ed.D. Students at the University of Southern California. Your
participation is voluntary. Please take as much time as you need to read this document,
which explains information about this study. You should ask about anything that is
unclear to you.

PURPOSE OF THE STUDY
The current study is a program evaluation of Elementary Afterschool STEM Program’s
year one implementation and a needs assessment of their second year goals. We hope to
identify strengths within the year one implementation that can continue to be built upon
as well as identify areas of need that we can provide support and resources to aid in a
successful second year. You are invited to be in the study because you are an
administrator or Line Staff where the research is being conducted.

PARTICIPANT INVOLVEMENT
If you agree to participate in this study, you may be asked to teach STEM lessons in your
regular Elementary Afterschool STEM Program “classroom” with your assigned class.
Researchers will be observing your classroom and taking notes. We may ask you to
complete a questionnaire and survey that should take no more than 10-15 minutes of your
time. It will include questions on your experiences teaching and some demographics
information such as how long you have been teaching. You do not have to answer any
questions that you don’t want to. You can move on to the next question or withdraw
from the study at any time without any negative consequences whatsoever.

CONFIDENTIALITY
Researchers may observe your classroom and take notes. Observation data will not be
associated with teachers’ personal information. We will not use your name in anything
that we produce.

To maintain your confidentiality, we will transcribe your questionnaire and destroy the
electronic copy as soon as possible.

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD
126
All the data we gather from this study will be stored in a computer, which will be
protected with a password.

The members of the research team and the University of Southern California’s Human
Subject’s Protection Program (HSPP) may access the data. The HSPP reviews and
monitors research studies to protect the rights and welfare of research subjects.

The data will be stored for future use, but all identifiable information will be deleted as
soon as possible.

When the results of the research are published or discussed in conferences, no identifiable
information will be used.

An executive report will be created for SUSD and Elementary Afterschool STEM
Program at the conclusion of this project. Information obtained from this project will be
included in the report. However, no identifiable information will be included.

ALTERNATIVES TO PARTICIPATION
Your alternative is to not participate. Your relationship with your employer or the
researchers will not be affected whether you participate in this study.

INVESTIGATOR CONTACT INFORMATION
Rebecca Acosta at rmacosta@usc.edu and Joseph Calmer calmer@usc.edu
Ed.D. Candidates
Rossier School of Education
University of Southern California
3470 Trousdale Parkway
Waite Phillips Hall
Los Angeles, CA 90089-4036

IRB CONTACT INFORMATION
University Park Institutional Review Board (UPIRB), 3720 South Flower Street #301,
Los Angeles, CA 90089-0702, (213)821-5272 or upirb@usc.edu

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD
127
Appendix N: Student survey results
#
Question
This
describes
me
This
mostly
describes
me
This sort
of
describes
me
This
does
not
describe
me
Total
Responses
Mean
1 “Science makes me
feel like I am lost in
a jumble of
numbers and words.
42 41 75 125 283 3.00
2 I am able to
complete my
science homework
independently.
120 74 55 32 281 2.00
3 I earn good grades
in Science
activities.
115 91 58 16 280 1.91
4 Do you agree with
the statement,
“Science is fun.”?
162 49 46 23 280 1.75
5 My afterschool
program has made
me interested in
Science.
108 62 47 58 275 2.20
6 I am interested in
Science activities.
175 64 18 22 279 1.59
7 I like my Science
class.
152 58 36 33 279 1.82
8 If I perform well in
Science, it will help
me get a job I
Science in the
future.
107 61 62 50 280 2.20
9 Do you agree with
the statement: “I
would enjoy being
a scientist.”?
77 53 67 85 282 2.57
10 I am comfortable
using technology.
190 48 26 8 272 1.46
11 I know some tools
that scientists use.
95 52 75 54 276 2.32
12 I know some tools
that engineers use.
87 57 65 67 276 2.41
13 My after-school
program has helped
123 65 53 36 277 2.01
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD
128
me in my daily
classroom.
14 My after-school
program has helped
me in my daily
classroom,
especially in
science.
93 72 57 55 277 2.27
15 I understand that
science uses a
process of
observing,
collecting data, and
explaining
evidence.
125 67 45 41 278 2.01
16 I am able to ask my
own questions
about science.
124 60 60 37 281 2.04
17 The activities I do
allow me to think
about what I am
doing, rather than
simply follow
directions.
113 79 52 34 278 2.03
18 The activities in the
program allow me
to use my hands
and my mind.
156 71 31 21 279 1.70
19 I get to use a lot of
tools and materials
during STEM
instruction.
147 63 39 24 273 1.78
20 I know what the
letters STEM stand
for.
200 32 19 21 272 1.49
21 Completing the
activities in STEM
allows me to think
like a scientist.
123 65 44 45 277 2.04
22 I am interested in
participating in a
Science Fair now
that I have
participated in
Elementary
Afterschool STEM
116 71 45 49 281 2.10
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD
129
Program.
23 I have been able to
earn higher grades
on my homework
as a result of
participating in
Elementary
Afterschool STEM
Program.
99 81 55 43 278 2.15
24 I am interested in
pursuing answers to
scientific or
engineering
questions.
91 62 81 44 278 2.28
25 I feel comfortable
with science topics.
134 71 36 33 274 1.88

Running head: AFTERSCHOOL STEM PROGRAM AT SUSD
130
References:
Alkin, M. C. (2011). Evaluation Essentials: From A to Z. New York, NY: The Guilford
Press.
Altschuld, J. W., & Kumar, D. D. (2010). Needs Assessment: An Overview: Sage
Publications, Inc.
Altschuld, J. W., & Witkin, B. R. (2000). From Needs Assessment to Action:
Transforming Needs Into Solution Strategies. Thousand Oaks, CA: Sage
Publications, INC.
American Association for the Advancement of Science. (1993). Benchmarks for Science
Literacy. New York, NY: Oxford University Press.
Anderson, L. W., Krathwohl, D. R., & Bloom, B. S. (2001). A Taxonomy for Learning,
Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational
Objectives. New York: Longman.
Au, K. H. (2003). Balanced Literacy Instruction: Implications for Students of Diverse
Backgrounds. In J. Flood, D. Lapp, J. Squire, & J. Jensen (Eds.), Handbook of
Research on Teaching the English Language Arts (pp. 955-966). New Jersey:
Lawrence Erlbaum Associates.
Aydeniz, M., & Kirbulut, Z. D. (2014). Exploring challenges of assessing pre-service
science teachers' pedagogical content knowledge (PCK). Asia - Pacific Journal of
Teacher Education, 42(2), 147-120. doi: 10.1080/1359866X.2014.890696
Ball, D. L., & Forzani, F. M. (2009). The Work of Teaching and the Challenge for
Teacher Education. Journal of Teacher Education, 60(5), 497-511. doi:
10.1177/0022487109348479
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD
131
Ball, D. L., Thames, M. H., & Phelps, G. (2008). Content Knowledge for Teaching: What
Makes It Special? Journal of Teacher Education, 59(5), 389.
Bellflower Unified School District. (2014a). Bellflower Unified School District Website.
Retrieved 7/26, 2014, from http://www.busd.k12.ca.us/Index.html
Bellflower Unified School District. (2014b). Early Childhood Education. Retrieved
7/1/2014, from http://www.busd.k12.ca.us/projectApple.htm
Brooks, J. S., & Normore, A. H. (2009). Educational Leadership and Globalization:
Literacy for a Glocal Perspective. Educational Policy, 24(1), 52-82. doi:
10.1177/0895904809354070
Brown, R., Brown, J., Reardon, K., & Merrill, C. (2011). Understanding STEM: Current
Perceptions. Technology and Engineering Teacher U6 - ctx_ver=Z39.88-
2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-
8&rfr_id=info:sid/summon.serialssolutions.com&rft_val_fmt=info:ofi/fmt:kev:mt
x:journal&rft.genre=article&rft.atitle=Understanding+STEM%3A+Current+Pe
rceptions&rft.jtitle=Technology+and+Engineering+Teacher&rft.au=Ryan+Bro
wn&rft.au=Joshua+Brown&rft.au=Kristin+Reardon&rft.au=Chris+Merrill&rft.
date=2011-03-
01&rft.pub=International+Technology+Education+Association&rft.volume=70
&rft.issue=6&rft.spage=5&rft.externalDocID=2297799381¶mdict=en-US
U7 - Journal Article U8 - FETCH-proquest_dll_22977993811, 70(6), 5.
Bybee, R. W. (1997). Achieving scientific literacy: from purposes to practices.
Portsmouth, NH: Heinemann.
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD
132
Bybee, R. W. (2010a). Advancing STEM Education: A 2020 Vision. Technology and
Engineering Teacher U6 - ctx_ver=Z39.88-
2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-
8&rfr_id=info:sid/summon.serialssolutions.com&rft_val_fmt=info:ofi/fmt:kev:mt
x:journal&rft.genre=article&rft.atitle=Advancing+STEM+Education%3A+A+2
020+Vision&rft.jtitle=Technology+and+Engineering+Teacher&rft.au=Rodger+
W+Bybee&rft.date=2010-09-
01&rft.pub=International+Technology+Education+Association&rft.volume=70
&rft.issue=1&rft.spage=30&rft.externalDocID=2273092421¶mdict=en-US
U7 - Journal Article U8 - FETCH-LOGICAL-e441-
76deadebd58f1f6deb9620796192019ce8ef8f65146cb0aa4c2aa364202a8af11,
70(1), 30.
Bybee, R. W. (2010b). EDITORIAL: What Is STEM Education? Science, 329(5995),
996-996.
Bybee, R. W. (2010c). What Is STEM Education? Science, 329(5995), 996. doi:
10.1126/science.1194998
Bybee, R. W. (2013a). The Case fo STEM Education: Challenges and Opportunities.
Arlington,VA: NSTA press.
Bybee, R. W. (2013b). Translating the NGSS for Classroom Instruction. Arlington, VA:
NSTA Press.
California Afterschool Network. (2013). Readiness and Needs Assessment (T. P. o. D.
STEM^2, Trans.): The Power of DIscovery: STEM², .
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD
133
California Department of Education. (2000). Science Content Standards for California
Public Schools: Kindergarten Through Grade Twelve Retrieved from
http://www.cde.ca.gov/be/st/ss/documents/sciencestnd.pdf.
California Department of Education. (2014). Data & Statistics: Race and Ethnicity for
BUSD. from http://www.ed-
data.k12.ca.us/App_Resx/EdDataClassic/fsTwoPanel.aspx? -
!bottom=/_layouts/EdDataClassic/profile.asp?Tab=1&level=07&reportnumber=1
6&county=19&district=64303&school=6011654#studentsbyraceethnicity
Carney, M., & Indrisano, R. (2013). Disciplinary literacy and pedagogical content
knowledge. Journal of Education, 193(3), 39.
Clark, R. E., & Estes, F. (2008). Turning research into results: a guide to selecting the
right performance solutions. Charlotte, N.C: Information Age Pub Inc.
Committee on Conceptual Framework for the New K-12 Science Education Standards.
(2011). A Framework for K-12 Science Education: Practices, Crosscutting
Concepts, and Core Ideas. Washington, DC: The National Academies Press.
Committee on Highly Successful Schools or Programs in K-12 STEM Education. (2011).
Successful K-12 STEM Education: Identifying Effective Approaches in Science,
Technology, Engineering, and Mathematics. Washington, DC: The National
Academies Press.
Committee on Highly Successful Schools or Programs in K-12 STEM Education, Board
on Science Education, Board on Testing and Assessment, Division of Behavioral
and Social Sciences and Education, & Council, N. R. (2010). Successful K-12
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD
134
STEM Education: Identifying Effective Approaches in Science, Technology,
Engineering, and Mathematics: National Academies Press.
Committee on Integrated STEM Education, National Academy of Engineering, &
National Research Council. (2014). Integration in K-12 STEM Education: Status,
Prospects, and an Agenda for Research. Washington: National Academies Press.
Creswell, J. W. (2008). Research Design: Qualitative, Quantitative, and Mixed Methods
Approaches. Thousand Oaks, Calif: Sage Publications.
DeBoer, G. E. (2000). Scientiic Literacy: Another Look at Its Historical and
Contemporary Meanings and Its Relationship to Science Education Reform.
Journal of Research in Science Teaching, 37(6).
Donovan, S., & Bransford, J. (2004). How Students Learn: Science in the Classroom:
National Academies Press.
Drake, S., & Burns, R. (2003). Meeting Standards Through Integrated Curriculum:
Association for Supervision & Curriculum Development (ASCD) U6 -
ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-
8&rfr_id=info:sid/summon.serialssolutions.com&rft_val_fmt=info:ofi/fmt:kev:mt
x:book&rft.genre=book&rft.title=Meeting+Standards+Through+Integrated+Curri
culum&rft.au=Drake%2C+Susan&rft.au=Burns%2C+Rebecca&rft.date=2003-
12-
01&rft.pub=Association+for+Supervision+%26+Curriculum+Development+%28
ASCD%29&rft.externalDocID=10051970¶mdict=en-US U7 - eBook.
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD
135
Evans, B. R. (2011). Secondary Mathematics Teacher Differences: Teacher Quality and
Preparation in a New York City Alternative Certification Program. The
Mathmatics Teacher, 20(2).
Fang, Z., Lamme, L. L., & Pringle, R. M. (2010). Language and Literacy in Inquiry-
Based Science Classrooms; grades 3-8. Thousand Oaks, Ca: Corwin: A SAGE
Company.
Fensham, P. J. (2009). Real world contexts in PISA science: Implications for context-
based science education. Journal of Research in Science Teaching, 46(8), 884-
896. doi: 10.1002/tea.20334
Flammer, L. (2006). The Importance of Teaching the Nature of Science. The American
Biology Teacher, 68(4), 197-198. doi: 10.2307/4451964
Fullan, M., Hill, P., & Crévola, C. (2006). Breakthrough. Thousand Oaks, Calif.
:[Melbourne, Vic] :[Toronto]: Corwin Press ;NSDC ;Ontario Principals' Council.
Gee, J. P. (2001). Reading as Situated Language: A Sociocognitive Perspective. Journal
of Adolescent & Adult Literacy, 44(8), 714-725.
Gomez, A., & Albrecht, B. (2013). True STEM education. Technology and Engineering
Teacher, 73(4), 8.
Gupta, K. (2007). A Practical Guide to Needs Assessment (D. F. R.-E. Catherine M.
Sleezer Ed. 2nd ed.). Pfeiffer & Co.
Hand, B., Noton-Meier, L., Staker, J., & Bintz, J. (2009). Negotiating Science: The
Critical Role of Argumentation in Student Inquiry. Portsmounth, NH: Heinemann.
Herman, B. C. (2010). Teaching the nature of science: Practices and associated factors.
(3413935 Ph.D.), Iowa State University, Ann Arbor. Retrieved from
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD
136
http://libproxy.usc.edu/login?url=http://search.proquest.com/docview/750175385
?accountid=14749
http://zb5lh7ed7a.search.serialssolutions.com/?ctx_ver=Z39.88-
2004&ctx_enc=info:ofi/enc:UTF-
8&rfr_id=info:sid/ProQuest+Dissertations+%26+Theses+Full+Text&rft_val_fmt
=info:ofi/fmt:kev:mtx:dissertation&rft.genre=dissertations+%26+theses&rft.jtitle
=&rft.atitle=&rft.au=Herman%2C+Benjamin+C.&rft.aulast=Herman&rft.aufirst=
Benjamin&rft.date=2010-01-
01&rft.volume=&rft.issue=&rft.spage=&rft.isbn=9781124155951&rft.btitle=&rft
.title=Teaching+the+nature+of+science%3A+Practices+and+associated+factors&
rft.issn=
http://libproxy.usc.edu/login?url=https://usc.illiad.oclc.org/illiad/illiad.dll/OpenURL??ctx
_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-
8&rfr_id=info:sid/ProQuest+Dissertations+%26+Theses+Full+Text&rft_val_fmt
=info:ofi/fmt:kev:mtx:dissertation&rft.genre=dissertations+%26+theses&rft.jtitle
=&rft.atitle=&rft.au=Herman%2C+Benjamin+C.&rft.aulast=Herman&rft.aufirst=
Benjamin&rft.date=2010-01-
01&rft.volume=&rft.issue=&rft.spage=&rft.title=Teaching+the+nature+of+scien
ce%3A+Practices+and+associated+factors&rft.issn= ProQuest Dissertations &
Theses Full Text; ProQuest Dissertations and Theses A&I: The Humanities and
Social Sciences Collection database.
Hess, K. (2013). Working Knowledge: STEM Essentials for the 21st Century. New York,
NY: Springer New York.
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD
137
Highly Successful Schools or Programs for K-12 STEM Education. (2011). Successful
K-12 STEM Education: Identifying Effective Approaches in Science,
Technology, Engineering, and Mathematics. Washington, D.C.: The National
Academies Press.
Hilton, M. (2010). Exploring the Intersection of Science Education and 21st Century
Skills: A Workshop Summary. Washington, DC: The National Academies Press.
Hoachlander, G., & Yanofsky, D. (2011). Making STEM real (Vol. 68, pp. 60).
Alexandria: Association for Supervision and Curriculum Development.
Huk, T., & Ludwigs, S. (2009). Combining cognitive and affective support in order to
promote learning. Learning and Instruction, 19(6), 495-505. doi:
10.1016/j.learninstruc.2008.09.001
Jensen, E., Dabney, M., Markowitz, K., & Selsor, K. (2003). Environments for learning:
Brain Store.
John, W. M. (2007). Critical Needs of STEM Education. Journal of Chemical Education,
84(12), 1895.
Kier, M. W., Blanchard, M. R., Osborne, J. W., & Albert, J. L. (2014). The Development
of the STEM Career Interest Survey (STEM-CIS). Research in Science
Education, 44(3), 461-481. doi: 10.1007/s11165-013-9389-3
Krapp, A., & Prenzel, M. (2011). Research on Interest in Science: Theories, methods, and
findings. International Journal of Science Education, 33(1), 27-50. doi:
10.1080/09500693.2010.518645
Kuhn, T. S. (1970). The Structure of Scientific Revolutions (Vol. 2, no. 2). Chicago:
University of Chicago Press.
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD
138
Lee, O. (2002). Promoting Scientific Inquiry with Elementary Students from Diverse
Cultures and Languages. Review of Research in Education, 26, 23-69.
Lee, O. (2005). Science Education with English Language Learners: Synthesis and
Research Agenda. American EDucational Research Association, 75(4), 491-530.
Lee, O., Quinn, H., & Valdés, G. (2013). Science and Language for English Language
Learners in Relation to Next Generation Science Standards and with Implications
for Common Core State Standards for English Language Arts and Mathematics.
Educational Researcher, 42(4), 223-233. doi: 10.3102/0013189X13480524
Matthews, M. R. (1998). Foreword and Introduction. In W. F. McComas (Ed.), The
Nature of Science in Science Education: Rationales and Strategies. AA
Dordrecht, The Netherlands: Kluwer Academic Publishers.
Mayer, R. E. (2011). Applying the Science of Learning. Boston, MA: Pearson Education.
McComas, W. F. (2004). Keys to Teaching the Nature of Science. The Science Teacher,
71(9), 24-27.
McComas, W. F. (2006). Science Teaching Beyond the Classroom. The Science Teacher,
73(1), 26-30.
McComas, W. F. (2014). STEM: Science, Technology, Engineering, and Mathematics. In
W. McComas (Ed.), The Language of Science Education (pp. 102-103):
SensePublishers.
McComas, W. F., Clough, M. P., & Almazroa, H. (1998). The Role and Character of the
Nature of Science in Science Education. In W. F. McComas (Ed.), The Nature of
Science in Science Education: Rationales and Strategies (pp. 3-39). Dordrecht,
The Netherlands: Kluwer Academic Publisgers.
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD
139
Moje, E. B., Ciechanowski, K. M., Kramer, K., & Ellis, L. (2004). Working toward third
space in content area literacy: An examination of everyday funds of knowledge
and Discourse. Reading Research Quarterly, 39(1), 38.
Muhammad, A. (2009). Transforming School Culture: How to overcome staff division.
Bloomington, IN: Solution Tree Press.
National Academy of Engineering and National Research Council. (2014). STEM
Integration in K-12 Education: Status, Prospects and an Agenda for Research.
Washington, DC: The National Academies Press.
National Council for Accreditation of Teacher Education. (2006). What makes a Teacher
Effective?: What Research Says About Teacher Preparation. In N. C. f. A. o. T.
Education (Ed.).
National Governors Association Center for Best Practices. (2010). Common Core State
Standards. Washington, D.C.: National Governors Association Center for Best
Practices, Council of Chief State School Officers,.
NGSS Lead States. (2013). Next Generation Science Standards: For States, By States.
Washington, D.C.: Achieve, Inc. on behalf of the twenty-six states and partners
that collaborated on the NGSS Retrieved from
http://www.nextgenscience.org/next-generation-science-standards.
Noam, G. G., Barry, S., Moellman, L. W., Dyken, L. v., Palinski, C., Fiore, N., &
McCouch, R. (2004). The Four Cs of Afterschool Programming: A New Case
Method for a New Field. Afterschool Matters, Occasional Paper Series.
OECD. (2010). PISA 2009 Results: What Students Know and Can Do – Student
Performance in Reading, Mathematics and Science (Volume 1) (Vol. 1).
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD
140
Organisation For Economic Co-Operation Development. (2007). PISA 2006: Science
Competencies for Tomorrow's World Volume 1Analysis. FR: OECD Publishing.
Osborne, J., Simon, S., & Collins, S. (2003). Attitudes towards science: A review of the
literature and its implications. International Journal of Science Education, 25(9),
1049-1079. doi: 10.1080/0950069032000032199
Peterson, K. D., & Deal, T. E. (2002). The Shaping School Culture Fieldbook. San
Francisco: Jossey-Bass.
Program in Education Afterschool and Resiliency. (2014a). Dimensions of Success.
Retrieved 7/26, 2014, from http://www.pearweb.org/tools/dos.html
Program in Education Afterschool and Resiliency. (2014b). DoS Program Planning Tool:
A Guiding Framework for Successful Afterschool STEM Programs. Cambridge,
MA: Harvard Medical School Teaching Hospital.
Read, T. (2013). STEM Can Lead The Way Report: Rethinking Teacher Preparation and
Policy. In C. S. L. Network (Ed.).
Regional Educational Laboratory. (2014). Education Logic Model Application. from
http://relpacific.mcrel.org/ELM.html
Rueda, R. (2011). The Three Dimensions of Student Performance. 978-0807752401:
Teachers College Press.
Sanders, M. (2009). STEM, STEM Education, STEMmania. The Technology Teacher.
Schein, E. H. (2010). Organizational Culture and Leadership. San Francisco: Jossey-
Bass.
Schiavelli, M. (2010). STEM education needs long-term commitment. Central Penn
Business Journal, 26(37), 12-12,15.
Running head: AFTERSCHOOL STEM PROGRAM AT SUSD
141
Shulman, L. S. (1986). Those Who Understand: Knowledge Growth in Teaching.
Educational Researcher, 15(2), 4-14. doi: 10.3102/0013189X015002004
Smith, H. A. (1969). Science: Trends and Dilemmas. In D. G. Hays (Ed.), Britannica
Review of American Education (Vol. 1). Chicago, Il: Britannica Reviews.
Spiegel, J., Quan, A., & Shimojyo, Y. (2014). Planning Professional Learning Using the
NGSS Implementation Pathway Model. California Classroom Science, 27(3).
Spring, J. (2008). Research on Globalization and Education. Review of Educational
Research, 78(2), 330-363. doi: 10.3102/0034654308317846
Tomlinson, C. A. (2009). Intersections between differentiation and literacy instruction:
Shared principles worth sharing. New England Reading Association Journal,
45(1), 28.
Valencia, S. W., & Wixson, K. K. (2013). CCSS-ELA: Suggestions and Cautions for
Implementing the Reading Standards. The Reading Teacher, 67(3), 181.
Vasquez, J. A. (2013). STEM lesson essentials, grades 3-8: integrating science,
technology, engineering, and mathematics. Portsmouth, NH: Heinemann.
W.K. Kellogg Foundation. (2001). Logic model development guide: Using logic models
to bring together planning, evaluation, & action.
Washington, D. C. (1996). National Research Council (NRC). . National science
education standards. National Academy Press.
Zuger, S. (2012). The State of STEM (Vol. 33, pp. 38). San Bruno: New Bay Media
LLC.

Abstract (if available)

Abstract This project was a needs assessment and evaluation of an elementary afterschool STEM program in a small urban school district. The organization we studied will be called Elementary Afterschool STEM Program to protect the identity of the organization that resides is a small urban school district. Three framing questions were created to organize the data: 1) Does the staff at Elementary Afterschool STEM Program have a positive attitude toward STEM and their LIAS principles? 2) To what extent is Elementary Afterschool STEM Program implementing the four components of STEM in their daily plans to achieve their program’s goals? and 3) To what extent does Elementary Afterschool STEM Program provide opportunities for student voice and choice in their learning? To answer these questions, data from the afterschool program’s students, tutors, site leads, and administration was sought. To support recommendations, literature was reviewed on the history of STEM education, STEM instruction, STEM programs practices, and effective STEM pedagogy. Also, literature about the literacy, the CCSS, and the NGSS were investigated as possible solutions as enhancements to their afterschool STEM program. Data was collected and analyzed to produce recommendations for program improvement. Data from student, tutor, and site leads along with an administrative questionnaire, classroom observations and document analysis was conducted to support the recommendations. The analysis included quantitative analysis on the surveys and qualitative analysis on the observations and document analysis. These methods were used to triangulate data and generate recommendations. Recommendations included creating SMART goals, increase staff training, implementing a full STEM curriculum, increasing students’ affective domain, measuring student outcomes, increase student voice and choice, and enhancing their STEM curriculum.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

A needs assessment of an urban elementary school district's afterschool program

PDF

A needs assessment and program evaluation of an urban school district's after-school STEM program

PDF

A needs assessment of Meadows Elementary School District STEM education after school program

PDF

A needs assessment for Hillview Middle School's STEM education program

PDF

A needs assessment of an after school science, technology, engineering, and mathematics program

PDF

A needs assessment of an after school, science, technology, engineering, and mathematics program

PDF

College and career readiness through high school STEM programs: an evaluation study

PDF

An evaluation of Urban Compass after-school program

PDF

Promoting STEM integration, interest and identity among elementary school students

PDF

STEM + design thinking training: investigation of perceived changes in self‐efficacy, pedagogy, and conceptual development at the K-5 level

PDF

Integration of STEM and gardening for urban elementary youth

PDF

A capstone project: closing the achievement gap of English learners in literacy at Sunshine Elementary School using the gap analysis model

PDF

Preservice teacher preparation for engineering integration in K-5

PDF

Strengthening community: how to effectively implement a conflict resolution and peer mediation program on a secondary school campus: an evaluation study

PDF

The relationship between teacher assessment practices, student goal orientation, and student engagement in elementary mathematics

PDF

Analysis of STEM programs in Oregon public high schools

PDF

Employee churn in afterschool care: an evaluation study of manager influences on employee retention and turnover

PDF

Elementary principal leadership and learning outcomes for low socioeconomic status Hispanic English learners

PDF

Effective STEM initiatives in high-poverty elementary schools

PDF

Elementary STEM policies, practices and implementation in California

Asset Metadata

Creator Calmer, Joseph Malcolm (author)

Core Title A needs assessment and evaluation of an elementary afterschool STEM program in a small urban school district

School Rossier School of Education

Degree Doctor of Education

Degree Program Education (Leadership)

Publication Date 04/23/2015

Defense Date 03/23/2015

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag afterschool education,afterschool STEM program,Education,elementary,needs assessment,OAI-PMH Harvest,program evaluation,STEM,STEM education

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Rueda, Robert (committee chair), Sinatra, Gale M. (committee chair), Freking, Frederick W. (committee member)

Creator Email calmer@usc.edu,jmcalmer@gmail.com

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c3-559505

Unique identifier UC11301266

Identifier etd-CalmerJose-3387.pdf (filename),usctheses-c3-559505 (legacy record id)

Legacy Identifier etd-CalmerJose-3387.pdf

Dmrecord 559505

Document Type Dissertation

Format application/pdf (imt)

Rights Calmer, Joseph Malcolm

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA