University of Southern California Dissertations and Theses

A needs assessment and program evaluation of an urban school district's after-school STEM program

(USC Thesis Other)

A needs assessment and program evaluation of an urban school district's after-school STEM program

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 1

A NEEDS ASSESSMENT AND PROGRAM EVALUATION OF AN URBAN SCHOOL
DISTRICT’S AFTER-SCHOOL STEM PROGRAM

by

Rebecca Marie Acosta

A Dissertation Presented to the
FACULTY OF THE USC ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF EDUCATION

May 2015

© 2015 Rebecca Marie Acosta
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 2
Acknowledgements
This dissertation would not have been possible without the guidance and mentorship of our
faculty committee members; the collaboration and support from my colleagues at USC; the
accommodation and cooperation of the staff at Elementary Afterschool School STEM Program
and Small Urban School District; and the frequently tested patience of and encouragement from
my family members and friends. Thank you all.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 3
Statement of Co-authorship
This document reports the results of a capstone project that was completed as part of the
Ed.D culminating program requirements. It was designed as a dissertation of practice that targets
authentic problems of practice and that provides an opportunity for students to demonstrate skills
and competencies that will be required in future career activities. The present project was a needs
assessment/evaluation of a specific program, and was designed to address concerns of this
specific site rather than being designed as a generalizable research project. In line with the goal
of reflecting real world practice, this project was carried out collaboratively between Joseph
Calmer and myself. To accurately reflect the division of work on this project, some of this
document is co-authored, and the specific chapters are labeled to reflect the collaborative
authorship where appropriate.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 4

Table of Contents
List of Tables………………………………………………………………………………. 7
List of Figures ……………………………………………………………………………... 8
Abstract……………………………………………………………………………………. 9
Chapter One: Overview of the Problem……………………………………………. ………10
Background of the Problem………………………………………………………... 10
Statement of the Problem …………………………………………………………. 11
Purpose of the Project……………………………………………………………… 11
Chapter Two: Literature Review……………………………………………………………13
Introduction………………………………………………………………………… 13
Partnership between SUSD, Elementary Afterschool STEM Program
and USC……………………………………………………………………. 13
History of STEM in Education……………………………………………. 13
The Importance of STEM Education………………………………………. 14
Literacy in STEM education………………………………………. 14
The Role of NGSS…………………………………………………………. 14
Areas of Focus for the Evaluation…………………………………………. 14
Evaluation………………………………………………………….. 14
Assessment…………………………………………………………. 15
DoS Observational Tool…………………………………………………….15
Organization…………………….…………………………………. 15
Materials…………………….……………………………………... 16
Space utilization…………………….……………………............... 16
Participation…………………….………………………………….. 16
Purposeful activities…………………….…………………………. 16
Engagement with STEM…………………….……………............... 17
STEM content learning…………………….……………................. 17
Inquiry …………………….………………………………………...17
Reflection…………………….…………………….………………. 18
Relationships…………………….………………………………… 18
Relevance…………………….…………………….………………. 18
Youth voice…………………….…………………………………... 19
Summary…………………….…………………….……………………….. 19
Chapter Three: Methodology…………………….…………………….…………............... 20
Purpose…………………….…………………….…………………………. ………20
Site Description…………………….…………………….………………………… 20
Small Urban School District………………………………………………. 20
Sites…………………….…………………………………... ……... 21
Site staff…………………….……………………………............... 22
Instrumentation…………………….…………………….………………………… 23
Student Career Awareness in STEM Survey………………………………. 23
Staff Knowledge of STEM Survey…………………….…………............... 24
Questionnaire…………………….…………………….…………............... 24
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 5
Document Analysis…………………….…………………….…………….. 25
Observations…………………….…………………….…………………… 25
DoS Observation Tool…………………….………………………………. 26
Data Collection…………………….…………………….………………………… 28
Sequence of Completed Work for this Project…………………….............. 28
Initial Meeting …………………….……………………….............. 28
Site Visit…………………….…………………….……………….. 29
Scope of Work…………………….………………………………. 29
Conference Call…………………….……………………………… 30
8/13/14 Meeting…………………….……………………………… 30
IRB…………………….…………………….…………….............. 30
Survey administration…………………….………………………... 30
Questionnaire…………………….………………………………… 31
Document analysis…………………….………………….... ………31
Observations…………………….…………………………………. 32
Chapter Four: Findings…………………….…………………….………………………… 33
Introduction…………………….…………………….…………………………….. 33
Participant Demographics…………………….……………………………………. 34
Study Participants…………………….…………………….……………………… 34
Tutors…………………….…………………….…………………............... 34
Site leads…………………….…………………….……………….. ………37
Site supervisors…………………….………………………………………. 38
Students…………………….…………………….………………………… 38
Framing Question Analysis…………………….…………………………... ……... 39
Elementary afterschool STEM program question one: staff
attitude toward STEM……………………………………………………… 39
Tutor survey…………………….………………………………….. 39
Site lead survey…………………….……………………………..... 43
Elementary afterschool STEM program question two: amount
of STEM implemented………………………………………….................. 44
Tutor survey…………………….…………………….……………. 45
Site lead survey…………………….……………………………..... 46
Supervisor questionnaire…………………….…………………....... 47
Elementary afterschool STEM program question three: student
voice and choice…………………………………………..... ………49
Student survey…………………….………………………………... 49
Student observations…………………….…………………………. 50
Relationships…………………….…………….…………………… 51
Relevance…………………….……………………………………. 51
Youth voice…………………….…………………….……………. 52
Summary…………………….…………………….……………………………….. 52
Question One: Staff Perceptions of STEM and LIAS Principles………….. 52
Question Two: Implementation of STEM for Elementary Afterschool
STEM Program Goals…………………….…………….............................. 53
Question Three: Student Voice and Choice in Activities………………….. 54
Chapter Five: Discussion…………………….…………………….………………………. 55
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 6
Summary…………………….…………………….……………………………….. 55
Discussion of Findings…………………….………………………………………..56
Unexpected Outcomes…………………….………………………………. 56
Recommendations…………………….……………………………. ……... 57
Develop or revise organizational goals…………………...... ………57
Action item………………………………………………….57
Measure student outcomes directly………………………………… 58
Action item…………………………………………………. 58
Increase training…………………….……………………................ 59
Action item…………………………………………………. 59
Create tiered staff training…………………………………………. 60
Action item…………………………………………………. 60
Implement STEM instruction……………………………………… 60
Action item…………………………………………………. 61
Student voice and choice in learning………………………………. 61
Action items………………………………………………... 62
Enhance current curriculum………………………………………... 63
Action item…………………………………………………. 63
Increase staff members with STEM interests…………………….... 64
Stimulate students’ affective domains……………………………... 64
Limitations…………………….…………………….……………............... 65
References…………………….…………………….……………………………………… 67
Appendix A: Elementary Afterschool STEM Program/USC
Scope of Work……………………………………………………………............... 71
Appendix B: DoS Observer Rating Sheet…………………….……………………………. 75
Appendix C: IRB—Child Assent Form…………………….……………………………… 79
Appendix D: IRB—Information Sheet for Parents………………………………………… 80
Appendix E: Elementary Afterschool STEM Program Student Survey…………………… 82
Appendix F: IRB—Information Sheet for Adults…………………………………………. 84
Appendix G: IRB—Elementary Afterschool STEM Program Tutor Survey……………… 86
Appendix H: IRB—Elementary Afterschool STEM Program Site Lead Survey………….. 92
Appendix I: IRB—Elementary Afterschool STEM Program Supervisor
Questionnaire………………………………………………………………………. 95
Appendix J: SUSD’s Strategic Action Plan…………………….………………………….. 96
Appendix K: Elementary Afterschool STEM Program Staff Descriptions…....…............... 97
Appendix L: STEM Career Interest Survey………………………………………... ……... 99
Appendix M: Secondary Teacher’s Pedagogical Content Knowledge Instrument................102
Appendix N: Student Survey Results……………………………………………………….106

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 7
List of Tables
Table 1: Hope Elementary School Demographic Information……………………………. 22
Table 2: Summary of survey responses……………………………………………………. 34
Table 3: Elementary Afterschool STEM Program’s Tutor percentage and
participation in this project………………………………………………………… 35
Table 4: Descriptive Statistics for Elementary Afterschool STEM
Program’s Tutor Survey……………………………………………………. ……... 41
Table 5: Results from the Site Lead’s Survey……………………………………............... 44
Table 6: Descriptive Statistics for Elementary Afterschool STEM Program
student surveys……………………………………………………………………... 49

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 8
List of Figures
Figure 1: An Overview of the Dos Observation Tool ……………………………………... 26
Figure 2: Tutor Age Distribution at Elementary Afterschool STEM Program……………. 36
Figure 3: Tutor Current Undergraduate Degree Matriculation Plan…………….................. 37
Figure 4: Students’ School Attended for Elementary Afterschool STEM
Program…………………………………………………………………….. ……... 39
Figure 5: Statistics from Tutors Perspective on Elementary Afterschool STEM
Program’s Resources and Supplies, Training and Instructional Space…………….. 42
Figure 6: Percent of Tutors and their perspective on their training in STEM,
science, and engineering practices and Elementary Afterschool STEM
Program’s mission statement………………………………………………………. 43
Figure 7: Average Results From Tutors Responses About Their Daily Activities
Completed in Classrooms at Elementary Afterschool STEM Program……. ……... 46
Figure 8: Site Lead Responses to the Planning of Daily Activities………………............... 47
Figure 9: Results of Youth Development in STEM Domain……………………..... ……... 51

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 9
Abstract
Out of School Time (OST) science, technology, engineering and mathematics (STEM)
programs are on the rise as the need for global competitiveness increases in STEM-related fields.
This project was a needs assessment and program evaluation for Small Urban School District’s
afterschool STEM program, Elementary Afterschool STEM Program. It focused on evaluating
Elementary Afterschool STEM Program’s first year of implementation while assessing their
needs for year two and beyond. We collected and analyzed data from 293 Elementary
Afterschool STEM Program students, 24 tutors, 15 site leads and 8 supervisors using surveys,
observations and previous program documents. Our findings from this triangulation of data
indicate a need for increased professional development, longevity in program staff, and an
increase in student voice. This project provides a foundation for Elementary Afterschool STEM
Program to use when making decisions critical for future student learning.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 10
CHAPTER ONE: OVERVIEW OF THE PROJECT
Background of the Problem
According to Herbert Smith (1969), science curricula was written in a way that
treated science as a discipline, thus leading to its lack of success in traditional educational
settings. However, science remains a unique content area that has somehow uncovered
techniques to thrive in today’s classrooms (Bybee, 2013a; Smith, 1969).
The problem many classrooms in our country face today is that science is often
taught as a set of facts to simply memorize. However, simply relying on rote
memorization does not foster deep understandings of scientific concepts (Brown, Brown,
Reardon, & Merrill, 2011). California currently uses state standards for science that were
adopted in 1997 for kindergarten through twelfth grade. However, a new set of
standards, Next Generation Science Standards (NGSS) has been developed and adopted
and will be required across the state by next year. The NGSS approaches science from a
process-orientated, inquiry-based and integrated approach (NGSS Lead States, 2013).
The NGSS were created with the goal of enhancing the current understanding of science
by students and increase students’ knowledge and interest in STEM related fields
(OECD, 2010).
Science, technology, engineering and mathematics curricula have been
interwoven to create an academic content area known as STEM. Since afterschool
programs allow content knowledge to be presented in various ways, it seems fitting for
NGSS to be implemented within an afterschool program setting.
While STEM education research has increased over the past decade, there still remains a
need for support of effective STEM implementation within afterschool programs. Elementary
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 11
Afterschool STEM Program, Small Urban School District’s (SUSD) afterschool program is just
one program that has implemented STEM and is in need of such support. Data collection and
analysis will be one method for helping Elementary Afterschool STEM Program and SUSD meet
their students’ STEM needs.
Statement of the Problem
Elementary Afterschool STEM Program is currently an afterschool program within Small
Urban School District. They have completed their first year of implementation and wanted to
evaluate the level of completion of their year one goals. Additionally, as they embark on their
second year of implementation, they wanted to have a needs assessment of their year two goals
completed.
One of Elementary Afterschool STEM Program’s goals was to increase students' interests
in STEM fields so students would become aware of and interested in potential STEM-oriented
fields of work for their future. To measure this Elementary Afterschool STEM Program goal,
both an evaluation of current and past practices as well as a needs assessment to align goals and
expected outcomes was conducted.
Purpose of the Project
The purpose of this project was to evaluate Elementary Afterschool STEM Program’s
current and past goals and practices to generate a series of recommendations that would help
enhance the future of their STEM program. Data was collected on SUSD’s Elementary
Afterschool STEM Program and then analyzed to provide formative information on their current
status with respect to their program goals. The data analysis was then used to help build
recommendations for effective strategies that Elementary Afterschool STEM Program so their
goals and outcomes could be realized (Altschuld & Witkin, 2000). The data was collected,
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 12
analyzed, and shared for future developments to occur within SUSD’s Elementary Afterschool
STEM Program as well as provided information as to Elementary Afterschool STEM Program’s
current effectiveness as an afterschool STEM program.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 13
CHAPTER TWO: LITERATURE REVIEW
Introduction
Partnership between SUSD, Elementary Afterschool STEM Program and USC
In order to successfully compete this we meet with SUSD Elementary Afterschool STEM
Program leadership and agreed on a Scope of Work (see Appendix A)., This chapter provides
context on STEM as well as current STEM evaluation tools, as this knowledge, as well as these
tools, helped aide in the data collection, analysis, and interpretation of SUSD’s Elementary
Afterschool STEM Program.
History of STEM in Education
Problems with science instruction in the U.S. have been highlighted for some time.
Historically, science instruction was focused on the cognitive development of students at the
detriment of their affective domain. This caused a decrease in student interest in science
(Osborne, Simon, & Collins, 2003). According to McComas (2006), school programs can serve
the role of providing affective stimulation for students in science beyond content. His
description helps to explain how science can activate students’ affective domain and foster an
interest in STEM fields.
Benchmarks for Science Literacy first described and highlighted the importance of
science education by stating that science, mathematics, and technology are so tightly
interconnected, that it is hard to differentiate them from one another when their ideas and
practices are taught in isolation (American Association for the Advancement of Science, 1993).
These benchmarks articulated the integration of the nature of science, math, and technology.
However, the original standards did not reflect an integrated relationship between disciplines.
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 14
Therefore, the Next Generation Science Standards (NGSS) were created in an effort to bridge the
gap of the unknown.
The Importance of STEM Education
Literacy in STEM education. Coupled with basic literacy skills lies the increased need
to become literate in science. According to the Committee on the Conceptual Framework for the
New K-12 Science Education Standards (2011), every citizen should be literate in science as
well as have a general understanding of science. Adaptability, complex communication skills
and the ability to solve non-routine problems are said to be part of 21
st
century science skills.
These skills are imperative to one being ready for the working world.
The Role of NGSS
The NGSS were created to foster an understanding of the current disparities that exist in
science education today (Bybee, 1997; Committee on Conceptual Framework for the New K-12
Science Education Standards, 2011). One way to improve science instruction is to enhance the
standards. It is only after standards are improved that instruction can follow suit. The NGSS
were designed to frame science content as well as to inform science instruction (Bybee, 2013b;
Committee on Integrated STEM Education et al., 2014). Learning science through proper science
education is the goal of NGSS (Committee on Conceptual Framework for the New K-12 Science
Education Standards, 2011).
Areas of Focus for the Evaluation
Evaluation. Organizations often conduct evaluations to gather data that can help inform
future decisions (Alkin, 2011). Small Urban School District has a number of existing programs
within Elementary Afterschool STEM Program. Therefore, an evaluation of these programs and
their effectiveness was conducted to aid in the alignment of current program goals.
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 15
Program in Education Afterschool and Resiliency (PEAR), the creator of DoS, designed a
method of measuring and evaluating STEM programs. Their method relies on four domains.
Each domain has three dimensions. The four elements and dimensions are:
1. Features of a Learning Environment- Organization, Materials, Space Utilizations
2. Activity Engagement- Participation, Purposeful Activities, Engagement with STEM
3. STEM Knowledge and Practice- STEM Content, Inquiry, Reflection
4. Youth Development in STEM- Relationships, Relevance, and Youth Voice
Through observation each dimension of each element can be measured and analyzed for its
impact on STEM learning for participants (Program in Education Afterschool and Resiliency,
2014a).
Assessment. Noam, Barry, Moellman, Dyken, Palinski, Fiore, & McCouch, (2004)
describes the social and academic needs for afterschool programs, or out-of-school programs, as
a place for academic growth and youth development. They also argue that afterschool programs,
or out-of-school programs, can be used to enhance students’ learning. The tool, Dimensions of
Success (DoS) was developed by Program in Education, Afterschool and Resiliency (PEAR) to
evaluate afterschool and out-of-school learning in 12 different dimensions (Program in
Education, Afterschool and Resiliency, 2014b). This tool was used to facilitate the data analysis
of the classroom observations at Small Urban School District.
DoS Observational Tool
Organization. Organization is assessed on the degree to which the activity is planned and
prepared by the facilitator. Supporting evidence for organization includes having the appropriate
materials for the number of participants that will be completing the activity, the ability to be
flexible in changing situations as well as having smooth transitions in a manner that prevents loss
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 16
of time and chaos within the learning environment. The time aspect is related to whether or not
enough time was allotted to complete the particular activity.
Materials. The materials domain focuses on whether materials are appropriate and
appealing. Appropriate materials are culturally sensitive and developmentally appropriate,
provide support toward the STEM learning goals and are safe. Appeal is related to whether or
not students find the materials interesting.
Space utilization. Space utilization has three specific foci. First, the space should be
utilized in a way that is conducive to STEM learning in out-of-school settings (OST). For
example, the setting should be informal and incorporate a generous amount of student input
versus teacher directed instruction. Students should have sufficient room to move around to
work in both small and large groups, the ability to have whole group discussions and to interact
with the materials in hands-on ways.
Participation. The participation domain focuses exclusively on the extent to which students
are participating in a specific activity. It refers only to explicit participation and not engagement
in STEM thinking. Additionally, participation considers whether all or just a subset of the
students are participating, if students are participating and following along with the facilitator,
and whether or not students are up and roaming around the classroom interrupting the learning of
other students.
Purposeful activities. The purposeful activities domain focuses on the structure of the
activities being taught by the facilitator. The three major factors to focus on are whether or not
the activities are structured so students understand the purpose and goals of the activity, if the
activity goals are structured to support a cohesive focus on a concept or set of related topics in
STEM and if the facilitator uses time productively to support STEM learning goals.
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 17
Engagement with STEM. Engagement with STEM focuses on the extent to which students
are engaging in hands-on activities that allow them to actively construct their understanding of
STEM content. Students should have the opportunity to engage in hands-on activities. They
should also have an active role in their learning. This means they should be doing most of the
cognitive work, rather than the facilitator. (Students are “Hands-on”, meaning constructing their
understanding physically and “Minds-on”, meaning using their prior knowledge to increased new
learning.) Furthermore, each of the hands-on activities needs to aid students’ understanding
STEM content and not simply provide them with superficial experiences with vocabulary words
and disconnected facts.
STEM content learning. Students need support in order to build an understanding of
science, technology, math, and engineering concepts through STEM activities. One important
component of this dimension is whether or not the STEM content is being presented accurately
by the facilitator. The facilitator must also make connections across content areas to deepen
students’ understanding of ideas. Students’ should be making comments and questions that
reflect their understanding of the content being presented. Additionally, students must be given
the opportunity to apply their knowledge beyond superficial memorization and repetition.
Inquiry. Inquiry involves pursuing a question through investigation using the tools and
practices within a given field. When observing lessons for inquiry, one should see that the
activities are providing opportunities for students to engage in STEM practices such as making
observations, asking questions, developing and using models, planning and carrying out
investigations, analyzing and interpreting data, engaging in argument from evidence and
constructing explanations as well as sharing findings with peers. Also, the activities should
provide opportunities for students to participate in STEM practices in authentic ways.
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 18
Reflection. The ability for students to reflect on what they have learned and discovered
following an activity allows the facilitator to know that learning has taken place. When looking
for student reflection, two ideas to keep in mind are that activities should support explicit
reflection on the STEM concepts in which the students have engaged and student reflections
should be meaningful and connected to other greater ideas rather than be superficial and
meaningless.
Relationships. Having positive relationships between the facilitator and student, facilitator
and facilitator, and among students has a positive effect on learning outcomes. Positive
relationships are characterized by students working cooperatively and sharing ideas and
materials with each other. They may also be seen taking turns and listening to one another. No
criticism or judgments are passed from one student to another. Equally important are the
facilitator relationships. The facilitator should be making eye contact when speaking with
students and make a genuine effort to address students by their names. All interactions should be
respectful.
Relevance. Making connections between STEM activities and students’ lives and personal
experiences makes learning more meaningful. Additionally, when activities help student link
STEM concepts to careers and community concerns, they are more likely to understand and
make meaning. A key element of the standards and their acceptability is their relevance.
Students learn more when the subject matter is relevant and connected to the real-world
(Fensham, 2009). An essential feature of STEM teaching is the nature of connecting content to
students’ lives. Afterschool programs have a duty to demonstrate the relevance of the subjects of
their instruction. Moreover, afterschool programs have the adaptability and structure to provide
that relevance through its instruction and facilitation.
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 19
Youth voice. A key component to the DoS is youth voice. This focuses on the extent to
which the STEM activities encourage students to have a voice by taking active roles that allow
for personal responsibility and having their ideas, concerns, and opinions acknowledged and
acted upon by others and the community. The facilitator should help students make important
and meaningful choices within specific limits defined by the facilitator.
Summary
Instructional practices of an afterschool STEM program should have inquiry, relevancy,
and engagement in their structure. Developing and integrating literacy in STEM instruction can
supplement and enhance STEM program effectiveness. To ensure program effectiveness,
programs need to be assessed and the Dimensions of Success tool enables that assessment
component.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 20
CHAPTER THREE: METHODOLOGY
Purpose
The purpose of this project was to provide a needs assessment and program evaluation for
Small Urban School District (SUSD) on their STEM component of their afterschool program,
Elementary Afterschool STEM Program. The data collected was analyzed to measure
Elementary Afterschool STEM Program’s progress toward achieving their goals. Furthermore,
the analysis of this data was used to create strategies for Elementary Afterschool STEM Program
and Small Urban School District.
Site Description
Small Urban School District
Small Urban School District’s mission is to “provide the pathway for all students to attain
the expertise and develop skills of academic excellence that will empower them to: become
active lifelong learners; demonstrate respect for themselves and others in a dynamic and global
society; become responsible, informed, productive, independent and contributing citizens;
perform successfully in their chosen field and in society.” (Small Urban School District, 2014).
Small Urban School District is located at 1234 Education Way, Higher Learning, CA. There are
fourteen schools, nine of which are elementary. Throughout the 2013-2014 school year, SUSD
educated 13, 401 students.
Small Urban School District is unique in educating students in an afterschool program
called Elementary Afterschool STEM Program. Elementary Afterschool STEM Program stands
for Afterschool Program Promoting Learning and Enrichment. There are more than 700 students
at seven sites within the district that participate in this afterschool opportunity. The unique
mission of Elementary Afterschool STEM Program is to “engage students in enrichment courses
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 21
and activities to explore and develop their talents while gaining critical thinking skills needed for
success in school and beyond” (Small Urban School District, 2014b). Elementary Afterschool
STEM Program delivers STEM instruction in collaboration with the Los Angeles County Office
of Education (LACOE).
Sites. The elementary school site that I observed was Hope Elementary. Only one site
was chosen at this time. The data collected was then generalized to be representative of all
participating sites. Elementary Afterschool STEM Program indicated that their goal was to
create a model that could be replicated and implemented within SUSD’s other elementary sites in
the future. Therefore, we decided to focus on one school to collect data. Elementary Afterschool
STEM Program was then able to use the recommendations from the data that was presented to
inform decisions that will be useful when making future decisions on STEM programs in the
district and Elementary Afterschool STEM Program.
Hope Elementary served grades Kindergarten through sixth grade and had 861 students
total. Table 3 has a breakdown of the students that attended Hope Elementary by ethnicity and
grade (California Department of Education, 2014). Hope Elementary enrolled 100 of their
students in the Elementary Afterschool STEM Program afterschool program.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 22
Table 1
Hope Elementary School Demographic Information
Ethnicity
K Grade
1
Grade
2
Grade
3
Grade
4
Grade
5
Grade
6

Ethnicity
Hispanic
or Latino

76 57 104 74 82 89 81 65.38 %
America
Indian

0 1 0 0 2 0 0 0.34 %
Asian 6 5 2 6 4 4 4 3.60 %
Pacific
Islander

0 2 3 1 1 0 0 0.81 %
Filipino 4 5 2 3 2 2 2 2.32 %
African
American

24 18 30 25 21 30 19 19.39 %
White 9 6 10 3 5 7 9 5.69 %
Two or
more
races

2 1 6 3 4 3 2 2.43 %
Totals 121 95 157 115 121 135 117 99.96%
Note. The raw number represents the number of students in each grade level.
Elementary Afterschool STEM Program had seven classrooms that were used at Hope
Elementary. While most classrooms were taught independently, some classrooms were used by
two grades simultaneously. The afternoon was broken into three segments where the students
rotated between three classrooms during each of these three segments. Each classroom was
organized by activity being taught.
Site staff. Elementary Afterschool STEM Program had various staff members that
formed its organizational structure within Small Urban School District. Jane Doe was the Project
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 23
Coordinator and director and worked exclusively with Small Urban School District. John Doe
was the Curriculum Specialist and worked for both Elementary Afterschool STEM Program and
Small Urban School District. Along with the district staff, each site had a site coordinator that
oversaw instruction and the site staff members. Each site had various numbers of tutors to
accommodate their instructional needs. Elementary Afterschool STEM Program worked with
LACOE in various capacities. One way in which Elementary Afterschool STEM Program and
LACOE collaborated was through overseeing the progress of Elementary Afterschool STEM
Program in their STEM education development. Additionally, LACOE helped with establishing
the Scope of Work (see Appendix A) with the University of Southern California’s (USC) Rossier
School of Education.
Instrumentation
Student Career Awareness in STEM Survey
Kier, Blanchard, Osborne, & Albert, (2014) developed a STEM career interest survey for
middle school students. This survey was used to measure the awareness of future careers in
STEM. The goal of STEM career awareness is to create interest and innovations, while having
students recognize their academic strengths. Kier et al. (2014) argue that interest in STEM begins
in middle school. However, this survey attempted to assess student awareness of possible
careers in STEM (Kier et al., 2014). While the survey was originally created for middle school
students, it was adapted to meet the needs of Small Urban School District and Elementary
Afterschool STEM Program’s student population of first through sixth grade students.
Additionally, the survey was modified to meet Elementary Afterschool STEM Program’s year
two goals.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 24
Staff Knowledge of STEM Survey
In order to accurately analyze the effectiveness of a program, the extent of the staff’s
prior knowledge in a specific area needed to be considered. Therefore, we assessed the staff’s
STEM knowledge to better support Small Urban School District and Elementary Afterschool
STEM Program’s needs. Aydeniz and Kirbulut (2014) identified problems of measuring
Pedagogical Content Knowledge (PCK) in teachers in light of reform efforts. As a result, they
designed an instrument called Secondary Teachers’ Science Pedagogical Content Knowledge,
for measuring teacher a teacher’s prior content knowledge. Their tool categorizes responses into
three categories: naive, developing, and sophisticated (Aydeniz & Kirbulut, 2014). This tool was
used with modifications to support in the completion of our study for Small Urban School
District and Elementary Afterschool STEM Program.
The questions included on the survey were taken from valid sources and from the project
evaluator. Some items on the survey needed to be modified in order to reflect the needs and
interests of Elementary Afterschool STEM Program. The focus of the questions centered on the
areas of needs addressed by Small Urban School District. We chose these instruments to be used
because research has demonstrated their reliability and validity. The outcomes of these survey
results were analyzed quantitatively.
Questionnaire
A questionnaire was created and sent to the Elementary Afterschool STEM Program
coordinator who then distributed it to the proper participants. All questionnaires were
anonymous and completed voluntarily. The purpose of these questionnaires was to collect data
that was different from the data collected on surveys.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 25
Document Analysis
The analysis of documents from Small Urban School District was a form of data
collection of artifacts from Elementary Afterschool STEM Program. The uses of their documents
demonstrated an alignment of goals via curriculum, agendas, and calendars. These resources
were a reflection of the work and pedagogy of Elementary Afterschool STEM Program. The
purpose of document analysis was to measure the program’s goals with the tools and resources
they implemented.
Observations
Observations took place to see the impact Elementary Afterschool STEM Program’s
STEM program in action. These observations were one way to collect data on environments that
allowed for detailed and specific data to be collected. In education, observational data collection
is a method for collecting data on classrooms, pedagogy, and facilities.
The DoS measurement tool was created by Dr. Gil Noam to be used in a systematic way
for assessing quality in a range of information STEM out-of-school (OST) programs (Program in
Education, Afterschool and Resiliency, 2014). Funding was used from the National Science
Foundation, the Program in Education, Afterschool, and Resiliency (PEAR) as well as partners at
the Educational Testing Service and Project Liftoff collaborated to create the DoS. This tool was
intended to be used in afterschool and summer programs that focus on STEM content in informal
learning environments such as nature centers, afterschool clubs, universities, boys and girls
clubs, museums and science centers, YMCA and community centers as well as hospital-
sponsored programs.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 26
DoS Observation Tool
The DoS is comprised of twelve dimensions with four levels of quality (See Figure 1
below).

Figure 1.
An overview of the DoS observation tool.
These twelve dimensions were originally based on the five Impact Categories in the 2008
NSF Framework for Evaluating Impacts of Informal Science Education Projects. The features of
the learning environment include organization, materials, and space utilization. Activity
engagement includes participation, purposeful activities, and engagement with STEM.
Reflection, Inquiry, and STEM Content Learning make up the STEM knowledge and practices
while the youth development in STEM is comprised of relationships, relevance and youth voice.
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 27
In order to observe and rate an out-of-school STEM lesson using DoS, one must
complete DoS training and earn certification by the PEAR team. Participants in the certification
process are introduced to the twelve dimensions with video examples and cases from the field.
Next, they practice rating using video simulations and then discuss the rating until a consensus is
reached. Participants are then asked to complete two practice ratings in the field and receive
personal feedback from the DoS team before becoming DoS certified.
On Monday, June 22, 2014 and Tuesday, June 23, 2014, I began my DoS certification
process. Prior to those two days, I was required to watch a preselected video, entitled Flubber, to
gain an understanding of what I would be evaluating in the future. Additionally, I received a
packet in the mail that had the DoS handbook as well as rating sheets.
Our training took place in a virtual classroom through Adobe Connect. There were three
facilitators, Dr. Ashima Shah, Keri Carvalho, and Jeff Buehler. While each of the three
facilitators took turns presenting the twelve domains, Dr. Shah took responsibility as lead
facilitator and clarified main points throughout the training. Over the course of the two days,
each domain was described in detail. After discussing each domain, we watched preselected
videos and took field notes that included specific details and evidence that described the specific
domain. When the video was over, we used our rubric to assign a rating. The final step was to
cite specific evidence that supported the rating that we assigned.
At the completion of day two, I was mailed a post-training packet electronically. This
gave me instructions on four preselected lessons that I needed to watch and rate on different
domains. Once I watched the videos, took notes, assigned a rating and provided evidence to
support our rating, I uploaded my rating and evidence via Survey Monkey. The next step in the
certification process was a calibration call with DoS and awaiting individual feedback from Dr.
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 28
Shah on my lessons. I completed my calibration call on Monday, July 7, 2014. This was an
hour-long phone call with Dr. Shah and five other participants. Those of us present on the
calibration call reviewed any common themes among our ratings and discussed the next steps in
the Dos certification process.
The next step was to complete two field observations and upload them to Survey Monkey
for feedback from Dr. Shah and the Dos team. I conducted my first observation at the Tiger
Woods Learning Center, Orange County (TWLC-OC) on Wednesday, July 16, 2014 for a
robotics lesson and my second observation at Disneyland on Wednesday, July 23, 2014 for a
lesson on kinetic and potential energy. After those two lessons were observed, I assigned a
rating, provided evidence to support my rating and uploaded them to Survey Monkey one last
time. I received my confirmation for my Dos Certification on August 8, 2014.
Data Collection
Sequence of Completed Work for this Project
Initial meeting. There was initial meeting on April 23, 2014 that was conducted over the
phone to interview and introduce the project goals and expectations to the participating members
(Small Urban School District, LACOE, and USC). During the meeting, Small Urban School
District described their three-year plan for their current STEM program that described its
development and implementation to date. They also described a few measurable goals that
allowed them to monitor their progress, i.e. surveys, student-learning outcomes, utilization of
Bloom’s Taxonomy and Anderson and Krathwohl’s Depth of Knowledge (DOK) during
instruction, and staff confidence. A lack of content knowledge among Small Urban School
District and Elementary Afterschool STEM Program’s staff as well as a lack in confidence about
STEM lesson implementation was also acknowledged. Small Urban School District mentioned
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 29
that each site had a limited supply of resources available for its participants. It was identified
that their program’s participants were first through sixth grade students at seven sites within
Small Urban School District with approximately 100 students at each site.
Among their goals, Small Urban School District wanted to 1) increase staff and students
positive “attitudes” towards STEM fields, 2) Participate in the science fair, 3) Teach
metacognitive skills to their participants, 4) Train their staff in the use of the scientific method
and its application to engineering and the curriculum’s activities, 5) Incorporate more technology
in their programs, 6) Connect the STEM activities to real-world, “STEM career clusters.” 7)
Support staff in gaining confidence in the use of STEM lessons from their curriculum, SCI-
Gineering.
Site visit. In May, there was a site visit at Hope Elementary to establish the guidelines for
the project and Small Urban School District’s expectations of the USC research group. During
this site visit, it was determined that an initial needs assessment was already completed for their
year one implementation of Elementary Afterschool STEM Program. This initial needs
assessment survey was administered by UCI. UCI’s data from this survey will be included in our
analysis of documents.
From the initial meetings, it was established that further data needed to be collected in
regards to the initial needs of the program. The data was used to determine future strategies and
recommendations for Elementary Afterschool STEM Program. The types of data collected were
in the form of surveys, observations, interviews, and document analysis.
Scope of work. After the initial meetings, a Scope of Work was created in collaboration
with LACOE between Small Urban School District’s Elementary Afterschool STEM Program
and the USC Rossier School of Education (See Appendix A).
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 30
Conference call. A conference call took place on July 24, 2014 to calibrate the parties
involved with this project and update the partnership on progress that had been made. There was
a discussion about next steps and a timeline created for those next steps. During this call it was
established that the preliminary UCI needs assessment data would be collected on July 30, 2014
and follow-up meeting would occur on August 13, 2014 at Small Urban School District.
8/13/14 Meeting. A meeting was held at Small Urban School District’s annex office. In
the meeting, there was a discussion the topics and plans for the project. It was determined that
through the scope of the project, we would examine the current goals of Elementary Afterschool
STEM Program. While specific goals and data were not identified, topics of STEM attitudes,
Elementary Afterschool STEM Program’s alignment, and modes of data collection were
discussed. Additionally, an overview of the Scope of Work was addressed to clarify any
questions that existed with all partners involved in the project.
IRB. This project was designed in collaboration with Small Urban School District, the
Los Angeles County Office of Education and USC to collect observational data from human
subjects. Therefore, university research approval was needed for data collection to take place.
In order to gain approval, there were a few steps that took place. First, I designed a study
through iStar. The next step required the department and faculty advisor to signoff of our
proposal to help ensure that the proposal was adequate. Once our study was signed-off, it was
sent to the IRB office for approval. The approval from the Institutional Review Board (IRB) was
achieved on 10/15/2014.
Survey administration. In order to have collected data for the survey, students from
Hope Elementary School were instructed to complete the computer survey during a class session.
Data was collected through Qualtrics, a private research software company that creates online
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 31
survey software and hosts an insight platform. The results of the survey were analyzed for
comparison of the different groups that were predetermined: administration, tutors, and students.
As requested by Small Urban School District, Joseph Calmer and myself looked for STEM
career interest in the students. The surveys indicated the level of student awareness of STEM.
Questions regarding the career focus of the students were also included since an increase in
STEM career focus was a goal. Furthermore, the knowledge of staff pedagogical and content
knowledge will need to be surveyed. The survey results were used to determine the initial level
and trajectory for our recommendations.
Questionnaire. A questionnaire was designed by my research partner, Joseph Calmer, as
a method for collecting data from administrators in the form of open-ended questions about
Elementary Afterschool STEM Program. The results were analyzed qualitatively to determine if
there were any recurring themes amongst staff members.
Document analysis. Joseph Calmer and I collected the curriculum, meetings, and
calendar documents from Small Urban School District and Elementary Afterschool STEM
Program for analysis. These documents were analyzed for STEM content, alignment towards
their identified goals, and for recommendations of improvement.
The documents that were collected were their curriculum, “Sci-Gineering,” the
Elementary Afterschool STEM Program organizational chart, meetings, and various forms of
organizational related materials. The organization’s curricula, calendars, and correspondences
were collected, reviewed and analyzed for themes of structure of the program, academic content,
rigor, and authenticity of the learning environment for their program.
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 32
We collected UCI’s initial needs assessment and evaluated the items to determine if it
was useful for our subsequent evaluation. The items that were pertinent and yielded results were
reported in the analysis chapter.
Observations. A classroom observation occurred at a time that was agreed upon by
Small Urban School District and myself. I conducted an observation within two different
classrooms during an hour-long period; thirty minutes was spent in each classroom. One third
grade classroom and one fifth grade classroom was observed at Hope Elementary using the DoS
observation tool.
During the lessons, I took field notes on the lesson being presented. I observed the
lessons looking for evidence to support each of the twelve domains that comprise the DoS
observation tool. The data collect was then analyzed using the methods described by DoS
(Program in Education Afterschool and Resiliency, 2014a).

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 33
CHAPTER FOUR: FINDINGS
Rebecca Acosta and Joseph Calmer coauthored this chapter
Findings
Introduction
This chapter presents the findings obtained from the evaluation of Small Urban School
District’s Elementary Afterschool STEM Program. Specifically, this project sought to examine
three areas of Elementary Afterschool STEM Program: 1) evaluating feedback on current STEM
programming during STEM implementation; including staff attitudes toward STEM
implementation and 2) feedback on professional development for STEM professionals;
specifically, those that are delivering the instruction to the students who participate in
Elementary Afterschool STEM Program. This chapter is divided into several sections including
the demographics of both the survey respondents and questionnaire participants, as well as a
discussion of the research findings from the quantitative and qualitative data collected. The
analysis of our findings of this needs assessment and evaluation were centered around the
following framing questions:
1. Does the staff at Elementary Afterschool STEM Program have a positive attitude
toward STEM and their LIAS principles?
2. To what extent is Elementary Afterschool STEM Program implementing the four
components of STEM in their daily plans to achieve their program’s goals?
3. To what extent does Elementary Afterschool STEM Program provide opportunities
for student voice and choice in their learning?

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 34
Participant Demographics
The participants for this needs assessment were Small Urban School District Elementary
Afterschool STEM Program tutors, site leads, site supervisors, and students (grades 3-6).
According to Elementary Afterschool STEM Program’s documents, Elementary Afterschool
STEM Program employed 39 tutors, 7 site leads, and 3 coaches with approximately 700 students
total throughout the program. Table 2 displays the number of surveys that were distributed,
returned, and used for the project analysis. Only surveys that were 100% complete were
considered valid and used for analysis in this project.
Table 2.
Summary of survey responses.

Sent Returned Analyzed Response rate
Site Leads N=15 N=15 N=15
100%
Site Supervisors N=8 N=8 N=8
100%
Students
N=293 N=293 N=266 91%
Tutors
N=24 N=24 N=20 86%

Study Participants
Tutors. Twenty-four tutors responded and completed the tutor survey, which
corresponds to an 86% completion rate. The participants who responded to the tutor survey were
from five of the six Elementary Afterschool STEM Program sites. Rainbow Elementary and
Shady Elementary tutors did not participate in this survey (see Table 3). Sixty-seven percent of
tutor survey respondents indicated that they have been at Elementary Afterschool STEM
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 35
Program for fewer than two years. Only two participants indicated that they have been employed
with Elementary Afterschool STEM Program for more than five years.
Table 3.
Elementary Afterschool STEM Program’s Tutor percentage and participation in this project
from each of their sites
Tutor school site Responses (N=20) Percent of total
Granada Elementary 2 10
Grace Elementary 7 35
Faith Elementary 2 10
Rainbow Elementary 0 0
Shady Elementary 0 0
Hope Elementary 4 20
Washington Elementary 5 25
Note. Only 20 tutors responded to this question in the survey. Rainbow and Shady Elementary
did not have any respondents.
Forty-four percent of tutors indicated that their age was between 18-21 years old and 44%
indicated that there age was between 22-25 years. Also, the age groups of 26-29 and above 30
are represented at 8% and 4%, respectively (see Figure 2).
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 36

Figure 2.
Tutor age distribution at Elementary Afterschool STEM Program.
All the tutors in Elementary Afterschool STEM Program were in enrolled in a college
and identified their undergraduate matriculation plans. The tutors had a range of degree
aspirations. Although 48% of the participants had an educational focus, only five respondents
indicated their degree focus as a science-related. Similarly, 28% indicated that their future career
goal was STEM-related. Figure 3 displays tutors’ responses about their future degree. According
to the tutor survey responses, 71% of Elementary Afterschool STEM Program tutors are in non-
STEM related fields.
18-‐21
years
old

22-‐25
years
old

26-‐29

years
old

>30
years
old

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 37

Figure 3.
Tutor current undergraduate degree matriculation plan
Site leads. Fifteen site leads responded to the site lead survey and nine completed the
entire survey for a 60% completion rate. The site leads are responsible for managing their tutors,
communicating site effectiveness to supervisors, and ensuring that Elementary Afterschool
STEM Program is implemented properly at their site. Full descriptions of site leads’ duties are
in Appendix K. The site leads’ survey responses indicated that the respondents were 63%
Hispanic and 38% Black or African American, which is a similar representation of the
demographic data of the tutors and students in Elementary Afterschool STEM Program. Seventy-
five percent of the respondents reported their highest degree attained was a high school diploma
while 25% reported that they earned a Masters degree. One hundred percent of the respondents
indicated their highest degree was not STEM related. The degree areas reported were
exclusively in Social Science and Education.
17%

21%

4%

3%

41%

14%

Science
(Biology,
Chemistry,
etc.)

Social
Science
(sociology,
history,

cultural
studies)

Arts

Engineering

Education

Undecided

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 38
Site supervisors. Site supervisors were identified as persons who work in the
administrative offices at Small Urban School District or among one of Elementary Afterschool
STEM Program’s contracts. A questionnaire was sent to Small Urban School District’s
Elementary Afterschool STEM Program coordinator, who then sent the questionnaire to any
person fitting the data collection criteria, as determined by Elementary Afterschool STEM
Program. There were a total of five respondents. Demographic information was not collected on
this group in order to protect the anonymity of the respondents and to maintain confidentiality of
the results.
Students. Elementary Afterschool STEM Program students were identified as those
students who participated in Small Urban School District’s Elementary Afterschool STEM
Program. The program is comprised of seven elementary schools within Small Urban School
District. For the purpose of this project, we analyzed the survey data from students at six schools
who were in fourth, fifth, and sixth grades. Additionally, fourth-grade classrooms at Hope
Elementary were observed. Rainbow Elementary was not included in this analysis because none
of the respondents indicated Rainbow Elementary as their current school site. The distribution of
respondents by grade level can be seen in Figure 4 below.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 39

Figure 4.
Students’ School Attended for Elementary Afterschool STEM Program
Framing Question Analysis
Elementary afterschool STEM program question one: staff attitude towards STEM.
The first framing question was: How does the staff at Elementary Afterschool STEM Program
perceive STEM and their LIAS principles? These survey questions were designed to determine
and assess the staffs’ attitude towards STEM, their understanding of STEM, and their
commitment to the LIAS principles (Learning in Afterschool and Summer) of Elementary
Afterschool STEM Program. This question was based on the following Elementary Afterschool
STEM Program goal to: “Focus on attitude about STEM; embed LIAS principles in STEM
programming” (see Appendix A).
Tutor survey. The tutors’ survey asked questions about tutors’ experiences working at
Elementary Afterschool STEM Program and the amount of training the tutors received on STEM
topics. According to the survey, 95% of the tutors responded that Elementary Afterschool
15%

23%

20%

0%

20%

22%

Granada
Elementary

Grace
Elementary

Faith
Elementary

Rainbow
Elementary

Shady
Elementary

Hope
Elementary

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 40
STEM Program has “always” provided a safe environment, improved students’ academic
performance, helped develop relationship with peers, and helped students develop new skills (see
Table 4 for statistics). According to their responses, tutors’ attitudes are committed to
Elementary Afterschool STEM Program’s LIAS principles. From the data, tutors responses
demonstrated a positive feeling towards the impact of Elementary Afterschool STEM Program
and its purpose. Tutors consistently felt positive about the educational environment at
Elementary Afterschool STEM Program (See Table 4). However, an area that was not consistent
with Elementary Afterschool STEM Program’s LIAS principles was the tutor’s perception of
their students’ desire to be a part of the Elementary Afterschool STEM Program. The mean
response for the tutors was 1.48 (SD=0.59). This indicates that tutors perceived few students had
a desire to be apart of Elementary Afterschool STEM Program.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 41
Table 4
Descriptive Statistics for Elementary Afterschool STEM Program’s tutor survey items about the
environment of Elementary Afterschool STEM Program
Tutors M SD
Our program provides a physically safe learning
environment for students.
1.04 0.21
Our program provides an emotionally safe learning
environment for students.
1.04 0.21
Our program supports improvement in student academic
performance.
1.00 0.00
The curriculum includes activities and approaches aimed at
improving the leadership and character development of
students.
1.13 0.34
I believe students like coming to the program. 1.48 0.59
Our program helps students to develop self-confidence. 1.09 0.29
Our program provides opportunities for students to build
friendships with peers.
1.04 0.21
Our program helps students develop new skills. 1.09 0.29

Note. Tutors’ responses to questions about Elementary Afterschool STEM Program’s
environment from the tutor survey. Not all tutors responded to this question (N=23).
The majority of tutors (82%) responded “always” to the questions about their preparation
and training for success in Elementary Afterschool STEM Program (M=1.17, SD=0.38). Despite
tutors reporting that they felt that their preparation and training was adequate, their responses
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 42
about activity space and program resources were varied. The results indicated that a significant
number of tutors (56%) “always” felt that space and resources were an impediment to their
assigned duties. Figure 5 presents the tutors’ responses to items concerning training, resources,
and program space.

Figure 5.
Statistics from tutors perspective on Elementary Afterschool STEM Program’s Resources and
Supplies, Training, and Instructional Space.
Tutors were asked if they received enough training on various topics, including STEM
content knowledge (SCK), science and engineering practices (SEP), and Elementary Afterschool
STEM Program’s mission. The tutor surveys indicated that 82% and 91% of the respondents felt
that they received enough training on Science and Engineering Practices (SEP) and Elementary
Afterschool STEM Program’s mission, respectively (see Figure 5). However, the tutors
perception of the amount of training of STEM Content Knowledge (SCK) and lab safety
responses were more varied, (82% and 65%, respectively). This means that tutors recognize and
0

10

20

30

40

50

60

70

80

90

Resources
and
supplies
Received

proper
training
Adequate
space

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 43
report that their level of content knowledge varied across Elementary Afterschool STEM
Program.

Figure 6.
Percent of tutors and their perspective on their training in STEM, science and engineering
practices and Elementary Afterschool STEM Program’s mission statement.
Site lead survey. The site lead survey covered topics related to the logistics and attitudes
toward Elementary Afterschool STEM Program. There were only 9 completed surveys.
Therefore, the sample size was smaller and more varied than the tutors’ surveys.
One question asked the site leads about their needs and priorities. The results showed a
variance among respondents (see Table 5). Training was reported 50% of the time as the highest
area of need, M= 1.75, SD=0.89. Conversely, site visits were reported 62% of the time as the
least area of need, M= 4.5, SD=0.76. The notion of coaching and consultation was only reported
76
78
80
82
84
86
88
90
92
94
96
STEM content knowledge Science and Engineering Processes Program Mission/ Vision
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 44
12% of the time as an area of need. These results indicated a desire to improve STEM pedagogy
through more training.
Table 5
Results from the site leads’ survey that identified the needs in Elementary Afterschool STEM
Program.
Statistic Percent Identifying
a ‘Pressing Need
M SD
Training 50 1.75 0.89
Information and
Resources
25 3 1.6
Coaching 12 2.13 0.64
Consultation 12 3.63 1.19
Site Visit 0 4.5 0.76

Note. Site Leads’ response about their sites’ needs and priorities from the site lead survey (N=8).
Elementary afterschool STEM program question two: amount of STEM
implemented. The framing question for this section was: To what extent is Elementary
Afterschool STEM Program implementing the four components of STEM in their daily plans to
achieve their program goals? This question was designed to assess the staff’s practice and
implementation of science and engineering content during the week, it is based on the
Elementary Afterschool STEM Program goal of building a strong science and engineering
focused curriculum (see Appendix A).
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 45
Tutor survey. Figure 7 shows the responses of the tutors regarding daily activities in
Elementary Afterschool STEM Program. The questions from this instrument had the options
“1= never, 2=less than once a month, 3= once a month, 4= 2-3 times a week, 5=once a month,
6=2-3 times a week, and 7= daily”. A mean of 6 or greater indicated that activities were
completed daily. The tutor survey indicated that tutors did not implement technology and
science in the same frequency across Elementary Afterschool STEM Program sites. The use of
their “Sci-Gineering” curriculum once a week was consistent with Elementary Afterschool
STEM Program’s goals of STEM lessons and tutor responses. These responses were consistent
with the Elementary Afterschool STEM Program’s curriculum/ activities planning calendar. The
tutor survey indicated that the promotion of healthy behaviors activities “always” (82%)
occurred and the use of Microsoft products “never” occurred (43%), M= 6.38, SD= 1.53 and
M=2.46, SD 1.77. Similarly, Elementary Afterschool STEM Program’s weekly plan indicated
that 2 days a week were spent with the curriculum “Kids Lit” and “Kids Math”, which are ELA
and math curriculums. The results indicated that sites are using these later forms of curricula 2-3
times a week, as verified by classroom observations.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 46

Figure 7.
Average results from tutors responses about their daily activities completed in classrooms at
Elementary Afterschool STEM Program
Site lead survey. Along with asking the tutors how much time they spent on the
classroom-based learning activities, site leads were asked about the amount of time spent
planning and supporting tutors with their classroom activities. The results of site leads responses
to STEM lesson planning are represented in Figure 8. The results show that 25% of site leads
spend 2-3 times per week planning a science activity (M= 2.63, SD= 0.52). Meanwhile, 12%
reported that math activities are planned daily (M=1.88, SD=0.35). Interesting 12% of site leads
indicated that engineering activities were not planned. The results reflected a varied
understanding of the curriculum and the nature of STEM programs.
0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Never
Less

than

Once
a

Month

Once
a

Month

2-‐3

Times
a

Month

Once
a

Week

2-‐3

Times
a

Week

Daily

Use
of
Microsoft
products

Kidz
Math

Kidz
Lit

Scigineering
curriculum

Science
based
activities

Technology
based
activities

Activities
promoting
healthy

behaviors

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 47

Figure 8.
Site Lead Responses to the Planning of Daily Activities
The site leads were asked about the amount of time supporting the components of STEM
(science, technology, engineering, and mathematics). According to the site lead survey, 87% of
the site leads supported math activities 2-3 times a week at their sites. Also, 62% of site leads
supported science and technology activities weekly. In contrast, site leads had conflicting reports
about engineering activities. Twelve percent reported that engineering activities did not occur,
while 87% state that they occurred monthly (M=3.25, SD= 0.71). This result indicated that the
understanding of the activities is not explicitly identified as STEM activities by the site leads.
Supervisor questionnaire. From the questionnaire responses, the site leads described
the development of Elementary Afterschool STEM Program’s STEM program as an evolving
process that relied on feedback from the previous year’s implementation. For example, a theme
0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Everyday
2-‐3
times/

week

Once
a
week
This
activity
is

not
offered

A
MATH
related
activity

An
ENGINEERING
related
activity

A
TECHNOLOGY
related
activity

A
SCIENCE
related
activity

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 48
emerged that building on the program’s strength is a goal of Elementary Afterschool STEM
Program’s improvement plan. One respondent stated, “Throughout this process we are aiming to
be mindful of youth choice and development as we identify and expand upon our partnerships in
education.” This suggests that Elementary Afterschool STEM Program site leads understand
that growth areas exist in the program. To build on their needs, the respondents consistently
reported that they primarily collected data on tutors and staff in the form of observations by
outside consultants and teachers on special assignment.
Elementary Afterschool STEM Program used curricular materials that are created by
venders and consultants. Respondents indicated that their role as a lead is not to generate the
STEM curriculum, but to focus on researching, facilitating, and discovering partnerships that
will enhance Elementary Afterschool STEM Program’s afterschool program. The facilitators
brought materials and activities from existing curriculum to their sites. The respondents all
indicated that the currently used materials and curriculum were “highly effective” and “very
beneficial” in student learning about STEM.
Two of the five respondents indicated that pre and post surveys were used to evaluate
Elementary Afterschool STEM Program students. One respondent stated that the UCI online
toolbox (survey) was used to measure student and tutor growth. From the data collected and site
administrator’s responses, the measurement of student growth was not done in an explicit and
systematic way. Eighty percent of respondents indicated that observations were used to measure
accountability in students’ understanding of the curriculum. From that, one respondent said that
collected observational data were used to inform subsequent meetings with staff.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 49
Elementary afterschool STEM program question three: student voice and choice.
Student survey. Students were asked to respond to a series of statements and identify
which best described them using a 4-point Likert Scale (1= This describes me, 2= This mostly
describes me, 3= This sort of describes me, 4= This does not describe me). Most students (64%)
responded positively to being able to ask their own questions about science (M=2.04). 44% of
students answered, “This describes me” while 21% of students answered, “This mostly
describes me” (See Table 6). Forty percent of students answered, “This describes me”
(M=2.03), when stating that the activities that they completed during Elementary Afterschool
STEM Program allowed them to think about what they were doing rather than simply following
directions. Fifty-six percent of the students (M=1.7) stated that the activities allowed them to
use their hands and mind. (See Appendix N for student results in its entirety.)
Table 6
Descriptive Statistics for Elementary Afterschool STEM Program student surveys.
Students N M SD
I am able to ask my own questions about science. 281 2.04 1.09
The activities I do allow me to think about what I am doing,
rather than simply follow directions.
278 2.03 1.04
The activities in the program allow me to use my hands and my
mind.
279 1.7 0.94
The data that was collected and analyzed on the student surveys contradicted the
observational data from the classroom. Possible explanations for this discrepancy will be
discussed in Chapter 5.
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 50
Student observations. In an effort to gain a better understanding of students’ role in
Elementary Afterschool STEM Program, the Dimensions of Success (DoS) observation tool was
used to pinpoint strengths and weaknesses as well as opportunities for students to interact with
STEM practices (PEAR, 2010). This instrument was used to collect qualitative data and assign a
rating in twelve different dimensions: organization, materials, space utilization, participation,
purposeful activities, engagement with STEM, STEM content learning, inquiry, reflection,
relationships, relevance and youth voice. For this project, students were observed in their
classroom on November 4, 2014. The lesson was observed for an hour. During the observation
tutor and student behaviors were recorded. The notes were then used as evidence to assign a
rating based on DoS’s 4-level rubric for increasing quality.
The lesson observed was part of Elementary Afterschool STEM Program’s Sci-Gineering
curriculum; Module #4, Build and Test, lesson 8. The purpose of the lesson was to continue
student learning about acceleration and how it affected the travel of a marble on a roller coaster.
Sixteen fourth grade students participated in the lesson: eight boys and eight girls.
Students were inside a general education classroom. Desks were all faced forward in a
traditional setting where the direct-instruction took place.
The results of this lesson focused on the fourth domain of the DoS Observational Tool:
Youth Development in STEM. This domain was selected because it mirrors Elementary
Afterschool STEM Program’s third essential question for this project: “To what extent does
Elementary Afterschool STEM Program provide opportunities for student voice and choice in
their learning?” (See Figure 9). It rated the facilitator-student interactions as well as student-
student interactions. More specifically, it magnified the way interactions were encouraged or
discouraged during participation in STEM activities for the duration of the observation, whether
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 51
or not the activities made STEM relevant and meaningful to students' everyday lives, and how
the interactions allowed youth to make decisions and have a voice in the learning environment
and community (www.pearweb.org/tools/dos.html, 2015). The DoS Observer Rating Sheet can
be found in its entirety in Appendix B.

Figure 9.
Results of Youth Development in STEM Domain
Relationships. Weak evidence existed to show that the relationships among students,
student-facilitator, and facilitator-facilitator were positive. Several examples of facilitator
comments included: “Demetrius!” “Get your hands off that!” “That sounds like an argument,
not a compliment.” “Boys. Boys. We’re not outside.” “Around! Around! That is over, not
around!” A few positive comments included: “Yours looks good,” while walking away. “You
did this right the first time.”
Relevance. Limited evidence of relevance was observed during the activity.
Additionally, limited effort was observed to make the lesson relevant to students’ lives. The
0
1
2
3
4

Youth
Voice

Relationships

Relevance

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 52
facilitator made an isolated reference to acceleration while peddling a bike when students were
copying the definition onto their paper but not during the hands-on part of the activity. The
conversation did not continue beyond the teacher’s reference to the bicycle.
Youth voice. Little evidence of activities encouraging student voice occurred while the
observation took place. The first tutor told a group of boys, “Don’t share with anyone. We’ll
share at the end,” while the second facilitator said to the students, “Remember what I said.” The
students immediately stopped talking and little conversation between students took place after
that.
Summary
Question One: Staff perception of STEM and LIAS principles.
The results from the tutor survey and staff lead survey indicated that the staff had a
positive attitude towards Elementary Afterschool STEM Program’s LIAS principles at the time
of this report, however their attitude towards STEM was not directly measured. Tutors felt that
Elementary Afterschool STEM Program helped students gain content knowledge and skills,
indicating program effectiveness. However tutors’ attitudes about STEM could only be inferred
from their positive outlook about Elementary Afterschool STEM Program’s mission and
principles. Elementary Afterschool STEM Program was able to increase their staffs’ attitudes
from their first year of implementation, based on the evidence. This shows that participants
perceived that Elementary Afterschool STEM Program and its consultants had increased their
effectiveness at training their tutors. The survey did not allow for the respondents to elaborate.
Therefore, an additional survey should be conducted to identify current and future needs.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 53
Question Two. Implementation of STEM for Elementary Afterschool STEM Program
Goals.
According to Elementary Afterschool STEM Program tutors, the data collected supported
the practices of Elementary Afterschool STEM Program completing STEM activities once a
week at the participating sites. Conversely, the site leads indicated that they spend 2-3 days a
week on planning a science and technology activity. Elementary Afterschool STEM Program’s
current calendar has STEM- based activities once a week, which does not support the research-
supported practices for an effective STEM program. Elementary Afterschool STEM Program
had a discrepancy with what activities were planned and what was actually delivered to the
students. The results showed that activities were not categorized equally or consistently across
the sites.
The respondents identified the activities on which they spent their afterschool academic
time. Elementary Afterschool STEM Program’s program calendar outlines STEM activities
occurring once a week. From the tutor surveys, site lead surveys, and questionnaire, STEM
activities occur once a week. Conversely, not all respondents agreed on which specific activities
are done or how they are categorized. The data support the notion that sites do differ in the types
of activities they perform in the classroom. From the data, it is clear that the nature of the
activities supported in the classrooms were not all identified as STEM. According to the results,
each site does engage in simultaneous activities and instruction, but the activities are minimal in
STEM or were not identified as such. Respondents of the tutor survey consistently reported that
they were doing certain activities, such as math, ELA, and “healthy activities,” but varied on the
how much they engaged the students in STEM activities. The student outcomes of Elementary
Afterschool STEM Program were not measured in student knowledge growth and their resources
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 54
were not assessed for effectiveness. The results supported the notion that all four components of
STEM were not practiced equally across the sites.
Question Three. Student Voice and Choice in Activities
The data collected on the student surveys indicated that the majority of the students had
positive self-efficacy related to science. Most students felt that they could ask questions about
science while in class. Additionally, students felt that they were able to use their hands and
minds during their Elementary Afterschool STEM Program time. While students responded
positively on the survey, the classroom observations provided some conflicting data. Students
were not given the opportunity to ask questions and have them answered. Most students were
actively engaged on the project but received minimal guidance and reference to the day’s
objective. The discussion of these questions and recommendations from these findings are
presented in the next chapter.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 55
CHAPTER FIVE: DISCUSSION
Rebecca Acosta and Joseph Calmer coauthored this chapter.
Summary
Elementary Afterschool STEM Program is an afterschool program that provides STEM
education to elementary students in the Small Urban School District. This project sought to
evaluate Elementary Afterschool STEM Program’s first year of implementation as well as a
needs assessment for year two and beyond. Surveys were distributed and completed by
Elementary Afterschool STEM Program tutors, site leads and students (grades 4-6 only).
Additionally, site supervisors completed a questionnaire. The data collected from these surveys
and questionnaires were combined with site observations and document analysis for
triangulation.
Our findings indicated that the tutors agreed with the mission and principles of
Elementary Afterschool STEM Program while simultaneously having ambiguous feelings
toward the actual structure of the academic activities within the program. The tutors felt they
received enough training to be effective. Moreover, the site leads identified training as a need.
Site leads also stated that STEM instruction occurred once a week. Additionally, site leads
stated that ELA and math activities occurred twice a week. The frequency of these subjects
being taught aligned with the program structure, but not Elementary Afterschool STEM
Program’s STEM learning goals. Elementary Afterschool STEM Program’s site leads also
reported, via questionnaire, that the only tutor evaluation and program evaluation that occurred
was through observations with a protocol for the next evaluative step.
All data that was collected helped provide evidence to support our recommendations for
Elementary Afterschool STEM Program’s effectiveness of STEM instruction and
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 56
implementation. Unexpected outcomes that were discovered throughout the project are also
discussed below as well as limitations that could have impacted the data collected.
Discussion of Findings
Unexpected Outcomes
During the analysis of results, some unexpected outcomes were identified. All of the site
leads indicated that their professional education and university degrees were not STEM related.
However, research supports the notion that content knowledge supports one’s pedagogical
knowledge (Carney & Indrisano, 2013; DeBoer, 2000). Therefore, Elementary Afterschool
STEM Program would benefit by having staff members that have formal knowledge of education
and policy, as well as content knowledge in STEM fields. Currently, California is in the process
of transitioning science standards (Spiegel, Quan, & Shimojyo, 2014). The NGSS are structured
in a way to utilize modes of scaffolding Crosscutting Concepts and Disciplinary Core Ideas
throughout a student’s educational career (NGSS Lead States, 2013). Elementary Afterschool
STEM Program can use these standards to frame their program and foster STEM learning in
their students.
In Chapter 4, the discrepancy between the results from student surveys and the student
observation were not expected. We would have expected the observation to validate student
responses on the survey. Students responded positively when asked about opportunities to add
voice to their learning, whereas in the observation, students were not given the opportunity to
express the learning that had taken place and make connections to their everyday life.
Furthermore, during the data collection phase of the project, it became clear that the
perception of respondents suggested that the curriculum being used was not meeting the
students’ needs or the program’s instructional needs. Although the students were being exposed
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 57
to many concepts in science, the perception was that retention of those lessons was not occurring
with the existing curriculum. However, the cause could be curricular, pedagogical, or
instructional. It is not known at this time.
Recommendations
After carefully analyzing the data collected from surveys, questionnaires, classroom
observations and existing data for this project, several recommendations have been made. These
recommendations are focused around the three framing questions: (1) Does the staff at
Elementary Afterschool STEM Program have a positive attitude toward STEM and their LIAS
principles? (2) To what extent is Elementary Afterschool STEM Program implementing the four
components of STEM in their daily plans to achieve their program’s goals? (3) To what extent
does Elementary Afterschool STEM Program provide opportunities for student voice and choice
in their learning?
Develop or revise organizational goals. Research has identified the characteristics of
effective organizational goals. At present, the goals in Elementary Afterschool STEM Program’s
strategic action plan do not fit the measures of effective goal development. Therefore,
Elementary Afterschool STEM Program should consider creating performance goals, short-term
objectives that are used for specific tasks, that support their intermediate and global goals
(Rueda, 2011). Once the performance goals are created, the gaps can be identified and the path
towards goal attainment can begin. These goals can be constructed through the creation of a
logic model, which would aid in measuring outcomes over time (Regional Educational
Laboratory, 2014; W.K. Kellogg Foundation, 2001).
Action item. The development of SMART goals needs to occur before the remaining
recommendations. A smart goal is specific, measurable, action-oriented, realistic and timely
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 58
(Smith, 1994). This can be accomplished with meetings and professional development for the
various stakeholders associated with Elementary Afterschool STEM Program. The meetings
should be focused on specific outcomes for developing a long-term vision and/or mission for
Elementary Afterschool STEM Program.
A logic model can also be developed to organize the vision and goals for Elementary
Afterschool STEM Program. The OC STEM Initiative has a long history of STEM program
implementation. Therefore, meetings with their leadership should be made to aid in the
development of Elementary Afterschool STEM Program’s vision and goals to ensure alignment
and effectiveness.
Measure student outcomes directly. The analysis of the first framing question
illuminated the need for Elementary Afterschool STEM Program to measure student knowledge
objectively, rather than by observation only. It was stated that teacher observations and other
anecdotal methods were used for student measurement, but students’ knowledge was not
measured directly. It is recommended that Elementary Afterschool STEM Program develop an
instrument to measure students’ content knowledge throughout the program. Collection of this
data could also be used to measure the effectiveness of curriculum and therefore aid in making
more informed program decisions in the future.
Action Item. 44% of students felt that science makes them feel “lost and in a jumble”.
Elementary Afterschool STEM Program needs to decrease this number. One way to help is to
analyze the way in which instructional time is being used. Tutors can collect data quickly
through the use of exit cards, increased time for student reflection, and the use of technology
applications on tablets and iPads. A suggested app is called Near Pod (www.nearpod.com). On
this app, tutors would generate quizzes and all students would answer simultaneously through
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 59
their own device. Another suggestion would be using a Senteo Interactive Response System.
With this, the tutor would pose a question and students would answer by using a clicker. Tutors
would have immediate feedback on student understanding and be able to adjust future
instruction.
Furthermore, the DoS observational tool should be used more frequently at all
Elementary Afterschool STEM Program school sites, and across all grade levels. This would
ensure continuity with measuring identified student outcomes. Increasing the use of the DoS
observational tool would give Elementary Afterschool STEM Program a better indication of the
quality of their program over time (Program in Education, Afterschool & Resiliency, 2015).
With an increase in observational frequency, Elementary Afterschool STEM Program would
build a database of lessons taught. Therefore, they could receive and develop customized reports
on trends and student outcomes across their sites.
Increase training. The analysis supports the need for more training in identified areas.
Training needs to be targeted and aligned with the organizational outcomes identified by
Elementary Afterschool STEM Program. Site leads have indicated that training was needed.
Therefore, a survey should be constructed to identify the specific needs of these tutors. This
recommendation would satisfy the action item on Elementary Afterschool STEM Program’s
current SAP.
Action item. It is recommended to use the National Academies Press as a resource to
gather research and training materials on the specific items of interest to Elementary Afterschool
STEM Program. Professional development should be created using the materials of existing
programs with effective professional development at this time. One such resource is the
Association of Supervision and Curriculum (ASCD). The ASCD offers professional
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 60
development in STEM, literacy, student engagement, and many others via recognized
professionals in educational research such as Douglas Fisher, Nancy Frey, Mike Schmoker, and
Robert Marzano.
Create tiered staff training. All staff could benefit from receiving regular training and
professional development. It should be noted that training in content and STEM-related fields
would be beneficial. Tiered staff training would consist of sessions that build upon prior content
knowledge as previous concepts had been mastered. It would enable Elementary Afterschool
STEM Program staff members to learn new strategies and concepts that are unknown to them.
Research has shown that proper content knowledge will increase staffs’ self-efficacy and
motivation in their organization (Schein, 2010). Despite the tutors indicating that they have
received enough training and the site leads indicating that training is a need, the cogent thing to
do for Elementary Afterschool STEM Program is to identify the training topics that need to be
expanded and progress from there.
Action item. Elementary Afterschool STEM Program needs to identify specific areas of
training based on current tutor content knowledge and pedagogies. Elementary Afterschool
STEM Program supervisors could create a questionnaire or survey with topics that meet their
program’s goals and current Strategic Action Plan. Based on the responses received, training
could be scheduled and implemented to increase tutor competency.
Implement STEM instruction. The full implementation of STEM instruction in
Elementary Afterschool STEM Program is recommended. While completing this needs
assessment, Elementary Afterschool STEM Program was delivering STEM lessons once a week
for one hour. The full actualization of STEM pedagogical practices would incorporate CCSS
Math and ELA. Therefore, all students’ learning would be positively affected. Based on the
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 61
surveys and observations, STEM instruction was not practiced in accordance with the
recommended research-based perspective at Elementary Afterschool STEM Program. However,
according to research, increasing STEM instruction should help ensure students learn science
and essential practices at a faster rate (Schiavelli, 2010).
Increased training and intentional Professional Development is also recommended.
Elementary Afterschool STEM Program’s weekly schedule should incorporate more STEM
activities that would compliment Common Core State Standards Math and ELA instruction. The
discrepancy between tutors and site leads understanding of STEM could potentially be
eliminated by increased training. Literature supports and recommends that STEM-based
activities occur daily in order to be effective (Gomez & Albrecht, 2013; Program in Education
Afterschool and Resiliency, 2014; Schiavelli, 2010).
Action item. Build upon the intended practices of the Next Generation Science Standards.
Practice the strategies from other partnership programs in the Orange County Science
,Technology, Engineering and Mathematics (OC STEM Initiative) and reevaluate their academic
calendars. According to the National Science Teachers Association, in order to achieve full
STEM Implementation, a program should include instruction, curriculum, assessment, teacher
preparation and professional development (National Science Teacher’s Association, 2015).
Additionally, we suggest that Elementary Afterschool STEM Program build relationships with
its stakeholders to help generate financial resources and increased public support to enhance
student learning.
Student voice and choice in learning. While the results of the student survey were
encouraging, the observational data from the DoS observational tool did not confirm the student
survey results. In fact, the results contradicted one another. Therefore, more research is
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 62
necessary to validate the results. It is recommended that Elementary Afterschool STEM
Program have a staff member trained in DoS so that they can maintain ongoing observations to
compare the delivery of instruction and quality of student learning over time (Program in
Education, Afterschool & Resiliency, 2015). Another option for Elementary Afterschool STEM
Program is to take advantage of the pre-established relationship with the OC STEM Initiative in
which regular observations can be made from an OC STEM

Initiative member and the quality of
instruction and student learning can be compared not only within Elementary Afterschool STEM
Program, but also across the participating schools in Orange County, California.
Furthermore, results from the student survey and what was noted during the classroom
observations indicated a need for an increase in student engagement. It is recommended that
Elementary Afterschool STEM Program build in an increased amount of time for students to ask
questions and reflect on their learning at the conclusion of each lesson.
Action item. Students need to be given the opportunity to explain their learning
experience and share their ideas and opinions about structuring the activities to complete the
project at hand (Program in Education, Afterschool & Resiliency, 2015). Elementary
Afterschool STEM Program students should also be given multiple opportunities to make
decisions and choices with tutor-selected constraints placed upon them. While completing the
daily STEM activity, students must be provided the environment where they feel they can take
personal and group responsibility for important aspects of their learning and participation in
activities. Additionally, giving students the opportunity to share their ideas outside of
Elementary Afterschool STEM Program to school and community members such as at school
board meetings and community gatherings would increase their voice.
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 63
Enhance current curriculum. Increased STEM pedagogy can be developed by the
implementation of various STEM resources. Currently, Elementary Afterschool STEM Program
uses Sci-Gineering; a curriculum that was designed for regular classroom teachers. Curriculum
is only effective when delivered properly by trained facilitators. It is recommended that
Elementary Afterschool STEM Program increase their pedagogical training for its staff, so that
the curriculum can be utilized more fully. The new NGSS standards have many identified and
vetted resources for STEM program development that can be used to support their current
curriculum (Committee on Integrated STEM Education, National Academy of Engineering, &
National Research Council, 2014; NGSS Lead States, 2013). Also, the NGSS standards are
congruent with the CCSS, so Elementary Afterschool STEM Program can increase their Math
and English by strongly adopting the practices identified by the NGSS.
Action item. Elementary Afterschool STEM Program should consider an alternative
STEM curriculum. An OST STEM curriculum needs to be student centered. It should be
focused on science and engineering practices rather than content alone. Students need
opportunities to be active and thinking about the activity they are engaged in. They should also
be given opportunities to collaborate and engage in habits of mind that enhances their
understanding and meaning-making of real-world phenomena. It is recommended that
Elementary Afterschool STEM Program look at the Consumers Guide to Afterschool Science
Resources for direction on choosing a curriculum that fits their program’s goals. The Science
After School (SAS) Consumer’s Guide was developed by the Coalition for Science After School
to help afterschool programs find high-quality materials to enhance their existing program. This
guide will provide the opportunity to read expert reviews of products, curriculum, activity kits,
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 64
instructor guides and more. You can find this website at:
http://www.sedl.org/afterschool/guide/science/.
Increase staff members with STEM interests. Along with the implementation of
NGSS and aligning NGSS tenets to their organizational goals, Elementary Afterschool STEM
Program should develop a recruitment plan to hire more STEM-degree focused undergrads.
Having staff that has the content background and interest will enhance their STEM program by
building interest in staff and students.
Stimulate students’ affective domain. It is recommended that more emphasis on
inquiry and constructive practices will increase engagement and increase students’ affective
domain. For example, adopting the Science Writing Heuristic would help students build content
knowledge while engaging them in the practices of science and engineering (Hand, Noton-Meier,
Staker, & Bintz, 2009). The development and integration of NGSS practices will explicitly make
the activities engaging and attractive to the students. The reestablishment of the partnership with
OC STEM Initiative should be used as a source for modeling the effective STEM program. The
identified stakeholders should visit and observe OC STEM Initiative programs for ideas and
networking.
Another way to stimulate students’ affective domain in STEM is through the use of trade
books. The use of trade books can cover a wide range of topics. Most importantly, they can be
used for science, engineering, literature, and any non-fiction-based topic. The National Science
Teacher’s Association produced a list of the year’s best trade book, organized by topic and
grade, each year. It can be found on their website

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 65
Limitations
This project collected data from students at six of seven elementary schools that
participated in Small Urban School District’s Elementary Afterschool STEM Program. This
could have impacted the outcome of data with over 100 student responses missing as well as
several tutors.
The site observations only took place at one participating school, Hope Elementary.
While this was Small Urban School District and Elementary Afterschool STEM Program’s
decision, it is difficult to assume that the results from the DoS observation tool were reflective of
all schools that participate in Elementary Afterschool STEM Program. Therefore, the results
cannot be generalizable.
Additionally, the tutors, those individuals who teach the lessons to Elementary
Afterschool STEM Program students, may have been nervous about being observed for fear of
being evaluated, among other reasons. This may have impacted the delivery of their lessons.
Finally, it was difficult to measure whether the participants of the surveys and questionnaires
answered honestly. The tutors may have felt pressure to answer the questionnaires but feared
that their job was on the line had they answered truthfully.
The wording of the questions on the surveys and questionnaires may have also affected
the results. For example, when most tutors’ responses varied on the situation of space and
resources, it may have been better to have had a follow-up, open-ended response section to hear
tutors’ feedback in the matter rather than only a selection of answers to choose from.
Furthermore, the questions were worded in a way that seemed to convey the investigator’s
intentions. However, during the project implementation and analysis, that intention was not
realized. This became apparent while analyzing the questionnaire. The responses seemed to
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 66
reflect the convoluted nature of the questions; therefore, the results may not be as precise and
reliable as would be hoped.
Elementary Afterschool STEM Program was very interested in ascertaining data about
the improvement of their staff’s attitudes towards STEM from year one. While analyzing the
tutor survey, it was realized that most tutors had only been employed with Elementary
Afterschool STEM Program for less than one year. Therefore, a measurement of growth could
not be reliable or valid. The results can only be applicable to this current year of
implementation.
Lastly, the survey and questionnaire items were generated by the authors and were
influenced by Elementary Afterschool STEM Program’s organizational goals and documents.
The data collected for this project was from Elementary Afterschool STEM Program’s site leads,
tutors, and students. Therefore, the recommendations are specific to Elementary Afterschool
STEM Program in Small Urban School District. The recommendations have limited
generalizability because of limited and targeted data collection. The data collected was on
Elementary Afterschool STEM Program’s specific goals and measures.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 67
References
Alkin, M. C. (2011). Evaluation Essentials: From A to Z. New York, NY: The Guilford Press.
Altschuld, J. W., & Witkin, B. R. (2000). From Needs Assessment to Action: Transforming
Needs Into Solution Strategies. Thousand Oaks, CA: Sage Publications, INC.
American Association for the Advancement of Science. (1993). Benchmarks for Science
Literacy. New York, NY: Oxford University Press.
Anderson, L. W., Krathwohl, D. R., & Bloom, B. S. (2001). A Taxonomy for Learning,
Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives.
New York: Longman.
Aydeniz, M., & Kirbulut, Z. D. (2014). Exploring challenges of assessing pre-service science
teachers' pedagogical content knowledge (PCK). Asia - Pacific Journal of Teacher
Education, 42(2), 147-120. doi: 10.1080/1359866X.2014.890696
Brown, R., Brown, J., Reardon, K., & Merrill, C. (2011). Understanding STEM: Current
Perceptions. Technology and Engineering Teacher, 70(6), 5-9.
Bybee, R. W. (1997). Achieving scientific literacy: from purposes to practices. Portsmouth, NH:
Heinemann.
Bybee, R. W. (2013a). The Case fo STEM Education: Challenges and Opportunities.
Arlington,VA: NSTA press.
Bybee, R. W. (2013b). Translating the NGSS for Classroom Instruction. Arlington, VA: NSTA
Press.
California Department of Education. (2014). Enrollment Data for Hope Elementary. Retrieved
6/4/2014, from
http://dq.cde.ca.gov/dataquest/Enrollment/EthnicGrade.aspx?cType=ALL&cGender=B&
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 68
cYear=2013-14&Level=School&cSelect=CRAIG%5EWILLIAMS%5EELEME--
HOPE%5EUNIF--1964303-6011654&cChoice=SchEnrAll
Carney, M., & Indrisano, R. (2013). Disciplinary literacy and pedagogical content knowledge.
Journal of Education, 193(3), 39.
Committee on Conceptual Framework for the New K-12 Science Education Standards. (2011). A
Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core
Ideas. Washington, DC: The National Academies Press.
Committee on Integrated STEM Education, National Academy of Engineering, & National
Research Council. (2014). Integration in K-12 STEM Education: Status, Prospects, and
an Agenda for Research. Washington: National Academies Press.
DeBoer, G. E. (2000). Scientiic Literacy: Another Look at Its Historical and Contemporary
Meanings and Its Relationship to Science Education Reform. Journal of Research in
Science Teaching, 37(6).
Fensham, P. J. (2009). Real world contexts in PISA science: Implications for context-based
science education. Journal of Research in Science Teaching, 46(8), 884-896. doi:
10.1002/tea.20334
Gomez, A., & Albrecht, B. (2013). True STEM education. Technology and Engineering
Teacher, 73(4), 8.
Hand, B., Noton-Meier, L., Staker, J., & Bintz, J. (2009). Negotiating Science: The Critical Role
of Argumentation in Student Inquiry. Portsmounth, NH: Heinemann.
Kier, M. W., Blanchard, M. R., Osborne, J. W., & Albert, J. L. (2014). The Development of the
STEM Career Interest Survey (STEM-CIS). Research in Science Education, 44(3), 461-
481. doi: 10.1007/s11165-013-9389-3
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 69
McComas, W. F. (2006). Science Teaching Beyond the Classroom. The Science Teacher, 73(1),
26-30.
National Academy of Engineering and National Research Council. (2014). STEM Integration in
K-12 Education: Status, Prospects and an Agenda for Research. Washington, DC: The
National Academies Press.
National Science Teachers Association. (2015). NSTA Position Statement. Retrieved 3/23,
2015, from nsta.org
NGSS Lead States. (2013). Next Generation Science Standards: For States, By States.
Washington, D.C.: Achieve, Inc. on behalf of the twenty-six states and partners that
collaborated on the NGSS Retrieved from http://www.nextgenscience.org/next-
generation-science-standards.
Noam, G. G., Barry, S., Moellman, L. W., Dyken, L. v., Palinski, C., Fiore, N., & McCouch, R.
(2004). The Four Cs of Afterschool Programming: A New Case Method for a New Field.
Afterschool Matters, Occasional Paper Series.
OECD. (2010). PISA 2009 Results: What Students Know and Can Do.
Osborne, J., Simon, S., & Collins, S. (2003). Attitudes towards science: A review of the literature
and its implications. International Journal of Science Education, 25(9), 1049-1079. doi:
10.1080/0950069032000032199
Program in Education Afterschool and Resiliency. (2015). Dimensions of Success. Retrieved
3/22, 2015, from pearweb.org
Program in Education Afterschool and Resiliency. (2014a). Dimensions of Success. Retrieved
7/26, 2014, from http://www.pearweb.org/tools/dos.html
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 70
Program in Education Afterschool and Resiliency. (2014b). DoS Program Planning Tool: A
Guiding Framework for Successful Afterschool STEM Programs. Cambridge, MA:
Harvard Medical School Teaching Hospital.
Regional Educational Laboratory. (2014). Education Logic Model Application. from
http://relpacific.mcrel.org/ELM.html
Rueda, R. (2011). The Three Dimensions of Student Performance. 978-0807752401: Teachers
College Press.
Schein, E. H. (2010). Organizational Culture and Leadership. San Francisco: Jossey-Bass.
Schiavelli, M. (2010). STEM education needs long-term commitment. Central Penn Business
Journal, 26(37), 12-12,15.
Small Urban School District. (2014a). Small Urban School District Website. Retrieved 7/26,
2014, from http://www.SUSD.k12.ca.us/Index.html
Small Urban School District. (2014b). Early Childhood Education. Retrieved 7/1/2014, from
http://www.SUSD.k12.ca.us/projectApple.htm
Smith, H. A. (1969). Science: Trends and Dilemmas. In D. G. Hays (Ed.), Britannica Review of
American Education (Vol. 1). Chicago, Il: Britannica Reviews.
Spiegel, J., Quan, A., & Shimojyo, Y. (2014). Planning Professional Learning Using the NGSS
Implementation Pathway Model. California Classroom Science, 27(3).
W.K. Kellogg Foundation. (2001). Logic model development guide: Using logic models to bring
together planning, evaluation, & action.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 71
Appendix A
Elementary Afterschool STEM Program/ USC: Scope of Work
07/29/14

Elementary
Afterschool
STEM
Program/USC
Rossier
School
of
Education

Scope
of
Work

This agreement ("Agreement") establishes a Scope of Work between the following parties:
Elementary
Afterschool
STEM
Program,
Small
Urban
School
District

1234
Education
Way

Higher
Learning,
CA
43210

And

University of Southern California, Rossier School of Education
3470
Trousdale
Parkway

Los
Angeles,
CA
90089

The purpose of this Scope of Work is to outline the details of the partnership between
Elementary Afterschool STEM Program and the USC Rossier School of Education. Through the
partnership, USC Doctoral students will provide consultative support to Elementary Afterschool
STEM Program Afterschool programs around the topic of Science Technology Engineering and
Math (STEM) as part of their dissertation program.
Elementary Afterschool STEM Program afterschool STEM needs

The Elementary Afterschool STEM Program team identified several areas of need that they
would like assistance with:
• List of effective STEM programs
• Evaluative feedback on current STEM programming
• Feedback on PD for STEM professionals
• Suggestions and recommendations for future directions
• A final report that includes an executive summary of findings
• Access to the complete findings in dissertation form

USC capacity

USC agrees to provide the following supports to Elementary Afterschool STEM Program:

Support options
• Help analyze pre-existing needs assessment
• Develop an evaluation of 2014-1015 STEM programming
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 72

In addition to those supports, the graduate students have also offered additional added value
based upon their own professional expertise.

• Provide recommendations on professional development opportunities and impact
(Joseph)

Timeline for rollout and implementation

May-June 2014

USC team will provide Elementary Afterschool STEM Program with:
• Biographies for the students involved in the project
• A sample letter for requesting approval for research to be distributed to appropriate
district representatives.
• A list of data or information they will need to complete their recommendations.

Elementary Afterschool STEM Program will provide the USC team with:
• Copies of your needs assessment (pre and post data)
• Information about professional development efforts up until this point (providers and
content)
• Elementary Afterschool STEM Program STEM inventory (products, curriculum, kits,
etc.)

Both USC Team and Elementary Afterschool STEM Program
• Develop a Memorandum of Understanding defining your partnership
USC Students at this time will also start to develop their dissertation proposals and prepare to
defend them.
Elementary Afterschool STEM Program team should identify any administrative barriers to
USC students collecting data and developing a needs assessment. Please secure a letter of
approval from the appropriate source (Superintendent, Board, etc.) to continue with the project
that addresses and removes those barriers.
July-August 2014
USC
Team

• Applying
for
IRB
approval

• Defending
dissertation
proposals

• Provide
feedback
on
needs
assessment

• Developing
evaluation
plan
and
tools
(interviews,
surveys,
focus
groups,
etc.)

• Provide
recommendations
on
blended
learning

• Provide
recommendations
on
professional
development

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 73

Please
keep
in
mind
that
the
USC
students
will
be
providing
those
recommendations
based
upon

data
that
Elementary
Afterschool
STEM
Program
has
already
collected.

The
more
robust

comprehensive
evaluation
report
will
be
delivered
at
the
end
of
the
year.

Elementary
Afterschool
STEM
Program
Team

• Planning
for
programs
opening

• Securing
curriculum

• Development
of
professional
development
plan

• Solidifying
partnerships/aligning
resources
based
on
feedback

•
Preparing
stakeholders
for
implementation
of
STEM
initiative

September-‐
October
2014

USC/Elementary
Afterschool
STEM
Program
Team

• Implementation
of
2
nd

year
of
Elementary
Afterschool
STEM
Program
STEM
strategic

initiative

• Survey
distribution

• Focus
groups

• Interviews

November
–December
2014

USC/Elementary
Afterschool
STEM
Program
Team

• Collection
of
surveys

• Analyzing
data

January-‐April
2015

USC
team

• Creation
of
Evaluation
report

• Defense
of
work

May
2015

USC
Team

• Delivers
final
evaluation
report
to
Elementary
Afterschool
STEM
Program.

Elementary
Afterschool
STEM
Program
Team

• Hosts
meeting
to
review
results
and
recommendations.

Afterschool
Technical
Assistance
Unit
Recommendations

1.

We
recommend
that
the
USC
team
clearly
define
the
evaluation.
The
"evaluation"
tool
(or
suite

of
tools)
and
process
should
be
operationalized
explicitly
so
that
all
parties
are
clear
about
what

will
occur
when
and
with
whom.

For
example,
some
questions
that
should
be
addressed

include:

(1)
Is
this
evaluation
process
going
to
involve
surveys,
focus
groups,
and
interviews?,
(2)

Who
will
be
the
target
audience
(e.g.,
Program
Director,
Site
Coordinator,
Instructional
day
staff)

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 74
for
each
of
the
data
collection
instruments?,
(3)
which
school
sites
will
participate
in
the
study
as
a

whole
(e.g.,
elementary,
middle,
high),
and
(4)
how
will
the
results
be
aggregated
and
reported?

2.

We
recommend
you
pre-‐schedule
conference
calls
for
the
entire
year
to
occur
at
least
once
a

month
to
maintain
momentum
and
alignment.

3.

We
also
recommend
you
plan
and
schedule
at
least
2
in
person
meetings
that
include
all
parties

sometime
between
now
and
the
end
of
the
2015
school
year.

4.

Each
team
should
begin
to
gather
the
documents
as
stated
in
the
timeline
and
prepare

deliverables.

This
partnership
will
end
after
the
final
meeting
to
review
the
results
of
the
evaluation
and
to

discuss
recommendations.

That
meeting
should
occur
in
May
of
2015.

Onward:
• List of effective STEM programs
• Direction of STEM education and integration

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 75
Appendix B
DoS Rating Sheet

OBSERVER RATING SHEET

Date of Observation:
Site/Location Name:
Observer Name:
Co-Observer Name:

STEM Focus (science, technology, engineering, or mathematics program?):
•
Curriculum/Module Name (e.g., Building Bridges, Lego Robotics, Fun with Food):
•
Activity Name/Description (e.g., students are racing their Lego cars on a track):
•
Facilitator/Activity Leader/Teacher Name (First, Last):
•
Context/Background of Lesson:
•
What happened last time?
•
Where does this lesson fall in the overall unit (beginning, middle, end)?
•
Description of Setting:
•
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 76
Description of Program Participants:
•
How many participants?
•
Female/Male breakdown
•
Age
•

FIELD NOTES

Dimension Evidence Overall
Rating
Organization
Materials
Space
Utilization

Participation
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 77
Purposeful
Activities

Engagement
with STEM

STEM
Content
Learning

Inquiry
Reflection
Relationships
Relevance
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 78
Youth Voice

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 79
Appendix C
IRB-Child Assent Form
University of Southern California
Rossier School of Education
Waite Phillips Hall, 3470 Trousdale Parkway, Los Angeles, CA 90089

CHILD ASSENT FORM

Title: Needs Assessment and Program Evaluation for SUSD’s Elementary Afterschool STEM
Program

Rebecca Acosta and Joseph Calmer are graduate students at USC and want to learn about
students’ interest in science, technology, engineering and mathematics (STEM). We hope to find
out what interests students in these STEM subjects. One way to learn about it is to conduct a
research study; the people conducting the study are called researchers.

Your mom/dad/Legal Guardian have told us we can have you answer questions about the study.
You can also talk this over with your mom or dad. It is up to you if you want to take part. You
can say “yes” or “no.” No one will be upset if you don’t want to take part.

This study will be a regular part of your Elementary Afterschool STEM Program activities.

You will receive your regular STEM lessons. You may be asked about what you learned in your
lesson or whether you liked the activity. You may also be asked questions about what you
learned from the activity.

Researchers do not always know what will happen to people in a research study. We don’t
expect anything to happen to you, but you might not like to answer the questions.

Your teacher or parents will not have access to your answers. You will not be graded on answers
you provide in the study.

If you have any questions, you can as one of the researchers.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 80
Appendix D
IRB-Information Sheet for Parents
University of Southern California
Rossier School of Education
Waite Phillips Hall, 3470 Trousdale Parkway, Los Angeles, CA 90089

INFORMATION/FACTS SHEET FOR NON-MEDICAL
RESEARCH—Information Sheet for Parents

Title: Needs Assessment and Program Evaluation for SUSD’s Elementary Afterschool STEM
Program

PURPOSE OF THE NEEDS ASSESSMENT/EVALUATION
This study explores your child’s learning in Small Urban School District’s Elementary
Afterschool STEM Program after-school Science, Technology, Engineering, and Mathematics or
STEM program. We hope to find out more about your child’s motivation and efficacy for
STEM. Your child is invited to be in this study because he or she is a student in the after-school
program where the evaluation is being conducted.

This project will be a regular part of your child’s regular instruction. In other words, the
instruction will take place during the after-school instruction as part of the planned instruction.
There will be no added requirements outside of the classroom. You child’s participation in the
study is completely voluntary.

PARTICIPANT INVOLVEMENT
Your child has received lessons about STEM. Your child may be asked about what he or she has
learned in the after school STEM program, if he or she is interested in science, and if he or she
knows what careers in Science are available to him or her. The study will take place one day and
will only take 10-15 minutes to complete a survey.

ALTERNATIVES TO PARTICIPATION
You and your child’s relationship with the school will not be affected, whether or not your child
participates in this study. Your child will be asked to continue with his or her regular class
activities, and will not be asked to complete the survey or participate in the focus group, if s/he
doesn’t want to participate.

CONFIDENTIALITY
All data will be coded by assigning a numerical code. No names will be attached to the
information collected. We will not use the name of your student in anything that we produce.

Your child will not be graded on participation in the study. Your child’s teachers will not know
whether or not your child was one of the participants.

Data will be stored in a computer which is password protected.
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 81

An executive report will be created for SUSD and Elementary Afterschool STEM Program at the
conclusion of this project. Information obtained from this project will be included in the report.
However, no identifiable information will be included.

The members of the research team and the University of Southern California’s Human Subjects
Protection Program (HSPP) may access the data. The HSPP reviews and monitors research
studies to protect the rights and welfare of research subjects.

The data will be stored for future use. When the results of the research are published or
discussed in conferences, no identifiable information will be used.

INVESTIGATOR CONTACT INFORMATION
Rebecca Acosta at rmacosta@usc.edu and Joseph Calmer calmer@usc.edu
Ed.D. Students
Rossier School of Education
University of Southern California
3470 Trousdale Parkway
Waite Phillips Hall
Los Angeles, CA 90089-4036

IRB CONTACT INFORMATION
University Park Institutional Review Board (UPIRB), 3720 South Flower Street #301, Los
Angeles, CA 90089-0702, (213)821-5272 or upirb@usc.edu

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 82
Appendix E
IRB- Elementary Afterschool STEM Program Student Survey
Surveys
Students:

(Please note: The follow questions will have a Likert Scale: “I strongly agree, I agree, Somewhat
agree, I do not agree, Not at all.”)

Grade 4-6

Understanding of STEM tools of practice
1. I am comfortable with using technology.
2. I know the tools that scientists use.
3. I know the tools that engineers use.
Participation in STEM learning opportunities
1. My afterschool program has help me in my daily classroom
2. My afterschool program has help me in my daily classroom, especially in science

Scientific method
1. I understand that science uses a process of observing, collecting data, and explaining
evidence
2. I think the process of science can be used in other fields
Student generated focus
1. I get to create my own questions about science
2. The activities I do allow me to think about what I am doing, rather than simply follow
directions
Active: hands on
1. The activities in the program allow me to use my hands and my mind
2. I get to use lots of tools and materials during STEM instruction

STEM content knowledge
1. I can name all the parts of STEM
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 83
2. Completing the activities really lets me think like a scientist
3. I am more interested in participating in a Science Fair, it is was available, now that I have
participated in Elementary Afterschool STEM Program
4. I have been able to get better grades on my homework because of Elementary
Afterschool STEM Program
5. I am interested in pursuing answers to scientific or engineering questions
6. I feel comfortable with science topics

Demographics:
1. What is your grade?
a. 4
th
grade
b. 5
th
grade
c. 6
th
grade
2. What is the name of your school?
a. Albert Baxter Elementary
b. Grace Elementary
c. Faith Elementary
d. Rainbow Elementary
e. Washington Elementary
f. Hope Elementary
g. Frank Woodruff Elementary

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 84
Appendix F
IRB-Information Sheet for Adults
University of Southern California
Rossier School of Education
Waite Phillips Hall, 3470 Trousdale Parkway, Los Angeles, CA 90089

INFORMATION/FACTS SHEET FOR NON-MEDICAL
RESEARCH—Adult Permission Form

Title: Needs Assessment and Program Evaluation for SUSD’s Elementary Afterschool STEM
Program

You are invited to participate in a needs assessment/evaluation conducted by Rebecca Acosta
and Joseph Calmer, Ed.D. students at the University of Southern California. Your participation
is voluntary. Please take as much time as you need to read this document, which explains
information about this study. You should ask about anything that is unclear to you.

PURPOSE OF THE STUDY
The current study is a program evaluation of Elementary Afterschool STEM Program’s year one
implementation and a needs assessment of their second year goals. We hope to identify strengths
within the year one implementation that can continue to be built upon as well as identify areas of
need that we can provide support and resources to aid in a successful second year. You are
invited to be in the study because you are an administrator or Line Staff where the research is
being conducted.

PARTICIPANT INVOLVEMENT
If you agree to participate in this study, you may be asked to teach STEM lessons in your regular
Elementary Afterschool STEM Program “classroom” with your assigned class. Researchers will
be observing your classroom and taking notes. We may ask you to complete a questionnaire and
survey that should take no more than 10-15 minutes of your time. It will include questions on
your experiences teaching and some demographics information such as how long you have been
teaching. You do not have to answer any questions that you don’t want to. You can move on to
the next question or withdraw from the study at any time without any negative consequences
whatsoever.

CONFIDENTIALITY
Researchers may observe your classroom and take notes. Observation data will not be associated
with teachers’ personal information. We will not use your name in anything that we produce.

To maintain your confidentiality, we will transcribe your questionnaire and destroy the electronic
copy as soon as possible.

All the data we gather from this study will be stored in a computer, which will be protected with
a password.
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 85

The members of the research team and the University of Southern California’s Human Subject’s
Protection Program (HSPP) may access the data. The HSPP reviews and monitors research
studies to protect the rights and welfare of research subjects.

The data will be stored for future use, but all identifiable information will be deleted as soon as
possible.

When the results of the research are published or discussed in conferences, no identifiable
information will be used.

An executive report will be created for SUSD and Elementary Afterschool STEM Program at the
conclusion of this project. Information obtained from this project will be included in the report.
However, no identifiable information will be included.

ALTERNATIVES TO PARTICIPATION
Your alternative is to not participate. Your relationship with your employer or the researchers
will not be affected whether you participate in this study.

INVESTIGATOR CONTACT INFORMATION
Rebecca Acosta at rmacosta@usc.edu and Joseph Calmer calmer@usc.edu
Ed.D. Candidates
Rossier School of Education
University of Southern California
3470 Trousdale Parkway
Waite Phillips Hall
Los Angeles, CA 90089-4036

IRB CONTACT INFORMATION
University Park Institutional Review Board (UPIRB), 3720 South Flower Street #301, Los
Angeles, CA 90089-0702, (213)821-5272 or upirb@usc.edu

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 86
Appendix G
IRB-Elementary Afterschool STEM Program Tutor Survey
Elementary Afterschool STEM Program Facilitator Survey
This survey is designed to gather data on Elementary Afterschool STEM Program’s
implementation and alignment of its goals, as perceived by its Line Staff.

1. Please respond to the following statements, as you have experienced so far while working at
Elementary Afterschool STEM Program
Statement Always Sometimes Never Not sure
Our program
provides a
physically safe
learning
environment for
students

Our program
provides an
emotionally
safe learning
environment for
students

Our program
provides
exciting and
engaging
enrichment
opportunities
for students

Our program
supports
improvement in
student
academic
performance

The curriculum
includes
activities and
approaches
aimed at
improving the
leadership and
character

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 87
development of
students
I involve
students in
decision
making about
program
activities

I believe
students like
coming to the
program

Our program
helps students
to develop self-
confidence

Our program
provides
opportunities
for students to
build
friendships with
peers

I have close
relationships
with students in
the program

Our program
helps students
develop new
skills

I keep parents
informed about
their child's
participation in
the program

I keep the
Principal
informed about
the program

I keep the
teachers
informed about
student
participation in
the program

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 88
I receive the
support I need
from the
Principal

I receive the
support I need
from teachers

I respond
quickly and
appropriately to
any concerns
that the
Principal might
have

2. Please indicate how often each of the following activities occur at your site:
Statement
1-2 times/
month
1 time/ week
2-3 times/
week
Every day Not at all
Activities
promoting
healthy
behaviors
(e.g., physical
activity,
gardening)

Technology
based
activities
(e.g., using
iPads,
multimedia
projects)

Science based
activities
(e.g., robotics,
environmental
science
projects)

Instrumental
music (e.g.,
drum,
keyboard,
clarinet,
guitar)

Sci-Gineering
Curriculum

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 89
Kidz Lit
Kidz Math
Use of
Microsoft
products;
Office,
PowerPoint,
etc.

Another,
optional club
that was
created for
your site

3. Please answer the following questions regarding your training for Elementary Afterschool
STEM Program
Statement Always Sometimes Never Not sure
I have the
resources and
supplies to do
my job well

I receive the
training that I
need to be
successful

I receive the
coaching that I
need to be
successful

I have the
confidence to
do my job well

I have the space
to do my job
well

I have close
relationships
with my
colleagues

I get the
support that I
need from the
Site Lead

I get the
support that I
need from the

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 90
leadership team
(e.g. Program
Director)

4. Please indicate the level of knowledge of the following items
Statement I received enough training
I need more training/
information
Classroom management
STEM content
Science and Engineering
Processes

Pedagogy
Lesson design
Assessing student
knowledge

Lab Safety

Demographic Information

Please answer the following so we can gather data about Elementary Afterschool STEM
Program in SUSD

1. Please indicate the site where you work:

Albert Baxter Elementary
Grace Elementary
Faith Elementary
Rainbow Elementary
Washington Elementary
Hope Elementary
Frank Woodruff Elementary

2. Please indicate how long you have worked at Elementary Afterschool STEM Program in your
current position:

Less than 1 year
1-2 years
More than 3

3. What is your age

18-21 22-25 26-29 >30

4. What level of school have you received?

Community College
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 91
4 year University
Completed Undergraduate Degree
Competed Teacher Credential

5. Is your (future) degree STEM-related:

Yes
No

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 92
Appendix H
IRB-Elementary Afterschool STEM Program Site Lead Survey
Site Lead Survey

1. Please identify the amount of time spent on the following activities:
Statement Everyday
2-3 times/
week
Once a week
1-3 times a
month
This activity
is not
offered
A SCIENCE
related activity

A
TECHNOLOGY
related activity

An
ENGINEERING
related activity

A MATH
related activity

2. Please indicate who is responsible for delivering the following activities to the students:
Statement
Site
coordinator
Facilitator/
tutor
Credentialed
teacher
Field Expert Vendor
A SCIENCE
related activity

A
TECHNOLOGY
related activity

An
ENGINEERING
related activity

A MATH
related activity

3. In the past year, how many of the following STEM related support services have you received
as a Site Lead?

Training on how to integrate elements of STEM into my program activities

Coaching on how to implement STEM training content into my program activities

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 93

4. Please rate how the following factors have impacted the implementation of STEM related
activities within Program APPLE?

Statement Not a challenge
A small
challenge
Somewhat of a
challenge
Very
challenging
Adequacy of
space/facilities

Availability of
trained staff

Access to
resources,
curricula,
and/or materials

Time (e.g.,
competition
from other
activities)

Access to the
Internet
connected
computers

Student interest
other

Please rate the following:
5.The goals of SUSD are aligned with Elementary Afterschool STEM Program.
Completely
agree
Mostly agree
Somewhat
agree
Don’t Agree/
Not Aligned

6. The California Educational Center supports the educational goals of Elementary Afterschool
STEM Program (i.e. Rigorous instruction, being culturally relevant, and engaging)
Completely
agree
Mostly agree
Somewhat
agree
Do not agree

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 94
Please respond to the following questions:

7. Moving forward, what are your most pressing needs related to implementing STEM activities
in your within Elementary Afterschool STEM Program?

Training (e.g., integrating STEM into program activities, evaluating STEM activity outcomes)

Information and resources (e.g., existing STEM curricula, developing partnerships with outside
agencies, funding opportunities, evaluation tools) that I may be able to use in Elementary
Afterschool STEM Program

Coaching (e.g., how to integrate and/or improve STEM components in program activities)

Consultation (e.g., how to integrate and/or improve STEM components in program activities)

Site Visits (to sites who have already implemented effective STEM activities)

Other

8. What are your goals for students participating in STEM activities in Elementary Afterschool
STEM Program?
• Increased knowledge of STEM disciplines
• Increased interest in and positive attitude toward STEM disciplines
• Increased awareness of and interest in pursuing STEM careers
• Other (please specify)

Demographics
1. Please indicate the level of your role in Elementary Afterschool STEM Program:
District
Site Lead
California Educational Center

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 95
Appendix I
IRB-Elementary Afterschool STEM Program Supervisor Questionnaire
Administration Questionnaire:

1. What do you think was the most beneficial meetings, trainings, or other support materials
that assisted you in supporting your staff?
2. How effective do you find partnerships in your work?
3. From your perspective, how effective at creating STEM understanding in the students, do
you think “Sci-Gineering” was in the past year? Give specific examples.
4. How is the student knowledge gained from Elementary Afterschool STEM Program
measured?
5. What are the most productive systems or products that you use?
6. What do you see as your role in the creation of science, technology, engineering, or math
activities?
7. What is being done to implement the Common Core State Standards in Elementary
Afterschool STEM Program? (to support the daily instruction in SUSD)
8. What is the biggest lesson you learned about STEM program implementation this year?
Give a specific time and what you learned from it.
9. What will you do differently for year 2 or year 3?
10. What barriers do you foresee about year 2 implementation?
11. How is student achievement measured in Elementary Afterschool STEM Program?
12. How is the curriculum measured for accountability?
13. How are you evaluating any STEM activities offered in your program?
14. How are families invited and involved with the Afterschool program?

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 96
Appendix J
SUSD’s Strategic Action Plan
Elementary Afterschool STEM Program
Strategic Action Plan 2014-2015
Instructional

Strategic Action – Support the District’s Plan to Measurably Improve Student
Achievement
A Analyze Data from 2013-2014 Surveys for Program Improvement
B Improve Kidz/Lit , Kidz/Math, and STEM Practices
C Implement Basic Skills Practice 3X weekly (1
st
-3
rd
HFW, and 4
th
-6
th
Math Facts)
D Identify and Implement Common Core Practices in Afterschool
Strategic Action – Support the District’s Plan to Improve the Use of Technology
A Connect “Type to Learn” to School Assignments
B Implement Project-Based Learning Assignments through Technology Clubs with Student
Leaders
C Purchase Big Universe On-Line Leveled Reading Program

Human

Strategic Action – Support Improved Student Guidance Services (Youth Development
in APPLE)
A Increase Opportunities for Student Voice and Choice
B Implement Student Led Project-Based Learning and Clubs
C Continue with Student Behavior Agreements
D Pilot Girl Scout Project for Anti-Bullying and Self Esteem
E Continue Healthy Living Practices Through AmeriCorps Coaches and NEOP Partnership
Strategic Action – Increase Public and Teacher Confidence in APPLE
A Provide Opportunities for Tutors to Connect Routinely with and Learn from Teachers of
Their Students
B Provide Tutor Training for Communicating Effectively with Families and Teachers
C Include APPLE Activities in School Calendar
D Increase Parent and Teacher Participation at APPLE Showcases
E Increase Publication of APPLE Newsletter
F Identify Additional Partnerships
Strategic Action - Highly Skilled Staff
A Implement “Tiered” Staff Development Based on Tutor Evaluations
B Introduce Tutors to DII, Thinking Maps, and Nancy Fetzer Practices (Awareness of Best
Practices)
C Provide Program-Specific Leadership and Coaching Training to Site Supervisors
D Provide Site Supervisors with Opportunities to Participate in District Staff Development
and/or Committees

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 97
Appendix K

Elementary Afterschool STEM Program Staff Descriptions

Tutors (Elementary Afterschool STEM Program):
Complete all Team Member responsibilities above with increased responsibilities as assigned
including, but not limited to:
• Create and maintain a room environment that best facilitates instruction and class
ownership;
• Enthusiastically participate with students, managing behavior throughout program
duration and communicating with team to effectively handle situations in an appropriate
manner;
• Actively engage students throughout the classroom and program remembering to make
meaningful connections with students equally;
• Provide guidance and participate with students during academic instruction; which
includes but not limited to STEM, Kidz Lit, Kidz Math, PBL, Game Day, Type to Learn
4, and or district driven initiatives.
• Make a clear objective to check homework if homework is complete and correct.
• Be a resource for students to discover what they know and need to know, helping
students to understand processes of solving and not just give answers;
• Direct volunteers to group students effectively, utilizing all resources provided to keep
students engaged and interested;
• Be accountable for student attendance procedures, being mindful of the number of
students in their group;
• Assist with rotation process of students from one activity to another;
• Assist with attendance procedures, including sign-in and sign-out processes at the
beginning and the ending of the day;
• Assist with distribution & cleanup of snacks to students;
• Evaluate all students fairly and consistently; praising for students for their efforts and
guiding them through their challenges.
• Communicate absences and call-ins as detailed in Staff Attendance Policy and
Procedures; committing to evaluation expectations and areas for improvement as
opportunities to grow and learn.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 98
Lead Tutors (Elementary Afterschool STEM Program):
Complete all respective Team Member, Tutor, and Instructor responsibilities above with
increased responsibilities as assigned including, but not limited to:
• Be a beacon of communication for team members, supervisors, volunteers, managers and
other concerning parties;
• Attend all Lead Trainings and Meetings with the intent to contribute and benefit from
discussion and information presented;
• Facilitate discussion, training reminders, and announcements with team; taking initiative
to find solutions rather than excuses,
• Respond to emails consistently, relaying information to site staff, and hold them
accountable for given information;
• Work with site supervisors to ensure program effectiveness and communicate needs
immediately;
• Ensure deadlines are met team-wide;
• Email weekly reports to management, noting site accomplishments and challenges;
keeping log of students who need extra attention;
• Acknowledge team members successes and share in their challenges; talk to team
members about how you would handle decisions;
• Arrive early and remain on site to triple-check that all opening and closing duties are
completed;
• Communicate supply orders and arrange pick-up/delivery

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 99
Appendix L
STEM Career Interest Survey

STEM Career Interest Survey (STEM-CIS)
Optional Demographic Questions
1. Date

2. First name

3. Last name

4. Grade

5. Gender

6. Teacher

7. Race

8. Period

9.School

Directions: Students will complete the STEM-CIS online via iPod Touches or computers. Each
question is a Likert scale with the following choices:

Strongly Disagree (1), Disagree (2), Neither Agree nor Disagree (3), Agree (4), Strongly Agree
(5)

Science

S1 I am able to get a good grade in my science class.

S2 I am able to complete my science homework.

S3 I plan to use science in my future career.

S4 I will work hard in my science classes.

S5 If I do well in science classes, it will help me in my future career.

S6 My parents would like it if I choose a science career.

S7 I am interested in careers that use science.
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 100

S8 I like my science class.

S9 I have a role model in a science career.

S10 I would feel comfortable talking to people who work in science careers.

S11 I know of someone in my family who uses science in their career.

Mathematics

M1 I am able to get a good grade in my math class.

M2 I am able to complete my math homework.

M3 I plan to use mathematics in my future career.

M4 I will work hard in my mathematics classes.

M5 If I do well in mathematics classes, it will help me in my future career.

M6 My parents would like it if I choose a mathematics career.

M7 I am interested in careers that use mathematics.

M8 I like my mathematics class.

M9 I have a role model in a mathematics career.

M10 I would feel comfortable talking to people who work in mathematics careers.

M11 I know someone in my family who uses mathematics in their career.

Technology

T1 I am able to do well in activities that involve technology.

T2 I am able to learn new technologies.

T3 I plan to use technology in my future career.

T4 I will learn about new technologies that will help me with school.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 101
T5 If I learn a lot about technology, I will be able to do lots of different types of careers.

T6 My parents would like it if I choose a technology career.

T7 I like to use technology for class work.

T8 I am interested in careers that use technology.

T9 I have a role model who uses technology in their career.

T10 I would feel comfortable talking to people who work in technology careers.

T11 I know of someone in my family who uses technology in their career.

Engineering

E1 I am able to do well in activities that involve engineering.

E2 I am able to complete activities that involve engineering.

E3 I plan to use engineering in my future career.

E4 I will work hard on activities at school that involve engineering.

E5 If I learn a lot about engineering, I will be able to do lots of different types of careers.

E6 My parents would like it if I choose an engineering career.

E7 I am interested in careers that involve engineering.

E8 I like activities that involve engineering.

E9 I have a role model in an engineering career.

E10 I would feel comfortable talking to people who are engineers.

E11 I know of someone in my family who is an engineer.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 102
Appendix M

Secondary Teacher’s Pedagogical Content Knowledge Instrument

Secondary teachers’ scientific pedagogical content knowledge (STSPCK)
Instructions
The purpose of this study is to measure pre-service secondary school teachers’ PCK. This
questionnaire consists of three parts and 30 statements in total.

Part A. Measures your knowledge of curriculum (n = 7).
Part B. Measures your knowledge of instructional strategies (n = 9).
Part C. Measures your knowledge of assessment strategies (n = 14).

There are multiple statements in each of these three categories. Please provide an example of
how you would address each statement as a science teacher. Your honest and detailed responses
can potentially help teacher educators to develop targeted professional development
opportunities in the area of physical science to improve science teachers’ PCK.

PART A. Curriculum
Statement
Provide examples and
comments for each statement.
1. The teacher uses an inquiry-based science
curriculum.

2. The teacher uses problem-based science
curriculum.

3. The teacher pays attention to the relevancy of
curriculum to the students’ everyday lives.

4. The teacher pays attention to the relevance of
curriculum to his/her students’ cultural
backgrounds.

5. The teacher uses a curriculum that emphasizes
the nature of science.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 103
PART A. Curriculum
Statement
Provide examples and
comments for each statement.
6. The teacher uses a curriculum that emphasizes
the history of science.

7. The teacher uses a curriculum that is not gender-
biased. (It provides examples from and biographies
of both male and female scientists.)

PART B. Instruction
Statement
Provide examples and
comments for each
statement.
1. The teacher starts the lesson by helping children to
discover what they already know about the concepts
taught in the lesson.

2. The teacher identifies the limitations and strengths of
his/her students’ prior conceptions.

3. The teacher uses probing questions to help students
retrieve relevant information and experiences.

4. The teacher uses guiding questions to help students
integrate relevant information and experiences into what
they are currently learning

5. The teacher provides experiences for the students to
understand the limitations of their initial ideas.

6. The teacher engages students in sharing and discussion
to help them consider their individual ideas in relation to
the ideas of others.

7. The teacher provides experiences (supports lecture
notes with static or interactive visuals, models) for the
students to understand the plausibility/intelligibility of
scientifically accurate conceptions.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 104
PART B. Instruction
Statement
Provide examples and
comments for each
statement.
8. The teacher supports her explanation of scientific
concepts by using analogies or metaphors.

9. The teacher supports her explanations with
demonstrations to make the concepts intelligible to her/his
students.

PART C. Assessment
Statement
Provide examples and
comments for each
statement.
1. The teacher poses open-ended questions.
2. The teacher uses problems that require students to
engage in methods of inquiry.

3. The teacher asks problems that require students to
explain a concept.

4. The teacher asks problems that require students to
justify their understanding of the concept.

5. The teacher uses problems that require the students to
communicate their understanding of the concept through
multiple means.

6. The teacher uses problems requiring students to
develop their own methods of investigation.

7. The teacher uses problems that emphasize students’
understanding of the relationships among ideas under
study.

8. The teacher challenges his/her students to interpret
graphical representations of scientific data.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 105
PART C. Assessment
Statement
Provide examples and
comments for each
statement.
9. The teacher requires his/her students to explain their
reasoning when giving an answer to a question.

10. The teacher challenges his/her students to evaluate
each other’s ideas and answers.

11. The teacher challenges his/her students to consider
alternative methods for solutions to the problems.

12. The teacher asks his/her students to explain concepts
to one another.

13. The teacher asks questions that require students to
explain a scientific phenomenon through a model.

14. When a student asks a question, which of the
following action does the teacher take:

1. ____She/he just tells the answer.
2. ____She/he wants the student to find the answer on
his/her own by asking leading questions.

3. ____She/he wants her students to find the answer by
asking his/her peers.

Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 106
Appendix N
Student Survey Results
#
Question
This
describes
me
This
mostly
describes
me
This sort
of
describes
me
This
does
not
describe
me
Total
Responses
Mean
1 “Science makes me feel
like I am lost in a jumble of
numbers and words.
42 41 75 125 283 3.00
2 I am able to complete my
science homework
independently.
120 74 55 32 281 2.00
3 I earn good grades in
Science activities.
115 91 58 16 280 1.91
4 Do you agree with the
statement, “Science is
fun.”?
162 49 46 23 280 1.75
5 My afterschool program
has made me interested in
Science.
108 62 47 58 275 2.20
6 I am interested in Science
activities.
175 64 18 22 279 1.59
7 I like my Science class. 152 58 36 33 279 1.82
8 If I perform well in
Science, it will help me get
a job I Science in the
future.
107 61 62 50 280 2.20
9 Do you agree with the
statement: “I would enjoy
being a scientist.”?
77 53 67 85 282 2.57
10 I am comfortable using
technology.
190 48 26 8 272 1.46
11 I know some tools that
scientists use.
95 52 75 54 276 2.32
12 I know some tools that
engineers use.
87 57 65 67 276 2.41
13 My after-school program
has helped me in my daily
classroom.
123 65 53 36 277 2.01
14 My after-school program
has helped me in my daily
classroom, especially in
93 72 57 55 277 2.27
Running head: NEEDS ASSESSMENT OF A STEM PROGRAM 107
science.
15 I understand that science
uses a process of
observing, collecting data,
and explaining evidence.
125 67 45 41 278 2.01
16 I am able to ask my own
questions about science.
124 60 60 37 281 2.04
17 The activities I do allow
me to think about what I
am doing, rather than
simply follow directions.
113 79 52 34 278 2.03
18 The activities in the
program allow me to use
my hands and my mind.
156 71 31 21 279 1.70
19 I get to use a lot of tools
and materials during STEM
instruction.
147 63 39 24 273 1.78
20 I know what the letters
STEM stand for.
200 32 19 21 272 1.49
21 Completing the activities in
STEM allows me to think
like a scientist.
123 65 44 45 277 2.04
22 I am interested in
participating in a Science
Fair now that I have
participated in Elementary
Afterschool STEM
Program.
116 71 45 49 281 2.10
23 I have been able to earn
higher grades on my
homework as a result of
participating in Elementary
Afterschool STEM
Program.
99 81 55 43 278 2.15
24 I am interested in pursuing
answers to scientific or
engineering questions.
91 62 81 44 278 2.28
25 I feel comfortable with
science topics.
134 71 36 33 274 1.88

Abstract (if available)

Abstract Out of School Time (OST) science, technology, engineering and mathematics (STEM) programs are on the rise as the need for global competitiveness increases in STEM-related fields. This project was a needs assessment and program evaluation for Small Urban School District’s afterschool STEM program, Elementary Afterschool STEM Program. It focused on evaluating Elementary Afterschool STEM Program’s first year of implementation while assessing their needs for year two and beyond. We collected and analyzed data from 293 Elementary Afterschool STEM Program students, 24 tutors, 15 site leads and 8 supervisors using surveys, observations and previous program documents. Our findings from this triangulation of data indicate a need for increased professional development, longevity in program staff, and an increase in student voice. This project provides a foundation for Elementary Afterschool STEM Program to use when making decisions critical for future student learning.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

A needs assessment and evaluation of an elementary afterschool STEM program in a small urban school district

PDF

A needs assessment of an urban elementary school district's afterschool program

PDF

A needs assessment of an after school, science, technology, engineering, and mathematics program

PDF

A needs assessment of an after school science, technology, engineering, and mathematics program

PDF

A needs assessment of Meadows Elementary School District STEM education after school program

PDF

An evaluation of Urban Compass after-school program

PDF

A needs assessment for Hillview Middle School's STEM education program

PDF

Analysis of STEM programs in Oregon public high schools

PDF

College and career readiness through high school STEM programs: an evaluation study

PDF

Promoting STEM integration, interest and identity among elementary school students

PDF

Math achievement and self-efficacy of linguistically and ethnically diverse high school students: their relationships with English reading and native language proficiency

PDF

Examining the relationship between Latinx community college STEM students’ self-efficacy, social capital, academic engagement and their academic success

PDF

Personnel resource allocation strategies in a time of fiscal crisis: case study of elementary schools in a California school district

PDF

Elementary STEM policies, practices and implementation in California

PDF

Capacity building for STEM faculty and leaders: supporting university students with ADHD in earning STEM degrees

PDF

Assessing the effectiveness of an inquiry-based science education professional development

PDF

Effective STEM initiatives in high-poverty elementary schools

PDF

First-year persistence of rural high school graduates at four-year, urban colleges and universities

PDF

Improving college participation success in Glendale Unified School District: An application of the gap analysis model

PDF

Nursing students' perceptions of formal faculty mentoring

Asset Metadata

Creator Acosta, Rebecca Marie (author)

Core Title A needs assessment and program evaluation of an urban school district's after-school STEM program

School Rossier School of Education

Degree Doctor of Education

Degree Program Education

Publication Date 04/20/2015

Defense Date 03/13/2015

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag after-school,DoS,OAI-PMH Harvest,STEM,urban school

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Mannion, William (committee chair), Rueda, Robert (committee chair), Sinatra, Gale M. (committee chair)

Creator Email beccaandjason@gmail.com,rmacosta@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c3-554648

Unique identifier UC11297813

Identifier etd-AcostaRebe-3346.pdf (filename),usctheses-c3-554648 (legacy record id)

Legacy Identifier etd-AcostaRebe-3346.pdf

Dmrecord 554648

Document Type Dissertation

Format application/pdf (imt)

Rights Acosta, Rebecca Marie

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA