University of Southern California Dissertations and Theses

A needs assessment of an urban elementary school district's afterschool program

(USC Thesis Other)

A needs assessment of an urban elementary school district's afterschool program

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Running
head:
STEM
NEEDS
ASSESSMENT

1

A NEEDS ASSESSMENT OF AN URBAN ELEMENTARY SCHOOL
DISTRICT’S AFTERSCHOOL PROGRAM

by

Linda Moon

A Dissertation Presented to the
FACULTY OF THE USC ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF EDUCATION

May 2015

Copyright 2015 Linda Moon
STEM
NEEDS
ASSESSMENT

2

Acknowledgements

This dissertation would have failed to exist without the critical guidance and mentorship
of our faculty committee members; the collaboration and support from my colleagues at USC;
the accommodation and cooperation of the staff at the project sites; the frequently tested patience
of and encouragement from my family members and friends; and the steadfast, undeserved
mercies of God that has given me the strength to press on towards the finish line. Thank you all.

STEM
NEEDS
ASSESSMENT

3

Statement of Co-Authorship
This document reports the results of a capstone project that was completed as part of the
Ed.D culminating program requirements. It was designed as a dissertation of practice that targets
authentic problems of practice and that provides an opportunity for students to demonstrate skills
and competencies that will be required in future career activities. The present project was a
needs assessment/evaluation of a specific program, and was designed to address concerns of this
specific site rather than being designed as a generalizable research project. In line with the goal
of reflecting real world practice, this project was carried out collaboratively between Nsoah Abu-
Rasool and Linda Moon, listed alphabetically to indicate equal contributions. To accurately
reflect the division of work on this project, some of this document is co-authored, and the
specific chapters are labeled to reflect the collaborative authorship where appropriate.

STEM
NEEDS
ASSESSMENT

4

Table of Contents
ACKNOWLEDGEMENTS 2
STATEMENT OF CO-AUTHORSHIP 3
LIST OF TABLES 5
LIST OF FIGURES 6
ABSTRACT 9
CHAPTER ONE: OVERVIEW OF THE PROJECT 10
Background of the Problem 10
Statement of the Problem 11
Purpose of the Project 13
Organization of the Dissertation 14
CHAPTER TWO: LITERATURE REVIEW 15
Issues in STEM Education 16
STEM and OST Programming 18
Instructor Capacity 19
Knowledge 20
Motivation 21
Professional Development 22
Curriculum Adequacy 24
The Needs Assessment Model 25
Past Evaluation 26
Summary 28
CHAPTER THREE: METHODOLOGY 29
Site 30
District 30
RNH Program 31
Participating Stakeholders 32
Instrumentation 33
Survey 34
Instructor Capacity Questions 35
Professional Development Questions 36
Curriculum Adequacy Questions 36
Additional Questions 37
Focus Groups 37
Interviews 38
Observations 40
Procedures 40
Preassessment 41
Initial Contact 41
Preliminary Site Visit 41
Scope of Work 42
Subsequent Meetings 42
IRB and District Approval 42
Assessment 43
Survey Distribution 44
STEM
NEEDS
ASSESSMENT

5

Focus Groups 44
Interviews 44
Observations 45
Document Review 46
Postassessment 46
Data Analysis 46
Executive Summary 46
CHAPTER FOUR: RESULTS 48
Participants 48
Question 1 51
Survey Responses 51
Focus Groups with Program Leaders 54
Knowledge 55
Motivation 56
Interviews with Program Supervisors 57
Knowledge 58
Motivation 58
Observations 59
Question 2 62
Survey Responses 62
Focus Groups with Program Leaders 64
Interviews with Program Supervisors 65
Observations 67
Question 3 68
Survey Responses 68
Focus Groups with Program Leaders 69
Interviews with Program Supervisors 70
Document Review 72
Summary 73
CHAPTER FIVE: DISCUSSION 74
Discussion of Findings 75
Instructor Capacity 75
Knowledge of Program Leaders 75
Motivation of Program Leaders 77
Professional Development 77
Curriculum Adequacy 80
Recommendations 81
Instructor Capacity 82
Knowledge 82
Assess the Task at Hand 83
Evaluate Knowledge and Skills 83
Plan An Appropriate Approach 84
Apply Strategies and Monitor Progress 84
Reflect and Adjust 84
Motivation 85
Increase Situational Interest 85
STEM
NEEDS
ASSESSMENT

6

Add Intrinsic Value 85
Add Utility Value 86
Other Considerations 86
Professional Development 87
Gather Input About Perceived Training Needs 87
Increase Frequency of Trainings 88
Restructure STEM Training Workshops 89
Involve Outside Experts 89
Curriculum Adequacy 90
Prevent Supply Shortages 91
Realign Project-Based Learning with STEM Instruction 91
Familiarize RNH Staff with Standards 91
Further Considerations 92
Limitations of Project 94
REFERENCES 98
Appendix A: Survey Questionnaire Items 104
Appendix B: Focus Group Protocol 107
Appendix C: Interview Protocol 108
Appendix D: Observations Protocol 109
Appendix E: IRB Approval Letter 111
Appendix F: Data Collection Plan 112
STEM
NEEDS
ASSESSMENT

7

List of Tables
Table 1: Methods of Data Collection for Each Needs Assessment Focus 34
Table 2: Summary of Survey Responses 49
Table 3: Descriptive Statistics for RNH Staff 50
Table 4: Responses to Instructor Capacity Survey Questions 52
STEM
NEEDS
ASSESSMENT

8

List of Figures
Figure 1: Mean instructor capacity responses 53
Figure 2: Mean professional development responses 63
Figure 3: Mean curriculum adequacy responses 69

STEM
NEEDS
ASSESSMENT

9

Abstract
Science, technology, engineering, and math (STEM) education has been a hot button
issue in recent years, especially concerning its implementation in schools and application to the
growing global economy in need of STEM professionals. This project was designed for
Meadows Elementary School District (MESD)’s Reaching New Heights (RNH) afterschool
program (pseudonyms), and was aimed at examining the issues of STEM-related education in
out-of-school time programs, especially with respect to instructor capacity (knowledge and
motivation), professional development, and curriculum adequacy. Using a needs assessment
model, we collected data from 45 program leaders (afterschool instructors) and 6 program
supervisors (site administrators) at six different RNH sites, utilizing surveys, focus groups,
interviews, observations, and document reviews. The results suggest that program leaders feel
they have sufficient knowledge and motivation to teach STEM material, although program
supervisors conveyed that there is room for improvement. Both program supervisors and
program leaders value professional development and feel that the RNH program requires more
frequent training sessions, preferably with the assistance of experts outside of RNH. Finally, the
STEM curriculum in use at the RNH sites seems to be adequate for promoting STEM learning,
but both program leaders and program supervisors feel that the curriculum alone will not suffice
and they find the need to supplement with alternative resources. To address these concerns, the
researchers have compiled a list of recommendations for the purposes of enhancing STEM
instruction in RNH, as well as suggestions for further study. Throughout this dissertation, the
names of the school district and afterschool sites have been changed to preserve their anonymity.

STEM
NEEDS
ASSESSMENT

10

CHAPTER ONE: OVERVIEW OF THE PROJECT
To provide an overview of the project, a needs assessment of an urban afterschool
program, this chapter will begin by introducing the background of the problem of declining
STEM literacy in the United States, followed by a statement of the problem undergirding the
current project. Next, I will outline the purpose of the project and supply a summary of how this
dissertation has been organized.
Background of the Problem
As the global technological climate progresses, there exists the constant need for
individuals who are highly trained and competent in the fields of science, technology,
engineering, and math (STEM). STEM education prompts students to investigate their world
and contribute to it using 21
st
century learning skills, such that students well-versed in STEM are
better equipped in problem-solving situations (Havice, 2009; Salinger & Zuga, 2009). STEM
literacy and achievement is imperative to remain competitive at both an individual and broader
levels in a growing global economy (Wang, Moore, Roehrig, & Park, 2011). STEM education is
therefore important in building a society that thrives, contributes profoundly towards the future,
and ultimately supplies students with a need to achieve (Havice, 2009).
However, the United States as a whole seems to be lacking somewhat in the STEM fields
as compared to other countries. In recent years, there has been a stark decline in students
pursuing careers in the STEM disciplines (Simpkins, Davis-Kean, & Eccles, 2006). The World
Economic Forum ranked the United States 52
nd
in quality of mathematics and science education,
and 27
th
among developed nations in the number of college students majoring in a STEM-related
field ("How does the U.S. compare to other countries in STEM education?," 2014). Moreover,
only 16 percent of American high school seniors are proficient in mathematics and also
STEM
NEEDS
ASSESSMENT

11

interested in a STEM-related career ("Science, technology, engineering, and math: Education for
global leadership," 2014). Statistics such as these spurred the creation of the Committee on
STEM Education (CoSTEM), which will facilitate a national strategy for promoting STEM
education, as outlined in the President’s 2015 budget proposal ("Science, technology,
engineering, and math: Education for global leadership," 2014).
Despite such initiatives, the task of promoting STEM learning is a complex one,
especially because there are ongoing debates about how STEM instruction should be executed.
An even more rudimentary problem is that there are mixed perceptions of what STEM really is,
and there is no universal or operationalized definition of STEM even among STEM professionals
(Breiner, Harkness, Johnson, & Koehler, 2012). Some argue that it is just a composite of the
four subjects taught separately, or in “silos,” while others argue that it is the integration of all
four subjects into everyday learning (Sanders, 2009; Bybee, 2010; Honey et al., 2014). While
many champion the latter definition, and even though there have been concerted efforts to infuse
the individual STEM subjects as one interrelated entity, the reality is that students are still
exposed to STEM subjects as individual courses (e.g., chemistry, physics, math). This poses a
challenge because STEM professionals, such as chemists or engineers, do not compartmentalize
their work into individual disciplines but rather their jobs are derived from an integration of these
subjects—a concept that is not fortified in most academic settings (Breiner et al., 2012).
Statement of the Problem
Despite the elusiveness and manifold difficulties surrounding STEM, the persisting
downward trend of U.S. STEM competitiveness has invigorated efforts to integrate STEM into
K-12 learning. The culmination of such efforts would be to attain successful STEM education,
which, as described by Salinger and Zuga (2009), is as follows:
STEM
NEEDS
ASSESSMENT

12

The principles of science and the analysis of mathematics are applied to technological
problems of benefit to society. The learning is in relevant contexts and uses hands-on
activities to engage students. Twenty-first century skills of teamwork, communications,
and leadership are all practiced in the development of a solution to a problem. This is
STEM (p. 8).
Accordingly, current goals in STEM education are building STEM literacy and 21
st

century competencies, developing a future workforce capable in STEM, and boosting STEM
interest and engagement (Honey, Pearson, & Schweingruber, 2014). The scarcity of STEM-
literate professionals as a result of declining interest and opportunity adds to the urgency of these
goals.
Currently, there are numerous deficiencies in K-12 STEM education, the most prominent
among them being qualified instructors, professional development, and effective curricula
(Kuenzi, 2008; Bybee, 2010). These shortcomings are amplified when the same pressures of
implementing STEM in school environments are placed on voluntary settings, such as out-of-
school time (OST) programs. First, there tends to be high turnover in staff and among the more
stable members, both knowledge and motivation may be lacking or existent in low amounts.
Second, professional development may not always be available and if it is, it may be inadequate
to properly equip the staff or the staff may not see value in receiving training. And third, the
curriculum may not satisfy the project-based learning model that effective STEM environments
require.
Given this context, the Meadows Elementary School District has conveyed interest in
improving their STEM program by focusing on improving instructional quality, professional
STEM
NEEDS
ASSESSMENT

13

development, and curriculum. It is for a context where complex issues such as these are
prevalent that a needs assessment model may be best applicable.
Purpose of the Project
The purpose of this project was to conduct a comprehensive needs assessment of an
urban STEM afterschool program, located in Meadows, California called “Reaching New
Heights” (RNH). This project, developed with the assistance of the Los Angeles County Office
of Education, focused on gathering data in order to identify RNH’s specific needs toward
improving their current programs for the 2014-2015 academic year. In particular, the areas of
focus were on the instructors, or program leaders, and administrators, or program supervisors, at
the six elementary school RNH sites. Each of these interest groups were asked questions
regarding the knowledge and motivation of program leaders (instructor capacity), opportunities
available for professional development, and whether or not they feel the STEM curriculum being
used in RNH is sufficient for promoting STEM learning among students (curriculum adequacy).
The needs assessment was a collaborative project involving another doctoral student who
assessed the same sites simultaneously. Together, we addressed the areas of concern mentioned
above that have been identified by RNH and are also constant across out-of-school time (OST)
programs: instructor capacity, professional development, and curriculum adequacy. My co-
evaluator conducted focus group sessions with the program leaders at each of the RNH sites,
while I interviewed the program supervisor at each site. Additionally, the RNH program has
asked for assistance in evaluating existing data from an evaluation executed in the previous year,
so we have related our findings to this existing data to augment this needs assessment. The
needs assessment process consisted of using interviews, surveys, observations, and document
review to assemble the necessary data. Upon data collection and subsequent analysis, the
STEM
NEEDS
ASSESSMENT

14

evaluation team at USC organized the findings in order to report back to the district and RNH
program through an executive summary.
Organization of the Dissertation
This dissertation is comprised of four major sections: a review of relevant literature
(chapter 2), description of the methodology for the project (chapter 3), results from data
collection (chapter 4), and a discussion of findings and future considerations (chapter 5).
Chapter 2 discusses in further detail through extant literature the background and need for STEM
education in out-of-school time settings, honing in on the areas of instructor capacity,
professional development, and curriculum adequacy. Chapter 3 describes the research site,
participants, instrumentation, and procedures that make up the needs assessment. Chapter 4
reveals the qualitative and quantitative findings with respect to the central questions, and finally
Chapter 5 delves deeper into implications of the findings, discusses limitations to the project, and
makes recommendations for future direction, both for the RNH program specifically and for
further research.

STEM
NEEDS
ASSESSMENT

15

CHAPTER TWO: LITERATURE REVIEW
This chapter provides an overview of STEM education and out-of-school time (OST)
programming, including reasons for their importance, current initiatives in STEM education, and
relevant work that serves as background to the project. A main program goal is to cultivate a
balanced, holistic approach to learning that is based on student needs and, additionally, to
empower the staff to teach and have an awareness of STEM learning practices and principles.
Initial conversations indicated that the areas that are essential to achieving these goals and that
are important for effective program functioning include: instructional quality; organizational
factors; and curriculum. Because the Meadows Elementary School District (MESD) afterschool
program has expressed a need for assistance in these specific areas, the research team at USC has
conducted an applicable needs assessment. For the scope of this project, the focus in each of
these areas was as follows:
1. Instructor capacity – instructors’ knowledge of and motivation in teaching STEM
2. Professional development – opportunities for training and availability of resources
3. Curriculum adequacy – the extent to which the curriculum promotes STEM
learning.
Examining these three components in existing research will help formulate the next steps
for Meadows as well as assess their current progress when compared to indicators of effective
programs. To provide relevant background, the chapter first provides a general description of
STEM and current issues in STEM education and OST programming. The chapter next briefly
covers relevant work and issues of concern with respect to the three research foci. Third, a
description of a needs assessment model and is provided along with, lastly, a discussion of a past
evaluation conducted at the research site and its role in shaping this current project.
STEM
NEEDS
ASSESSMENT

16

Issues in STEM Education
STEM is the acronym designated by the National Science Foundation for science,
technology, engineering and mathematics, but should not be defined as merely the sum of its
parts. STEM education is the intentional practice of integrating STEM content into learning
environments such that it increases students’ understanding and application of STEM (Wang et
al., 2011; Roberts, 2012).
As a general trend in STEM, students are demonstrating a lack of knowledge,
understanding, and skills, which will ultimately close doors for the future (Axtell, McCallum,
Mee Bell, & Poncy, 2009). Even at the end of their K-12 learning career, many students tend to
lack even fundamental knowledge of STEM. Because many students opt out of taking more
rigorous math or science courses, a vast number of high school students graduate with relatively
low science and math ability, or fail to graduate at altogether (Havice, 2009; Sanders, 2009).
Continuing in this snowball effect, this could lead to socioeconomic deficits such as lesser
paying jobs, poorer health, and increased likelihood of delinquency (Havice, 2009). Based on
this observation, the Committee on a Conceptual Framework for the New K-12 Science
Education Standards recommended that K-12 science education incorporate the following three
premises: 1) scientific and engineering practices, 2) crosscutting concepts that unify STEM
subjects through their common applications across fields, and 3) core ideas across STEM fields
(Schweingruber, Keller, & Quinn, 2012).
The installation of Common Core Standards and Next Generation Science Standards in
the K-12 setting is aimed at increasing STEM competency across the board. STEM disciplines
will help students develop 21
st
century learning skills such as creativity and innovation, critical
thinking and problem solving, communication, collaboration, information management, effective
STEM
NEEDS
ASSESSMENT

17

use of technology, career and life skills, and cultural awareness (NRC, 2010; Beers, 2012).
However, conventional teaching of STEM subjects has led to students losing interest in STEM
early on, and falling out of the “STEM pipeline” (Freeman, Dorph, & Chi, 2009; Sanders, 2009).
This results in students not pursuing careers in STEM-related fields, mainly because these fields
have been previously portrayed to them as dry and boring (Dierking, 2007).
To address this phenomenon in part, The Power of Discovery: STEM
2
(2013), a
consortium made up of the California Afterschool Network and California STEM Learning
Network, outlines the STEM initiatives of their out-of-school time (OST) programs as the
following:
• Increase the quality and depth of regional and statewide partnerships in support of
OST STEM learning opportunities.
• Increase frequency, intensity, duration, and quality of STEM learning opportunities
for youth in OST programs.
• Increase staff competence, confidence, and motivation to facilitate STEM learning
opportunities.
• Increase student interest, engagement, and knowledge of STEM processes and
concepts.
These initiatives suggest that OST programs are important and necessary vehicles for
delivering STEM content in conjunction with traditional classroom contexts. Prevalent issues
such as quality of instruction and increasing student interest should therefore be addressed in
non-traditional settings, such as OST programs, in order to contribute to STEM education across
the board, and ensure that enough students are being regularly exposed to the STEM disciplines,
encouraging them to remain in the STEM pipeline.
STEM
NEEDS
ASSESSMENT

18

STEM and OST Programming
While schools aim to address paucity in STEM proficiency and equity in the context of
the school day, schooling alone is insufficient in supporting lifelong STEM literacy (Dierking,
2007). Thus it is becoming increasingly apparent that out-of-school time interventions may be a
valuable component of addressing this issue. This is evidenced by the growing number of
students enrolling in OST programs, and rising awareness that OST experiences do in fact play a
role in generating students’ interest, engagement and future involvement with STEM (Freeman
et al., 2009).
Out-of-school time (OST) programs could range from a variety of activities, including
homework help, subject tutoring, clubs, sports, religious activities, and other recreational
activities, such as art and music. OST programs do not necessarily have to be purely academic;
in fact, in many situations, greater gains were seen for students who participated in programs that
involved a variety of activities (Lauer et al., 2006; Mahoney, Parente, & Lord, 2007; Huang &
Cho, 2009). With respect to the academic component, it is beneficial for students to be given
ample time and resources for homework and instruction of study skills. In addition, pairing
instruction with career or college skills allows for students to have more positive responses to
learning (Lauer et al., 2006). However, the programs that were deemed as higher quality tended
to devote more attention to enrichment activities and less to homework and non-skill-building
exercises (Mahoney et al., 2007). Norland (2005) views such activities as “nonformal”
education, characterized by inconsistent attendance and leadership, variable leadership skills and
backgrounds, and curriculum flexibility.
The employment of OST programs is not a question of whether or not these programs
positively impact children and youth, but rather why and to what extent these programs are
STEM
NEEDS
ASSESSMENT

19

effective (Vandell, Reisner, & Pierce, 2007; Sheldon, Arbreton, Hopkins, & Grossman, 2010;
Springer & Diffily, 2012). Out-of-school time has historically been viewed as constructive for
students by extending the school day and providing a platform for practice of skills and
reinforcement of classroom learning. Enrollment in OST programs, such as afterschool
programs, often prevents delinquency, enriches academic performance, and impacts positive
social and behavioral outcomes. These outcomes are more readily achieved when out-of-school
environments produce stable, relational staff and providing a comprehensive environment that
focuses on both academic and other enrichment activities.
Such positive influences from OST programming could be utilized for enhancing student
engagement and achievement in STEM subjects, as well as piquing student interest in STEM-
related careers for the future (Bevan et al., 2010). For example, a longitudinal study by Simpkins
et al. (2006) showed that students who had once been enrolled in an out-of-school math and
science program in the 5
th
grade were more inclined to pursue a related career by the 12
th
grade.
To this end, there is a general consensus among extant literature that highly qualified staff,
adequate professional development, and effective curricula are factors that constitute successful
STEM OST programming.
Instructor Capacity
Instructional quality is often the primary scapegoat in any learning environment. But
beyond failure to employ sound pedagogy, the quality of the instructors themselves needs to be
reassessed. At minimum, staff needs to be stable, which is an ongoing challenge for OST
programs where the staff are generally transient and turnover is high (Mahoney et al., 2007;
Gottfredson, Cross, Wilson, Rorie, & Connell, 2010; Sheldon et al., 2010). Furthermore, staff
should foster productive and positive relationships with the students in the program (Miller,
STEM
NEEDS
ASSESSMENT

20

2010). Establishing these kinds of relationships have the dual effect of enforcing positive
behavior and preventing delinquency. A mentoring relationship with positive adult role models
allows for students to be more engaged with the program and could influence learning outcomes.

While these are the generic prerequisites for effective staff, assessing the capacity of the
instructors for STEM education goes beyond being stable, relational role models. As informal
educators, OST instructors are rarely highly qualified in teaching STEM material; that is, they
are neither adequately skilled nor motivated. Therefore, two dimensions by which instructor
capacity should be measured are through their knowledge of and motivation in teaching STEM
material.
Knowledge. Because STEM knowledge is also continually changing as technology
advances and new scientific discoveries are made, instructors of STEM—professionals and
paraprofessionals alike—need to constantly update their understanding of STEM (Dierking,
2007). Quality STEM instruction entails generating an environment that fosters inquiry, critical
thinking, and engagement (Little, Wimer, & Weiss, 2008). There are two components of
instructors’ STEM knowledge that need to be addressed to ensure high instructional quality:
content knowledge and pedagogical content knowledge. The former asks whether or not
instructors are aware of what STEM education is in general whereas the latter is more specific to
the instructors’ preparedness to know and apply the STEM material.
In OST programs, the prior knowledge of instructors vary significantly as they are greatly
diverse in age, education level, and work experience. Moreover, they are usually involved in a
part-time or volunteer basis. A survey by Freeman et al. (2009) showed that approximately 76
percent of OST programs did not have a dedicated science person on staff, the implications being
the majority of STEM programs are led by people with no STEM background.
STEM
NEEDS
ASSESSMENT

21

Motivation. Apart from being knowledgeable, instructors should also be highly
motivated in order to be effective. This is a challenging criterion as OST programs, by
definition, are low stakes environments. While this is beneficial for students because there is
more flexibility in activities and the environment is less restrictive, it is less incentivizing for
instructors because the program by nature is non-evaluative (Bevan et al., 2010). As a result,
employee turnover is usually high, with instructors leaving an organization due to a variety of
factors, such as low wages and limited opportunities for advancement (Freeman et al., 2009).
Given this nature of OST programs, cultivating motivation in program staff is a critical endeavor.
Motivation, or motivated behavior is a composite of motivational factors, such as
expectancy, value, and affective components (Thoonen et al, 2011). Expectancy factors are
mostly comprised of instructors’ self-efficacy and teaching beliefs, both of which are
instrumental in improving teacher performance as it demonstrates the confidence teachers have
in their ability to teach (Bandura, 1993). Lin, Chuang, and Hsu (2014) found in their research
that teaching beliefs were inextricably linked to instructional innovation; that is, instructors are
more likely to develop more creative and deliberate methods of instruction if they feel that
student outcome is reflective of their input. Thoonen et al. (2011) found instructors’ sense of
self-efficacy to be the strongest motivational predictor of learning and teaching practices.
Perceived value of a task is also a large motivational factor that needs to be considered.
Subjective task-value, according to Chow, Eccles, and Samela-Aro (2012) is comprised of four
components: intrinsic value, attainment value, utility value, and perceived cost. The first three
components of task-value refer to the positive, attracting attributes of a task, whereas the fourth
component, perceived cost, highlight its negative attributes. Finding value in the task of
STEM
NEEDS
ASSESSMENT

22

instruction is an important predictor of overall motivation, which is a necessary stepping-stone to
implementing change (Thoonen et al., 2011).
Task-value is closely predictive of interest, and vice versa. While there is overwhelming
research on student interest in academic settings, currently there exists surprisingly little
evidence for teacher interest and its effect on their instructional output. Regardless, the concept
of interest is also largely applicable to instructors because just as for students, “the potential for
interest is in the person but the content and the environment define the direction of interest and
contribute to its development” (Hidi & Renninger, 2006, p. 112). Thus, in addition to one’s
individual and situational interests in teaching a given subject matter, evaluating teacher interest
also should be compounded with didactic interest, or one’s interest in the domain of the
instructional process itself (Schiefele, Streblow, & Retelsdorf, 2013).
Overall, there is a lack of sufficient research on instructors’ motivation in OST settings
(Bevan et al., 2010). However, STEM education is undoubtedly enhanced by motivated
instructors who consider themselves to be facilitators of the learning process rather than merely
distributors of information (Roberts, 2012). Freeman et al. (2009) discuss that one possible way
of motivating instructors might be to have their participation in the OST program be counted as
credits for future professional pursuits, such as entry into a teacher credential program.
Professional Development
Given that instructional quality is often in question, providing adequate professional
development in order to improve students’ achievement outcomes is an ongoing challenge in any
given learning environment (Thoonen et al., 2011; Lin et al., 2014). Staff members of OST
programs do not always have the qualifications as are required for school teachers and
administrators, so they are limited in their exposure to proper instructional strategies and
STEM
NEEDS
ASSESSMENT

23

pedagogy. Professional development, consequently, is the main vehicle by which the
administrators and instructors in OST programs are trained in STEM and teaching in general.
Most professional development targets specific outcomes, such as child development,
instructional strategies, and mentoring models. Some also target the quality and sustainability of
the OST workforce by enriching providers’ marketable skills. However, in many cases, a
majority of the STEM instructors are not given ample professional development opportunities or
if they are, do not find value in participating in them (Freeman et al., 2009).
Successful professional development is important for the individual, the students, and the
organization (Bowie & Bronte-Tinkew, 2006). Development should first and foremost expose
OST staff to materials, supplies, equipment, and curricular resources that they can use with the
student participants (Freeman et al., 2009). Yet effective professional development should be
targeted to not only improving immediate job performance, but also to building long-term career
and life skills that could be applied beyond current job descriptions. Sheldon et al. (2010) further
suggested that targeted staff training should take place throughout the year, including regular
observations and coaching of staff, and resulting data should be used to measure progress.
Collaboration is crucial in professional development. The NSF Knowledge Synthesis
found that instructors who participated in STEM professional learning communities (PLCs)
understood math and science better and furthermore, felt better prepared to teach the content,
utilizing more research-based methods and diverse modes of learning to engage students (Fulton
& Britton, 2011). Wang et al. (2011) found that teachers who were a part of PLCs believed they
had more opportunities and connections on how to integrate STEM into their teaching. In the
same vein, professional development should be an inclusive and respectful collaborative process
STEM
NEEDS
ASSESSMENT

24

that brings all the stakeholders together to develop a shared vision for improving STEM
education in their organization (Dierking, 2007; Bevan et al., 2010).
In addition to having these necessary components of successful professional
development, OST settings may require revising the conventional training model altogether to
maximize effectiveness. Thoonen et al. (2011) advocated professional learning activities as
recourse for traditional models of professional development such as workshops, seminars, and
conferences. Professional learning is the idea of embedding lifelong professional development
as a natural and ongoing part of organizational culture, with activities such as keeping up to date,
experimenting, and reflection (Thoonen et al., 2011). Integrating this type of approach into OST
programs may be beneficial for fostering continued learning postures in their instructors.
Moreover, engendering this kind of atmosphere will encourage learning and practice, which is
considerably more effectual than “one-shot” training events.
Curriculum Adequacy
Many of the issues surrounding curriculum adequacy is that gap between what is
intended to be taught and what is actually taught, thereby making it critical to evaluate alignment
between instruction, curriculum, and assessment (Martone & Sireci, 2009). Curriculum should
not be evaluated as a stand-alone venture but in alignment, or triangulation, with instruction and
assessment. Accordingly, Porter and Smithson (2001) suggest three indicators by which to
evaluate curriculum alignment: 1) topic coverage, 2) cognitive demand, and 3) mode of
presentation. In addition, STEM curriculum used in OST programs need to be reviewed in terms
of their effectiveness and potential to engage students’ interest. Typically, OST programs offer
remedial content or homework help rather than profound experiences with STEM subjects; while
STEM
NEEDS
ASSESSMENT

25

the former may help them achieve acceptable school grades, this kind of approach does little to
spark students’ interest in the subject in the present and for the future (Dierking, 2007).
Although project-based learning (PBL) has had supporters and cynics alike in the
classroom context, PBL is the primary approach in STEM curriculum (Mills & Treagust, 2003).
Project-based learning, though often used interchangeably with inquiry-based or problem-based
learning, differs from the latter two as it focuses more on the application of knowledge rather
than acquisition of knowledge, and self-regulation is much more imperative (Mills & Treagust,
2003). Bell (2010) simply defines PBL as a student-driven, teacher-facilitated approach that
involves learners pursuing answers to a question that has naturally piqued their curiosity and
interest. Similarly, Bybee (2010) proposes model instructional units where students are
presented with a problem or challenge that is of interest to them, and work toward a solution by
gleaning knowledge and skills from each of the STEM disciplines. The basic components of the
PBL approach are: the problem as starting point; small group collaboration; flexible guidance of
a tutor; minimal direct instruction; student-initiated learning; and time for self-study (Schmidt,
Loyens, Van Gog, & Paas, 2007). This approach emphasizes competency in addressing the
problem, issue, or situation as a whole, and not just knowledge of concepts and processes
(Bybee, 2010). PBL looks different in OST programs than in school classrooms because it is not
as verbal and abstract but rather more tactile and connected to everyday settings (Bevan et al.,
2010). Despite the notable differences, Noam (2003) suggests that PBL pedagogy bridged with
school contexts is the most ideal for students to have a more holistic experience.
The Needs Assessment Model
As noted earlier, the purpose of this project is to conduct a needs assessment. Exemplary
assessment is a key requisite of successful STEM program implementation (Bybee, 2010). This
STEM
NEEDS
ASSESSMENT

26

project is based on a three-phase needs assessment model: 1) preassessment, 2) assessment, and
3) postassessment (Altschuld & Kumar, 2010). Phase 1 consists of narrowing the scope of the
assessment to the most important needs as well as gathering the information that is already
known about those needs. Among the needs that were articulated by the RNH director, the focus
of this study will be on instructor quality, professional development and curriculum evaluation.
The second phase, assessment, involves building on existing knowledge by conducting further
research about the identified needs, culminating in a full assessment of those needs. The job of
the needs assessment committee (NAC) is to employ various methods of data collection (i.e.,
surveys, interviews, document reviews, etc.) in order to pinpoint possible causes and trends of
the needs; this is considered an “epidemiological” approach, because it integrates both existing
knowledge and probable outcomes (Altschuld & Kumar, 2010). In the final part,
postassessment, the NAC formulates an action plan to implement the necessary changes within
the organization.
Past Evaluation
The afterschool program that is targeted for this needs assessment was evaluated just the
year before (Fletcher & Cantrell, 2014). The evaluation was formulated in a pre- and post-test
format, where the same set of survey questions were administered first in December 2013 then
again in June 2014. This evaluation surveyed program supervisors, program leaders, activity
specialists, students, and parents involved in the RNH program. In addition, the evaluation
included certificated teachers who answered questions about the students in their class that were
in the RNH program.
The questions were all related to STEM, although the wording varied drastically
depending on the respondent (i.e., student, program leader, etc.). Program supervisors, program
STEM
NEEDS
ASSESSMENT

27

leaders, and activity specialists were asked to describe to what extent they agreed to statements
such as, “Students are able to use computers effectively,” and “Students are excited about STEM
activities and projects.” The response choices were “not so much,” “sometimes,” and “most of
the time.” Certificated teachers were given the same statements with “Students who attend
RNH” preceding each of the questions. Students were asked more simplified statements such as,
“I would like to be a scientist when I grow up,” “I like learning new things,” and “I think math is
fun.” Parents were given similar statements about their perceptions of their children’s
achievement and beliefs (e.g., “My child thinks math is fun”).
Although this evaluation compiled a large amount of data from a variety of stakeholders,
it was difficult to determine RNH’s actual progress in STEM programming. For one, the
evaluators did not provide a background or literature to qualify the questions being asked to each
of the participants. Secondly, all of the results were presented in percentages, without listing the
actual means or even the exact number of people surveyed, which made for unclear results. An
example of such a statement is as follows:
In December, two-thirds of Program Supervisors reported that students were enthusiastic
about learning new things and were becoming more knowledgeable and skilled almost all
or all of the time. Surprisingly, by June this figure dropped to 38% (Fletcher & Cantrell,
2014, p. 2).
Third, the recommendations made at the conclusion of this evaluation were broad and
moreover ambiguous about what data supported a particular recommendation. For example, one
recommendation was, “Consider changing site directors and/or program leaders where this is
needed, based on the site-based findings provided at the conclusion of this report” (Fletcher &
Cantrell, 2014, p. 29).
STEM
NEEDS
ASSESSMENT

28

The evaluation was ambitious in terms of addressing various aspects of STEM and how it
affects a wide spectrum of people involved with RNH. However, much of the raw data and
ensuing recommendations were far too broad and unsupported by the results. Because this
evaluation was inconclusive about the afterschool program’s progress with STEM in the end, the
current needs assessment project was deemed necessary to supplement the existing data.
Summary
In summary, STEM education is a veritable need across learning environments in the
present global economy. In the context of OST programs, instructor capacity, professional
development, and curriculum assessment are three factors that contribute to successful STEM
learning. In terms of instructor capacity, OST STEM instructors generally lack adequate
knowledge of STEM content, and in addition to limited expertise, instructors may not be
motivated to teach STEM content. To address this gap, effective professional development is
needed that equips staff with both job-specific knowledge and skills, accompanied by
development of skills applicable to broader and future contexts. A third component is effective
curriculum that successfully integrates STEM as a project-based learning model.
To address these issues, a needs assessment model will be used to synthesize the above
challenges in STEM education with specific areas of concern for the Meadows RNH program.
The research team at USC has decided to conduct a needs assessment because of the limitations
of previous evaluations, such as the one mentioned above. The methodology for this project, as
will be discussed in Chapter 3, was formulated by building on the types of questions and topics
covered in the previous evaluation, but also by addressing specific areas of interest—instructor
capacity, professional development, and curriculum adequacy.

STEM
NEEDS
ASSESSMENT

29

CHAPTER THREE: METHODOLOGY
Authors: Linda Moon and Nsoah Abu-Rasool
The research context for this project is the Reaching New Heights (RNH) program of
Meadows Elementary School District (MESD). According to their website, their objective is
“using a project based learning model as a framework, we promote an inquiry based learning
environment with an emphasis on 21st century learning skills.” Based on these objectives,
dialogue with the RNH program director, and extant literature, the needs assessment of the RNH
programs will be structured around the following questions:
1. Are the instructors knowledgeable about and motivated in teaching STEM material?
2. Are instructors provided sufficient opportunities for professional development?
3. Is the current STEM curriculum adequate for promoting STEM learning?
In order to provide as thorough of an assessment as possible, this project has been
executed as a collaborative effort, where each of the co-evaluators have examined these issues—
instructor capacity, professional development, and curriculum adequacy—simultaneously from
differing lenses. Nsoah Abu-Rasool directed the focus groups with the program leaders (RNH
instructors) while Linda Moon conducted the interviews with the program supervisors (RNH site
coordinators). Differentiating the work in this way allowed us to cover more ground in the time
allotted for this project. Based on our findings, we have provided Meadows RNH with a more
holistic needs assessment that takes into account data from the juxtaposition of two different but
interdependent stakeholders. This chapter is organized by first providing an introduction to the
site and participants, followed by an overview of the instruments and procedures.
1

1
Linda Moon wrote this section.

2

Nsoah Abu-Rasool wrote this section. Linda Moon made contributions.

STEM
NEEDS
ASSESSMENT

30

Site
2

District
Meadows Elementary School District (MESD) began operation over 100 years ago in
1906. Many of the current schools first opened in the 1950s to educate the expanding population
after World War II. In the 1960s, the district consisted of six elementary schools and one middle
school up until September 2006, the year in which the district was celebrating its 100
th

anniversary in education. That year they decided to convert one of the elementary schools into a
second middle school and move sixth grade classes to be part of the middle schools. To
effectively meet the demands of the restructuring, new school boundaries were established to
accommodate the opening of the new schools, which were “Adams” Middle School (formerly an
elementary school) and “Lincoln” Elementary School (pseudonyms). In the early 2000s, thanks
to a local school bond supported by matching funds from the State of California, all of
Meadows’s schools were modernized and extensively refurbished.
Meadows Elementary School District is geographically located in the South Bay region
of Los Angeles County. According to the California Department of Education website, more
than 50 percent of its enrolled students are of Hispanic or Latino origin. Currently, MESD
serves students from kindergarten to eighth grade, and consists of eight campuses—six
elementary schools (for grades K-5) and two middle schools (for grades 6-8), all of which have
instigated RNH programs. For the purposes of this project, we will only collect data from the six
elementary school programs, which we will identify only by letters: Site A, Site F, Site G, Site
M, Site S, and Site T. Average Academic Performance indices (API) from 2011-2013 according
to the California Department of Education are 796 for Site A, 772 for Site F, 816 for Site G, 807
for Site M, 847 for Site S, and 879 for Site T. The API reports also show that a vast majority of

2

Nsoah Abu-Rasool wrote this section. Linda Moon made contributions.

STEM
NEEDS
ASSESSMENT

31

the students come from socioeconomically disadvantaged backgrounds, accounting for
approximately 87% of the district in 2013. Roughly fifty percent of MESD students are also
categorized as English Language Learners.
RNH Program
It is within this context that the Reaching New Heights afterschool program has been
implemented. The program has been around since 1999 and is primarily funded through the
After School Education and Safety (ASES) and 21
st
Century Community Learning Centers.
Additionally, RNH maintains several partnerships that help provide resources to the program and
a connection to the community. The Meadows RNH partnerships consist of Loyola Marymount
University (LMU), Inside Arts, Press friends, NASA’s Best, and Trash for Teachers. A few
more contributors for the Meadows RNH training resources include, Local Learning Community
(LLC), Los Angeles County Office of Education (LACOE), the LACOE After School Technical
Assistance Unit (ASTAU), Mindworks, LEGO Mindstorms, Pearson Science Curriculum, and
Discovery Science.
RNH started mainly with homework help, arts and crafts, recreational activities and other
supplemental activities but has evolved into more intentional programming, which now includes
fitness, healthy behaviors, STEM, and 21
st
century learning skills. Moreover, they are trying
now to place kids in more leadership roles and have them facilitate as well. The key element
about RNH is that it is not an intervention or just an enrichment program but a program that
“extends the day” to provide students with more opportunities to learn outside of the classroom
without replicating what’s done in the classroom. The overarching mission of RNH is to provide
abundant opportunities and experiences for all, and their goals to that end are to provide holistic,
student-driven enrichment comprised of academics, sports and fitness, community partnership
STEM
NEEDS
ASSESSMENT

32

and leadership opportunities. Being both a before and after school program, each RNH site is
open an hour before school officially starts, and from school dismissal until 6:00 p.m. daily,
except for holidays. For the scope of this project, we only assessed the programming that takes
place after school dismissal.
Participating Stakeholders
3

Together with the RNH director, we have identified the stakeholders of the RNH program
to be the program supervisors (site administrators), program leaders (instructors), parents,
students, schools, district specialists, union (classified), and school board. Among these
stakeholders, the most salient are the administrators, instructors and students. The program
supervisors have the largest influence as they deal directly with the policies and programming of
RNH. They have a vested interest in this needs assessment as they will need to justify to their
board both past and future funding to maintain the program. As for the program leaders, while
they may not have as pronounced an impact on the organization, they are the ones interacting
with the students on a regular basis and thus have a better understanding of how effective the
program is for the students themselves. Students are the most pertinent beneficiaries of an
effective STEM afterschool program, because the outcome would be an increase in their
academic achievement in STEM subjects, which in turn will provide educational and
professional opportunities for them in the future. RNH serves more than 1,800 students five
days a week at all eight MESD school sites.
For the purposes of this project, we are mostly interested in the stakeholders at the staff
level—program supervisors and program leaders—because they are both the recipients and
agents of change for implementing strategies for improving STEM education. There are 120
full-time and part-time staff members with varying levels of experience with STEM. The full-

3

Linda Moon wrote this section. Nsoah Abu-Rasool made contributions.

STEM
NEEDS
ASSESSMENT

33

time administrators, or program supervisors, oversee the program leaders. There are eight
program supervisors in MESD, one at each site. There are approximately ten part-time
instructors, or program leaders, at each site who work approximately three and a half hours a day
after normal school hours. Their average age is about 20-25, ranging from recent high school
graduates to retired practitioners.
We have narrowed the scope of this project to include only the elementary school sites,
such that our participant pool consisted of 6 program supervisors and 50 program leaders. To
provide a more comprehensive needs assessment, we have divided up the qualitative work such
that one of us interviewed program supervisors while the other led the focus groups with
program leaders.
Instrumentation
4

We used a combination of interviews, focus groups, surveys, observations, and document
review to gather data. The rationale for the multi-method approach was to triangulate the data to
obtain the most accurate reflection of MESD’s RNH programming. Table 1 shows which
methods of data collection were applied to each of the three needs assessment foci—instructor
capacity, professional development, and curriculum adequacy. Much of the instrumentation was
the same for both program leaders and program supervisors, both parties having participated in
aggregated research (i.e., answering survey questions) and disaggregated research (i.e.,
interviews or focus groups). However, the questions had slight differentiation in wording such
that it is tailored to the participants’ roles in the RNH program. For example, if a question asked
about program leaders’ motivation, program leaders would supply their responses from their own
perspective whereas program supervisors would discuss their perceptions of the program leaders’
motivation.

4

Linda Moon and Nsoah Abu-Rasool wrote separate subsections of this section, as indicated by subsequent notes.

STEM
NEEDS
ASSESSMENT

34

Table 1
5

Methods of Data Collection for Each Needs Assessment Focus
Focus Survey
Interviews/
Focus Groups Observations
Document
Review
Instructor capacity X X X
Professional development X X X
Curriculum adequacy X X

Survey
6

Two surveys, one for program leaders and one for program supervisors, were distributed
via Qualtrics. The survey link was distributed by the program director to each of the participants
and the survey was made available online for two weeks. Both groups received the same set of
questions, with a few revisions on wording to best fit each category of respondents (see
Appendix A). The survey was constructed by adapting from existing instruments such as the
Teacher Efficacy Scale, Teachers’ Attitudes About Professional Development Scale, Curriculum
Emphasis Survey, and the Power of Discovery STEM
2
’s Readiness and Needs Assessment Tool.
We also collaborated with the RNH program director in order to ensure that we were asking the
kinds of questions he was interesting in learning more about, in addition to revising some of the
wording so the questions can be more accessible to the population. The researchers also supplied
additional questions as deemed necessary such as demographic questions and other pertinent
questions specific to the site that was not covered by the existing instruments (e.g., “How often is
professional development such as staff training offered at your site?). The questions were

5

Linda Moon created this table.

6
Linda Moon wrote this section.

STEM
NEEDS
ASSESSMENT

35

formatted for rating on a 6-point Likert scale (1 = “strongly disagree,” 2 = “disagree,” 3 =
“somewhat disagree,” 4 = “somewhat agree,” 5 = “agree,” and 6 = “strongly agree”). The entire
survey had a total of 63 items (41 questions with some of them involving multiple parts): 24
instructor capacity items (11 for knowledge and 13 for motivation), 9 professional development
items, and 8 curriculum adequacy items.
Instructor capacity questions. To measure instructor capacity, the questions were about
the extent of the program leaders’ knowledge and motivation. The 11 knowledge questions
consisted of both STEM content and pedagogical content knowledge where respondents had to
indicate to what extent they agree with statements such as, “I am aware of the basic principles of
STEM teaching and learning,” and “I know how to use effective teaching strategies to guide
student thinking and learning in STEM.” The same questions were posed to program supervisors
with the pronoun “I” replaced with “program leaders,” such that the question would be aimed at
gauging program supervisors’ perceptions of their program leaders’ knowledge. We ran
reliability analyses on these survey questions in order to determine the internal consistency of all
the items, where a reliability coefficient higher than .7 is favorable. After running a reliability
analysis on the knowledge questions, the Cronbach’s alpha (α) for program leaders was .91, and
the alpha for program supervisors was .94.
The 13 motivation questions were also the same for both program leaders and program
supervisors, but since the latter group was reporting their perceptions of program leaders’
motivation, the measurement scales were adjusted accordingly. Overall, the motivation scale
was highly reliable for program leaders (α=.95). The motivation questions were further divided
into the subscales of self-efficacy (3 items), teaching beliefs (7 items), value (2 items) and
interest (1 item). Cronbach’s alpha for the self-efficacy, teaching beliefs, and value subscales
STEM
NEEDS
ASSESSMENT

36

were .88, .93, and .90, respectively. Reliability analysis was not conducted on the interest
segment, as it was only one question. For the program supervisors, the self-efficacy, value, and
interest questions were grouped together to measure overall perception of motivation (α=.85).
The teaching belief questions were converted to measure program supervisors’ perceptions of
their program leaders’ influence on students, creating an outcome expectancy subscale, which
yielded a reliability alpha of .95.
Professional development questions. The professional development questions were
aimed at determining whether or not program leaders were provided sufficient opportunities for
training and resources. There were nine items in total, four of which were dedicated to
ascertaining the availability of professional development, and five pertained to how important
program leaders and program supervisors believed professional development was to the program.
An example of an availability item was, “I (Program leaders) receive the training I (they) need to
be successful” and a sample importance item would be, “Professional development workshops
are vital to helping program leaders develop new teaching techniques.” Each of these statements
were rated by the respondents on the same 6-point Likert scale ranging from 1=”strongly
disagree” to 6=”strongly agree.” Reliability analysis for these questions yielded a Cronbach’s
alpha of .92 for availability and .87 for importance.
Curriculum adequacy questions. The final portion of the survey were questions about
the current STEM curriculum being utilized at RNH in order to determine its adequacy in
promoting STEM learning among students. Sample items include identifying competencies that
students can glean based on the curriculum (e.g., “use scientific vocabulary and principles in
everyday discussions”) and whether or not STEM-centered processes (i.e., inquiry skills,
problem solving, critical thinking) are incorporated into the current STEM curriculum. There
STEM
NEEDS
ASSESSMENT

37

were 9 items in all for this section, all of which were also answered according to the 6-point
Likert scale. The Cronbach’s alpha for the curriculum adequacy questions was .94.
Additional questions. At the end of the survey, we included several demographic
questions for the purposes of descriptive statistics and accounting for external factors and
covariates. The demographic questions consisted of respondents’ gender, age, level of
education, number of years teaching/administering at RNH, intended or completed
undergraduate major. Additionally, we added a few questions that were not directly relevant to
the needs assessment but included anyway in order to obtain as rich a data set as we could. An
example of such a question would be, “Our program supports improvement in student academic
performance.”
Focus Groups
7

The focus group interviews with the program leaders were structured to identify their
knowledge of STEM principles as well as their motivation for teaching STEM classes. We
developed a focus group interview protocol to keep track of interview times, locations, and to
capture response data (see Appendix B). In addition, the prepared questions, in the protocols,
helped to establish discussions and to stay focused on the interview topic. The focus group
interviews provided the researcher a personal view of the program leader’s thought processes as
it related to STEM. The focus group protocol was designed with the expectation of generating a
candid discussion among the program leaders that would deliver valuable data related to their
STEM capacity and motivation. In addition, the focus group questions were also designed to
capture the information processing method of the program leaders by asking them to give
examples of how they used motivation and STEM concepts in the classroom. The focus group
interviews were conducted at each RNH site during the three month data collection period.

7
Nsoah Abu-Rasool wrote this section.

STEM
NEEDS
ASSESSMENT

38

In order to thoroughly investigate the program leaders’ needs, in relation to professional
development, questions were asked during the focus group interview that targeted how they felt
about receiving adequate support and professional development training from the administrative
staff. The questions in the focus group interviews were framed so that the program leaders
would be encouraged to reveal his or her point of view about professional development and how
it has impacted their instruction directly. In addition, inquiries about the need for more resources
and other types of professional development support that could help the program leaders become
more efficient and prepared for teaching STEM were also asked and discussed. The final needs
assessment question, covered in the focus group interviews, centered on the effectiveness of the
RNH program’s curriculum. Questions concerning PBL and 21st century learning were asked in
order to find out how the program leaders felt about the current state of the RNH program’s
curriculum effectiveness. Additionally, the program leaders were asked if they thought that the
curriculum was in alignment with the RNH program’s goals. The focus group questions were
also designed to foster an environment, for the program leaders, that would trigger open
discussions about their experiences with the different curriculum vendors used within the RNH
program. Additionally, the response data from the program leaders could then be analyzed and
used by the RNH program director to ensure that the most efficient curriculum would be used
that offered the most value and benefit to the student’s STEM learning experience.
Interviews
8

The program supervisors participated in one-on-one interview sessions with the USC
team during this study. An interview protocol was designed for the program supervisors that was
similar to the focus group protocol for the program leaders (see Appendix C). However, the
interview questions were created specifically for the program supervisors in the project.

8
Nsoah Abu-Rasool wrote this section.

STEM
NEEDS
ASSESSMENT

39

The questions were first in regard to the program leader’s STEM capacity and the
program supervisor’s perceptions of how knowledgeable and efficacious the program leaders
seemed to be with respect to teaching STEM material. Since the program supervisors were
responsible for scheduling and implementing professional development, the next set of interview
questions were framed around what they thought professional development should look like and
what should be the desired result upon completing training. For example, some of the questions
covered during the individual interviews with the program supervisors asked, how well is the
professional development training attended, what is your perception of what a program leader
should know after they attend training, and if there are any observable changes in the program
leaders as a direct result of professional development, what should it look like.
Different curriculum vendors within the RNH program are used by the program leaders
to teach STEM courses. The program supervisors were asked, during the interviews, what was
the need for such a variety of curriculum vendors to teach STEM and which one seemed to be
the most beneficial to the RNH students? Again, questions were designed to see if there was an
alignment between the program leaders and the program supervisors in their understanding of the
curriculum vendors that provided the most benefit to the RNH students.
The interviews with the program supervisors were conducted individually to avoid
conflict on sensitivity issues or anything unforeseen that could have occurred during a combined
focus group interview with the program leaders. In addition, we wanted to create a nonintrusive
or threatening environment in order to allow the most freedom, for all participants within the
study, with the intent of getting the most candid response. Additionally, the interviews were
designed to see the point of view of the program supervisors in relation to the actions or
STEM
NEEDS
ASSESSMENT

40

perceptions of the program leaders as they navigated through the same three needs assessment
questions.
Observations
9

Observations were conducted at each RNH site in order to view program leaders in the
classroom setting as they taught STEM content to their students. An observation protocol was
developed to capture descriptive data of the RNH sites and actions of the program
leaders. Sections in the protocol included adding details about the physical space of the site,
demographics of participants, and the sequential actions of the program leaders (see Appendix
D). The aim of these observations was to further illuminate findings from the focus group
sessions and in the Qualtrics survey regarding their views on teaching STEM. Access to the
various school sites, where the observations were conducted, was arranged and authorized
through prior coordination with the MESD RNH program director.
A total of seven observations occurred, six of which were mainly of the program leaders,
at each one of the participating elementary schools in the RNH program. The program leaders’
interaction with students, their teaching of STEM subjects, use of the STEM curriculum, along
with classroom environment were documented during the course of the observations. The final
observation was of a professional development session at which the same protocol was used to
assess the STEM workshops offered.
Procedures
10

The needs assessment for this project follows the framework of the three-phase model
designed by Altschuld and Kumar (2010). This model was used because it offers the opportunity
to do a preassessment, assessment, and postassessment of the sites pertaining to this project.

9
Nsoah Abu-Rasool wrote this section.

10

Linda Moon wrote this section. Nsoah Abu-Rasool made contributions.

STEM
NEEDS
ASSESSMENT

41

Following this model, we were able to determine the most pressing needs in RNH during the
preassessment phase, conduct further research on these needs during the assessment phase, and
formulate recommendations based on those results in the postassessment phase.
Presassessment
To carry out this preassessment phase, we held meetings with a liaison at STEM
2
(a
consortium of STEM afterschool programs in California), a project coordinator from Los
Angeles County of Education’s (LACOE), the RNH program director at Meadows, and our
faculty committee chairs. From these conversations, we were able to determine and prioritize
RNH’s needs that have driven this project.
Initial contact. The representatives from STEM
2
and LACOE were our primary contacts
that helped set up many of the meetings to ensure collaboration between the University of
Southern California (USC) and MESD. It was through their networks that the USC team was
able to get in contact with MESD to make RNH the focus of this project. They were both
instrumental in helping us get the project going by initiating phone meetings and site visits with
the RNH director. Our first meeting was held via phone conference on March 20, 2014, with the
RNH director and our committee chairs in attendance. During this meeting, we were given a
general overview of the RNH program, including its goals. In addition, the history of the RNH
program within MESD and STEM-related concerns were also discussed during the meeting. The
RNH director felt at this time that a needs assessment would be most beneficial to understand
how to further their STEM-focused structures.
Preliminary site visit. Our second point of contact was a site visit on May 9, 2014,
attended by my co-evaluator and committee chairs. In addition to observing the site facilities,
the program director gave the group an overview of their curriculum, activities, and the different
STEM
NEEDS
ASSESSMENT

42

partnerships they have developed over the years. Afterward, the group discussed the next steps
for the needs assessment by talking through what RNH perceived their needs were and what our
research team could assist with that was within our scope. Upon listing these items, the group
formulated a timeline for actions, roles and responsibilities to be upheld by both parties.
Scope of work. Based on these conversations, our liaison from LACOE’s Technical
Assistance Unit drafted a memorandum that detailed the nature of our relationship with the
Meadows RNH site and distributed it to the USC team on June 3, 2014. This document was later
titled “Scope of Work” as to avoid any contractual implications otherwise. Upon receipt, the
USC team clarified the capacity to which we were able to participate in the project, since the
RNH program director had originally identified eight needs to investigate. It was at this point
that we narrowed our focus to providing recommendations for instructor capacity, professional
development, and curriculum adequacy, along with assistance with analyzing existing data from
an evaluation conducted the year before by an outside group (Fletcher & Cantrell, 2014). The
resulting Scope of Work document outlined RNH’s objectives, needs, stakeholders, and
resources as well as the support capacity that could be provided by our USC research team.
Subsequent meetings. Another phone meeting on July 25, 2014 was held at which we
confirmed the contents of the Scope of Work document. We reviewed the proposed timeline of
events and responsibilities of both USC’s research team and Meadows RNH. A follow-up on
site meeting was scheduled for August 12, 2014 where we planned to further discuss how to
proceed in detail regarding the needs assessment process, as well as receive hard copies of the
2014 RNH evaluation for analysis.
IRB and district approval. We completed our Collaborative Institutional Training
Initiative (CITI) certification through the CITI program website in February 2014. This online
STEM
NEEDS
ASSESSMENT

43

training involved modules that explained various facets of research involving human subjects.
We submitted an application for approval of our project from the USC Institutional Review
Board (IRB) on September 22, 2014. We submitted for approval as an exempt study because our
research will only involve adults over the age of 18. The USC IRB committee approved our
project on September 27, 2014 (see Appendix E).
As for working with the Meadows Elementary School District (MESD), we were not
required to fill out any additional contracts other than submit the Scope of Work to the RNH
program director. Following the necessary protocol set by the MESD Department of Human
Resources, we each completed TB testing and background checks and were cleared for the
project by the district on October 9, 2014.
Assessment
In the assessment phase, the research team collected data using various methods in hopes
of identifying root causes and trends of the needs. We utilized similar instruments but will be
addressing administrators (program supervisors) and instructors (program leaders) separately.
We will employed more than one method of data collection for each of the three needs to be
assessed in order to triangulate the data and provide more accurate feedback in the
postassessment phase. We created a data collection plan that included dates and times, which the
RNH program director and the assistant director then distributed to all the schools involved in
this project (see Appendix F). The rationale for conducting our research only on Thursdays was
because all of Meadows had early dismissal on Thursdays (1:00pm), so the RNH program is
longer and there is more flexibility. Also, Thursdays were the days each site usually held their
weekly meetings so we were able to conduct interviews and focus groups during their regular
meetings time as not to take away from program time.
STEM
NEEDS
ASSESSMENT

44

Survey distribution. After creating a questionnaire using Qualtrics, we made the survey
available through an online URL that we sent to the RNH program director and the assistant
director. From there, these RNH administrators took the prerogative to forward the link to the
six elementary school RNH sites. The link remained active for two weeks during which time
program leaders and program supervisors were required to complete it. While the program
leaders and program supervisors were strongly encouraged to participate, participation was
completely voluntary and there was no compensation for completing the survey.
Focus groups. Similarly, we conducted in-person focus groups with program leaders to
garner their opinions and views regarding their experiences within RNH, professional
development opportunities, and the STEM curriculum in use. The program director asked each
site to assemble three to four program leaders to participate in the focus group. The resulting
group was a well-balanced mixture of veteran as well as newer program leaders, including at
least one activity specialist—a former program leader and assistant to the program supervisor at
each site. Each focus group approximately 40 minutes long, and Nsoah Abu-Rasool was the
main interviewer during these sessions, although Linda Moon was also in attendance. We met
with the group at Site F on October 23, 2014, Site M on October 30, 2014, Site S on November
13, 2014; Site A on December 4, 2014, Site G on December 11, 2014, and Site T on December
18, 2014. For transcription and note-taking purposes, we asked each of the participants for
permission to record the audio of the focus group. For confidentially purposes, our results
indicate only the content of the focus groups, and the identity behind specific statements is kept
anonymous.
Interviews. We conducted in-person interviews with each of the RNH program
supervisors in MESD, or administrators, to garner their opinions and views regarding their
STEM
NEEDS
ASSESSMENT

45

program leaders, professional development, and the STEM curriculum in use. Each interview
was 30 to 40 minutes long, and Linda Moon was the main interviewer during these sessions,
although Nsoah Abu-Rasool was also in attendance. We met with the program supervisor from
Site F on October 23, 2014; the supervisors from Site S and Billy Site M on November 13, 2014;
the supervisor from Site A on December 4, 2014; and the supervisors from Site G and Site Ton
December 11, 2014. For transcription and note-taking purposes, we asked each of the
interviewees for permission to record the audio of the interview. For confidentially purposes,
our results indicate only the content of the interviews, and the identity behind specific statements
is kept anonymous.
Observations. We also spent 90-120 minutes on observations at each site on the same
days we held interviews with the program supervisors. We visited two to three RNH classrooms
during the time we had allotted for observations. Each of the sites accommodated our schedules
by rearranging their schedules so that the classrooms we visited would be focused on STEM. All
of the sections we observed used the MindWorks curriculum, and following our makeshift
observation protocol, we jotted notes on the classroom environment and the events that
transpired during that time. We did not interact with the students or intervene with the integrity
of the class in any way. The main purpose of these observations was to validate or contradict the
information gleaned from the surveys and interviews or focus groups.
We also attended a professional development session took place at a neighboring
district’s middle school on January 31, 2015. This was a series of workshops hosted by the
Local Learning Community (LLC), which is a consortium of five different South Bay school
districts. We sat in on the two STEM workshops that were offered both of which were about
how to apply scientific thinking.
STEM
NEEDS
ASSESSMENT

46

Document review. At the request of the RNH director, we reviewed two major
documents for the purposes of this study—the 2014 evaluation of RNH (Fletcher & Cantrell,
2014) and samples of the MindWorks curriculum. The purpose of reviewing these documents
was predominantly to guide the current project and make recommendations in Chapter 5.
Postassessment
In this third and final phase, all of our findings have culminated in an action plan for the
RNH program. We have coalesced in Chapter 4 the qualitative and quantitative data about
program leaders’ and program supervisors’ views regarding the three foci of this study and
provide detailed feedback and recommendations in Chapter 5.
Data analysis. We have transcribed and coded qualitative data, such as interviews, focus
groups, and observations, to identify transcending themes and patterns. All quantitative data,
such as survey responses, were inputted and analyzed using SPSS software. We have focused
predominantly on descriptive analyses–means and standard deviations–and simple comparative
analyses where appropriate.
Executive summary. We have summarized our findings and have reported back to the
Meadows district and RNH program by constructing a ten-page executive summary in the form
of a needs assessment report, as outlined by Altschuld and Kumar (2010), that includes,
1) Scope of the report,
2) Front material, authorship and acknowledgements,
3) The three phases and what was done in each,
4) Results for phase I,
5) Results for phase II,
6) Results for phase III, and
STEM
NEEDS
ASSESSMENT

47

7) Appendices (p. 145).
By addressing instructor capacity, professional development, and curriculum adequacy
from the lenses of both the program leaders and program supervisors, our hope is to have
provided a more comprehensive needs assessment from which RNH can enact necessary changes
while maintaining the strong components of the program.

STEM
NEEDS
ASSESSMENT

48

CHAPTER FOUR: RESULTS
Authors: Linda Moon and Nsoah Abu-Rasool
We collected data from each of the six RNH sites of Meadows Elementary School
District using surveys, focus groups, interviews, and observations in order to answer the
following three questions for this needs assessment:
1) Are the instructors knowledgeable and motivated in teaching STEM material?
2) Is each site providing sufficient opportunities for professional development?
3) Is the current curriculum adequate for promoting STEM learning?
The use of qualitative and quantitative methods enabled us to gather rich and meaningful
data from both the program leaders and the program supervisors. In addition, we were able to
create a more holistic understanding of the various needs of the STEM portions within the RNH
program at each site.
11

Participants
12

Table 2 shows a summary of survey response rates, based on the initial distribution to 50
program leaders, and six program supervisors. We initially had a 96% response rate (N=48) for
program leaders, but three of the response sets were eliminated because the survey was not
completed. There was no indication or known reason why three respondents did not complete
the survey. Thus, the Qualtrics survey analysis is based on 45 program leaders, with ages
ranging from 18 to 60, with an average age of 25 (SD=7.667). Female participants (N=29)
outnumbered the male participants (N=15), while one respondent did not indicate a gender. All
six program supervisors completed the survey, with their ages ranging from 28 to 42, and
average age of 32 (SD=5.62). There were more female participants (N=5) than male participants

11
Linda Moon wrote this section. Nsoah Abu-Rasool made contributions.

12

Linda Moon wrote this section.

STEM
NEEDS
ASSESSMENT

49

Table 2
13

Summary of survey responses.
Participant Sent Returned Valid Response rate
Program leaders N=50 N=48 N=45 90%
Program supervisors N=6 N=6 N=6 100%

13

Linda Moon created this table.

STEM
NEEDS
ASSESSMENT

50

Table 3
14

Descriptive statistics for RNH program leaders (N =45) and program supervisors (N=6).
N M SD
Program Leaders
Gender 44* 1.66 .48
1 = Male 15
2 = Female 29
Age 45 2.23 .70
1 = Ages 18-21 11
2 = Ages 22-24 19
3 = Ages 25+ 15
Previous teaching experience 45 2.49 1.60
1 = None 22
2 = Less than 1 year 8
3 = 1 to 3 years 7
4 = More than 3 years 8
Years at RNH 45 2.44 1.10
1 = Less than 1 year 11
2 = 1 to 3 years 13
3 = 3 to 5 years 11
4 = More than 5 years 10
Program Supervisors
Gender 6
Male 1
Female 5
Age 6 32 5.62
Years in current position 6 6.83 3.06
Note. *1 missing value in this category

14

Linda Moon created this table.

STEM
NEEDS
ASSESSMENT

51

(N=1), and the average amount of time they have been in their current positions is almost seven
years (M=6.83, SD=3.06). Table 3 provides a summary of these statistics.
In addition to partaking in the online survey, only the program leaders participated in the
focus groups and only the program supervisors participated in the individual interviews. Three
to four program leaders at each site participated in the 30-minute focus group session, for a total
of 24 program leaders. One or two activity specialists, who were all formerly program leaders,
also sat in on the focus group sessions but they did not partake in the online survey. We also
interviewed all six program supervisors in 30- to 40-minute sessions.
Question 1:
Are the Instructors Knowledgeable and Motivated to Teach STEM?
Survey Responses
15

To measure program leaders’ knowledge and motivation to teach STEM, both program
leaders and program supervisors were given survey questions comprised of Likert scale items
(1=strongly disagree; 6=strongly agree). The motivation portions for the program leaders were
further broken down to assess program leaders’ self-efficacy, teaching beliefs, value, and
interest. The program supervisors, in addition to being asked about their perceptions of their
program leaders’ knowledge and motivation, were also asked about outcome expectancies—
whether or not they felt that the program leaders were directly influencing STEM learning
among the RNH students.
The average values for the two components of instructor capacity, knowledge and
motivation, as reported by program leaders and program supervisors are summarized in Table 4.
The overall mean for program leaders’ knowledge as reported by program leaders was 4.75

15

Linda Moon wrote this section.

STEM
NEEDS
ASSESSMENT

52

Table 4
16

Responses to instructor capacity survey questions by RNH program leaders (N =45) and
program supervisors (N=6).
N M SD
Program leaders
Knowledge 45 4.75 .77
Motivation 45 5.03 .86
Self-efficacy 45 5.01 1.02
Teaching beliefs 45 4.93 .85
Value 45 5.24 .93
Interest 45 5.05 1.07
Program supervisors
Knowledge 6 3.82 .56
Motivation 6 4.24 .65
Outcome expectancy 6 4.36 .78

16

Linda Moon created this table.

STEM
NEEDS
ASSESSMENT

53

(SD=.77), while the mean for program supervisors’ perceptions of program leaders’ knowledge
was slightly lower at a value of 3.82 (SD=.56). For motivation, the overall means as reported by
program leaders and program supervisors were 5.03 (SD=.86) and 4.24 (SD=.65), respectively.
For the program leaders, the motivation questions were further analyzed by separating into four
components: self-efficacy (M=5.01; SD=1.02), teaching beliefs (M=4.93; SD=.85), value
(M=5.24; SD=.93), and interest (M=5.05; SD=1.07). Program supervisors’ perceptions of their
program leaders’ influence on students was measured as the outcome expectancy scale, which
yielded a mean of 4.36 (SD=.78).

Figure 1. Mean instructor capacity responses. This figure illustrates the differences in responses
to knowledge and motivation questions by 45 RNH program leaders and 6 program supervisors.
The mean scores along with the standard deviations are shown.
17

In this way, we were able to assess both program leaders’ and program supervisors’
views on the two critical components of instructor capacity—knowledge and motivation—using
the same survey questions. Program leaders answered questions pertaining to their own

17

Linda Moon created this figure.

STEM
NEEDS
ASSESSMENT

54

knowledge and motivation, whereas program supervisors provided their perceptions of their
program leaders’ knowledge and motivation. Program leaders reported somewhat higher means
for both knowledge and motivation than did program supervisors (see Figure 1). The
significance of these reported values will be further discussed in Chapter 5.
Focus Groups with Program Leaders
18

Focus group interviews were conducted at all six of the RNH sites in order to determine
the program leader’s knowledge and motivation of STEM centered content. The aim was to ask
questions that would allow the program leaders to elaborate on the interview topic and respond
as a participant in a casual discussion. For example, when the program leaders were asked about
their understanding of STEM education there was no universal response that would show
communication or a common philosophy of a STEM concept within the organization. However,
the program leaders did respond with their own individual account of what STEM education is
and what it means from their perspective.
The focus group sessions occurred about one hour prior to the program leaders start time
for their STEM classes, most took place in the RNH program supervisor’s office, except for two,
which were held in classrooms used by the program leaders prior to the start of class. The focus
group usually consisted of three to five program leaders and one activities specialist. The
activities specialist at each site managed the different educational clubs within the RNH
program, which are designed to enhance student interest and give them a sense of autonomy in
their learning. The educational clubs are chosen by the students, this dynamic gives them the
chance to learn more or participate in a subject which they have interest and they have requested.
The design of the focus group interviews consisted of a combination of structured and
semi-structured questions which allowed the researcher to stay on target with questioning as well

18

Nsoah Abu-Rasool wrote this section.

STEM
NEEDS
ASSESSMENT

55

as provide the flexibility consistent with an open dialogue, which encouraged the participants to
express their point of view, and understanding of STEM and motivation related topics. The
environment at all sites appeared to be relaxed and the program leaders seemed to be candid in
their responses without any noticeable uncooperative vibes or tensions. Descriptive data, such as
the amount of years they have worked in the RNH program and educational levels, was collected
from the program leader’s responses to the focus group interviews. Additionally, the accounts of
the program leaders motivation and STEM knowledge capacity provided valuable details for data
collection.
Knowledge. To get a sense of the program leaders’ STEM knowledge a series of
questions were developed for the focus group interview. The intent of the questioning was to
find out the STEM related background of the program leader, and their overall knowledge of
STEM. For instance, when asked, “What is STEM education?” one program leader laughed and
simply recited what the letters in the acronym stood for, while another replied, “STEM education
leans toward innovation so that kids can have an experience that will inspire them to someday
become scientist.” At all six sites, where the program leaders participated in the focus group
interviews, there appeared to be no common theme or consistent response of the definition or
perception of STEM education. Overall, the answers were quite different and it seemed that
each program leader had their own vision of what STEM actually meant to them and how much
they actually knew about the subject.
With respect to STEM content knowledge, the program leaders generally felt that they
did not have sufficient background knowledge of the material, but felt they knew enough to teach
the material to their students. When asked about specific STEM-related skills, such as project-
based learning or inquiry skills, the responses varied significantly. For instance, one program
STEM
NEEDS
ASSESSMENT

56

leader stated, “ Problem based learning is a base but sometimes because of lack of materials we
have to improvise when we are low on materials.” Another program leader, when asked about
project based learning, responded, “Let me try to remember, it’s not lecturing it’s more like
hands on, so you spend like five minutes giving instructions and teaching the rest of the time.”
Especially with respect to project-based learning, because this is the name for the form they fill
out for their clubs, the discussions about PBL were mostly about their processes about how they
fill out the form, rather than the broader concept of PBL as related to STEM.
In terms of prior STEM knowledge, all of the program leaders revealed they did not have
extensive background knowledge or previous experience with STEM. One program leader
stated that, “I didn’t know anything about STEM until about two or three years ago when I
started working here in the RNH program.” Another program leader revealed that they had taken
a robotics class in high school and that was their only experience to STEM prior to working in
the RNH program.
Motivation. Although the program leaders collectively admitted to having a limited
knowledge of STEM content, this did not consequently translate to any ineptitude or lack of
confidence and motivation in their ability to teach STEM subjects effectively. Moreover, even
though the program leaders admitted, in the focus group interviews, that their knowledge outside
of the given curriculum was limited, they still seemed motivated to teach because they felt that
they were having a positive effect on the student’s academic growth.
The motivation section of the focus group interview included questions aimed at finding
out why the program leaders taught STEM and what was the motivation behind wanting to teach
STEM. Together the program leaders felt that they were achieving some sort of goal by teaching
STEM to the students. For example, one of the program leaders stated,
STEM
NEEDS
ASSESSMENT

57

The excitement in the kids, I feel that is the biggest motivation for me, to see them
excited about learning, the smiles on their faces when they are building something, and
knowing I had a part, that is my motivation.
During another focus group interview, one of the program leaders revealed that their
daily goal is to enhance the lives of their student’s in the RNH program.
The program leaders responses, to the motivation questions, seemed to follow a constant
theme of wanting to teach and enhance the academic lives of their students. In addition, they
seem to be very motivated to teach their students and help them achieve academic growth. For
instance, a program leader stated, “the reality is that I’m supposed to be teaching them something
but in actuality they are teaching me something and they don’t even realize it, that is really
motivating to me and it can’t get any better than that.” Another program leader said, “I think it is
just interesting to see how they take everything in, it makes me think back to when I was
younger and how I enjoyed taking in information.” Although the program leaders expressed
excitement about their motivation to teach and to see the progress of their students, there did not
seem to be any mention of them having motivation to engage or learn more about subjects
related to STEM education on their own during the focus group interviews.
Interviews with Program Supervisors
19

Each of the program supervisors participated in an interview with the researchers that
lasted for thirty to forty minutes. One portion of the interview focused on the program
supervisors’ perceptions and assessments of their program leaders’ knowledge and motivation,
both overall and specifically regarding STEM content. Some general expectations that were
mentioned of the program leaders included characteristics such as punctuality, energy,
passionate, responsibility, flexibility, creativity, and communication. Most of the program

19

Linda Moon wrote this section.

STEM
NEEDS
ASSESSMENT

58

supervisors emphasized that their program leaders must be a “kid-magnet,” in addition to being
willing to make mistakes and learn from them.
Knowledge. When asked to assess program leaders’ knowledge when it comes to
teaching STEM and project-based learning, program supervisors stated that the level of
knowledge undoubtedly varied from individual to individual. Program supervisors were
confident that the program leaders had sufficient knowledge to lead the clubs to which they were
assigned, but not many actually have any more background knowledge than what is provided in
the STEM curriculum used in RNH. In addition to content knowledge, one factor that was
discussed is that program leaders may not have a lot of pedagogical knowledge, since more than
half of the program leaders have little to no experience with teaching, let alone teaching STEM
material (see Table 3). All of the program supervisors stated that their staff could definitely
benefit from additional professional development, and more focused training for STEM.
Motivation. When asked if they thought that program leaders were generally motivated
when they taught STEM, program supervisors again said that level of motivation varied among
individuals. Some program leaders are naturally more motivated and energetic in everything that
they do, but others’ motivation could easily be affected by several factors. One critical factor is
the program leaders’ personal interest in a particular topic that day or week. If it was a topic that
is inherently interesting to them, then program leaders were much more motivated to teach that
topic, and also do additional research, such as finding online videos, to present to their
students. Another factor, as also reported by program leaders, was the motivation level of the
students themselves. They feed off the excitement and engagement of the students, especially if
the students are retaining information or actively participating. As one program supervisor
observed,
STEM
NEEDS
ASSESSMENT

59

The best motivation is self-motivation. Most staff are motivated when they see how their
work is impacting their kids. One program leader reported that his instruction regarding
healthy behaviors seemed to be working because one of his students was starting to
noticeably slim down. He was very excited that he seemed to be making a difference in
that student’s life.
A third factor that program supervisors expressed helps program leaders’ motivation is
feedback and affirmation. One program supervisor conveyed that she found it effective when
she provided feedback to her program leaders, and stressed the fact that she was there to support
them in whatever they needed. Along the same vein, another program supervisor attested to
working hard in order to collect program leaders’ buy-in to their collective visions and goals, that
by taking such ownership, they will be more motivated to work hard. Yet another program
supervisor said she usually gives little gifts for her program leaders, not purely as incentives but
to show her appreciation for their hard work. By maintaining open lines of communication and
providing constructive feedback and support, program leaders are more motivated in their jobs.
Observations
20

An observation of the program leaders, at all six of the RNH program sites, provided
valuable information in regards to their STEM capacity in the classroom, and teaching
styles. The goal of the observations was to observe the program leaders in a classroom setting
and collect data with the aid of an observation protocol. However, upon completion of the
observations, the relationship between what was observed and what was stated during the focus
group interviews and Qualtrics survey will be further discussed in Chapter 5.
The physical makeup of each classroom in the MESD was very similar. The rooms all
had adequate natural and artificial lighting, which allowed for unobstructed views of the teaching

20

Nsoah Abu-Rasool wrote this section.

STEM
NEEDS
ASSESSMENT

60

and reading needs of the students. The walls were covered with educational content that
matched the elementary school setting and various grade levels, such as the whole number scale,
world maps, and the alphabet. The participants in the classroom consisted of the program leader
and a student population that ranged from 20-30 in class size. The grade level of the students
ranged from kindergarten to 5
th
grade, and the instructors seemed well-equipped at making the
adjustment to the different grade levels by their approach and engagement techniques during
instruction.
A consensus of what learning looked like, from the teaching styles observed, consisted of
four primary components. The first component consisted of a STEM-focused lecture taught by
the program leader. Moreover, the program leaders usually positioned themselves in the front of
the class and the students sat at either their desk or in a semi-circle on the floor. The program
leaders maintained eye contact with the students during the lectures to see if they were engaged
in the lessons or had questions. Second, the program leaders provided a demonstration, either
through an actual display of the upcoming activity or through a video on the classroom television
or on a tablet (i.e., iPad). The third component was an activity that consisted of some sort of
participation or practical application of the lesson by the students. The observed student activity
usually had components of the lesson that had been explained during the lecture and
demonstration component of the learning session. The fourth and final component was centered
on a reflection of what had just occurred from the lecture, demonstration, and the student
activity. The program leaders would give feedback and ask the students questions pertaining to
the, “how, what, and why,” of the learning experience to see what they had retained. The
reflective questions, asked by the program leaders, seemed to trigger within the students an
ability to recall what they had learned from the previous three components of the program
STEM
NEEDS
ASSESSMENT

61

leaders teaching style. In addition, the program leaders were viewed providing guidance,
facilitation, and student engagement techniques, which seemed to foster an environment that
allowed the students to be more involved in the learning process.
The program leader’s feedback, in particular, seemed to be very helpful to the
students. During the observations, the facial expression of the students was a big giveaway of
how they seemed to feel about the feedback from the program leaders. For example, when a
smile appeared the student was usually receiving positive feedback for working correctly on an
activity and when the, “ah-ha I understand,” expression was visible they were usually receiving
corrective feedback for guidance and gaining a deeper understanding of the
subject. Additionally, the interaction between the program leaders and the students showed that
the program leaders had a grasp of the content outlined in the curriculum. However, they did not
seem to provide more in-depth answers or feedback beyond what was explicitly scripted in the
curriculum.
Within the four components of teaching displayed by the program leaders, the sequence
of events usually followed the same pattern at all sites observed. For example, during an
observation, two components of STEM were noticed that featured science and engineering. The
observation protocol contained a section with the sequence of events separated into three frames
of reference, the beginning, middle, and the end. The initial sequence of events, the beginning,
the program leader gave a lecture on the science component, severe weather and the damages it
can cause to man made structures. In addition, the program leader explained to the students that
they would be participating in an activity that would allow them to build a house, the engineering
component, with materials provided to withstand severe weather. The second sequence of
events, the middle, consisted of a demonstration followed by an activity of the students building
STEM
NEEDS
ASSESSMENT

62

an actual severe weather resistant home. As the students worked together in groups the program
leaders walked around the classroom to monitor their progress and facilitate questions or concur
with students who were achieving success within the guidelines of the project. Finally, in the last
sequence of events, the end, the program leader asked reflective questions in an effort to gain a
deeper understanding of the students work and to see what the student had learned during the
period of instruction. Some of the paraphrased questions asked by the program leader were,
“why did you chose to build in a given location, why did you chose a certain material, and what
severe weather were you trying to defend against in your design?” Overall, the students seemed
happy, engaged in their activities, interacted respectfully to one another, and appeared to have a
close relationship with the program leaders.
Question 2:
Is Each Site Providing Sufficient Opportunities for Professional Development?
Survey Responses
21

Both program leaders and supervisors were asked questions regarding both the
availability and access to professional development and their views on its importance (see Figure
2). The responses were based on a 6-point Likert scale (1=strongly disagree, 6=strongly
agree). Program leaders reported a mean of 4.78 (SD=1.1) for professional development
availability, as in trainings and resources they have been and are currently supplied with, and a
mean of 5.42 (SD=.74) for professional development importance. For the same questions,
program supervisors reported somewhat lower but not statistically significant means for both
sections: for availability, the mean was 4.58 (SD=.75) and for importance, the mean was 4.13
(SD=.12). Two of the original questions, “I have been enriched by the training events I have
attended,” and “Without the training I receive in professional development, I would not be able

21

Linda Moon wrote this section.

STEM
NEEDS
ASSESSMENT

63

to teach STEM effectively,” were discarded because there were errors in the construction of the
response choices in Qualtrics.

Figure 2. Mean professional development responses. This figure depicts the responses from
program leaders (N=44 for availability, and N=43 for importance) and program supervisors
(N=6) regarding the availability and importance of professional development at RNH. The
means and standard deviation for each categorization are shown.
22

In addition to the Likert scale items, both program leaders and program supervisors were
asked two open-ended questions: 1) In the past year, how many of the following STEM-related
support services have your received—training and coaching, and 2) Please rank the following to
what you think are the most pressing needs related to implementing STEM activities in your
program—training, information and resources, coaching, site visits, and consult with outside
experts. Program leaders reported a mean of 1.3 (SD=1.11) training events and 1.24 (SD=1.3)
coaching events, whereas program supervisors reported means of 1.83 (SD=1.72) and 2.33

22

Linda Moon created this figure.

STEM
NEEDS
ASSESSMENT

64

(SD=2.5), respectively. Only 12 program leaders answered the question regarding pressing
needs, and all of them listed training, information, and resources, as the most pressing need.
Four out of six program supervisors supported this as the most pressing need, while the
remaining two indicated that coaching and consult with outside experts were the most pressing
needs.
Focus Groups with Program Leaders
23

The objective for this section of the focus group interview was to find out the frequency
of professional development trainings offered and attended by the program leaders, and if they
were perceived to be addressing their needs. According to the program leaders, professional
development is offered in the RNH program to the program leaders and it is often taught by the
program director and program supervisors. Moreover, the program leaders indicated that there
are usually three major professional development sessions for the RNH program within the
school year. The first professional development is held in January, which is called Local
Learning Community (LLC), it consists of a two-day forum of rotating subjects, including
STEM, which last about an hour per subject. It is held off-site with trainers from various
education centers in the region and includes all of the personnel in five neighboring districts’
afterschool programs. The second professional development is held in the summer at the end of
the school year in June, and the third is held prior to the beginning of the next school year in
August. The second and third professional development sessions are held at on-site locations
within the MESD.
The specific topics covered in the professional development sessions ranged from
organization and management of the classroom to behavioral management of students who may
have challenges in those areas. One of the program leaders stated,

23

Nsoah Abu-Rasool wrote this section.

STEM
NEEDS
ASSESSMENT

65

We received professional development training on the STEM curriculum from the
NASA’s Best vendor, which consisted of training on how to use the materials provided in
the curriculum box set, as well as, how it should be maintained.
In most cases the program leaders felt that the professional development training was
helpful but thought that it should include more emphasis on STEM-related subjects and
organizational goals. In fact, one of the program leaders stated,
I would like to see someone show us exactly how to do a PBL for consistency, because
sometimes when we turn them in, some things are not right, or we need to make
corrections. Once, at a training session, they gave us a PBL already filled out but they
did not show us how to do it so it was unclear to us how to fill it out correctly.
Interviews with Program Supervisors
24

As stated previously, program supervisors agreed that additional training would prove
beneficial for their staff. The main modes of training for all of the Meadows RNH staff are
through the mass training events held once before the school year begins and once during winter
break. These sessions are typically led by the program director and all the program
supervisors. Program leaders also participate in a professional development workshop held by
the Local Learning Community (LLC) that is comprised of afterschool programs throughout
Meadows, Lynnwood, Lennox, Hawthorne, and Compton. In addition to training in STEM, the
training sessions cover various topics such as behavior management, curriculum training, and
project-based learning. A few program supervisors stated that they would like to see STEM
workshops that were more experiential in nature, as opposed to a more didactic model, and also
workshops that focused less on how to make STEM better but even more fundamentally, what
STEM actually is and why it is important.

24

Linda Moon wrote this section.

STEM
NEEDS
ASSESSMENT

66

Starting last year, the training sessions were organized by tiers, where the program
supervisors categorized each of the program leaders into one of four tiers—beginning,
intermediate, advanced, and applying—based on years of experience and general performance
evaluations. Program supervisors found that this method was beneficial because for a while the
program leaders had been saying that there was a lot of repetition at training sessions. By
organizing them into tiers, they were able to sort information based on its suitability for veteran
or newer staff members.
In addition to these large-scale events, each of the program supervisors initiates their own
training and coaching sessions at their site. As of late, however, program supervisors express
that they find it difficult to squeeze in time for much site-specific professional development.
One of the main reasons is that the staff are now unionized, so there are specific standards
regarding breaks and overtime schedules that need to be complied. All of the program
supervisors asserted that they try to hold meetings at least once a week, but a lot of the time, the
meeting time gets devoted to lesson planning and preparations. Due to this phenomenon, a few
of the program supervisors suggested that the program leaders need continual training, not just at
the district-wide RNH sessions, but perhaps through visiting another site apart from their
own. One supervisor suggested,
We need a system of coaching and staff development at the site level. If program
supervisors would form subcommittees on a particular subject…master it, and offer
training to program leaders throughout RNH, that would be ideal.
Additionally, program supervisors feel that program leaders would most likely want to
receive training from experts outside of their own site supervisors. The reason for this, as
inferred from the interviews, is that they sense program leaders are viewing the training sessions
STEM
NEEDS
ASSESSMENT

67

by the same people to be redundant, thus devaluing professional development altogether.
Moreover, program supervisors indicate that they need the time to recharge and be trained also.
Because they conduct their own versions of coaching at their respective sites, they feel much of
the large-scale professional development sessions should be relegated to outside experts.
Observations
25

We were able to attend one of the large-scale Local Learning Community (LLC) training
sessions held on January 31, 2015 at a neighboring school district. This professional
development event, as mentioned previously, is a collaboration between five districts located in
the South Bay region of Los Angeles County. They offered two back-to-back workshops
regarding STEM instruction, which was attended by three program leaders in the first session,
and eight program leaders in the second, with no attendees from Meadows at either
session. Both workshops were led by the same speaker, a retired K-12 educator who specializes
in afterschool programs.
The focus of both workshops was about how to engage students into the process of
scientific thinking. The first workshop focused on inductive versus deductive reasoning while
the second workshop focused on using elements of Dimensions of Success (DoS) and developing
learning goals. Both sessions started out with an icebreaker where participants had to try to
remove all but one card from a deck of cards balanced on top of a paper cup. The purpose of this
exercise was to integrate the scientific process into an activity seemingly unrelated to
science. During this exercise, participant used skills such as formulating a question, predicting,
planning, executing, reflecting and repeating, in order to achieve the task at hand. Afterwards,
program leaders in the first session practiced inductive and deductive reasons by categorizing
fruits and vegetables. The use of both pictures on cards and handling actual fruits and vegetables

25

Linda Moon wrote this section.

STEM
NEEDS
ASSESSMENT

68

that were brought in by the speaker made the learning experience more engaging. The second
session was centered around the question, “What degree of slope for a ramp allows a car to go
the furthest before stopping,” so participants took the materials provided (e.g., tracks, racecars,
protractors, measuring tape, baking sheets) to experiment with this question.
Throughout both sessions, the speaker was focused more on the process and the
conversations surrounding the process rather than whether or not the task was achieved. The
participants were asked to articulate what they did, what they learned, and how could they use
what they had learned in the future. Additionally, they were urged to encourage similar
discussions with their own students in each of their contexts.
Question 3:
Is the Current Curriculum Adequate for Promoting STEM Learning?
Survey Responses
26

Both program leaders and program supervisors were asked to evaluate the curriculum
they utilize to teach STEM in each of their RNH sites. The survey asked respondents to assess
how they believed the students were responding to the STEM curriculum, how STEM
components were being integrated into their programs, and whether or not their STEM
programming was compliant with general expectations of project-based learning. While the
surveys did not specifically asked to rate each of the curricula, from our interactions with the
program leaders and supervisors, the main STEM curricula in use at RNH are MindWorks,
KidzScience and Nasa’s Best. As depicted in Figure 3, program leaders rated curriculum
adequacy at a mean of 5.01 (SD=.71), and program supervisors at a mean of 4.62 (SD=.53).

26

Linda Moon wrote this section.

STEM
NEEDS
ASSESSMENT

69

Figure 3. Mean curriculum adequacy responses. This figure depicts the responses from program
leaders (N=42) and program supervisors (N=6) regarding adequacy of the current STEM
curriculum to promote STEM learning among students. The means and standard deviation for
each categorization are shown.
27

Focus Groups with Program Leaders
28

The aim for this section of the focus group was to identify the program leaders’
perception of the STEM curriculum used in the RNH program and to gather their opinions to
what extent they felt the curriculum was effective. The STEM curriculum used by the RNH
program consists of several vendors ranging from, MindWorks, KidzScience, Trash for
Teachers, and NASA’s Best. The program leaders revealed that the curriculum comes in box
sets and are stored in the program supervisor’s office or in a storage area on the school
site. Each curriculum vendor has a particular set of materials, components, and lesson plans used
for curriculum delivery. Some are equipped with materials and laminated instructions that can
be reused multiple times, however, there are also some that have materials designed for a one

27

Linda Moon created this figure.

28

Nsoah Abu-Rasool wrote this section.

STEM
NEEDS
ASSESSMENT

70

time use, which leaves shortages for other program leaders who share the curriculum box
sets. Instructions on how to use the curriculum are included from each vendor in all of the box
sets.
The RNH program also incorporates project-based learning (PBL) as part of their
curricular strategy. When asked about PBL, program leaders revealed that they were unsure
about what it is and how it should be used. One stated that, “I once heard of PBL but needed
more clarification on what it actually meant,” and another stated that, “I had seen a presentation
about PBL during a professional development session but I was still unclear about how it should
be used and on what occasion.” A contributing factor to this confusion may be that PBL is used
throughout RNH in a specific context, as it is the name of the lesson plan worksheet they fill out
for their clubs. Upon hearing PBL, they are focused more on the logistics of filling out the form,
and not so much on the broader construct of project-based learning. It appears that PBL has been
integrated into the programming but it has become more of colloquial way to refer to their lesson
plans, and evidence of a deeper understanding of the construct of PBL was lacking.
Interviews with Program Supervisors
29

While each of the RNH sites use MindWorks, KidzScience, and Nasa’s Best as their
STEM curriculum, the interview was mostly focused on each site’s experience with MindWorks,
as it was the most widely and frequently used curriculum. The program supervisors, for the most
part, are content with MindWorks, mainly due to its convenience. Since everything—supplies
and scripted lesson plan—comes in a box, the program leaders have little to prepare, which is
expedient since they do not have sufficient time to prepare as it is. As one program supervisor
described,

29

Linda Moon wrote this section.

STEM
NEEDS
ASSESSMENT

71

Because of the lack of time, [MindWorks] is convenient. It comes in a box with all the
supplies, all the copies are ready, there’s a script for newbies but there is room for
creative [program leaders] to make adjustments. Prior to this, we had to make up our
own STEM curriculum and the outcome was hit or miss… and MindWorks has already
been tested.
Regardless, all of the program supervisors state that they encourage their program leaders
to “think outside the box,” by searching for additional resources to supplement the boxed
curriculum, and to alter the suggested activities based on classroom size and amount of time they
have.
Despite its convenience, program supervisors do pinpoint some of the challenges in
utilizing the MindWorks curriculum. Each MindWorks box represents a 10-week unit, which
gets rotated throughout each site, among seven to eight different clubs. Because of this rotation,
and even though the box is theoretically designed for 125 students, a lot of the supplies run out
along the way, which was a chief complaint of the program leaders as well. In addition, while
the scripted lesson plans are helpful, they can also cause some program leaders to not prepare as
much, and end up reading the material for the first time in the classroom.
Regardless of the curriculum’s strengths and weaknesses, program supervisors all state
the importance of capturing student engagement by whatever means. A few of them used the
phrase, “disguised learning” as a vital component in the classroom; they believe in kids learning
subconsciously through something fun that seems like play. As one program supervisor stated,
“We are still moving forward and still learning and trying to make sure we are doing what we
can to be knowledgeable and have enough resources for the students.”
STEM
NEEDS
ASSESSMENT

72

Document Review
30

The main STEM curriculum is called MindWorks Resources. According to the
MindWorks website, the curriculum is designed in compliance with common core standards, pre-
and post- tests, project-based learning components, and hands-on learning (“MindWorks
resources,” 2013). There is one overarching theme per school year, which is divided into seven
different units. For the current school year, the theme is “Mission Possible” and the seven
different units are Myth Buster Kids, Whodunit, A Week in the Life, Sticky Situations, Storm
Chasers, Entrepreneurs and Inventors, and Future Builders. Every week in each of these units,
students explore a different subject matter, for a total of six subjects, or clubs, in six weeks: Math
Matters (math), Around the World (social studies), Art in Action (art), Exploration (science),
Bring on the Books (reading), and Spreading the Word (language). Each of these clubs has their
own box, complete with a step-by-step lesson guide and most of the materials needed for all the
activities.
Each lesson is formatted in the following manner: overview of the lesson, lesson
objectives, materials list, pre-test, pre-activity discussion, activity, extension activities, post-
activity discussion, and post-test, which contain the same questions given in the pre-test. The
lesson guide is scripted so that the instructors not only have instructions to follow but it includes
the dialogue they can read aloud to the students. The lessons are also differentiated into two age
groups: kindergarten to second grade, and third to fifth grade.
Depending on the RNH site, MindWorks is taught once or twice a week, lasting
anywhere between 45 to 90 minutes. While each unit is supposed to last for six weeks, RNH
typically uses each box for 10 weeks before it is rotated to a different group. Because of this,
some of the lessons are stretched out over two weeks to accommodate the schedule. A further

30

Linda Moon wrote this section.

STEM
NEEDS
ASSESSMENT

73

discussion of the MindWorks curriculum in light of interviews with program leaders and
program supervisors is presented in Chapter 5.
Summary
With regards to the three needs assessment concerns for the RNH program—instructor
capacity, professional development, and curriculum adequacy, both program leaders and
program supervisors see room for improvement. The program leaders believe that they lack a
strong foundation in STEM knowledge, but are confident in their ability to teach and their
motivation to teach. Additionally, they indicated that the enlightened look displayed on their
students faces, when they learn something new, is a motivational event that they look forward to
on a daily basis when reporting for work. Program supervisors feel that the program leaders
have sufficient knowledge for the task at hand, but further training would be useful. They also
feel that program leaders’ motivation is largely contingent on their personal interest in a
particular topic and the level of the students’ engagement.
In the area of professional development, the program leaders felt that overall, they are
receiving training that will help them succeed as STEM instructor, however, they would like to
learn more about STEM and other teaching methods that is in alignment with the RNH mission
and vision, such as PBL and 21
st
Century learning strategies. Program supervisors would like to
see continual, hands-on training for staff and if possible, conducted by outside experts.
In terms of curriculum adequacy, both program leaders and program supervisors attest to
the convenience of MindWorks, as everything comes in a box. However, program leaders are
encouraged to enhance the learning experience by “thinking outside the box,” and supplementing
the scripted lesson plans with additional resources.

STEM
NEEDS
ASSESSMENT

74

CHAPTER FIVE: DISCUSSION
Authors: Linda Moon and Nsoah Abu-Rasool
Engaging students in STEM during out-of-school time (OST) programs is a difficult but
necessary prerogative, as STEM proficiency becomes increasingly important for students and for
the economy at large. Even so, the reality is that OST programs cannot hope to provide
extensive academic direction in STEM as there are definite limitations in instructor capacity,
professional development, and adequate curricula. However, OST programs, such as
afterschool, can provide outlets for students to develop self-efficacy and interest, which in turn
may assist students’ overall performance in STEM (Dierkling, 2007).
In Meadows Elementary School District’s Reaching New Heights (RNH) program, there
is a concerted effort to promote STEM learning in their K-5 students. The purpose of this needs
assessment was to use multiple data collection methods and two interest groups to understand the
current state of RNH instruction, professional development, and curriculum in an effort to
determine what areas require further attention. The three central questions guiding this
assessment were:
1. Are instructors knowledgeable and motivated in teaching STEM?
2. To what extent are instructors provided opportunities for professional development?
3. Is the current curriculum sufficient for promoting STEM learning?
The previous chapter presented the results of the data analysis. In this chapter, these
results are discussed and implications and recommendations based on these findings are
presented. The discussion is organized as follows: 1) interpretation of the findings, 2) limitations
of the project, and 3) recommendations for RNH and possibly future research.
31

31

Linda Moon wrote this section. Nsoah Abu-Rasool made contributions.

STEM
NEEDS
ASSESSMENT

75

Discussion of Findings
Instructor Capacity
32

From our research, the RNH program exhibits strengths in the following areas when it
comes to instructor capacity. For one, the program leaders indicated, during the focus group
sessions and in their survey responses, that they were highly motivated and enthusiastic about
teaching STEM subjects to their students. They enjoyed being a part of the RNH program and
valued their contributions to STEM enrichment and learning as a whole. Although the program
leaders revealed that their background knowledge of STEM-related subjects was limited, they
indicated they had sufficient knowledge to teach the material at hand. In addition, the program
leaders’ passion to engage students in STEM learning was a positive theme that was noticed
throughout the focus group interviews and the classroom observations.
Beyond instruction in the classroom, the program leaders have built relationships with
their students which is a vital component of out-of-school time instruction (Miller, 2010).
Program leaders seemed to have a genuine concern for their students. As one program leader
stated, “I really enjoy the kids, to watch them learn something that they have never seen before
and to see the smile on their face, that’s my motivation, that’s what keeps me going.” This was a
sentiment that was echoed with all the RNH staff across all the sites we researched. Overall, the
program leaders seemed to be willing to learn and they believe that more STEM-related training
will help them become even more effective instructors within the RNH program.
Knowledge of program leaders.
33
According to the survey results, program leaders’
self-report of their knowledge (M=4.75) is noticeably higher than program supervisors’
perceptions of their knowledge (M=3.82). This could be because the program leaders know

32

Nsoah Abu-Rasool wrote this section.

33

Linda Moon wrote this section.

STEM
NEEDS
ASSESSMENT

76

more than the supervisors are aware of, or they have a higher sense of self-efficacy or confidence
in their teaching knowledge. The latter notion is supported by the data, as the program leaders’
self-efficacy score was high (M=5.01). Another possible explanation for this discrepancy is that
because pedagogical knowledge and content knowledge were combined in the knowledge score,
the score could reflect more on what’s visible in the program leaders’ teaching, rather than
assessment of content knowledge. This is supported by program supervisors’ comments
regarding program leaders not having sufficient teacher training in addition to observations made
by the researchers wherein some of the program leaders tended to read off of the scripted lesson
guide and appeared to have not prepared enough in advance.
Even though program leaders reported a higher knowledge score, the score was not a
strongly positive one, which was reflected in the focus groups and observations. With respect to
knowledge of specific STEM-related terminology and concepts, there was some confusion
among program leaders. While all of them knew what the acronym stood for, their
understanding of each of the components was fixated on the more traditional sense of science
instruction, for example, rather than the principles of science that can be applied across
disciplines. The same was true when asked about how their STEM instruction related to 21
st

century learning skills. Beers (2012) mentions that these skills encompass various competencies
ranging from creativity and innovation to critical thinking and collaboration, but program leaders
only focused on the technology as 21
st
century learning skills. Furthermore, when discussing
project-based learning (PBL), program leaders responded mostly about their lesson plans, also
entitled PBL, rather than the underlying principles.
STEM
NEEDS
ASSESSMENT

77

Motivation of program leaders.
34
Similar to the knowledge survey results, there was
also a divergence in perceptions of motivation, as program leaders reported an average level of
motivation as 5.03, whereas program supervisors perceived their motivation to be 4.24. This is
surprising because, as discussed in Chapter 2, afterschool programs and the like are inherently
low stakes environments that generally yield lower levels of motivation (Bevan et al., 2010;
Freeman et al., 2009). However, according to the data, program leaders are highly motivated in
teaching STEM. This is further supported because all of the sub-categories of motivation that we
surveyed had relatively high scores: self-efficacy (M=5.01), teaching beliefs (M=4.93), value
(M=5.24), and interest (M=5.05). A possible explanation for why program supervisors’
perceptions of program leaders’ motivation was lower could be that they have not directly asked
their program leaders about their confidence in teaching STEM, their teaching beliefs, values, or
interest. They answered the survey questions purely on their observations, which is not entirely
reliable because a lot of these motivational factors are not outwardly apparent.
Moreover, the overall mean for outcome expectancies was 4.36, which shows that
program supervisors only somewhat agree that program leaders directly influence students’
learning outcomes pertaining to STEM. This reflects program supervisors’ composite
perceptions regarding their program leaders’ knowledge and motivation. From their
understanding of their program leaders’ knowledge and motivation, it would seem program
supervisors believe that other factors in addition to program leaders’ influence have profound
impacts on students’ STEM learning.
Professional Development
35

There are also definite strengths in professional development within RNH. Professional

34

Linda Moon wrote this section.

35

Nsoah Abu-Rasool wrote the first part and Linda Moon wrote the second part of this section.

STEM
NEEDS
ASSESSMENT

78

development is a very useful tool for organizations looking to keep their personnel sharp and up
to date on the latest policies and procedures in a given industry. It is important to note that the
leadership within the RNH program understands the importance of professional development and
how it can be used to help train their staff become more efficient in all aspects of educational
growth including STEM-related subjects. Based on our surveys, interviews, and observation, the
RNH program has three types of professional development events that occur at different times
during the school year: 1) the Local Learning Community (LLC), a large-scale training event
involving five neighboring districts, 2) district-wide training event dedicated to all the RNH
sites, and 3) training and coaching occurring at the individual site level. The program leaders
conveyed their enjoyment in the professional development trainings and expressed their value for
such trainings to help them become effective instructors. They were generally pleased that the
RNH program is committed to making professional development a major part of their teaching
experience. Additionally, program leaders and program supervisors alike have responded
positively to the practice of offering “tiered” workshops, where program leaders are organized
into four different tiers (i.e., beginning, intermediate, advanced, applying) based on their
experience. Innovations such as these show promise for continued growth in the area of
professional development at RNH.
Based on our data, more occurrences of professional development may be a start for
improving this area. Program leaders and program supervisors show similar views about the
availability of professional development, which encompasses resources, training, and
coaching. In terms of whether or not they think professional development is readily available,
program leaders and program supervisors reported average scores of 4.78 and 4.58,
respectively. Although on the higher end of the spectrum, these scores do not reflect very high
STEM
NEEDS
ASSESSMENT

79

positive responses. This is aligned with responses garnered during focus groups and interviews,
where both program leaders and program supervisors expressed that there needs to be more
training and resources. Specifically, more focused training on STEM would be beneficial. The
researchers were only able to observe one STEM training session, but based on that experience,
we would agree that STEM training did not involve a heavy emphasis on the importance of
STEM and did not have a lot of applicable elements to take back to the classroom. A possible
reason for this is that the workshop instructor neither knew nor gauged the amount of prior
knowledge and experience that the attendees had regarding STEM, so the instructor tended to
assume program leaders already knew a lot of the information.
Perhaps due to fewer training events than desired, program staff, and program leaders in
particular, believe them to be highly important. The overall mean score for the importance of
professional development was remarkably higher for program leaders (M=5.42) than it was for
program supervisors (M=4.13). One possible explanation for this phenomenon is that program
supervisors find that a lot of the information disseminated at training events is not readily
implemented into the classroom, thereby construing the idea that program leaders do not deem
professional development to be important. Another reason for this could be that program
supervisors are more knowledgeable about the resources and training that are needed, as they
were all program leaders at one point. Even though the survey questions specifically asked them
about current professional development efforts, some of their answers may reflect what they
once thought about professional development when they were program leaders. For example, in
question 21 of the survey questionnaire, “Professional development workshops are vital to
helping program leaders develop new teaching techniques,” they might have answered from their
own perspectives on how they were not dependent on such things, possibly. These are just
STEM
NEEDS
ASSESSMENT

80

conjecture, but a variety of different factors are at play that could account for these discrepancies
in scores.
A surprising result, however, is that even though program leaders indicated through the
survey and during focus groups that they regarded professional development to be highly
important and necessary, there were no representatives from Meadows present at either STEM
workshop at the LLC training observed by the USC evaluators in January 2015. This may go to
show that the results regarding professional development may not have been specific to or even
related to STEM at all, but rather they found training in other areas (e.g., behavior management)
to be helpful. After all, literature shows that effective professional development is concerned
also with building long-term career and life skills (Freeman et al., 2009; Sheldon et al.,
2010). Regardless, given that they expressed the need for additional STEM training, the
program leaders might have sought out these workshops to supplement their STEM instructional
knowledge.
Curriculum Adequacy
36

Project-based learning is an essential tenet of STEM and of the RNH program, which is
reflected in their main STEM curriculum, MindWorks. The alignment of the MindWorks
curriculum with the RNH program teaching goals, helps maintain academic continuity within the
program. In addition, the MindWorks curriculum is designed for elementary and middle school
students enrolled in afterschool educational programs, including programs with a STEM
focus. RNH staff commented on the benefits of having the user-friendly MindWorks curriculum
to teach STEM, during the focus groups and interviews, when they explained how the step-by-
step instructions, pre- and post-test, and hands-on activities are all included as part of the
curriculum in each box set. The program leaders also indicated that the step-by-step instructions,

36

Nsoah Abu-Rasool wrote this section. Linda Moon made contributions.

STEM
NEEDS
ASSESSMENT

81

albeit a bit tedious, were very helpful for teaching STEM especially when they had limited
background knowledge or preparation time.
Regarding whether or not the STEM curriculum is sufficient for promoting STEM
learning among students, program leaders were slightly more affirmative (M=5.01) than were
program supervisors (M=4.62). This was an unexpected result because many of the program
leaders found MindWorks to be tedious in our focus group discussions. However, because the
survey only referred to the STEM curriculum more generically and did not specifically ask about
MindWorks, these responses could reflect their overall opinions about all the STEM-related
curricula used throughout RNH combined or about a single curriculum that was not MindWorks
(e.g., KidzScience, Nasa’s Best).
Even though most of the program leaders mentioned that they found the MindWorks
curriculum to be tedious because of its step-by-step format, it appears that this type of scaffolded
curriculum is necessary in this context. All of the program supervisors stated that there was not
enough time for program leaders to prepare their lessons each day. One of the main reasons for
the lack of preparation time is that the RNH staff are now unionized, which make it challenging
for program supervisors to ask program leaders to invest any additional time for which they will
not be compensated. In such a context, MindWorks seems to be a convenient curriculum to use
in that it requires minimum preparation. However, the program supervisors, based on their
response, do not seem overly enthusiastic about MindWorks’ prospects for promoting STEM
learning in their afterschool students.
Recommendations
37

Despite the limitations of this project that will be discussed later, this needs assessment
still identified major areas in which the MESD RNH program could enhance their STEM

37
Linda Moon wrote this section. Nsoah Abu-Rasool made contributions.

STEM
NEEDS
ASSESSMENT

82

activities and instruction at each of their sites. As evidenced in our interactions with the RNH
staff, program leaders and program supervisors are tasked with a myriad of roles and
responsibilities outside of STEM instruction. But given the rising importance of STEM at both
the individual and global level, we feel attention to STEM programming should be a top
priority. Hence, the following recommendations should be taken into account by the RNH staff,
for the purposes of preparing their students to develop proficiency and motivation in STEM-
related subjects, careers, and thinking processes.
Instructor Capacity
Knowledge. To enhance program leaders’ knowledge, it seems appropriate that program
supervisors take inventory of their program leaders’ background knowledge and also
pedagogical knowledge. While program leaders and program supervisors contend that program
leaders have sufficient knowledge for the task of teaching the MindWorks curriculum, it appears
that they may be lacking in sufficient metacognitive knowledge that would allow them to go
beyond scripted lesson plans. Developing metacognitive knowledge in program leaders would
be heuristic in nature, enabling program leaders to become self-aware of their knowledge and to
apply appropriate strategies to enhance their knowledge and consequent instruction (Anderson &
Krathwohl, 2001). Instructors with sufficient metacognitive knowledge about applying STEM
content will be able to assess the task at hand, evaluate their own knowledge and skills, plan an
appropriate teaching approach, apply and monitor various teaching strategies, and reflect and
adjust if necessary (Ambrose, Bridges, DiPietro, Lovett, & Norman, 2010). Based on these
criteria, we suggest that program leaders and program supervisors take the following actions to
achieve higher levels of knowledge in RNH program leaders.
STEM
NEEDS
ASSESSMENT

83

Assess the task at hand. Program supervisors should review each MindWorks unit with
all the program leaders prior to the start of said unit. Even though each program leader will
implement the lessons at different times, having reviewed it collectively will allow the program
leaders to know what to expect, rather than possibly diving into each unit without any
background. During this overview session, program supervisors can entertain questions, allow
collaboration and group brainstorming for ideas, and emphasize the desired outcomes of each
lesson.
Evaluate knowledge and skills. While using the MindWorks curriculum to teach STEM
in the RNH program does not require program leaders to be fluent in STEM content, assessing
knowledge is important for situations where one may possess incorrect prior knowledge that
subsequently impedes student learning. Moreover, assessing prior knowledge would also help
program leaders unlock relevant background information that can enhance their lessons.
A relatively simple way to conduct a STEM knowledge assessment in RNH would be to
distribute the list of key terms supplied by MindWorks that students are expected to know at the
end of a MindWorks lesson. Program leaders can indicate the depth of their knowledge of each
term with statements such as “I have never heard of this term,” “I have heard of this term,” “I
could define this term,” “I could explain it to someone else,” and “I can use it to solve problems”
(Ambrose et al, 2010, p. 29). The rationale for this assessment is that if the instructors
themselves only have cursory understandings of key terms, students cannot be expected to
develop deeper understandings, which is ultimately the goal of effective STEM
instruction. Following this assessment, program leaders should work to address the areas in
which they need more knowledge, and program supervisors should assist program leaders in
identifying and addressing these knowledge gaps accordingly.
STEM
NEEDS
ASSESSMENT

84

Plan an appropriate approach. The lesson plans are already provided in the STEM
curriculum, but program leaders can benefit from planning how to execute the lesson plans. Due
to time constraints, this step could be as simple as asking program leaders to orally explain how
they intend to implement a particular lesson. Being cognizant of all the components of the task
in this manner may yield more intentional and well-prepared instruction.
Apply strategies and monitor progress. At the end of each lesson, program supervisors
should require program leaders to do a brief self-assessment of how successful it was. The
assessment could range from program leader conduct to perceptions of student learning. A wide
range of self-assessment checklists can be found at http://teacher.scholastic.com/professional/
selfassessment/checklist/. In addition, program supervisors could sit in on a lesson periodically,
using the same checklists, and provide substantial and timely feedback to help program leaders
be more effective.
Reflect and adjust. An exercise that may prove beneficial in the reflection process is to
have program leaders think about what they could have done differently by brainstorming
multiple strategies (Ambrose et al., 2010). For example, a lesson might have required students to
form groups of four in order to create buildings out of scrap materials that could withstand the air
from a hair dryer to simulate hurricanes. Program leaders can reflect on various elements that
could have been changed, such as the number of students per group, the use of a hair dryer, the
use of different materials, the time given to complete the task, etc. While this may not help the
lesson that has already passed, it helps program leaders practice evaluating their strategies, and
moreover, these notes can be passed on to another program leader who may have to replicate this
lesson at a later time.
STEM
NEEDS
ASSESSMENT

85

Motivation. Because program leaders’ personal interest and value seem to be important
factors in determining motivation, there should be a concerted effort in increasing these factors
for the program leaders such that it similarly impacts their motivation. We suggest the following
strategies to increase interest and value for STEM instruction.
Increase situational interest. Hidi and Renninger’s (2006) research demonstrates that
individual interest can be triggered by situational interest, especially in a context like RNH
where there is little or no personal interest in STEM. Program supervisors at each site could
develop situational interest for the program leaders by offering freedom with instruction and
allowing them to make meaningful choices. Program leaders are already given free reign with
the material and are encouraged to improvise as needed, but the scope of this freedom is so vast
that it can be overwhelming. Providing meaningful choices will help program leaders develop
interest and engagement with the task at hand (Schraw, Flowerday & Lehman, 2001). Program
supervisors can enact this by infusing decision-making into the culture of RNH where program
leaders are routinely encouraged to make meaningful choices.
One way to promote this kind of organizational culture is for program leaders to choose
which subject they would like to teach. The current format obligates program leaders to rotate
through all of the subjects with the same group of students, but it might be more beneficial for
program leaders to select the subject they would like to teach and have the students rotate
instead. This would allow for each program leader to become somewhat of an expert in the
chosen subject which, in turn, may spur interest.
Add intrinsic value. Increasing intrinsic value for the task of STEM instruction is the
next critical action step, which is closely tied to interest. One way to add intrinsic value is by
catering to the program leaders’ interests, and allowing them to develop lessons with such vested
STEM
NEEDS
ASSESSMENT

86

interests in mind. For example, if a program leader is very skillful in music, that program leader
should be encouraged to draw from experiences and anecdotes related to music when teaching
the lesson. Program supervisors could scaffold this process by creating a simple worksheet for
each lesson plan that asks, “What are your areas of expertise or interest” and “What are some
ways you could relate this week’s lesson to those areas.”
Add utility value. Program leaders may also need increased utility value for their work
with RNH. Perhaps they feel they do not need higher order (i.e., metacognitive) knowledge for
the purposes of their jobs, but guiding them to see how instrumental their current experiences can
be for the future could have positive effects on their motivation. We suggest that RNH
administrators facilitate seminars with former program leaders that have now gone on to
different career paths. A panel consisting of such individuals can share with current program
leaders about the kinds of skills and knowledge that they have gleaned from RNH that have been
influential in their subsequent careers. While attendance cannot be compulsory, program
supervisors can offer incentives (e.g., entering attendees into a raffle for a prize, certificates, etc.)
to increase participation.
Other considerations. For future employment, it may be prudent to hire program
leaders with majors or career aspirations in the STEM field to spearhead STEM teaching. This
does not mean that RNH should only hire program leaders who have STEM interests or
affiliations, but to perhaps delegate STEM instruction to such a person. While the current STEM
curriculum does not require one to be an expert in any of the STEM subjects, having an
instructor who has an inherent interest and relation to the field may generate similar attitudes
among the students. While this reasoning may not be applicable in traditional classrooms
settings, in an OST setting such as RNH, the program leaders serve as positive adult role models
STEM
NEEDS
ASSESSMENT

87

with whom the students can build mentoring relationships that they may otherwise not have
opportunities for at school or home (Lauer et al., 2006; Mahoney et al., 2007; Gottfredson et al.,
2010). One way to gauge a program leader’s interest in the STEM field would be to utilize tools
such as the STEM Semantics Survey that measures interest in STEM and in STEM careers,
which is available at http://stelar.edc.org/ instruments/stem-semantics-survey.
Professional Development
38

Professional development training is supported by the RNH program and training session
are designated throughout the school year. The program leaders believe they were proffered
opportunities to receive professional development, but not overwhelmingly so. But even if they
had been exposed to enough training events, the challenge does not lie simply in
attendance. Effective professional development should allow for ample and ongoing practice, in
order for transfer of any skill learned in a workshop or other training event (Darling-Hammond,
Wei, Andree, Richardsons, & Orphanos, 2009). In coalescing literature such as these with our
findings, we propose the following solutions to enhance professional development at RNH.
Gather input about perceived training needs. In order to ensure the effectiveness of
professional development training, program supervisors should request input from program
leaders on specific areas of training which they would like to receive, especially regarding
STEM. Some of these topics might be: what STEM is, the core principles of STEM learning,
practical ways to implement STEM, skills learned in STEM, integrating STEM concepts into
non-STEM contexts, additional resources for STEM instruction. Based on feedback from the
program leaders, RNH administrators (i.e., program director and supervisors) should construct
training tailored to the more urgent and prevalent needs. Since the program leaders are
requesting specific trainings they would have a vested interest in them and would be all the more

38

Nsoah Abu-Rasool wrote this section. Linda Moon made contributions.

STEM
NEEDS
ASSESSMENT

88

likely to attend. It would also empower program leaders if they knew that their suggestions
were being considered and necessary changes were being made as a result of their feedback and
requests. Following this training event, program leaders should be given an assessment form that
clearly identifies that this training was offered in response to their requests. Essentially, this
form is asking if this particular training event satisfied their needs. An exemplar of an
assessment form can be found at http://www.usi.edu/science.
Increase frequency of trainings. When professional development training is given, it is
often met with failure because it is usually offered without a chance to continue practicing the
newly learned skill. According to Darling-Hammond et al. (2009), without support during this
phase, it is highly unlikely that teachers will persevere with the newly learned
strategy. Professional development should be ongoing, and scheduling at designated sites with
organized and structured training should be the objective. Remedial training with feedback
should also be included to insure that the participants are engaged and learning.
The frequency of professional development will ultimately be balanced according to the
managerial needs of the RNH program leadership, however, retaining the knowledge gained
during professional development must be continually practiced. We suggest that RNH
professional development sessions occur once every two months at minimum, in the months that
the inter-district LLC workshops are not offered. And instead of targeting a slew of various
topics, each professional development session should hone in on one or two topics, allowing
ample time for practice of an introduced skill or concept. Weekly training would be ideal but
staffing needs may result in scheduling conflicts. However, program supervisors should
continually coach program leaders at each of their sites by scheduling one-on-one meetings with
a different program leader each day, lasting as short as 15-20 minutes, where program leaders
STEM
NEEDS
ASSESSMENT

89

can voice areas in which they need additional support. This would also be a time for program
supervisors to ask whether or not information from training sessions has been helpful. Bridging
the information disseminated in trainings and the actual workplace will help engender an
environment of learning and practice (Thoonen et al., 2011).
Restructure STEM training workshops. Knapp (2003) reports that real teacher
learning occurs in professional development training when it: 1) focuses on instruction with a
strong emphasis on high learning standards, 2) develops teachers' content and pedagogical
knowledge, and 3) models effective instructional practices and activities in the training itself and
for future classroom use. We propose that every STEM workshop include both a pedagogical
and a preparation component. A pedagogical focus in professional development will prepare the
program leaders to be more efficient in the classroom with class management skills that will
enhance student learning. Examples of this would be program leaders learning how to develop a
lesson plan or learning how to facilitate a learning activity more effectively. Preparation
strategies would entail activities such as doing a trial run of the lesson prior to class in order to
get an idea of the amount of time it will take to teach or perform the activity and troubleshooting
the lesson to help better manage things that could go wrong.
A tiered training model, which is already in use, would maximize effectiveness by
targeting the appropriate audience for each training session by identifying beginning,
intermediate, advanced, and applying program leaders. This will help mitigate redundancy for
veteran program leaders, and ensure novice program leaders are not overwhelmed with
unfamiliar concepts.
Involve outside experts. A few of the program leaders and supervisors expressed that
they would like to see more outside experts, particularly in STEM. This may be a costly venture
STEM
NEEDS
ASSESSMENT

90

depending on how much such experts need to be compensated for their time and travel. RNH is
unique in their employment of full-time afterschool staff (i.e., program supervisors), and
therefore should maximize the potentials of their existing staff. Moreover, relegating training to
outside experts could be disadvantageous especially because said expert is most likely going to
be unfamiliar with all the academic, social, and political dynamics at RNH. Training conducted
by internal staff would most likely generate more specific and relevant information, rather than
generic best practices.
However, in order to achieve superior training by program supervisors and other related
RNH staff, they in turn need to be well-trained, which can more easily be executed with the
assistance of outside experts. One way this could be achieved is to invite masters or doctoral
students to host workshops during their district-wide training events, and offer a small
honorarium to that effect. Otherwise, perhaps Meadows could partner with another institution,
such as USC, where students working on a teaching credential or even a doctoral degree could
obtain credits by participating in these events.
Curriculum Adequacy
39

While our data did not provide a clear understanding of the MindWorks curriculum’s
capacity for promoting STEM learning, we found that there are some issues regarding
curriculum that do need to be addressed. We found that RNH stakeholders should be concerned
with the following areas: 1) having sufficient supplies, 2) reinforcing project-based learning
model, and 3) compliance with school instruction and state standards (i.e., Common Core State
Standards and Next Generation Science Standards). Accordingly, we provide the following
recommendations for RNH’s consideration.

39

Linda Moon wrote this section. Nsoah Abu-Rasool made contributions.

STEM
NEEDS
ASSESSMENT

91

Prevent supply shortages. Program leaders and program supervisors alike stated that
supplies in the MindWorks kits tended to run out because of the box being rotated from class to
class. To prevent supply shortages, program supervisor could have the disposable supplies for
each class removed and placed in separate bags or containers, such that whenever a class
receives their box, their supplies will remain intact. Some of the curriculum supplies are
reusable and do not need to be removed from the kits, but any material that can only be used
once, should be evenly divided among the program leaders for future use.
Realign project-based learning with STEM instruction. A deeper understanding of
project-based learning is an identified need of the program leaders. One way to ensure that all
involved stakeholders are aware of the principles of PBL, is to implement a systemic change
with respect to the protocols filled out by program leaders also called PBL. Even though the
principles of PBL are embedded in this form, it seems to get overlooked and is just seen as
obligatory sections of a lesson plan worksheet. The first step would be to change the name of the
worksheet/protocol and work to remove using “PBL” as the shorthand name for the worksheet
out of the organizational culture. The second would be to add a question that would have the
program leader indicate that all the components of a project-based learning model is being
adhered to: elements such as having the problem as a starting point, small group collaboration,
minimal direct instruction, and time for self-study (Schmidt et al., 2007; Bybee,
2010). Reframing and ingraining PBL in this way can standardize PBL instruction, which is a
core component of STEM, consistently throughout the RNH program.
Familiarize RNH staff with standards. While the MindWorks curriculum claims to be
in compliance with Common Core State Standards (CCSS) and Next Generation Science
(NGSS) standards, program staff do not seem to be all too familiar with what they actually are
STEM
NEEDS
ASSESSMENT

92

and how they impact instruction. First of all, both program supervisors and program leaders
need to be aware of what they are by utilizing online resources (see http://www.commoncore.
pearsoned.com for CCSS and http://www.weareteachers.com for NGSS). To account for there
being insufficient time for preparation and training, these modules can be broken down into
shorter segments as needed.
Moreover, the RNH administrators should designate a time when they can reassess the
organizational goals and objectives in reference to CCSS and NGSS. According to their website,
their program goals are the following: 1) learning that is active, 2) learning that is collaborative,
3) learning that is meaningful, 4) learning that supports mastery, and 4) learning that expands
horizons. In order to enrich the STEM components, program staff should consider creating
actionable objectives under each of these umbrella goals that are geared towards STEM learning
and in compliance with CCSS/NGSS. Additionally, each of these goals should be C
3
goals—
concrete, challenging, and current (Clark & Estes, 2008). For example, one of the Common
Core mathematics competencies is, “Construct viable arguments and critique the reasoning of
others” (http://www.cde.ca.gov/re/cc). This competency could easily translate into a C
3
goal
with a STEM focus under the larger goal of “learning that supports mastery.” By reworking
state standards into the program goals and enforcing these goals, RNH can not only be informed
about the standards being implemented currently, but also the program itself will already be in
alignment with these competencies.
Further Considerations
While the current project was a needs assessment of instructor capacity, professional
development, and curriculum adequacy in promoting STEM learning among RNH students, a lot
of program leaders and program supervisors conveyed that they are most excited when they see
STEM
NEEDS
ASSESSMENT

93

that they are impacting the kids directly. One possible study that can be conducted in the future
would be the efficacy of the STEM segment of RNH in predicting student involvement with
STEM in the future. This would most likely be in the form of a longitudinal study that would
follow the same group of students participating in the STEM program for a number of years,
possibly until senior year of high school. At that point, they might ask these students about their
career plans and their views on STEM-related careers. A simplified version of this study could
be for the RNH program to track down past students who had once been a part of the STEM
programming at RNH and ask them similar questions about how they felt the RNH program,
with regards to STEM instruction, had or had not shaped their views on STEM.
Another possible study could be to gather information regarding students from both the
daytime faculty (i.e., credentialed teachers) at each school site and the RNH after school
staff. While previous evaluations of RNH attempted to catalogue this relationship, more
descriptive and complete data could be compiled to determine a causal relationship, if any,
between student involvement in RNH and their academic performance. The current project
relied mostly on reflection and feedback given by the RNH program staff, but further research
could show, for example, if students’ grades have improved since enrolling in RNH or if students
in RNH are achieving higher than those who are not.
Another point of departure would be to assess how STEM is being integrated across
disciplines, or other clubs at RNH. While this needs assessment was focused on the MindWorks
clubs, true STEM learning would be for scientific or engineering thought processes to be applied
in all activities. This would require assessing how various aspects of a STEM-centered mindset,
such as critical thinking, problem solving, and inquiry skills, are being employed in a non-STEM
discipline, such as dance or journalism.
STEM
NEEDS
ASSESSMENT

94

Furthermore, given that the RNH staff have built and maintained relationships with the
students enrolled in RNH, serving as mentors and positive adult role models, they are in a unique
position to influence students out of the scope of teachers and parents. As nonformal educators,
students enter into voluntary relationships with the program staff so there is a lot of inherent
trust. Learning how to effectively utilize these relationships in promoting STEM learning
outcomes could be another source for further research. This kind of study would involve
interviewing both program leaders and students, and even parents, in order to identify attributes
of trustworthy and relational program leaders, along with the extent to which students adhere to
advice and instruction set for by such persons.
A final aspect for consideration is that there was no way of knowing how much of the
issues are related to budgeting. A more comprehensive assessment may require reevaluating
how funds are allocated and may be redistributed to bring about necessary changes that may on
the whole benefit students’ learning of STEM.
Limitations of Project
40

As comprehensive as we strived to be, this needs assessment had some limitations,
mainly in the areas of data collection. The first major aspect in which we faced limitations was
in that of survey design. For one, the survey was designed prior to meeting with any of the
program leaders or supervisors, so the survey questions were much more vague than
necessary. The instructor capacity questions could have separated knowledge into content and
pedagogical content knowledge so it could be more clear in which area of knowledge the
program leaders are stronger. For the motivation component of instructor capacity, we could
have added more items under interest and value for a more complete picture of how those two
components actually are relevant. Moreover, there could have been more questions that asked

40

Nsoah Abu-Rasool wrote this section. Linda Moon made contributions.

STEM
NEEDS
ASSESSMENT

95

about the excitement and energy, which is a factor of motivation that is most visible to the
supervisors, regardless of what their internal beliefs about motivation might be. The professional
development questions could have been separated to ask about the different training events
offered by RNH and by LLC. And as mentioned above, the curriculum questions could have
been more explicit in naming MindWorks as the STEM curriculum for the purposes of the
project. Additionally, we did not collect demographic data regarding race and ethnicity, which
we did not need for the purposes of this study but may have been instrumental for other
considerations.
Similarly, focus groups and interviews could have been more streamlined, with concise
and unambiguous questions. According to Merriam (2009), the way in which questions are
worded is a crucial consideration in extracting the type of information desired. An obvious place
to begin is by making certain that what is being asked is clear to the person being
interviewed. For instance, during focus group sessions, when participants were asked questions
about project based learning (PBL), they responded by giving descriptions of how they set up
their lesson plans, which coincidentally is also called PBL. While their lesson plan forms were
probably modeled after the broader construct of PBL, this underlying theory was not mentioned,
which was, in fact, the root of the question being asked. Sometimes, these kinds of unexpected
responses would lead to straying off topic and would consume a larger portion of the interview
sessions than we anticipated. This became a perceptible challenge, as we already had a very
limited amount of time allotted for these interviews, based on the schedules of the RNH staff.
For observations, it might have been beneficial to observe more classrooms and more
frequently, especially on different days of the week. Due to difficulties in scheduling, we were
only able to meet with the program leaders on one Thursday for each site. Perhaps having the
STEM
NEEDS
ASSESSMENT

96

ability to conduct more observations throughout the week at each site, would have provided a
more holistic picture of what takes place during the course of a school week.
In terms of data analysis, we only focused on descriptive data, but we could have also ran
some correlational analyses. For instance, we could have ran a correlation to see if there was a
relationship between program leader knowledge and motivation. We could have also compared
the results by gender, age, or years of experience to see if there were any significant findings
between the groups.
Another limitation is in sampling. We only had 45 survey respondents, which was a
significant number with respect to the 50 program leaders total, but still a relatively small
sampling size in terms of gleaning reliable data. Additionally, more descriptive data could have
been collected if we had also included students and parents in this project. Program leaders and
program supervisors are both on the administrative side of STEM teaching, but students and
parents are on the receiving end, thus their input would have provided an even more
comprehensive analysis, especially with regard to the extent to which STEM learning is actually
taking place in RNH.
Finally, there is the issue of bias. Most of the information was collected through self-
reporting, which yields inherent bias and sometimes erroneous responses. The Qualtrics survey
contained 58 questions, including demographic questions and all the embedded questions (e.g.,
4-a, 4-b, 4-c, 4d). Because of the lengthiness of the survey and no incentive to finish it,
respondents possibly did not think through each question thoroughly, or just picked a number all
the way down. In addition, a lot of the respondents skipped the open-ended questions. In the
focus group and interviews, there is the possibility that the respondents were not entirely
STEM
NEEDS
ASSESSMENT

97

forthcoming about their opinions due to the sensitivity of the topics and wariness of being
identified, even with assurances of anonymity.

STEM
NEEDS
ASSESSMENT

98

References
Altschuld, J. W., & Kumar, D. D. (2010). Needs assessment: An overview: SAGE Publications,
Inc.
Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M.C., & Norman, M. K. (2010). How
learning works: 7 research-based principles for smart teaching. San Francisco: Jossey-
Bass.
Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing:
A revision of Bloom’s taxonomy of educational objectives. New York: Addison Wesley
Longman, Inc.
Axtell, P. K., McCallum, R. S., Mee Bell, S., & Poncy, B. (2009). Developing math automaticity
using a classwide fluency building procedure for middle school students: A preliminary
study. Psychology in the Schools, 46(6), 526-538.
Bandura, A. (1993). Perceived self-efficacy in cognitive development and functioning.
Educational Psychologist, 28, 117-148.
Beers, S. Z. (2012). 21st century skills: Preparing students for their future.
Bevan, B., Michalchik, V., Bhanot, R., Rauch, N., Remold, J., Semper, R., & Shields, P. (2010).
Out-of-school time STEM: Building experience, building bridges.
Bowie, L., & Bronte-Tinkew, J. (2006). The importance of professional development for youth
workers. Child Trends: Research-to-Results(17).
Breiner, J. M., Harkness, S. S., Johnson, C. C., & Koehler, C. M. (2012). What is STEM? A
discussion about conceptions of STEM in education and partnerships. School Science and
Mathematics, 112(1), 3-11.
STEM
NEEDS
ASSESSMENT

99

Bybee, R. W. (2010). Advancing STEM education: A2020vision. Technology and Engineering
Teacher, 30-35.
Chow, A., Eccles, J. S. &, Samela-Aro, K. (2012). Task value profiles across subjects and
aspirations to physical and it-related sciences in the United States and Finland.
Developmental Psychology, 48(6), 1612–1628. doi: 10.1037/a0030194
Clark, R. E. & Estes, F. (2008). Turning research into results: A guide to selecting the right
performance solutions. Atlanta GA: CEP Press.
Darling-Hammond, L., Wei, R. C., Andree, A., Richardson, N., & Orphanos, S. (2009).
Professional learning in the learning profession: A status report on teacher development
in the United States and abroad. Washington, DC: National Staff Development Council.
Dierking, L. D. (2007). Linking after-school programs and STEM learning: A view from another
window: Oregon Sea Grant.
Fletcher, A. J., & Cantrell, K. (2014). [Year-end evaluation of Meadows Elemetntary School
District Reaching New Heights (RNH) Program]. Unpublished raw data. (pseudonyms)
Freeman, J., Dorph, R., & Chi, B. (2009). Strengthening after school STEM staff development.
Lawrence Hall of Science, University of California Berkeley.
Fulton, K., & Britton, T. (2011). STEM teachers in professional learning communities: From
good teachers to great teaching. National Commission on Teaching and America's
Future.
Gottfredson, D., Cross, A. B., Wilson, D., Rorie, M., & Connell, N. (2010). Effects of
participation in after-school programs for middle school students: A randomized trial.
Journal of Research on Educational Effectiveness, 3(3), 282-313. doi: 10.1080/
19345741003686659
STEM
NEEDS
ASSESSMENT

100

Havice, W. (2009). The power and promise of a STEM education: Thriving in a complex
technological world ITEA (Ed.) The Overlooked STEM Imperatives: Technology and
Engineering (pp. 10-17).
Hidi, S., & Renninger, K. A. (2006). The four-phase model of interest development. Educational
Psychologist, 41(2), 111-127.
Honey, M., Pearson, G., & Schweingruber, H. (2014). STEM integration in K-12 education::
Status, prospects, and an agenda for research: National Academies Press.
How does the U.S. compare to other countries in STEM education? (2014). Retrieved July 24,
2014, from http://www.lpfi.org/how-does-us-compare-other-countries-stem-education
Huang, D., & Cho, J. (2009). Academic enrichment in high-functioning homework afterschool
programs. Journal of Research in Childhood Education, 23(3), 382-392.
Knapp, M. S. (2003). Professional development as a policy pathway. Review of research in
education, 27, 109-157.
Kuenzi, J. J. (2008). Science, technology, engineering, and mathematics (STEM) education:
Background, federal policy, and legislative action.
Lauer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. S., Snow, D., & Martin-Glenn, M. L.
(2006). Out-of-school-time programs: A meta-analysis of effects for at-risk students.
Review of educational research, 76(2), 275-313.
Lin, M. H., Chuang, T. F., & Hsu, H. P. (2014). The relationship among teaching beliefs,
student-centered teaching concept and the instructional innovation. Journal of Service
Science and Management, 7, 201-210.
STEM
NEEDS
ASSESSMENT

101

Little, P., Wimer, C., & Weiss, H. B. (2008). After school programs in the 21st century: Their
potential and what it takes to achieve it. Issues and opportunities in out-of-school time
evaluation, 10, 1-12.
Mahoney, J. L., Parente, M. E., & Lord, H. (2007). After ‐school program engagement: Links to
child competence and program quality and content. The Elementary School Journal,
107(4), 385-404.
Martone, A., & Sireci, S. G. (2009). Evaluating alignment between curriculum, assessment, and
instruction. Review of educational research, 79(4), 1332-1361.
Merriam, S.B. (2009). Qualitative Research: A guide to design and implementation. San
Francisco, CA: Jossey-Bass.
Miller, P. M. (2010). Community-based education and social capital in an urban after-school
program. Education and Urban Society, 44(1), 35-60.
Mills, J. E., & Treagust, D. F. (2003). Engineering education—is problem-based or project-based
learning the answer? Australasian Journal of Engineering Education, 3, 2-16.
NRC. (2010). Exploring the intersection of science education and 21st century skills: A
workshop summary. Washington, D.C.: National Academies Press.
Roberts, A. (2012). A justification for STEM education. Technology and Engineering
Teacher(May/June).
Salinger, G., & Zuga, K. (2009). Background and history of the STEM movement. In ITEA
(Ed.), The overlooked STEM imperatives (pp. 5-9).
Sanders, M. (2009). STEM, STEM education, stemmania. The Technology Teacher, 68(4), 20-
26.

STEM
NEEDS
ASSESSMENT

102

Schiefele, U., Streblow, L., & Retelsdorf, J. (2013). Dimensions of teacher interest and their
relations to occupational well-being and instructional practices. Journal for Educational
Research Online/Journal für Bildungsforschung Online, 5(1), 7-37.
Schmidt, H. G., Loyens, S. M., Van Gog, T., & Paas, F. (2007). Problem-based learning is
compatible with human cognitive architecture: Commentary on kirschner, sweller, and
clark (2006). Educational Psychologist, 42(2), 91-97.
Schraw, G., Flowerday, T., & Lehman, S. (2001). Promoting situational interest in the
classroom. Educational Psychology Review, 13, 211–224.
Schweingruber, H., Keller, T., & Quinn, H. (2012). A framework for k-12 science education::
Practices, crosscutting concepts, and core ideas: National Academies Press.
Science, technology, engineering, and math: Education for global leadership. (2014). Retrieved
July 24, 2014, 2014, from http://www.ed.gov/stem
Sheldon, J., Arbreton, A., Hopkins, L., & Grossman, J. B. (2010). Investing in success: Key
strategies for building quality in after-school programs. American journal of community
psychology, 45(3-4), 394-404.
Simpkins, S. D., Davis-Kean, P. E., & Eccles, J. S. (2006). Math and science motivation: A
longitudinal examination of the links between choice and beliefs. Developmental
Psychology, 42(1), 70.
Springer, K., & Diffily, D. (2012). The relationship between intensity and breadth of after ‐
school program participation and academic achievement: Evidence from a short ‐term
longitudinal study. Journal of Community Psychology, 40(7), 785-798.
STEM
NEEDS
ASSESSMENT

103

Thoonen, E. E., Sleegers, P. J., Oort, F. J., Peetsma, T. T., & Geijsel, F. P. (2011). How to
improve teaching practices the role of teacher motivation, organizational factors, and
leadership practices. Educational Administration Quarterly, 47(3), 496-536.
Vandell, D. L., Reisner, E. R., & Pierce, K. M. (2007). Outcomes linked to high-quality
afterschool programs: Longitudinal findings from the study of promising afterschool
programs. Policy Studies Associates, Inc.
Wang, H., Moore, T. J., Roehrig, G. H., & Park, M. S. (2011). STEM integration: Teacher
perceptions and practice. Journal of Pre-College Engineering Education Research (J-
PEER), 1(2), 2.

STEM
NEEDS
ASSESSMENT

104

Appendix A
Survey Questionnaire for RNH Program Leaders and Program Supervisors

Each
of
the
following
questions
will
be
answered
on
a
6-‐point
Likert
scale
(1-‐strongly
disagree,
2-‐
disagree,
3-‐somewhat
disagree,
4-‐somewhat
agree,
5-‐agree,
6-‐strongly
agree),
unless
otherwise

specified.
[45
questions]

Construct
measured
Question
Source

Instructor

Capacity

Knowledge

My
(Program
leaders’)
academic
background
and/or
work

experience
has
given
me
(them)
the
necessary
skills
to
be
an

effective
instructor.

Knowledge
I
(Program
leaders)
are
aware
of
the
basic
principles
of
STEM

teaching
and
learning.

Knowledge
I
(Program
leaders)
am/are
aware
of
what
inquiry-‐based

learning
is.

Knowledge

I
(Program
leaders)
have
sufficient
knowledge
of
the
science

curriculum
used
in
the
program.

Adapted
from

the
Baseline

IFSS
Coach

Survey

Knowledge

I
(Program
leaders)
have
sufficient
knowledge
of
the
different

ways
technology
is

content
used
in
the
program.

Knowledge

I
(Program
leaders)
have
sufficient
knowledge
of
the
how

engineering
is
content
used
in
the
program.

Knowledge

I
(Program
leaders)
have
sufficient
knowledge
of
the
how
math

content
is
used
in
the
program.

Knowledge

I
(Program
leaders)
know
how
to
use
effective
teaching

strategies
to
guide
student
thinking
and
learning
in
science.

Knowledge

I
(Program
leaders)
know
how
to
use
effective
teaching

strategies
to
guide
student
thinking
and
learning
in
technology.

Knowledge

I
(Program
leaders)
know
how
to
use
effective
teaching

strategies
to
guide
student
thinking
and
learning
in
engineering.

Knowledge

I
(Program
leaders)
know
how
to
use
effective
teaching

strategies
to
guide
student
thinking
and
learning
in
math.

Motivation

(self-‐efficacy)

I
(program
leaders)
am/are
very
motivated
to
get
new
ideas
and

strategies
to
better
teach
STEM.

Motivation

(self-‐efficacy)

I
(Program
leaders)
have
the
confidence
to
teach
STEM
related

curriculum
and
activities
well.

Motivation

(self-‐efficacy)

I
(Program
leaders)
have
the
skills
to
teach
STEM
related

curriculum
and
activities
well.

Motivation

(teaching

beliefs;

outcome

expectancy)

Program
leaders
are
a
powerful
influence
on
student

achievement
in
STEM.

Adapted
from

Teacher

Efficacy
Scale

Motivation

(teaching

beliefs;

outcome

expectancy)

If
a
student
is
able
to
master
a
STEM
concept,
it
is
because
I

knew
implemented
the
necessary
steps
in
teaching
the

curriculum.

Adapted
from

Teacher

Efficacy
Scale

Motivation

(value)

I/Program
leaders
believe
that
it
is
important
for
students
excel

in
STEM.

Motivation

(value)

I
think
it
is
very
important
to
incorporate
STEM
into
all
program

activities.

Motivation

(interest)

My/Program
leaders’
personal
interest
in
STEM
directly
affects

how
well
I/they
teach
STEM
related
curriculum.

STEM
NEEDS
ASSESSMENT

105

Motivation

(teaching

beliefs;

outcome

expectancy)

I
believe
that
facilitating
related
activities
at
my
site
has
a
direct

impact
on...

-‐

my/program
leaders’
own
knowledge
of

STEM
related

content.

-‐

my/program
leaders’
own
interest
in
STEM
related
content

-‐
students’
knowledge
of
STEM
related
content

-‐
students’
interest
in
STEM
related
content

-‐
students’
awareness
of
an
interest
in
pursuing
STEM
related

careers.

Adapted
from

the
STEM

Survey
for
Site

Coordinators

Professional

Development

I
(Program
leaders)
have
the
resources
and
supplies
to
do
my
(their)
job
well.
Adapted
from

the
Baseline

IFSS
Coach

Survey

and
the
STEM

Survey
for
Site

Coordinators

I
(Program
leaders)
receive
the
training
I
(they)
need
to
be
successful.

I
(Program
leaders)
receive
the
coaching
that
I
(they)
need
to
be
successful.

I
(Program
leaders)
get
the
support
that
I
(they)
need
from
the
program

supervisor.

In
the
past
year,
how
many
of
the
following
STEM
related
support
services
have

you
received?
__________
Training
on
how
to
integrate
elements
of
STEM
into
program
activities
__________
Coaching
on
how
to
implement
STEM
training
content
into
program

activities

I
think
that
the
following
factors
have
greatly
helped
the
implementation
of
STEM

related
activities
within
my
program:
• adequacy
of
space/facilities
• availability
of
trained
staff
• access
to
resources,
curricula,
and/or
materials
access
to
internet-‐
connected
computers

Please
rank
the
following
according
to
what
you
think
are
the
most
pressing

needs
related
to
implementing
STEM
activities
at
your
site
(1=MOST
pressing,

5=LEAST
pressing).

• training
(e.g.,
integrating
STEM
into
program
activities,
evaluating
STEM

activity
outcomes)
• information
and
resources
(e.g.,
existing
STEM
curricula,
developing

partnerships
with
other
agencies,
funding
opportunities,
evaluation

tools)
that
I
might
be
able
to
use
in
my
program
• coaching
(e.g.,
how
to
integrate
and/or
improve
STEM
components
in

program
activities)

• consultation
outside
experts
(e.g.,
how
to
integrate
and/or
improve

STEM
components
in
program
activities)
site
visits
to
other
sites
implementing
effective
STEM
activities

Professional
development
workshops
are
vital
to
helping
program
leaders

develop
new
teaching
techniques.

Adapted
from

Teacher

Opinions

About

Professional

Development

Questionnaire

I
(Program
leaders)
have
been
enriched
by
the
training
events
I
(they)
have

attended.
Without
the
training
I
(they)receive
in
professional
development,
I(they)
would

not
be
able
to
teach
STEM
effectively.

Curriculum

Our
program
provides
exciting
and
engaging
enrichment
opportunities
for

students.

Adapted
from

the
Baseline

IFSS
Coach

Survey
and

STEM
Survey

for
Site

Coordinators

Our
program
supports
improvement
in
student
academic
performance.

The
curriculum
includes
activities
and
approaches
aimed
at
improving
the

leadership
and
character
development
of
students.

Please
indicate
how
often
you
facilitate
each
of
the
activities
occur
at
your
site
(1-‐
2
times
a
month;
1-‐2
times
a
week;
3-‐4
times
a
week;
every
day;
not
at
all)

• technology-‐based
activities
(e.g.,
using
iPads,
multimedia
projects)

• science-‐based
activities
(e.g.,
robotics,
environmental
science
projects)

STEM
NEEDS
ASSESSMENT

106

• math-‐related
activities
(e.g.,
creating
bar
graphs
to
display
data)

engineering-‐related
activities
(e.g.,
robotics,
building
devices)

Using
the
current
STEM
curriculum…Using
the
current
curricula…
• students
are
able
to
use
scientific
vocabulary
and
principles
principles
in

everyday
discussions.
• students
are
able
to
use
and
recognize
that
science
and
technology
are

developed
to
meet
societal
needs
and
expand
human
capability
• students
are
able
to
voice
their
own
opinions
about
STEM
• students
are
able
to
show
interest
in
STEM-‐related
questions
and
issues

• students
are
able
to

and
confidently
pursue
personal
interests
and

natural
curiosities
about
STEM

students
are
able
to
consider
and
career
possibilities
within
STEM-‐related
fields.

Adapted
from

Curriculum

Emphasis

Survey

The
following
skills
are
almost
always
built
into
our
program
activities:

• inquiry
skills
-‐based
learning
(posing
a
question
and
investigating
an

answer)

• problem
solving
(identifying
and
employing
strategies
to
resolve

problems)

• critical
thinking
(identifying
and
considering

multiple
aspects
of
an

issue/situation
to
make
an
informed
conclusion)

Adapted
from

STEM
Survey

for
Program

Directors/

Grant

Managers

I
think
our
current
STEM
curriculum
is
headed
in
the
right
direction
to
help

students
to
excel
in
STEM.

I
believe
the
students
enjoy
the
current
STEM
curriculum.

Demographic

information

Please
indicate
your
position
in
RNH:

[]
program
supervisor

[]
program
leader

How
many
years
have
you
been
in
your
current
position?

Please
indicate
your
gender.

[]
Female

[]
Male

What
is
your
age?

What
is
the
last
degree
you
have
obtained
or
are
currently
working
on?

[]
High
school
graduate,
diploma
or
the
equivalent
(for
example:
GED)

[]
Some
college
credit,
no
degree

[]
Trade/technical/vocational
training

[]
Associate
degree

[]
Bachelor’s
degree

[]
Master’s
degree

[]
Professional
degree

[]
Doctorate
degree

What
is
your
intended
or
completed
field
of
study
or
major
of
expertise,
if

applicable?

How
many
years
of
teacher
training
outside
of
RNH,
if
any,
do
you
have?

(PL
only)
Please
list
each
club
you
facilitate.

Indicate
your
site.

[]
Site
A

[]
Site
F

[]
Site
G

[]
Site
M

[]
Site
S

[]
Site
T

STEM
NEEDS
ASSESSMENT

107

Appendix B
Focus Group Protocol for RNH Program Leaders

Interviewer:
Date:

Interviewees:
Titles:

Contact
info:

Start
time:
End
time:

Introduction:

Hello,
my
name
is
XX,
and
I
am
a
doctoral

student
at
the
University
of
Southern
California’s

Rossier
School
of
Education
and
I
am
conducting
a

needs
assessment
for
the
RNH
after
school
STEM

education
program.

During
this
conversation,
we

are
hoping
to
learn
more
about
your
role
and

experiences
with
RNH
and
the
STEM
program.

I

want
to
assure
you
that
your
comments
will
be

strictly
confidential.

We
will
not
identify
you,
or

your
school
site,
by
name.

I
would
like
to
tape

record
this
interview
in
order
to
have
an
accurate

record
of
our
conversation.

Would
that
be
okay?

The
focus
group
session
should
take

approximately
30
minutes.

Do
you
have
any

questions
before
we
begin?

A.
Background

Before
I
ask
you
specific
questions
about
the
RNH

Program
I
would
like
to
start
by
asking
you
some

background
questions
about:

1) How
long
have
you
been
in
your
position?

2) What
is
your
prior
experience
and
training?

a) What
is
your
STEM
experience
and

training?

b) What
is
your
leadership
experience?

3) Talk
me
through
your
role
as
a
program

supervisor.

B.

Instructor
Capacity

1) What
is
your
understanding
of
STEM

education?

2) How
do
you
typically
prepare
to
teach
STEM

content?

3) What
do
you
think
would
be
the
best
way
for

students
to
learn
STEM
content?

a) Why
do
you
feel
that
this
method
is
the

best
way
to
teach
STEM?

b) How
do
you
know
students
are
learning?

4) What
is
your
understanding
of
21
st

century

learning
skills?

5) What
is
your
understanding
of
project
based

learning?

6) What
motivates
you
to
teach
STEM?

C.

Professional
Development

1) How
often
do
you
attend
professional

development
sessions?

a. How
long
do
they
usually
last?

b. Who
conducts
them?

2) Describe
a
typical
PD
session.

a. What
do
you
like
most
about
the

professional
development
session?

b. What
do
you
least
like?

3) Talk
me
through
how
you
apply
what
you

have
learned
from
a
professional

development
session
in
the
classroom.

a. How
do
you
think
it
is
working
for
the

students?

4) Do
you
have
an
opportunity
to
make

suggestions
from
your
classroom
experiences

during
professional
development
sessions?

a. If
so
how
are
the
received?

b. If
not
why?

5) What
would
you
like
to
see
changed
in
how

PD
is
conducted
at
your
site?

a. Why
would
you
choose
to
do
it

different?

b. What
would
you
keep
the
same?

6) Describe
the
resources
available
to
you.

a. What
additional
resources
would
you

like?

D.

Curriculum
Assessment

1) What
is
latest
STEM
curriculum
being
used
to

help
teach
students?

2) How
does
your
STEM
curriculum
address
21
st

century
learning
skills?

3) How
is
project
based
learning
being
used
in

the
STEM
after
school
program?

4) How
is
the
curriculum
generally
received
by…

a. Students?

b. Instructors?

5)

Why
do
you
think
the
RNH
STEM
curriculum

works
or
doesn’t
work?

Is
there
anything
else
that
I
didn’t
ask
that
you

would
like
to
mention?

Thank
you
for
taking
the
time
to
answer
my

questions.

STEM NEEDS ASSESSMENT

108

Appendix C
Interview Protocol for RNH Program Supervisors

Interviewer:
Date:

Interviewee:
Contact
info:

Start
time:
End
time:

Introduction:

Hello,
my
name
is
XX,
and
I
am
a
doctoral

student
at
the
University
of
Southern
California’s

Rossier
School
of
Education
and
I
am
conducting
a

needs
assessment
for
the
RNH
after
school
STEM

education
program.

During
this
conversation,
we

are
hoping
to
learn
more
about
your
role
and

experiences
with
RNH
and
the
STEM
program.

I

want
to
assure
you
that
your
comments
will
be

strictly
confidential.

We
will
not
identify
you,
or

your
school
site,
by
name.

I
would
like
to
tape

record
this
interview
in
order
to
have
an
accurate

record
of
our
conversation.

Would
that
be
okay?

The
interview
should
take
approximately
30

minutes.

Do
you
have
any
questions
before
we

begin?

A.
Background

1)
How
long
have
you
been
in
your
position?

2)
What
is
your
prior
experience
and
training?

a) What
is
your
STEM
experience
and

training?

b) What
is
your
leadership
experience?

3)
Talk
me
through
your
role
as
a
program

supervisor.

B.

Instructor
Capacity

1) What
do
you
look
for
in
a
program
leader’s

understanding
of
STEM
education?

2) What
expectations
do
you
have
of
program

leaders
in
how
they
prepare
and
teach
STEM?

3) How
do
you
evaluate
program
leaders’

performance?

4) How
do
you
address
program
leaders’

motivation?
(e.g.,
incentives)

C.

Professional
Development

1) How
often
are
professional
development

sessions
offered
at
your
site?

a. How
long
do
they
usually
last?

b. Who
conducts
them?

2) Describe
a
typical
PD
session.

3) Talk
me
through
how
you
prepare
for
a

professional
development
session.

a. How
do
you
think
it
is
working
for
the

program
leaders?

4) Is
there
anything
about
your
PD
sessions
you

would
like
to
change?

a. If
so,
why
would
you
choose
to
do
it

differently?

b. What
would
you
keep
the
same?

5) What
was
your
best
professional
development

experience
and
why?

a. Your
worst
and
why?

6) Where
do
program
leaders
get
resources

needed
to
teach
STEM
classes?

D.

Curriculum
Assessment

1) What
is
latest
STEM
curriculum
being
used
to

help
teach
students?

2) How
does
your
STEM
curriculum
address
21
st

century
learning
skills?

3) How
is
project
based
learning
being
used
in

the
STEM
after
school
program?

4) How
is
the
curriculum
generally
received
by…

a. Students?

b. Instructors?

5)

Why
do
you
think
the
RNH
STEM
curriculum

works
or
doesn’t
work?

Is
there
anything
else
that
I
didn’t
ask
that
you

would
like
to
mention?
Thank
you
for
taking

the
time
to
answer
my
questions.

STEM NEEDS ASSESSMENT

109

Appendix D
Observations Protocol

Name of Observer Date Time

Location Study

Brief Summary of Observation

Physical Space
Define the physical space.
• Geographical
• Temporal
• Physical
• Political

Utility: What is the purpose
of event/setting?

Participant reactions to
physical setting

Other

People/Participants
Who are the participants
taking place in
observation/event?
How many participated?

Demographical information
(e.g., racial, ethnic, gender,
class):

What does learning look like?
• How are activities organized- structure, content etc.?
• What materials are being used and by whom?
• Does the learning context appear to be teacher or student centered?
o When and in what ways?
What are students doing?
• Types of activities
• Ways of engaging in activities
• Ways of interacting with peers and staff
• How do they appear to be making choices about what they are doing- both activity type and tasks within an
activity?
What are the staff members doing?
• How do the staff members interact with kids?
o Do they ask questions? If so, what kind and in what circumstances?
• How do the staff members facilitate the activities?
• How are rules enforced?
• Do staff members demonstrate particular skill sets that appear effective for working with this age group?
Other observations about student behavior:
• Do students seem happy?
• Do students seem engaged in their activities?
• Do students seem to have close relationships with program staff?
• Do students interact with each other respectfully?
STEM NEEDS ASSESSMENT

110

Purpose of Events/Observation
Why is the event taking
place? Are there any
political contexts to be
discussed?

Who was invited to event?
Who was not?

Was there any discussion
of educational policy?
Why? How so?

What are the positions of
the various participants
involved?
• Power dynamics
• Roles

What is being discussed?

Sequence of Events
Beginning
Middle
End

Observer Role
What am I doing? What is
my role throughout the
observation?

Describe some of my
interactions with other
participants throughout the
observation.

How did my
interaction/presence affect
the observation
participants?

Other

Pictures

STEM NEEDS ASSESSMENT

111

Appendix E
IRB Approval Letter

UNIVERSITY OF SOUTHERN CALIFORNIA
UNIVERSITY PARK INSTITUTIONAL REVIEW BOARD
FWA 00007099

Exempt Review

Date: Sep 27, 2014, 08:46am
Principal Investigator: Robert Rueda
ROSSIER SCHOOL OF EDUCATION
Faculty Advisor:
Co-Investigators:
Gale Sinatra
ROSSIER SCHOOL OF EDUCATION
Nsoah Abu-Rasool
ROSSIER SCHOOL OF EDUCATION
Linda Moon
ROSSIER SCHOOL OF EDUCATION

Project Title: A STEM Needs Assessment
USC UPIRB # UP-14-00532

The iStar application and attachments were reviewed by UPIRB staff on 9/27/2014

The project was APPROVED.

Based on the information provided for review, this study meets the requirements outlined in 45 CFR 46.101(b)(1),
(2) & (4) and qualifies for exemption from IRB review. The study is not subject to
further IRB review. IRB exemption of this study was granted on 9/27/2014.

To access IRB-approved documents, click on the “Approved Documents” link in the study workspace. These
are also available under the “Documents” tab.

Researchers are reminded that some schools require permission to conduct research even if the research is exempt
from IRB review.

Sincerely,

RoseAnn Fleming, CIP

Approved Documents: view
Funding Source(s):
No Funding Sources

STEM NEEDS ASSESSMENT

112

Appendix F
Data Collection Plan

DATE
SITE
TIME
DATA
PARTICIPANTS

Thursday,

October
23

Site
F

12:15-‐12:45pm
Focus
group
Program
leaders

1:00-‐3:00pm
Observations

3:00-‐3:30pm
Interview
Program
supervisor

Thursday,

October
30

Site
M
12:15-‐12:45pm
Focus
group
Program
leaders

Thursday,

November
13

Site
S

12:15-‐12:45pm
Focus
group
Program
leaders

1:00-‐3:00pm
Observations

3:00-‐3:30
Interview
Program
supervisor

Site
M

4:00-‐5:30pm
Observations

5:30-‐6:00pm
Interview
Program
supervisor

Thursday,

December
4

Site
A

12:15-‐12:45pm
Focus
group
Program
leaders

1:00-‐3:00pm
Observations

3:00-‐3:30pm
Interview
Program
supervisor

Thursday,

December
11

Site
G

12:15-‐12:45pm
Focus
group
Program
leaders

1:00-‐3:00pm
Observations

3:00-‐3:30pm
Interview
Program
supervisor

Site
T

4:00-‐5:30pm
Observations

5:30-‐6:00pm
Interview
Program
supervisor

Thursday,

December
18

Site
T
12:15-‐12:45pm
Focus
group

Program
leaders

Abstract (if available)

Abstract Science, technology, engineering, and math (STEM) education has been a hot button issue in recent years, especially concerning its implementation in schools and application to the growing global economy in need of STEM professionals. This project was designed for Meadows Elementary School District (MESD)’s Reaching New Heights (RNH) afterschool program (pseudonyms), and was aimed at examining the issues of STEM-related education in out-of-school time programs, especially with respect to instructor capacity (knowledge and motivation), professional development, and curriculum adequacy. Using a needs assessment model, we collected data from 45 program leaders (afterschool instructors) and 6 program supervisors (site administrators) at six different RNH sites, utilizing surveys, focus groups, interviews, observations, and document reviews. The results suggest that program leaders feel they have sufficient knowledge and motivation to teach STEM material, although program supervisors conveyed that there is room for improvement. Both program supervisors and program leaders value professional development and feel that the RNH program requires more frequent training sessions, preferably with the assistance of experts outside of RNH. Finally, the STEM curriculum in use at the RNH sites seems to be adequate for promoting STEM learning, but both program leaders and program supervisors feel that the curriculum alone will not suffice and they find the need to supplement with alternative resources. To address these concerns, the researchers have compiled a list of recommendations for the purposes of enhancing STEM instruction in RNH, as well as suggestions for further study. Throughout this dissertation, the names of the school district and afterschool sites have been changed to preserve their anonymity.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

A needs assessment and evaluation of an elementary afterschool STEM program in a small urban school district

PDF

A needs assessment of Meadows Elementary School District STEM education after school program

PDF

A needs assessment and program evaluation of an urban school district's after-school STEM program

PDF

A needs assessment of an after school science, technology, engineering, and mathematics program

PDF

A needs assessment for Hillview Middle School's STEM education program

PDF

A needs assessment of an after school, science, technology, engineering, and mathematics program

PDF

Improving instructor skills (IIS): a Needs analysis

PDF

Assessing the effectiveness of an inquiry-based science education professional development

PDF

Systemic multilayered assessment of global awareness in undergraduate students: an innovation study

PDF

Examining the role of instructors and instruction in an after-school program

PDF

Service-learning and character development: an analysis of Up with People resulting in a model of global citizens for servant leadership

PDF

The increasing need for global education competencies in K-12 education

PDF

Improving early grade reading instruction in Ghana: a discrepancy gap analysis

PDF

Influence of globalization, school leadership, and students’ participation in science competitions on 21st-century skill development, instructional practices, and female students’ interest in sci...

PDF

Impact of technology on teaching and learning practices at high‐technology use K-12 schools: a case study

PDF

Employee churn in afterschool care: an evaluation study of manager influences on employee retention and turnover

PDF

Creating "excellent" learning experiences: a gap analysis of a university extension program

PDF

The development of 21st century skills among incarcerated youth: a needs assessment of teachers

PDF

Math achievement and self-efficacy of linguistically and ethnically diverse high school students: their relationships with English reading and native language proficiency

PDF

Girls in STEM: the underrepresented trajectory in Tennessee: an innovation study

Asset Metadata

Creator Moon, Linda (author)

Core Title A needs assessment of an urban elementary school district's afterschool program

School Rossier School of Education

Degree Doctor of Education

Degree Program Education

Publication Date 04/17/2015

Defense Date 03/13/2015

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag afterschool,curriculum,Knowledge,Motivation,needs assessment,OAI-PMH Harvest,OST,out-of-school time,professional development,project-based learning,STEM

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Rueda, Robert (committee chair), Seli, Helena (committee member), Sinatra, Gale M. (committee member)

Creator Email linda.moon@usc.edu,lmoon@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c3-552028

Unique identifier UC11297683

Identifier etd-MoonLinda-3327.pdf (filename),usctheses-c3-552028 (legacy record id)

Legacy Identifier etd-MoonLinda-3327.pdf

Dmrecord 552028

Document Type Dissertation

Format application/pdf (imt)

Rights Moon, Linda

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA