Close
About
FAQ
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
University of Southern California Dissertations and Theses
/
Examining the effectiveness of the intervention programs for English learners at MFC intermediate school
(USC Thesis Other)
Examining the effectiveness of the intervention programs for English learners at MFC intermediate school
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
EXAMINING THE EFFECTIVENESS OF THE INTERVENTION PROGRAM FOR
ENGLISH LEARNER STUDENTS AT MFC INTERMEDIATE SCHOOL
by
Michael C. Bachicha
A Dissertation Presented to the
FACULTY OF THE ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF EDUCATION
August 2008
Copyright 2008 Michael C. Bachicha
ii
Table of Contents
List of Tables iv
List of Figures vi
Abstract vii
Chapter 1:
Background 1
Problem Identification 3
Problem Analysis and Interpretation 13
Problem Solution 22
Purpose, Design and Utility 24
Chapter 2:
Literature Review 27
Chapter 3:
Design Summary 49
Participants/Sampling 51
Intervention Description 52
Instrumentation 53
Procedural Timeline 60
Data Analysis 61
Delimitations and Limitations of the Study 62
iii
Table of Contents
Chapter 4:
Results 65
Chapter 5:
Summary 82
Discussion 89
Recommendations 94
References 98
iv
List of Tables
Table 1: 2006 MFC EL/SED Students; % Scoring Proficient or 5
Above on CST
Table 2: 2006 MFC EL/SED Student Performance on CST ELA 6
Table 3: MFC 2006 EL/SED Student Performance on 7
CST Mathematics
Table 4: MFC 2006 CST Performance Band Performance by 8
English Speaking White Students
Table 5: MFC Performance on Title III Annual Measurable 11
Objectives (AMAOs)
Table 6: Academic Performance Index (API) Subgroup Progress 12
Table 7: MFC Demographic Data 15-16
Table 8: Stages of Second Language Acquisition 33
Table 9: 2006/2007 6
th
Grade CST ELA 57
Table 10: 2006/2007 7
th
Grade CST ELA 57
Table 11: 2006/2007 8
th
Grade CST ELA 58
Table 12: Pre-versus Post Intervention CST ELA Scale Score 67
Differences: Statistical Findings
Table 13: Pre-versus Post Intervention CST ELA Performance Band 68
Differences: Statistical Findings
Table 14: Pre-Post Intervention CST ELA Scale Score Differences: 69
Practical Significance
Table 15: Pre-Post Intervention CST ELA Performance Band 70
Differences: Practical Significance
v
List of Tables
Table 16: 2006 API Base Test Weights Grades 2-8 72
Table 17 API Performance Level Weighting Factors 73
Table 18: Comparison of 2007 API Results 75
Table 19: Staff Rating of Interventions Used During 2006-2007 78
vi
List of Figures
Figure 1: CST ELA Performance by Demographic Category 9
Figure 2: CST Mathematics Performance by Demographic Category 10
vii
Abstract
This study reviews the effect of two targeted intervention programs used in a
California middle school to improve English Learner achievement. The study uses an O
pre X O post design to compare the effects of the interventions for the English Learner
students that received the interventions to English Learner students, and native English
speaking students that did not receive the targeted intervention. The study uses matched
scores on the 2006 and 2007 California Standards Test in English Language Arts (CST
ELA) to compare the disaggregated subgroups described, (dependent groups) and also
to compare the schools results for English Learners to 2 similar schools and the state
results. The study measures for statistical significance and practical significance. The
literature review examines learning issues and best instructional practices for English
Learners as well as the two targeted interventions. The study concludes that the CST
ELA criterion referenced system creates internal validity issues that make it difficult to
determine significance in a study comparing two different years. Comparisons of the
groups show that there was a greater increase for the students that received the targeted
intervention although the difference was small. The study also concludes that while the
school attempted to implement many of the research based, best instructional practices
school-wide, the monitoring of the implementation must be increased to improve the
use and implementation of the effective instructional practices by the staff.
1
Chapter One
Background
The XYZ Unified School District, (XUSD), on the San Mateo County Coast,
California has an enrollment of approximately 3600 students in grades K-12. The
district has four K-5 elementary schools, a middle school, a high school, an alternative
high school, and an independent study program. Of the district’s 3600 students,
approximately 33% percent are identified as Latino, and 60% of the students are
identified as white, with the remaining 7% representing all other ethnic demographic
categories, including decline to state. Approximately 28% percent of the students are
English Learners, (EL). The population of socio-economically disadvantaged, (SED),
students is about 30% and there is high correlation between the EL students and the
SED students, 86% of the EL students are also SED. While the district has often been
referred to as very diverse, it would be more accurate to describe it as dichotomous,
there are two significant populations, middle class and affluent white students and the
Hispanic/Latino students with most of these families working in the local nurseries and
farms supporting the local agrarian economy.
No Child Left Behind, NCLB; accountability measures have highlighted the
demographic split in the district. There continues to be a significant gap in the
achievement between the two populations. The district has not met the benchmark
measures for performance on the English Language Arts, California Standards Test for
the EL and SED populations. The failure to meet the benchmark was most significant
at two schools; MFC Intermediate School and ABC Elementary School. Both schools
2
have been identified by NCLB as Program Improvement schools. Program
Improvement, (PI), schools are those that have failed to meet NCLB standards for at
least two consecutive years. ABC is a feeder school to MFC.
My focus was on MFC for several reasons. The first was that in my role as
Director of Categorical Programs, one of my responsibilities has been to monitor and
assist the implementation of programs that are funded by Title I, Title III and School
and Library Improvement Funds. MFC was the secondary school in our district that
receives funding through these programs so it has been a part of my job to evaluate the
effectiveness of the programs that were funded through the categorical funding received
by the school.
Another factor was the existence of the dual language Spanish/English
Immersion Program continues in the middle school as an elective that is directly under
my supervision. The immersion program is a draw for many affluent parents however if
MFC continues as a PI school, the immersion program may not be a sufficient draw for
these parents. It is important to remove the PI designation from MFC to help maintain
enrollment.
As a PI 3 school, the school had struggled to structure the intervention design
and implementation in order to exit PI status following the protocol of options given by
NCLB. I was a member of the District Site Leadership Team (DSLT), working with
Networks for Success, the San Mateo County Office of Education Intervention Team,
charged with designing an effective improvement plan.
3
Also as the only middle school in the district all of the EL students in the district
attend MFC as they move through the district. Because of the nature of high school,
where students track themselves onto the college and non-college tracks, the middle
school is the last, best chance to intervene to increase the representation of EL students
that are on the college track. As many who study adolescence have concluded, it is a
period of time that is critical to the establishment of an identity of self and self efficacy.
Therefore it is important to establish a positive attitude toward education and the
opportunities that it provides among the EL students while in middle school, and
academic success will help achieve this goal.
Possibly the most important factor in selecting MFC Intermediate School, for
the purposes of this study, was the ability to isolate the students that had received the
direct interventions. Much of the direct student intervention program occurred after
school and during the summer, some of the students that were a part of the EL
population were unable and or unwilling to participate due to the additional time
required. This provides another comparison group for the purposes of the study.
Problem Identification
The problem was to increase the achievement of the EL population in order to
exit Program Improvement status. I specifically examined the effectiveness of various
interventions that were funded through categorical funds. Some of the interventions
such as staff development were school-wide but all of the interventions were targeted at
students that are EL and SED. As mentioned, there continues to be a significant overlap
between these two groups, 86% of the EL students were also SED so a focus on the EL
4
subgroup should have also addressed the achievement gap for the SED students. I used
the data that was based on the California Standards Test and the existing state and
federal accountability programs.
There were three measures of student achievement that illustrate the problem at
MFC. The first measure was the achievement on the California Standards Tests and the
percentage of both the EL students scoring proficient or above on the language arts
portion of the test. The second was the Title III, Annual Measurable Achievement
Objectives, (AMAOs), which is the accountability measure of the progress of English
Learners. The third measure was California’s Academic Performance Index, (API)
score. For the year 2005-2006, MFC School failed to meet NCLB and Title III
objectives and posted negative API progress for Latino and SED students, while
showing gains for White students and a smaller gain for EL students.
California Standards Tests (CST’s) and Annual Yearly Progress (AYP)
In 2006-2007, MFC Middle School was in its third year of PI status. The school
did not meet NCLB’s AYP benchmark of at least 24.4% of the EL and SED students
achieving a score of proficient or above on the California Standards Test for English
Language Arts, (CST ELA), or the California Standards Test for Math, (CST MATH).
The chart below shows the performance of MFC on the CST ELA and CST MATH
relative to the NCLB AYP targets. It was important to note that the NCLB criterion
includes Reclassified Fluent English Proficient Students, (RFEP) among the EL
population until an RFEP student has scored at a level of proficient or above for three
consecutive years on the CST ELA. As a school in the third year of PI status MFC has
5
failed to meet the NCLB AYP targets for all sub-groups for four consecutive years,
although the school has shown achievement growth for all subgroups during that time
period.
Table 1
2006 MFC EL/SED Students % Scoring Proficient or Above
Year CST ELA
EL Student
Performance
CST ELA
SED Student
Performance
NCLB
ELA
Target
CST Math
EL Student
Performance
CST Math
SED Student
Performance
NCLB
MATH
Target
2006
18.5
21.6
24.4
21.1
23.7
26.5
Table 1 shows the results for EL and SED students (California Department of
Education, 2007). The following tables show the grade level performance on the CST’s
for 2006 for the EL and SED students. In the tables below, the RFEP students have been
pulled out of the EL calculation. In CUSD an EL student must score proficient or
above on the CST ELA test as well as score proficient or above on CELDT in order to
be considered for reclassification. Because the RFEPs have already achieved
proficiency on the CST’s they were not a part of the group targeted for intervention.
The difference between the inclusion and non inclusion of the RFEP students was
significant, instead of a whole school 18.5% of the EL students scoring proficient or
above the percentage dropped to 5% on the CST ELA. Table 2 shows the results for the
CST ELA test (California Department of Education, 2007). Table 2 and 3 also illustrate
the difference of effect of the inclusion of the RFEP students in the EL populations as
6
those students are still a part of the SED numbers shown while they have been
disaggregated from the EL population shown in the table.
Table 2
MFC 2006 EL and SED Student Performance on CST ELA
Grade / N
Far
Below
Basic
Below
Basic
Basic
Proficient
Advanced
6
th
EL / N= 56
36% 30% 29% 5% 0%
6
th
SED / N= 79
28% 25% 29% 14% 4%
7
th
EL / N= 64
16% 31% 45% 8% 0%
7
th
SED / N= 88
11% 30% 35% 20% 3%
8
th
EL / N= 55
16% 29% 53% 2% 0%
8
th
SED / N=91
8% 25% 49% 14% 3%
In Table 3 (California Department of Education, 2007) the math results are
shown for the same populations of students. The elimination of the RFEP students from
the disaggregated EL group dropped the percentage of EL students scoring proficient or
above for the school from 21.1% to 7.6% scoring proficient or above in math.
7
Table 3
MFC 2006 EL and SED Student Performance on CST Mathematics
Grade / N Far Below
Basic
Below
Basic
Basic Proficient Advanced
6
th
EL / N= 56
21%
54%
21%
4%
0%
6
th
SED / N= 79
20%
42%
28%
6%
4%
7
th
EL / N= 64
16%
31%
45%
8%
0%
7
th
SED / N= 88
12%
32%
28%
24%
2%
8
th
EL / N= 55
18%
35%
36%
9%
2%
8
th
SED / N= 91
11%
24%
33%
26%
5%
The depth of the problem that faces MFC and all of California was vividly
reflected by comparing the achievement results shown above with the achievement of
the white, English speaking population shown below in Table 4 (California Department
of Education, 2007). At 6
th
grade 78% of the English speaking-white students were
proficient or above on CST ELA as compared to 5% of the non-RFEP EL students, at
7
th
grade 67% of the students were proficient or above on CST ELA as compared to 8%
of the EL students and in 8
th
grade 79% of the students were proficient or above on CST
ELA as compared to only 2% of the non-RFEP, EL students in the 8
th
grade.
8
Table 4
2006 MFC CST Performance for English Speaking-White Students
Grade / Test
Far Below
Basic
Below Basic Basic Proficient Advanced
6
th
ELA
3% 4% 9% 30% 48%
6
th
MATH
2% 3% 11% 22% 56%
7
th
ELA
3% 3% 19% 31% 36%
7
th
MATH
2% 9% 19% 43% 20%
8
th
ELA
0% 3% 15% 36% 43%
8
th
MATH
1% 5% 23% 44% 25%
9
Figure 1 (California Department of Education, 2007) below graphically
illustrates the achievement gap that the staff faced on the CST English Language Arts
Test. Note that the EL and EL/RFEP are shown separately to illustrate the positive
effect of the reclassified students.
Figure 1
2006 CST ELA Performance by Demographic Category
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Schoolwide
Latino
SED
EL/RFEP
EL
White
Demographic Category
P ercen tag e P ro ficien t o r A b o ve
10
Figure 2 (California Department of Education, 2007) graphically illustrates the
achievement gap on the CST Mathematics test. While the EL students were clearly
performing better in mathematics than language arts, the achievement gap was
significant and the EL students were performing below NCLB targets.
Figure 2
CST Mathematics Performance by Demographic Category
0%
10%
20%
30%
40%
50%
60%
70%
80%
Schoolwide
Latino
SED
EL/RFEP
EL
White
Demographic Category
Percentage Proficient or Above
11
Title III Annual Measurable Academic Objectives (AMAO)
Title III is a federal program that provides funds for programs and strategies to
help EL students acquire English. The funds are divided into two categories, the larger
of the two are funds designated for Limited English Proficient (LEP) students, and the
second category targets immigrant EL students. There are three measures of
accountability and the school failed to meet two of the three, Title III AMAOs. The
first AMAO is overall improvement of EL performance in English as measured by the
California English Language Development Test, (CELDT), and MFC met that goal.
The second AMAO is an increase in the number of EL students that test as proficient in
English based on the CELDT. The school did not meet this goal. The third AMAO
refers to the same data as the AYP, and is percentage of EL students to meet the
benchmark percentage of proficiency on the CST ELA. The school did not meet the
third goal.
Table 5
MFC Performance on Title III Annual Measurable Academic Objectives (AMAOs)
Year
MFC
Performance
AMAO 1
Title III
Target
MFC
Performance
AMAO 2
Title III
Target
MFC
Performance
AMAO 3
Title III
Target
2005-06
69.2
62.6
39.0
44.0
18.5
24.4
12
California’s Academic Performance Index (API)
The API data for MFC, while not significant to NCLB accountability at this
time, was still an indicator of the lack of consistency in student achievement at the site.
MFC met NCLB requirements for a minimum API score of 590 with an overall score of
776. This score gave MFC a very good statewide ranking of 8 and a similar schools
ranking of 7. The API has remained fairly static and there hasn’t been a consistent
growth trend. For the last year while the schools overall API score showed a drop of 3
points and a 5 point increase for EL students, The SED targeted populations API’s fell
as did the overall score for Latino students. It was also important to note the greatest
benefit of professional development on standards based instruction seemed to accrue to
the White students who showed the greatest gains at MFC and at every school in the
district. Table 6 shows the sub-group API (California Department of Education 2006).
Table 6
API Sub-Group Progress
Subgroup
API 2006
Gain/Loss 2006-2005
MFC EL 625 +5
MFC Latino 646 -2
MFC SED 622 -9
MFC White 877 +14
MFC School-wide 776 -3
13
While the API in this case was not a part of the reason for MFC being a PI
school the data used to compute the API was an important measure of the progress that
the students were making. Unlike the AYP measure that is simply a look at the
percentage of students that have met the state’s definition of proficient, the API
calculates improvement and rewards schools for student improvement, with the greatest
weight given to the movement of students from the lowest achievement band of “Far
Below Basic.” This measure of improvement is valuable, and in some respects more
useful than the simple benchmark measure used by the AYP.
Problem Analysis and Interpretation
Obviously, based on the above description, the problem of EL student
achievement was a significant problem that must be approached on many fronts. There
were several factors that contributed to MFC’s failure to meet the required
accountability benchmarks. The factors were divided into four areas of focus,
knowledge, motivation, student, and organization factors. The student factors are
discussed first. While there was clear evidence that demographics and student home
factors had a predominate effect on the achievement and those factors will be discussed,
the focus was upon the factors that the school can control.
Student Factors
Robert Marzano (2001, p.123) said, “One of the perceived truisms in education
has been that student background characteristics are the most important determinants of
student achievement.” The students that were not meeting the benchmarks were the
students that one would expect would have difficulty meeting standards, the English
14
learners and the socio-economically disadvantaged. In the case of the students at MFC,
these two groups were very similar. There were two populations that combined to make
up over ninety percent of the students, the white made up fifty-one percent and the
Hispanic/Latinos made up forty percent, almost all of the Latino students had family
roots in Mexico. Approximately twenty-three percent of the students were on free and
reduced lunch. Most of the Latino students were from families where the primary
income was from work as farm laborers. The two largest employers in Half Moon Bay
are the Nurserymen’s’ Exchange and Bay City Flowers. Independent growers of a
variety of fruits and vegetables also employ many of the families. The Mexican
families often qualify for free or reduced lunch, so there was a high correlation between
the EL and SED populations.
The parents of the Latino students were generally first generation immigrants to
this country and most had not completed high school. The Mexican students that
enrolled in the district for the first time at MFC rarely had literacy skills in their native
language. For the students that begin their enrollment in kindergarten English usually
had become the dominant language by third grade. The vast majority of the EL students
that enrolled in CUSD in K and 1
st
grade arrive at MFC at the Basic level on both
CELDT and the CST ELA. For the students in the Immersion Program, Spanish
fluency was improved but CST scores show there was little difference between the
achievement of students in the Immersion Program and the students in the English only
program.
15
Table 7 shows the demographic information as reported to the California
Department of Education in October using the California Basic Educational Data
System (CBEDS). CBEDS is the annual data collection tool that determines a schools
demographic profile for the year. Additionally the percentage of EL students that were
also SED students at MFC was calculated to show the high correlation between the two
populations. Another reflection of the dichotomous nature of the student population
was the parent education level with an average that was in the middle between parents
that did not finish high school and parents that had completed graduate school.
Table 7
MFC Demographic Data Grades 6-8
Parent Education Level
1 = Not HS Graduate
2 = HS Graduate
3 = Some College / Trade School
4 = College Graduate
5 = Graduate School
(School Average 2.98)
Percent of White (not Hispanic) students
51%
Percent of Hispanic / Latino students
40%
Percent of Reclassified Fluent English Proficient
13%
Percent of English Learner students (EL)
24%
16
Table 7 (Continued)
Percent of students on free and reduced lunch (SED)
36%
The percentage of EL students that are also SED
86%
Student motivation for EL students at the middle school seemed to wane as
compared to students at the elementary level. There are several factors; one was that
the students in EL were required to have their language arts in an English Language
Development (ELD), class setting. The two-period ELD core course dictated much of
the rest of the schedule so the students EL students were together for much of the day
and they became very aware of being a separate group. The teachers for these students
were focused on the needs of the group, and the targeted instruction was educationally
sound but the teachers acknowledge that there were social problems associated with the
grouping of the ELD students. One other issue was the presence of a gang culture
among the Latino community that was a powerful draw to students that feel
disenfranchised. The staff does a good job of keeping gang colors and posturing off of
the campus, but it was present and middle school was often where much of the
attraction to gangs begins.
A significant student factor for the purpose of this study was that many of the
students that participated were those that demonstrated high motivation simply through
the act of participating in the intervention. This was due to the times that the
interventions were offered.
17
While few would argue with Marzano regarding the preponderance of influence
that student demographics have on student achievement, it was critical, that the student
demographic information was not used as an excuse, or that the staff blamed the
students for their own failure. The staff must recognize that they can have a significant
impact and that the student’s academic achievement was not determined by their birth
as poor, Spanish speakers.
Knowledge Factors
This area focuses upon the pedagogical and content expertise of the teachers and
principal. As previously discussed a skilled teacher can have a profound effect on
student motivation and student achievement. “Knowledge and skill enhancement are
required for job performance under only two conditions. First, they are required when
people do not know how to accomplish their performance goals, and second, when you
anticipate that future challenges will require novel problem solving” (Clark & Estes
2002).
I would maintain that both of these conditions were in place at MFC. Schools
throughout the nation have worked to solve the problem of English Learner
achievement. The challenge at MFC has grown, as the percentage of EL students
continues to increase and the academic demands have forced difficult choices regarding
the structure of offerings available at the school and the equity of opportunities offered
by the school.
The entire staff had been certified as “highly qualified teachers” by the district
according to NCLB guidelines. The entire staff has a Cross-cultural Language and
18
Academic Development certificate (CLAD), which indicates they have received
training in effective strategies for EL students as well as Specially Designed Academic
Instruction in English (SDAIE). The certification however, did not lead to best
practices in every classroom
Acknowledging that, “the more effective teachers use more effective
instructional strategies,” (Marzano, 2003, p.78), there had been a single focus to the
staff development for the year. A targeted staff intervention on teaching Academic
Language to EL students provided the staff with strategies to address the needs
vocabulary and cognitive needs of EL students. This had been a school-wide
intervention that hopefully had a positive impact on the instruction for all of the
students.
Motivation Factors
The staff motivation was both an asset and a liability. The staff really wanted to
exit PI status and really did care about educating the EL students. There were different
beliefs as to the best way to educate the EL students. Many on the staff believe strongly
that all EL students should be mainstreamed, placed in regular classrooms with English
speaking students, and that the English speaking students serve as models for the EL
students. Others on the staff recognize that specific targeted instruction for EL students
is necessary, and that leveled ELD groups and sheltered instruction are scaffolds to
build EL content knowledge. The staff was universal in their anger and resentment with
the burden of PI status and openly talks asks if they would really miss anything if they
simply did not accept Title I funds and were thus removed from accountability. The
19
staff was very defensive regarding changes to what they do and points to their high API
as evidence that they were not a failing school and in fact were doing well. While the
high API was true, the beneficial component to NCLB, that it demands achievement for
all students was not well accepted as possible by staff. At a school that had relied on
the high achievement of the already successful students the staff was required to view
themselves differently. This new view seemed to affect morale.
Student motivation varied, but most students did strive to do their best. MFC
was a very positive place with several incentive and participation programs that
engaged students. With regards to the direct intervention program, I must had to be sure
to account for the fact that many of these students may have been more highly
motivated as evidenced by their participation in the intervention program.
Organization Factors
I primarily relied on the county team’s assessment to describe the organization.
The audit was primarily based on teacher feedback sheets as well as scheduled
observations. In addition to the county team’s assessment, a key component of the
change in the organization was the instructional leadership of the principal who must
see himself as an agent of change and must move beyond the role of school manager.
This had been both an observation and directly stated to me by his administrative coach.
The idea of engaging change was a part of the new district Strategic Plan and needed to
be embraced by school leadership to positively impact student achievement.
20
The Academic Program Audit
Last year as a PI 2 school, MFC was reviewed by Networks for Success, a team
from the San Mateo County Office of Education. As mentioned earlier, I am a part of
the district team to monitor the program changes to increase student achievement, and
my plan was to measure and document the effectiveness of specific interventions that
took place as a result of recommendations. The audit completed last year identified
specific areas of concern and focus. The tool used for the assessment of the school is
the Academic Program Survey, a tool published by the California Department of
Education that identified features of a successful program into nine “Essential Program
Components” (Academic Program Survey 2001). The components were then grouped
into four clusters of related elements. A review of the review team’s findings helped to
identify the problems.
The first cluster looked at the instructional program, instructional time, student
achievement and the lesson pacing schedule. In this area, the team had the following
basic findings. “Some groups of students did not have access to the core curriculum.
Teachers did not collectively use common, curriculum-embedded assessments on an
ongoing basis to measure student achievement. The school did not have intervention
classes during the school day. There was no ELD program that differentiates from the
Reading/Language Arts Program” (Networks for Success, 2006).
The second cluster referred to teacher and administrator training. The group
made the following conclusions. The assistant principal had not received any
21
curriculum training, the principal only for language arts. Most teachers had not
received AB 466 training in either language arts or math. Teachers had not received
consistent staff development in State Board of Education (SBE) adopted materials.
Finally, staff had not received systematic, coherent professional development in ELD
and sheltered instruction, so that teachers had a common understanding and common
language (Networks for Success, 2006).
The third cluster combined, ongoing instructional assistance and support for
teachers and monthly collaboration by grade or program for teachers facilitated by the
principal. The group found: teachers did not have instructional support, such as
coaching, focused on the use of SBE materials; there was no coherent focused or
systematic intervention program in place for all students; teachers did not have time to
collaborate; ELD and sheltered teachers and all teachers who teach English Learners did
not meet consistently to discuss and strategize to address the instructional needs of EL
students (Networks for Success, 2006).
The final cluster is the fiscal support the school site received. The team
concluded that: teachers did not know how funds were allocated to support
implementation of their SBE-adopted materials; the school had not designated and
therefore had not purchased SBE-adopted intervention programs to be implemented
during the school day; the district and site had not funded a coherent staff development
program focused on SBE-adopted materials; resources had been minimally used for
professional development that was targeted toward improving EL student achievement;
22
Special Education teachers had not had sufficient mentoring to fully implement least
restricted environment (Networks for Success, 2006).
Problem Solution
The problem of low achievement for EL students was not unique to MFC and it
continued as a problem that the district must address. While there were many
grumblings about NCLB at the school, schools being held responsible for the
achievement of these students was viewed by some as a positive element of the law
even though these individuals still resented MFC being labeled a failing school. The
problem of the achievement gap was not a small one and has been approached on
multiple levels. As the person in charge of the funds that specifically address
underperforming students, Title I, Title III, Migrant Education, and Economic Impact
Aide (EIA), it was part of my job to evaluate the effective expenditure of these funds.
Most of the interventions were directly tied to the funding, while some were tied to
improving instructional leadership.
The first solution looked at the knowledge of the teachers. To increase the
effectiveness of teacher practice, there was ongoing staff development around a single
focus. Beginning on the staff development day just before the start of school and
continuing on six Tuesdays, throughout the school year, staff developer Kathleen
Kentfield presented Secondary Academic Language Training (SALT). The SALT
training presented clear, concise, applicable lessons, complete with plans to take back to
the classroom for the teachers on how to serve the EL students in the classroom. The
lessons were very well received. The presenter was a sought after presenter throughout
23
the state of California and the staff was very enthusiastic in their praise for the tools and
strategies that were taught. The focus of the program was developing academic
language for EL students. The staff development was funded through categorical
programs.
The language arts teachers had also participated in SB 466 training around the
Holt Reinhardt language arts text. The training took place in April of 2007 so the
possible effects of this training were not a significant part of the 2006-2007 student
achievement data. The teachers have used the program for five years.
Linked to the staff development was the use of walk-throughs by the principal
and assistant principal. The principal and the assistant principal received some training
around how to conduct walk-throughs with a single focus, for example to gather
evidence that teachers were checking for understanding, or that teachers were
frontloading vocabulary. The principal was a part of a coaching program that was
funded through categorical programs and the coaches at the county asked me to help the
principal to assimilate the walk-throughs and coaching feedback to the teachers as a part
of his repertoire of leadership skills.
I developed some spreadsheet tools to facilitate the tracking of data for the EL
and SED students. The tool is a simple spreadsheet that has current CST and CELDT
information, student demographic information was imbedded in the sheet through a
color coding system, and ongoing literacy measures were charted to measure student
progress. A key component was to provide this information to the community liaison, a
bilingual staff member to facilitate communication, which was funded through a
24
combination of categorical funds. The purpose was for the liaison to help families ask
questions when their student was not showing progress on the ongoing measures.
We also increased attendance in our EL parent education classes and instructed
the Spanish speaking parents in the meanings and importance of terms such as CST,
API, and AYP. My directive to my staff was that I want those parents to hold us
accountable by asking questions when there child was not making progress.
The intervention that was the most popular with the staff possibly because it was
the easiest to implement is the summer reading program using the Scholastic Read 180
as a tool. The program was funded through categorical program funds. The teachers
and counselors choose strategically targeted students for the intervention. Due to the
time commitment many of these students and their families must make, the group that
participated tends to be a motivated group of students and families.
The school also had affiliations with the Boys and Girls Club and the John
Gardner Center Community Schools Project which had been positively received
additions to the campus over the past three years, but were not a part of my examination
as they were not specific instructional intervention programs, the focus of these
programs was not academic improvement, the I programs did not receive categorical
funding and as the activities are open to all students they were part of the group of
school-wide interventions.
Purpose, Design and Utility
The purpose of this study was as stated in the title, to examine the effectiveness
of the interventions at MFC Intermediate School. This study attempted to quantify the
25
impact of the interventions undertaken by this PI 3 school. While it was obviously
impossible to isolate the effects of any one intervention, it was our hope that there were
student achievement gains sufficient to remove the Program Improvement Label from
MFC Intermediate School.
The design was a pre/post design using the data that was collected through the
states testing and accountability system. The California Department of Education was
the primary source of the comparison data, using the CST ELA test. I was able to
provide matched scores for the simple pre/post design and I was able to look at the
effectiveness of the direct student interventions in two ways. First of all I was able to
compare the effectiveness of the programs themselves through comparison of the
achievement gains for those EL/SED students that participated and those EL/SED
students that do not using the CST ELA performance band scores. I was also able to
use CST ELA scale scores as another comparison measure. This comparison provided
some insight into the effectiveness of the school-wide initiatives.
The utility of this study was that it was directly applicable to my job and role in
the district. As the Director of Categorical Programs it was my role not only to insure
that we received the money that the district was entitled to, but also to insure that the
district was compliant with the rules regarding the allocation of funds, and finally to
evaluate the effectiveness of the money spent. Previous to NCLB, the first two
components of my role had been in place in every school district for as long as there
have been categorical programs. The third element of the job has been a more recent
26
development and has been tied to the new levels of accountability in today’s world of
education. It also served the needs of the district as it was seen as critical to improve
the image of MFC in the community and an increase in student achievement is the
primary means to accomplish this goal.
27
Chapter Two
Literature Review
A profound issue facing public schools is the significant achievement gap
between white students and students of color that has persisted since long before the
initial authorization of Title I in 1966. With the current re-authorization of Title I,
popularly known as No Child Left Behind (NCLB), schools are being held accountable
for the achievement of the minority students. The growing challenge for teachers is the
number of students for whom English is their second language. An increasing
percentage of the minority students are also English Learner (EL) students, learning
English as a second language in addition to the content required in schools. Between
1979 and 2003 , the overall number of school age children increased by 19 percent,
during the same time period the number of children who spoke a language other than
English at home increased by 161 percent” (Hill & Flynn, 2006). One of the most
significant criticisms of NCLB is the way in which it treats EL students. “When it
comes to English language learners, NCLB defies logic. Common sense dictates that if
you administer a test to students in a language they don’t understand they probably
won’t do well on it” (Wright, 2007). While this may seem “common sense,” the
requirements of the NCLB program require all students to be tested regardless of their
language fluency. While NCLB regulations allow a school to not count a student in the
first 12 months of enrollment in U.S. schools in the Annual Yearly Progress measure for
language arts, the students are counted in the second year which is not a realistic time
period to allow for the development of academic fluency (Cummins, 1994).
28
“In California almost 80 percent of the students identified as English learners are
Spanish speaking” (Freeman, Freeman, & Mercuri, 2002). The increase in EL students
is true in Cabrillo Unified School District, where the percentage of EL students
progressively doubles from roughly 21 percent in the 11
th
grade to 42 percent in the 2
nd
grade. Due to the importance of addressing the learning needs of the EL student much
literature has been produced regarding best practices to raise EL achievement. The
importance is illustrated be the statistics that show the dropout rate in California is
highest for Hispanic students, “31 percent of Hispanic boys drop out, compared to 12.1
percent of African American boys and 7.7 percent of Anglo boys. Twenty-six percent
of the Hispanic girls drop out, compared with 13 percent of African American girls and
6.9 percent of Anglo girls” (Freeman, Freeman & Mercuri, 2002)
Recently a great deal of literature has been produced that address the
identification and differences between subgroups of EL students and the best teaching
practices to address their educational needs. For the purposes of this study the literature
examined will be divided to describe the two groups Long Term English Learner
(LTEL) students and Newcomer/Limited Formal Schooling (LFS) EL students who
make up the EL population at MFC Intermediate School. Also examined are the
research based practices to meet the needs of these populations with a closer look at the
strategies that are in place at the school. I will also look specifically look at the two
specific, targeted interventions that I was able to measure, summer school and the
Scholastic Read 180 program.
29
Long Term English Learners
The vast majority of the EL students at MFC Intermediate School have been in
Cabrillo Unified Schools for more than five years. Most have attended district schools
since kindergarten. Most of these students have demonstrated English Proficiency
through their performance on the California English Language Development Test
(CELDT) but have not been redesignated as Redesignated Fluent English Proficient
(RFEP) because most of these students have not scored proficient on the California
Standards Test for English Language Arts (CST ELA). This has resulted in the schools
failure to meet the No Child Left Behind (NCLB) benchmark proficiency percentages
for the EL subgroup.
The Long Term English Learner (LTEL) was defined by the following
characteristics (Olsen & Jaramillo, 1999):
• Seven or more years in U.S.
• Below grade level in reading and writing
• Adequate grades but low test scores
• ESL or bilingual instruction, but no consistent program
• Usually orally fluent in English
• Bi-modal academically: some doing well, others not
• Some have literacy in primary language, others not
• Mismatch between students perception of academic achievement (high) and actual
grades or test scores (low)
• Similar mismatch between language ability and reality
30
Most of these students have conversational fluency in English but do not have
the English language proficiency they need to compete with native English speakers
(Freeman, Freeman & Mercuri 2002). The LTEL students have acquired their Basic
Interpersonal Communications Skills (BICS) but have not acquired their Cognitive
Academic Language Proficiency Skills (CALPS) (Cummins, 1994). The conversational
communication skills often fool the teacher to assume greater English competency for
these students. Many of these EL students were born in the United States. The
mainstream teachers often don’t recognize the language acquisition needs of the
students, only that the students are struggling academically (Olsen & Romero, 2006).
As students demonstrate their oral proficiency communicating with teachers and peers
teachers expect the student to be able to use “English as a tool for learning” (Chamot &
O’Malley, 1994) but unfortunately the academic language demands, particularly in the
non-fiction, content areas have not been mastered. Secondary content language
demands increase significantly over the expectations in elementary school exacerbating
the problem for EL students (Calderon, 2007).
Newcomers with Limited Formal Schooling
The remaining population of EL students at MFC is small in comparison to the
LTEL students and consists of newcomer students that have had little formal schooling.
Almost all of the students have newly arrived from Mexico. The characteristics and
needs of the newcomer population differ significantly from the LTEL population.
31
The newcomers with limited formal schooling generally have the following
characteristics (Olsen & Jaramillo, 1999):
• In United States three years or less
• Little or no English language proficiency on arrival
• Some are well prepared in there own language others are below
• Steady progress through EL sequence
• Academic achievement in terms of grades similar to rest of school
• Difficulty passing CAHSEE within a four year frame
• Facing cultural transition to U.S.
These students are placed in ELD beginning level classes and sheltered content
instruction classes. Their access and engagement to the curriculum and the school are
deeply impacted by little or no English proficiency. The students that arrive in the U.S.
during the middle school years will have an enormous challenge to be able to pass the
California High School Exit Exam (CAHSEE) within the traditional four years of high
school. For many of these students the first cultural challenge is simply to adapt to the
routine of school (Olsen & Romero, 2006). This profile is typical of many EL students,
and their socio economic status is below that of the general population of students
(Lachet, 1999).
The diversity of the EL student population is not often recognized by those not
working in education. The assumption by most is that the EL population are
immigrants or new arrivals while, “according to a U.S. Department of Education report,
41% of EL students were born in the United States (Lachet, 1999). The differences
32
presented by the different learning needs of these populations presents a challenge for
schools and a variety of instructional strategies are required to address those needs.
Instructional Strategies
It is critical to understand the stages of second language acquisition in order to
best provide for the needs for EL students. While there are variables such as the level
of native language fluency or parent education level that affect the rate of second
language acquisition, there is a consensus among language specialists as to the stages of
language acquisition. The acquisition of language is a continuum beginning with the
silent reception of language, the pre-production phase, and progressing to the level of
advanced fluency. The work of Stephen Krashen and Tracy Terrell in defined the
stages of second language acquisition in their book, The Natural Approach (Hill &
Flynn, 2007). The following table describes the stages of second language acquisition,
the characteristics of the stage, the expected time spent at a stage, and recommended
teacher prompts for that stage.
33
Table 8
Stages of Second Language Acquisition Adapted from (Krashen & Terrell 1983)
Stage Characteristics Approximate
Time Frame
Teacher
Prompts
Preproduction
The student
• Has
minimal
comprehens
ion
• Does not
verbalize
• Nods “Yes”
and “No”
• Draws and
points
0-6 months
• Show me…
• Circle the…
• Where is…?
• Who has…?
Early
Preproduction
The student
• Has limited
comprehension
• Produces one- or two
word responses
6 months –
1 year
• Yes / no
questions
• Either / or
answers
Intermediate
Fluency
The student 3-5 years
• Has excellent
Comprehension
• Makes few
grammatical
errors
• What would
happen if ...?
• Why do you
think…?
Advanced
Fluency
The student has a 5-7 years
near-native level
of speech
• Decide if…
• Retell…
34
English learners have the double task of learning English in both a social and
academic setting while simultaneously learning the core curriculum. It is widely
acknowledged that second language learners progress through the stages defined by the
work of Krashen and Terrell and further defined by the work of the linguist Jim
Cummins and his distinction between social language and academic language. The
majority of the students examined in this study have attained at least the intermediate
level of fluency. The students have mastered the Basic Interpersonal Communication
Skills (BICS), the language required for daily social interaction as defined by Cummins.
He also defined another, higher level of language, Cognitive Academic Language
Proficiency (Cummins, 1979). The high degree of BICS fluency creates a perception
that the students have a higher CALP ability than there actual performance. California
developed the CELDT test to determine English proficiency, and most of the EL
students in the study have achieved proficiency, “Early Advanced” or “Advanced” on
the CELDT test. Unfortunately these students are still far below the native English
speakers on the CST ELA. This is true across the state (Olsen & Romero, 2006).
In addition to understanding the continuum of language acquisition there are
four other key component to understand about second language development to inform
instruction. “Language is developed in four interrelated domains, listening speaking,
reading and writing,” an English learner must develop in all of these areas in order to
become academically literate in English.
Another element of language acquisition is that, “Language is age and content
dependent” (Olsen & Romero, 2006). Obviously being fluent as a kindergarten student
35
will not require the same skills and mastery of language as a student in eighth grade. A
fluent student will also understand and use appropriate vocabulary based on the context
of the situation. Verbal language will have different expectations than written language.
The CELDT test does have different test based on the age of the students, and the scale
score to attain the various proficiency level is higher as the student is older (Olsen &
Romero, 2006).
First language and second language are interdependent. The greater the abilities
in the first language the easier it is to transfer those abilities to the second language. A
student that can read in their native language will more easily learn to read in the
second. Cross-language effects can be positive and students with a strong academic
background in their home language will likely develop higher level of English
proficiency than those with limited skill in the native language (Olsen & Romero,
2006).
It is important to be aware that socio-cultural factors are important to language
acquisition. Language is connected to cultural identity. The motivation and ability to
learn English can be affected by the perception of the value of the home language and
how welcoming and affirming the new school climate is for the EL student (Olsen &
Romero, 2006).
A critical component of effective instruction for EL students is to incorporate
higher order thinking skills into the lessons. The EL student’s ability to use critical
thinking skills must be respected and validated. One means of accomplishing this task
is provide instruction in the primary language; this allows the student to continue to use
36
their primary language as an asset to learning content. In California, primary language
education came under fire with Proposition 227, mandating English only education
except with parent waivers for an alternative education program. This law is based on
the belief that students immersed in English will learn the language more quickly than
those that use their primary language in school. Studies have shown that use of primary
language does not impede second language acquisition. In a study based on a total of
75 top performing schools where Spanish was the dominant language in the home it was
concluded that no single instructional strategy or program was a superior pathway to
success but the study did identify four practices that were deemed critical to success:
1. Highly qualified, committed staff collaborates and share leadership to promote
EL academic success.
2. A school-wide focus on English Language Development and standards-based
instruction to foster academic language and content literacy across the curriculum.
3. Shared priorities and high expectation in regard to educating EL students.
4. Systematic, ongoing assessment and data-driven decision making.
This study was conducted based on telephone interviews with staff regarding practices
used to enhance EL student academic achievement. In addition the study concluded
that there was no evidence of any benefit to English only instruction (Parrish, Perez &
Linquanti, et al 2006).
In a study using a multiple variable approach that controlled for variations in EL
population as well as school size and student socio-economic status, twelve exemplary
elementary and middle schools in Arizona were identified for an in-depth study.
37
Comparisons were made with schools that had similar student populations and
circumstances to identify the practices associated with the exemplary performance of
the twelve selected schools. This study identified six characteristics shared by
successful schools, as compared to those that were doing poorly (Waits, Campbell &
Gau et al., 2006)
1. Focus on achievement for all through responsibility for the performance of
every student in every classroom.
2. Ongoing assessment and other student performance data examined regularly to
inform instruction and improve student performance. Strong and steady
leadership from the principal that relentlessly focused the school on
performance.
3. Problem solving as a distributed function, not just the responsibility of school
leadership, through faculty collaboration.
4. No single program or approach seen as a magic bullet for improved performance
rather, staff picks proven methods and stick with them over time, adjusting as
needed.
5. Interventions are personalized to meet the needs of individual students.
The studies similar conclusions were further validated in a 2007 study conducted by
EdSource which examined effective school practices and EL academic achievement.
This study surveyed principals and teachers in 257 randomly selected elementary
schools in California in the spring of 2005. Schools were drawn from a small band of
California’s School Characteristics Index representing challenging student
38
demographics. The study controlled for factors such as mobility, parent education
level, ethnicity, poverty, and school size and teacher turnover. A total of 237 principals
and 4,700 teachers completed the survey. The study responses were analyzed in
relation to EL student performance EL API, and NCLB Title III AMAOs using
CELDT, to identify factors associated with higher levels of EL academic performance.
The study found that the implementation of a coherent standards based curriculum,
using assessment to monitor and improve instruction and learning, ensuring adequate
resources; and having a shared sense of responsibility were four factors strongly
associated with higher EL student achievement ( Williams, Hakuta & Haertal, 2007).
An essential aspect of standards based teaching is to teach thinking. This is
especially challenging and especially critical in the case of EL students who are often
met with accommodations and modifications of instructions that address their education
by asking them to accomplish tasks that involve little critical thinking or real world
problem solving. Because many students lack academic language skills that would
enable them to use English as a tool for learning, it is important, particularly at the
secondary level to engage students in ideas that will stimulate thinking even if the
students must demonstrate their thoughts in non-linguistic ways (Chamot & O’Malley
1994).
It is important to integrated language and content for EL instruction beginning in the
upper elementary grades as literacy and content are closely linked. The Center for
Applied Linguistics and the National Clearinghouse for Bilingual Education have
concluded that integrating language and content in English as a Second Language (ESL)
39
is necessary to prepare students for grade level material. It is also important to provide
explicit instruction in learning strategies as a principal method for delivering instruction
(Chamot & O’Malley 1994).
Lorin W. Anderson and David R. Krathwohl have developed a revision of Bloom’s
Taxonomy of Educational Objectives. They developed a two dimensional matrix of
educational objectives, linking the development of knowledge with the development of
cognitive processes. One dimension is the cognitive processes which are to remember,
understand, apply, analyze, evaluate, and create. Each move across this continuum is
assumed to be more complex than the previous one. These cognitive processes are
linked with the knowledge dimensions of factual knowledge, conceptual knowledge,
procedural knowledge, and meta-cognitive knowledge. The two dimensions create
categories into which ideas or experiences can be places. The criteria that are used to
sort objects into the categories are determined by a set of organizing principles which
differentiate among the categories. The characteristics of each category help teachers to
understand what is place into each category. This framework was developed to provide
teachers with a tool to organize the large number of state standards into an organizing
frame that increases the precision of learning objectives to promote understanding. The
framework is useful as a tool to help teachers organize learning tasks for EL students by
allowing teachers to devise non linguistic ways to demonstrate deeper learning
(Anderson & Krathwohl, 2001).
The idea of planning instruction for based on the demonstration of the target
knowledge using increasingly complex cognitive strategies has become a part of EL
40
best practices instruction. Researchers at the Mid-continent Research for Education and
Learning (McREL) worked to synthesize the volumes of educational research into a
practical guide that teachers could use. In their book, Classroom Instruction That
Works, Robert J. Marzano, Debra J. Pickering, and Jane E. Pollock have studied and
preformed meta-analysis on a multitude of studies to determine the average effect of a
given instructional strategy. They concluded by identifying nine broad teaching
strategies that if used effectively can have a positive effect on student achievement.
• Identifying similarities and differences
• Summarizing and note taking
• Reinforcing and providing recognition
• Homework and practice
• Nonlinguistic representations
• Cooperative learning
• Setting objectives and providing feedback
• Generating and testing hypotheses
• Questions, cues and advance organizers.
Significantly the researchers demonstrated the size of the difference that each strategy
had on achievement. For example homework that had teacher’s comments as feedback
showed an average effect size of .83 and a percentile gain of 30, graded homework
showed an average effect size .78 and a percentile gain of 28, while homework that was
neither graded or commented on showed a much smaller effect size of .28 and a
percentile gain of 11 (Marzano, Pickering & Pollock 2001).
41
Marzano’s book was used as the basis for a later book by McREL researchers
Jane D. Hill and Kathleen M. Flynn who realized the nine strategies articulated in
Marzano’s work were the same strategies that research have been shown to be effective
for EL students. In their book Classroom Instruction That Works for English Language
Learners the authors examine each strategy and suggest how to use the strategy
effectively with EL students.
Several of the strategies have been a part of the school-wide reform efforts taken
by the school. The strategies that are in place are homework, setting objectives and
providing feedback, using nonlinguistic representations, summarizing and note taking,
cooperative learning, and reinforcing effort and providing recognition are in place or
have been part of staff development efforts provided to staff.
Homework is a key element of the instruction and grading policy at the school.
Some teachers weight homework as fifty percent of a student’s grade. Homework is
necessary as students can’t master a skill without a significant amount of practice. “In
fact students generally do not reach 80 percent competency until the have practiced a
skill at least 24 times (Anderson, 1995; Newell & Rosenbloom, 1981 cited in Hill &
Flynn 2006). There are four generalizations that should guide teachers in the application
of homework (Marzano, Pickering & Pollock 2001).
1. The amount of homework assigned should be different from elementary to middle
school to high school.
2. Parent involvement in homework should be kept to a minimum.
42
3. The purpose of homework should be identified and articulated.
4. Homework is more effective when feedback is provided.
Setting objectives and providing feedback is another one of the key elements
identified as effective for students that were a targeted school-wide strategy that was
implemented at the school. The regular principal walk-throughs specifically observe
classrooms to see a specific standard that is a part of the lesson objective. The walk-
throughs by administrators do not examine the feedback. The range of effect size for
setting learning goals or objectives is between 1.37 and .46 with a range in percentile
gain of 41 to 18 (Marzano, Pickering & Pollock 2001). Marzano identifies three
generalizations regarding goals setting. The first is that the research shows that setting
goals help students when the goals narrow what the student needs to focus on. The
second is that goals should not be too specific. Caution must be used as the research
showed that a goal that is too specific can lead to a negative impact on performance.
The third is that students should adapt the goals to meet their personal learning needs.
Setting goals in contractual arrangements has shown positive effects (Marzano,
Pickering & Pollock, 2001).
The use of nonlinguistic representations has been an element of the staff
development trainings provided to address the needs of the EL students. The results of
the effect of nonlinguistic representation show an effect size range of 1.31 to .50 and
range in percentile gain from 40 to 19 (Marzano, Pickering & Pollock 2001). The use
of graphic organizers is a feature of instruction in many of the MFC classrooms. “The
use of nonlinguistic representations enhances students’ ability to represent and elaborate
43
on knowledge using mental images” (Hill & Flynn 2006). The use of graphic
organizers, physical models, generating mental pictures, drawing pictures and engaging
in kinesthetic activities generate nonlinguistic representations. The use of these
instructional strategies help students to understand the material better and to have
greater recall of learned material (Marzano, Pickering & Pollock 2001), (Hill & Flynn
2006).
Advancement Via Individual Determination (AVID) is an elective program at
MFC that targets students that have demonstrated academic success and would be the
first in their family to attend college. One of the elements of the program is to
specifically teach note taking skills. Many of the classes require students to take notes
however note-taking skills aren’t explicitly taught in each classroom. Teaching
summarizing and note-taking skills enhances the ability to process, organize and
synthesize information. Note taking is related to summarizing in that both skills ask a
student to be able to identify and restate information using their own words. Three
generalizations regarding note-taking are important to apply. First the notes should note
be verbatim, the notes should be specific to the learning goals be comprehensive but
with the opportunity for students to synthesize the information. Second the students
recognize the notes are a work in progress, they should be reviewed, revised, and added
to as knowledge and content is added. And finally the students should use their notes as
study guides (Hill & Flynn 2006).
Cooperative learning is also in place in many classroom settings at the school.
In fact the school uses the CST data to help create effective learning groups. Groups
44
are formed both heterogeneously and homogeneously depending on the instructional
outcome expected from the grouped setting. Marzano compares the effect of a variety
of grouping setting and interestingly also compares cooperative learning versus
competitive settings. He found that cooperative groups and inter-group competition
showed no difference in the impact on student learning while cooperative learning has
an effect size of .78 when compared to individual competition and the same effect size
when compared to students working individually without competition. Leading to the
conclusion that cooperative learning groups have a powerful effect on learning
((Marzano, Pickering & Pollock 2001). Three guidelines regarding cooperative
learning are suggested. First grouping by ability level should be done sparingly when
using cooperative groups. Second that groups should be small, three to four students is
ideal. Finally that cooperative learning is effective when used systematically and
consistently, at least once a week is ideal. As a part of the systematic consistent use of
cooperative learning, teachers must make sure that students still have time to practice
independently (Marzano, Pickering & Pollock 2001), (Hill & Flynn 2006).
“An important component of academic success is students’ motivation and
ability to take responsibility for their own learning (Denbo & Eaton, 2000). The school
prides itself on the positive reinforcement program that is an element of the school that
has existed for years. The power of positive rewards has been recognized by past
administrators who created programs that reinforce both academic and behavioral
success by students. Honor Role, Principal’s Honor Role, Student of the Week, Student
of the Month, quarterly recognition assemblies and classroom recognition are all part of
45
the culture of the school. However, the school must also recognize that D and F grades
are disproportionately given to EL students and these grades help reinforce belief in
failure. This is very important as belief in the ability to succeed is an important
motivating factor leading to achievement. Emotions are critical to learning, threat and
helplessness impair learning and appropriate challenge enhances learning. Research
indicates that disbelief that one has the ability to accomplish a task can lead to students
sabotaging their own success (Covington, 1985). Threat and high stress are
impediments to learning and prevent access to higher order thinking skill. Positive
reinforcement helps to achieve optimal brain functioning by providing students with
several elements that lead to success (Zadina, 2001).
• Ownership and a sense of control over learning
• Positive social bonding
• Hope and positive expectancy
• A world that makes sense
• Playfulness, joy
• Respect of students and teachers for themselves and each other
• Self discipline and capacity to delay gratification
• Sense of connectedness
(Caine, N., Caine, G. 1981)
The first element of effective use of the use of reinforcing effort identified in the
book Classroom Instruction That Works is that not all students realize the importance of
46
believing in effort. The second is that students can learn to believe that effort will be
rewarded (Marzano, Pickering & Pollock 2001), (Hill & Flynn, 2006)
Additionally positive reinforcement leads to improved performance. Sociologist
Alfie Kohn in his book Punished By Rewards argued that external rewards lead to a loss
of intrinsic motivation. Marzano’s meta-analysis led to three generalizations. The first
is that “Rewards don’t necessarily have a negative effect on intrinsic motivation”
(Marzano, Pickering & Pollock 2001). The research also showed that the rewards are
most effective if the require the students to reach a certain standard of performance.
Additionally the research showed that abstract symbolic recognition such as verbal
praise is more effective than tangible rewards (Marzano, Pickering & Pollock 2001),
(Hill & Flynn, 2006).
Summer School is the largest targeted intervention strategy used at MFC
Intermediate School for students that are below grade level standards. Summer-slide has
a disproportionate affect for English Learner students and students from low socio-
economic backgrounds. Disadvantaged learners display regression of a month or more
on academic skills during an inactive summer layoff. The regression has a significant
effect on literacy skills. An analysis of fall to spring comparisons of academic
achievement while school was in session showed all groups made substantially similar
gains for high poverty and low poverty students. Spring to fall comparisons for high
poverty and low poverty students demonstrated differential progress. “The differential
progress made during four summers between 2
nd
and 6
th
grade accounts for upwards of
80% of the achievement difference between economically advantaged and ghetto
47
schools” (Hayes & Greteher, 1983). Additionally a measurement of intervention
programs showed that gains made through intervention programs during the school year
were diminished by the summer vacation (Borman & D’Agostino, 1996). For the
students in this study, the impact of summer is compounded for the low income EL
students as in addition to a lack of literacy support there English skills receive little
reinforcement in a home where Spanish is the predominate language. Quality
supplementary instruction during summer that provides access to literature with a focus
on successful reading experiences can have an impact on closing the achievement gap
(Alligton & McGill-Franzen, 2003).
There are several commercial intervention programs available for purchase to
address the academic needs of low-performing students. Among the intervention
programs available MFC has purchased the Scholastic Read 180 program as a targeted
intervention for struggling EL readers. The program has been implemented in many
schools across the country and there have been many studies about the program’s
effectiveness.
The studies demonstrate that if the program is faithfully implemented that there is a
measurable growth in student achievement. A study on the implementation of the
READ 180 program in the Anaheim City School District compared the growth on CST
ELA for low scoring students using the READ 180 program to predicted growth for the
test population. Researchers concluded that the test students demonstrated greater than
one year’s growth and that the result was not likely due to random chance. The study
also compared READ 180 to another Scholastic product READ XL. The later product
48
was superior in the study (Brennan & Luere, et.al. 2006). Another study conducted in
Los Angeles Unified School District on middle school students by Dr. Rosemary
Papelweis focused on results for EL students. The study used the SAT-9 norm
referenced test in a pre-post. “Overall the READ 180 participants made significant
gains of over three (3) Normal Curve Equivalents in reading and almost two (2) Normal
Curve Equivalents in language arts (SAT-9)” (Papelweis, 2004). Richard Papelweis
studied the effects of READ 180 in Clark County Nevada Schools. His research
demonstrated similar gains for secondary students that participated in the program.
(Papelweis, 2003)
The research supports many of the efforts undertaken by the school to address the
low achievement of the EL learners. Many of the instructional strategies and
interventions described by Robert Marzano and the other researchers as effective have
been a part of the program to raise EL achievement.
49
Chapter Three
Design Summary
In this study there are four dependent groups from the school that are measured
using a pre-post design. The first experimental group consists of sixth and seventh grade
English Learner (EL) students that participated in the Scholastic Read 180 Intervention
Program. A second experimental group consisted of the students that participated in the
traditional summer school program. As a control, the data generated by the EL students
that received no targeted intervention are also examined. A final locally generated
group for comparison is the school-wide not EL population. The 2006 California
Standards Test, (CST), data are used as the baseline for comparison. The post test
results are the 2007 CST data. I use the following summative evaluation design: O pre
X O post. The effectiveness of the intervention programs on the experimental groups is
measured through a comparison of the 2006 and the 2007 years and looked at the
matched CST scores for the same students in both years. A dependent group’s t-test is
used to determine the statistical impact of the interventions. Cohen’s-d is calculated to
assess the practical significance of the changes.
As the Scholastic Read 180 Program and summer school were not used in
isolation as an intervention for MFC to exit Program Improvement status it is important
to look at the possible effects of the school-wide interventions. Using the same pre-post
design, the students that received the intervention are compared to a similar student
population that did not receive the targeted intervention program. The control group of
EL students that did not receive the targeted intervention may have also improved,
50
either due to other school-wide reforms or due to their own academic growth and their
increased English proficiency. Their growth is compared to the experimental groups.
An analysis of the pre-post scores for both the treatment population and the non-
treatment population using matched scores provides a comparison to determine the
effectiveness of the intervention programs. The pre-post design allows me to examine
the totality of the Program Improvement (PI) interventions as well as the possible
effectiveness of the direct intervention on EL student achievement.
As additional controls the API growth results of the EL subgroup and school-
wide are compared to two schools that are identified as similar schools by the California
Department of Education. The EL subgroup API is also compared to the statewide
results for the EL subgroup.
A qualitative component, in the form of an open-ended survey of the staff is
used. The qualitative component provides insight into the perceived efficacy of school-
wide initiatives that due to their global nature cannot be separated for measurement. It is
of interest in my role to compare the perceived efficacy with the measured efficacy of
the school’s improvement efforts.
My role with the school district is the Director of Categorical Programs.
Categorical Programs are programs such as Title I, the reauthorization of Title I is
known as No Child Left Behind, Title III which is a federal program to serve EL
students and Economic Impact Aide (EIA) which is designed to serve EL and Socio-
economically disadvantaged students. The definition of my role had been to ensure that
the district received the categorical funds it was entitled to and then to monitor the use
51
of those funds was compliant with the regulations attached to the funding source. I
have expanded the role to include the evaluation of the effectiveness of the
expenditures.
Participants/sampling
The participants in this study are the EL student subgroup of 6
th
and 7th grade
students. The achievement of all of the students in the subgroup is studied. From the
EL subgroup at the school a small portion had the opportunity to participate in the Read
180 intervention. Twenty students participated in the READ 180 summer program.
The students chosen were students that would have the best chance for success with the
program based on teacher evaluation of reading readiness skills. Students that were at
level one in the CELDT, signifying that they speak and understand very little English
were deemed not good candidates for the READ 180 intervention. The best candidates
for READ 180 those that have reached at least a level three, intermediate, on CELDT
and have demonstrated in the classroom the ability to read independently in English.
The teachers selected 6
th
and 7
th
grade students based on the achievement criteria and
the student’s availability to participate in the intervention program which takes place
outside of the regular school hours.
The other experimental group that received a targeted intervention was the EL
students that participated in the traditional summer school program. The students that
were recommended for summer school by staff are those students that have a grade
point average below 1.8 or have failed either their English/Language Arts or Math class.
52
This EL population attending summer school was larger than the Read 180 sub-group.
80 EL students attended the traditional summer school.
Intervention Description
The first targeted intervention that was measured was the Scholastic Read 180
summer program. The school offered this intervention in the form of a class of 28
students taught over the summer. The students that were targeted for the program are
EL and Socio-economically disadvantaged (SED) students. The program provides 90
minutes of reading skills instruction five days a week.
The program combined books, computers, videos and implementation training
in a program designed to provide high interest reading materials in a context that allows
the struggling reader to understand the story while developing the decoding skills
needed to improve reading. Research by Rosemary Papelewis (2002) demonstrated that
Read 180 had a positive affect on reading scores for students in Los Angeles Unified
School District that were similar to those that were a part of this study. Another similar
study by Richard Papelewis done in Clark County Nevada with middle school students
showed similar results. The available research was reviewed as a part of the literature
review.
The 28 students that participated in the Read 180 program will be the
experimental group for the purposes of this study. The Read 180 students were in the
class for two hours per day to insure meeting the program recommendation of a
minimum of ninety minutes of direct Read 180 instruction.
53
Traditional summer school was the other targeted intervention that is measured.
The summer school students participated in traditional grade level English Language
Arts and Math instruction during a four hour day. The school-wide test data and the
students that participated in summer school will form the two experimental comparison
groups for this study.
There were several other school-wide interventions that took place during this
time period that focused on addressing the needs of EL students. Some of these
interventions are described by Robert Marzano in his book Classroom Instruction That
Works (2001). The first intervention was the targeted school-wide staff development on
addressing the needs of the EL student this year. Another school-wide intervention was
the implementation of principal walk-throughs, short classroom visits to monitor the
implementation of a standards based curriculum and the setting of standards based
learning objectives for daily lessons. Recognition and rewards are features of the
school. Various classroom interventions practiced in some classrooms included using
cooperative learning strategies, and teaching of note taking skills.
Instrumentation
Quantitative
The instrument I use for the quantitative data was collected through the STAR
testing program. I use data from the California Standards Test (CST) results on the
English Language Arts (ELA) Test. I focus on the English Language Arts Test as the
school had previously exceeded No Child Left Behind Benchmarks (NCLB) in
54
mathematics. I use the district’s Cruncher Data System by Lighthouse Software
(Cruncher), to disaggregate the CST ELA test data for 2006 and 2007 by the sub-groups
established. One group I examined was the students that received the Read 180
intervention during the summer. Another comparison group is the students that
attended the traditional summer school. A third comparison group that served as one
control group was the EL students that did not attend any summer program. The data of
the experimental groups was also compared to the data generated by the school-wide,
non-EL students who were used as another control measure. The student achievement
data was available through the California Department of Education (CDE) website and
was accessed and disaggregated using the Cruncher data tool to identify individual
student data for the matched scores comparison.
The CST ELA Test is a criterion referenced test that is based on the California
State Curriculum Standards. Cut scores based on the scale scores were established to
determine proficiency bands. The proficiency bands are Far Below Basic, Below Basic,
Basic, Proficient, and Advanced. In this study one of the measures will be the
percentage of students in each performance band. The other measure is the scale score.
These measures will be used as they align with the accountability measures of NCLB,
which has set benchmarks for the percentage of students that are required to be
proficient each year. The scale scores were examined as the scale scores provided a
more precise and accurate measure of the achievement growth. The scale score range is
from 150 to 600 and a scale score of 350 equates to a proficiency band score of
proficient.
55
The sixth, seventh and eighth grade 2006 CST ELA exam was a multiple choice
test consisting of 75 multiple choice questions divided into five categories; Word
Analysis and Vocabulary Development, Reading Comprehension, Literary Response
and Analysis, Written Conventions, and Writing Strategies. The test at all three grade
levels 6
th
-8
th
was given in two-parts with a break of at least 10 minutes between the two
parts. The test was un-timed but the directions for administration estimate a time of 80
minutes to complete each part.
The 7
th
grade 2006 CST ELA had an additional component, the 7
th
grade test
includes a written essay that is scored on a 4-point rubric by two readers for a possible
score of eight (8). The written test was given during a separate administration in
March. The testing window for the multiple choice portion of the test begins in late
April, and most schools, including MFC, tested during the first week of May. The
writing score is also reported separately to schools and parents.
Aside from the addition of the essay component there were differences in the
test between 6
th
and 7
th
grade. While all of the tests have the same total number of
multiple choice questions the number of questions assigned to each sub-category and
the percentage required for proficient is different. Tables 9-11 on the following page
show the number of questions in each sub-category and the percentage of correct
answers required for proficiency in each of the sub-categories. It is important to note
the difference at each grade level. The data was pulled from the Cruncher data system
and is from the California Department of Education (Cruncher 2007) (California
Department of Education, 2007).
56
The change in the number of questions in each sub-category and the change in
the percentages of correct answers required to attain proficiency affects the internal
validity of the study. These test differences are one reason that CDE posts instructions
regarding the analysis of CST scores on the printed copies of the results that state that
cohort comparisons are not recommended (California Department of Education, 2007).
The lack of vertical integration was a problem for this study and any other that compare
cohort group growth on the CST tests. I did not have a statistical solution to the vertical
integration problem.
The 2007 CST ELA for all three grade levels has differences from the 2006 test
that further affect this study’s internal validity. Using the CDE data and the Cruncher
Data System it is noted that the state level for proficiency on the CST ELA is changed
from the previous year’s test. Tables 9-11 show the differences between the 2006 and
2007 CST ELA tests. These differences are significant and create an internal validity
problem that was not anticipated. I was not able to develop a solution to the absence of
vertical integration on the CST ELA however using the matched scores still provided an
important measure of individual student achievement and growth which is the goal of
the interventions.
57
Table 9: 2006/2007 6
th
Grade CST ELA
______________________________________________________________________
Sub-category # of questions % for proficient 2006 % for proficient 2007
Word Analysis and Vocabulary (13) 78% 69%
Reading Comprehension (17) 68% 57%
Literary Response and Analysis (12) 62% 63%
Written Conventions (16) 73% 68%
Writing Strategies (17) 59% 63%
TOTAL 6
th
CST ELA (75) 67.8% 63.7%
______________________________________________________________________
Table 10: 2006/2007 7
th
Grade CST ELA
______________________________________________________________________
Sub-category # of questions % for proficient 2006 % for proficient 2007
Word Analysis and Vocabulary (11) 64% 71%
Reading Comprehension (18) 66% 66%
Literary Response and Analysis (13) 53% 60%
Written Conventions (16) 71% 64%
Writing Strategies (17) 70% 64%
Writing Applications (8) 75% 67%
TOTAL 7
th
CST ELA (83) 66.3% 65.0%
______________________________________________________________________
58
Table 11: 2006/2007 8
th
Grade CST ELA
______________________________________________________________________
Sub-category # of questions % for proficient 2006 % for proficient 2007
Word Analysis and Vocabulary (9) 72% 73%
Reading Comprehension (18) 65% 70%
Literary Response and Analysis (15) 59% 65%
Written Conventions (16) 77% 66%
Writing Strategies (17) 62% 67%
TOTAL 8
th
CST ELA (75) 66.5% 67.8%
______________________________________________________________________
As before the test was administered in two parts with a minimum of a ten minute
break between parts also as before the 7
th
grade also had an essay portion to the CST
ELA that is administered separately. If a student is not finished s(he) was allowed the
time needed to complete the test. Some of the items were scored so the percentages
required for proficiency and the number of scored items in each category cannot be
determined until the results were received in August.
The CDE score report included the number possible, the raw score, a mean
scaled score and a performance band score for each subject area. Cut scores for the
proficiency bands were based on the mean scale score which range from 150 to 600. A
mean scale score above 350 was proficient at all of the grade levels.
The CDE also reported school-wide data on the % Advanced, % Proficient,
% Basic, % Below Basic, and % Far Below Basic. Schools were evaluated based on the
percentage of students in each grade level, as a total and in disaggregated groups.
59
Groups of 100 students or 10% of the total population are considered significant sub-
groups, and it was required that all significant sub-groups. NCLB requires that 100% of
students score proficient by the year 2014. For the 2006-2007 school year the NCLB
target percentage scoring advanced or above on CST ELA for all significant sub-groups
was 24.4% (California Department of Education 2006).
Qualitative
The formative assessment data was in the form of an open ended survey that
asked the staff to express their opinion on the variety of interventions that were a part of
this year’s effort to exit Program Improvement (PI) status. The staff was asked about
their opinion as to the effectiveness of the various interventions. As three of the
interventions, working with an outside evaluator, principal walk-throughs and staff
development were school-wide in nature, it was impossible to use only a summative
tool to ascertain the effectiveness of these interventions independently.
All staff members were asked to participate in the survey through staff meetings.
The purpose of the surveys was discussed and distributed. The surveys were collected
at the end of the meeting.
1. Describe the effectiveness of the staff development regarding “Strategies to Teach
EL Students?”
2. Describe the effects that walk-throughs had on instruction? Was there effective
feedback? Did they help to focus instruction on the standards for your subject area?
3. Are the students that participated in summer school known to the teachers?
60
4. Are the effects of summer school on student performance noticeable and/or
measurable? Describe.
5. Are the students that participated in Read 180 known to the teachers?
6. Are the effects of participation in Read 180 noticeable and/or measurable? Describe.
7. Describe how the “Networks for Success” (San Mateo County Office of Education)
contributed to school improvement.
8. In your opinion, what was the most effective intervention for the 2006 school year?
In addition to the open ended responses the survey asked the staff to rate the
effectiveness of the interventions using a four point scale. Four (4) equals very
effective, one (1) equals not effective. The ranking of the interventions provided a
context to the open ended questions and a means of numeric comparison.
Procedural Timeline
The literature review began once the topic was approved and continue throughout the
study.
February 2007: I collected the names of the students that participated in the
interventions of Read 180 and summer school.
March of 2007: I collected the 2006 6
th
and 7
th
grade CST ELA data for the groups in
this study.
May of 2007: I began to analyze the pre data from the 2006 test.
May of 2007: The students were administered the 7
th
and 8
th
grade tests.
August 2007: The California Department of Education (CDE) sent the scores to the
district on a CD-ROM and they were downloaded into Cruncher.
61
September 2007: The MFC staff was surveyed as to their perception of the efficacy of
the interventions.
September 2007: The CDE published the CST scores for the state on their website.
October 2007: Data analysis began analyzed and survey results were analyzed.
Data Analysis
Quantitative Analysis
The quantitative analysis used matched scores to examine the change in scale
scores as well as the change in the percentage of students in each performance band.
Matched scores will be used for the following populations: Read 180 students; summer
school students; EL students with no targeted intervention; and the school wide non-EL
students. Students without a score for both years will not be included in the study.
While using the change in scale scores would provide more accurate data on the growth
occurring between the test years, it is the performance band percentage that is the target
for exiting PI status, and is the measure used by NCLB and was also the measure that
was reported in the news media when the test results were published.
The quantitative data was analyzed using a 2 (pre vs. post) by 5 (performance
bands Advanced-Far Below Basic) display for each of the comparison groups. A
dependent group’s t-test will be used to assess the statistical significance as well as
differences between the treatment and non-treatment groups. Cohen’s d will be
computed to assess the effect size of the interventions. SPSS 15.0 for Windows will be
used to compute the statistical analyses.
62
An additional quantitative measure was to compare the Academic Performance
Index (API) growth of MFC School with schools that had been identified as similar
school through the California Department of Education (CDE) Similar Schools
program.
Qualitative Analysis
To analyze the qualitative data I followed the model described by Creswell in
the course book, Research Design: Qualitative, Quantitative, and Mixed Methods
Approaches (2003). The qualitative data was read and the ideas expressed by the staff
were recorded. The information was organized into categories and themes were
identified. I described the themes that were developed from the questions and I then
interpreted the themes. The information that I gained from this part of the study was the
staff perception of the most effective interventions.
Delimitations and Limitations of the Study
This study was limited in its scope to an evaluation of the effectiveness of the
interventions at a single school. While there was published data regarding the Read 180
intervention program this study was delimited from a broader application by the small
number of students in the experimental group and the lack of randomized selection of
the subjects.
While selection bias was somewhat mitigated by the fact that the same students
were used in the pre - post measurement of the students participating in Read 180 and
traditional summer school there was still an internal validity problem because
participation in either program was not based on a randomized selection process. Those
63
that participated in the interventions are demonstrating motivation or the ability to travel
to the site of the interventions. Additionally those that were selected to participate had
an achievement level below the level of the EL average for the school. Not every
student that is qualified to participate in the intervention programs participates. Internal
validity is further compromised by the nature of the pre-post design and the variability
of the state wide control group. An additional variable was the changes in the state
scoring of the CST ELA examination. Student history, student motivation, Type 1 error,
and variance of the instrumentation were all factors that introduced the question as to
whether or not the treatments had the desired effect.
The external validity refers to this study’s potential for generalization. As stated
above, this study was limited to an evaluation of a single school. The external validity
was limited because of the presence of multiple treatments. While it may serve as a
model for other evaluations, the data generated by this study was not applicable to other
schools. The fact that there was variance in the tests, the tests were not vertically
equated was also an effect on this and other studies that examined growth using the
CST Test results.
The formative portion of this study helped provide valuable data on the
perceptions of the staff and involved them in the process of improvement. The feeling
of being involved in the solutions tends to be a motivating factor for a staff and a
motivated staff is a critical component of continued improvement.
While the study is limited in its application to the specific school and district
studied, the study served as a valuable tool in the evaluation of the interventions that
64
were being funded using categorical funding. While the district collects a great deal of
test data, and I have access to the current data and test history for more than 3500
students, we have not as a practice used available quantitative and qualitative data to
evaluate the programs that we were funding. Our past practice has been to use
anecdotal attributions of effect to a variety of programs at the sites. In a district that is
the lowest funded per ADA of the Unified Districts in San Mateo County, and one of
the lowest funded unified districts in the state, it is important to begin a practice of
measurement of the programs that we are implementing to insure that every dollar is
well spent.
65
Chapter 4
Results
The results of the study are in three parts: an analysis of the comparative school
level data, a comparison of the school’s results to the results of a similar school and the
state using the Academic Performance Index, (API), and the school level qualitative
data regarding the interventions acquired from surveying the staff.
School Level Summative Evaluation
The summative evaluation of the study incorporated three dependent variables
and a pre-post independent groups design. The three dependent variables were: CST
ELA scale scores, the CST performance band scores, and the percentage of students
scoring proficient or advanced on the CST ELA. The possible range of scale scores on
the CST is 150 to 600. Performance band scores were coded as follows: 0 = Far Below
Basic; 1= Below Basic; 2= Basic; 3=Proficient; 4=Advanced. Scores were compared
using paired pre-post samples. Four independent groups were compared: EL students
that received the Read 180 targeted intervention; EL students that attended summer
school; EL students that received no targeted intervention; the school-wide results
minus the EL students. Matched scores were used in paired samples. Students with a
score for only one year were excluded.
The study also uses a non-equivalent comparison group design for MFC
Intermediate School. The non-equivalent group design was used to compare the results
of the English Learners in grades 6-8 at MFC Intermediate School (experimental group)
and the state-wide results for the English Learner population. The non-equivalent group
66
design compares the change in CST ELA percentage scoring proficient or advanced as
well as the change in the API score for the EL subgroup.
1. Pre-post dependent groups design. This design was used to analyze and
compare the change for the four dependent subgroups from the 2005-06 (pre-
intervention) to 2006-07 (post-intervention). The following statistics were used for both
the CST ELA scale scores and the CST ELA performance band scores: (1) A dependent
groups t-test to assess the statistical significance of the change (p<.15); (2) Cohen’s d to
assess practical significance (criterion for practical significance (d>.20)); (3) Raw
change from 2006 to 2007 to assess practical significance; and, (4) Percentage change
to assess practical significance (criterion for practical significance; change >.10).
2. Nonequivalent comparison group design. This design used the results of the
2005-06 and 2006-07 CST ELA. This design included the local experimental group of
EL students and the disaggregated results for the EL subgroup in California as reported
on the California Department of Education Data Quest web site as a control group. The
statistics examined are the raw change in the percentage of EL students achieving
proficient or advanced scores and the calculated change in the API for the EL subgroup
in the local (experimental group) and the state (control group).
Pre-Post Independent Group Results
Table 12 shows the pre-post significant test findings using CST ELA scale
scores (p<.15) for four independent groups of EL students. Two groups received
targeted interventions as noted and the third group of EL students received no targeted
67
intervention, but was part of school-wide reform efforts. The non EL subgroup was
also included in the study. Paired scores were used for each sub-group of students.
Table 12
Pre-versus Post Intervention CST ELA Scale Score Differences: Statistical Findings
______________________________________________________________________
Grouping Pre N
2006
Post N
2007
Pre M
2006
Post M
2007
Difference T-Ratio Observed
p
Read 180
Intervention
28
28
294.82
295.85
1.03
-.237
.814
Summer
School
80 80 292.08 298.29 6.21* -1.668 .099
No targeted
intervention
143 143 324.84 324.75 -.09 .038 .970
School-
wide
Not EL
361 361 381.29 383.29 2.0 -1.209 .227
*=p<.150
68
Table 13 shows the pre-post significant test findings using ELA Performance
Band Scores (p<.15) for the same four independent groups of EL students. Paired
scores were used for each sub-group. Using the performance bands no group
demonstrated statistical significance.
Table 13
Pre-versus Post Intervention CST ELA Performance Band Differences: Statistical
Findings
______________________________________________________________________
Grouping Pre N
2006
Post N
2007
Pre M
2006
Post M
2007
Difference T-Ratio Observed
p
Read 180
Intervention
28
28
1.21
1.35
.14
-1.072
.293
Summer
School
80
80
1.25
1.31
.06
-.60
.551
No targeted
intervention
143
143
1.93
1.97
.04
-.576
.566
Schoolwide
Not EL
361
361
4.03
4.05
.02
-.577
.564
69
Table 14 displays the practical significance using the CST ELA scale score for
the same disaggregated groups, using the same matched scores. Using Cohen’s d to
determine effect size a significant effect size is d>.20. No group met this threshold for
significance. Measuring the raw change and calculating the percentage change also
showed no practical significance, change >.10.
Table 14
Pre-Post Intervention CST ELA Scale Score Differences: Practical Significance
______________________________________________________________________
Grouping Pre M Pre SD Pre-Post
Change
Effect Size
(Cohen’s d)
%
Change
Read 180
Intervention
294.82
36.79
1.03
.03
.00
Summer
School
292.08
43.95
6.21
.14
.02
No Targeted
Intervention
324.84
48.76
-.09
.00
.00
Schoolwide
Not EL
381.29
56.98
2.0
.04
.01
70
Table 15 shows the practical significance using the CST ELA performance band
differences for the same disaggregated groups, using the same matched scores. Using
Cohen’s d to determine effect size a significant effect size is d>.20. No group met this
threshold for significance. Measuring the raw change and then calculating the
percentage change the Read 180 group showed practical significance, change >.10.
Table 15
Pre-Post Intervention CST ELA Performance Band Differences: Practical Significance
Grouping Pre M Pre SD
Pre-Post
Change
Effect Size
%
Change
Read 180
Intervention
1.21
.92
.14
.15
.12*
Summer
School
1.25
1.09
.06
.05
.05
No Targeted
Intervention
1.94
1.08
.04
.03
.03
School-wide
Not EL
4.03
1.06
.02
.02
.00
*= %> .10
71
API Comparison
“The Academic Performance Index (API) is the cornerstone of California’s
Public Schools Accountability Act of 1999” (CDE 2007). The Academic Performance
Index (API) is the system that California designed to measure the growth of a schools
academic performance on an annual basis. “It is a numeric index (or scale) that ranges
from a low of 200 to a high of 1000” (CDE 2007). Each school is measured in relation
to the statewide goal of 800 for every school. Schools that are below 800 have annual
goals calculated by the state to progress toward 800, while schools that are at or above
800 there is no established goal but continued growth is measured. The API growth is
measured using the previous year score as the “API Base.” A school’s base score is
subtracted from the current year, “API Growth,” to determine how much a school
improved. In addition to the school-wide API score, each numerically significant
subgroup has an API score. The target for each subgroup is also 800 and the goals for a
subgroup are calculated in the same manner as school-wide goals are established. A
numerically significant subgroup is at least 100 students with a valid STAR Program
score or 50 or more students that make up 15 percent of the total valid STAR Program
scores. In addition to seven ethnic subgroups, Socio-economically disadvantaged,
English Learners, and Students with Disabilities produce disaggregated scores
(California Department of Education, 2007).
The API score is a summary of the statewide test results derived from the
Standardized Testing and Reporting (STAR) Program. Each test or content area is
weighted or emphasized. “Each schools content area weights are determined based on
72
test weights established by the State Board of Education (SBE) and on the number of
valid test scores in each content area and grade level at a school. API calculations result
in content area weights that may be slightly different for each school (California
Department of Education, 2007). Table 16 shows the test weights that apply to MFC
Intermediate School.
Table 16
2006 API Base Test Weights Grades 2-8
____________________________________________________________________
Content Area 2006-07 API Test Weights
CST in English-Language Arts, Grades 2-8 0.48
CST in Mathematics, Grades 2-8 0.32
CST in Science, Grade 5 0.20
CST in History-Social Science Grade 8 0.20
CAT/6 Survey in Reading, Grades 3 and 7 0.06
CAT/6 Survey in Language, Grades 3 and 7 0.03
CAT/6 Survey in Spelling, Grades 3 and 7 0.03
CAT/6 Survey in Mathematics, Grades 3 and 7 0.08
CST in Science, Grade 8 0.20
Assignment of 200, CST in Mathematics, Grade 8 0.10
Note: The assignment of 200 weight is assigned as scores for students who did not take
the CST in mathematics, grade eight.
______________________________________________________________________
73
“The API calculation method determines the API as the weighted average of
student scores across content areas and tests results within the school” (CDE). Using
the California Standards Tests (CSTs) which are a criterion referenced test, the API
calculation uses the performance band scores, (Advanced, Proficient, Basic, Below
Basic, or Far Below Basic) to determine a school’s scores. For the CAT/6 Survey test,
which is norm referenced, the national percentile rank (NPR) is used for API
calculations. Severely disabled students take the California Alternative Performance
Assessment (CAPA) which produces a performance band score and is used as the CST
is used. Using the methodology described above, “Each student’s test result is assigned
a performance level weighting factor of 200, 500, 700, 875, or 1000 based upon the
level of the test score” (CDE 2007). Table 17 shows weighting factors.
Table 17
API Performance Level Weighting Factors (CDE)
CST or CAPA
Performance
Levels
CAT/6 Survey
Performance
Levels
CAHSEE
Score
API
Performance Level
Weighting Factors
Advanced
80-99
th
NPR
Pass
1000
Proficient
60-79
th
NPR
N/A
875
Basic
40-59
th
NPR
N/A
700
Below Basic
20-39
th
NPR
N/A
500
Far Below Basic
1-19
th
NPR
No Pass
200
74
Of note is the fact that a school’s API growth receives the greatest benefit from
moving a student from the bottom performance band, Far Below Basic, up one band to
Below Basic, a 300 point difference while an improvement from Below Basic to Basic
provides a 200 point reward and the benefit to a school declines with the improvement
between each subsequent performance band.
In addition to measuring a schools annual growth, California uses the API to
rank schools and to compare similar schools. For the statewide ranking, first schools
are divided by school type, elementary, middle, and high school. Then schools are
decile ranked in ten categories from one (lowest) to ten (highest). API scores are
organized from lowest to highest statewide. A schools statewide rank is the decile
where the API scores occur. For similar schools rankings schools are again separated
by type. Then a School Characteristics Index (SCI) is determined for each school. The
SCI is determined by demographic factors such as percentage of students on free and
reduced lunch, parent education level, and percentage of EL students. Schools with
similar SCI factors are grouped into a comparison group of 100 schools. The schools
within the comparison group are ranked into deciles fro one (lowest) to ten (highest). A
school’s similar schools rank is the decile where the school’s API occurs in the
comparison group. (California Department of Education, 2007)
Table 18 shows MFC Intermediate School’s API in comparison to the state and
two schools identified as similar schools in 2006 (California Department of Education,
2006). It includes the API Base, the API Growth, the statewide rank and the similar
schools rank. In addition the EL subgroup API Base and API Growth is shown. The
75
schools chosen to compare are the closest similar schools on the list that have a
significant subgroup of EL students. It is interesting to note that the two schools listed
just above and just below MFC school on the similar schools website did not have a
significant population of EL students so therefore did not generate a comparison group
of EL students. It was necessary to move up three and down three on the list to find two
schools that were determined to be similar by the state and also had a significant EL
subgroup for comparison purposes (California Department of Education, 2007). Similar
schools for 2006 were used as the official similar school rankings for 2007 are not yet
available. The delay in the release of the official API results is due to the fact that the
California Department of Education is currently working on a method of recalibrating
the API results from 2007 to account for the addition of the California test for 2008, for
students with minor learning disabilities.
Table 18: Comparison of 2007 API Results
School
2006
API
Base
2007
API
Growth
2006
Statewide
Rank
2006
Similar
Schools
Rank
2006 EL
Subgroup
API Base
2007 EL
Subgroup
API
Growth
MFC
780
772 (-8)
8
6
625
602 (-23)
School 1
727
717 (-10)
6
2
508
603 (+95)
School 2
761
768 (+7)
7
5
621
633 (+12)
California
Grade 7-8
716
720 (+4)
N/A
N/A
618
623 (+5)
76
The negative API results for MFC School are congruent with the results noted in
the statistical measurement of the subgroups and the interventions used during the 2006
school year.
Qualitative Analysis
The teachers and administrators and the school were surveyed to determine their
impressions and experiences with the interventions used during the school year. The
school staff was provided with a survey instrument in September 2007. The survey
asked for staff to rank their impression of the intervention on a four point scale and to
complete open ended responses to their impressions of the interventions. The response
to the scaled questions was tabulated. Questions with no response were simply
excluded from the tabulations as some staff did not feel they had enough experience
with particular interventions to be qualified to respond. Open ended responses were
grouped into themes for analysis. The survey questions were as follows:
1. Rate the impact of the following on student achievement or learning.
2. Describe the level of impact you fell each program had upon student achievement or
learning.
Additionally, as a part of the open ended response the staff was asked to respond
to the following questions:
1. Describe the effectiveness of the staff development regarding “Strategies to Teach
EL Students?”
2. Describe the effects that walk-throughs had on instruction? Was there effective
feedback? Did they help to focus instruction on the standards for your subject area?
77
3. Are the students that participated in summer school known to the teachers?
4. Are the effects of summer school on student performance noticeable and/or
measurable? Describe.
5. Are the students that participated in Read 180 known to the teachers?
6. Are the effects of participation in Read 180 noticeable and/or measurable? Describe.
7. Describe how the “Networks for Success” (San Mateo County Office of Education)
contributed to school improvement.
8. In your opinion, what was the most effective intervention for the 2006 school year?
The programs to respond to were as follows:
1. Scholastic Read 180
2. Staff Development (Language Development in the Content Areas)
3. Network for Success Program Audit
4. After School Homework Center
5. Summer School
6. High Point ELD Curriculum
7. Standards Based Teaching
8. Principal Walk-throughs (Administrative monitoring)
9. Extreme Learning (Outside Supplementary Education Service Provider)
The responses to the first question are tabulated in Table 19. The number of
respondents is noted and those that did not respond were not calculated in the average
rating. There were a variety of reasons given for a non-response in the open ended
portion of the survey. The most common reason for those that did not respond stated
78
reasons such as did not work with this program. For example Scholastic Read 180 is
used in the language arts department and those that teach in other content areas did not
feel qualified to comment on the effectiveness of the program.
Table 19
Staff Rating of the interventions used during the 2006-2007 school year
1 = no impact; 2 = very little impact; 3 = moderate impact; 4 = significant impact
Intervention
Average
Rating
Number of respondents
(35 possible)
Scholastic Read 180
3.23
17
Staff Development (Developing Content
Vocabulary for EL Students)
3.32 28
Network for Success Audit 1.80 15
After School Homework Club 3.67 34
Summer School 2.76 26
High Point ELD Curriculum 2.34 6
Standards Based Teaching 3.37 34
Principal Walk-throughs 3.41 32
Extreme Learning 1.40 10
79
Combining the ranking of the interventions and the commentary by the staff
regarding the interventions it was quite clear that the interventions divided into three
groups. The first group was the interventions associated with the school being
identified as a Program Improvement (PI) school. The second group is the interventions
that are perceived as district led interventions. The third group is the interventions that
are perceived as site based or site determined interventions.
There were two programs that were required by Title I, No Child Left Behind
(NCLB) regulations the Supplemental Education Service (SES) provider, Extreme
Learning, and the outside consultant, Networks for Success. There is a current of
resentment by the staff regarding the PI status of the school and a reflection of the
resentment is that the two intervention programs required by PI, received the lowest
rating by the staff. The SES provider is a tutoring service. It is required of PI schools
that they enter into a contract with an outside agency to provide tutoring or
supplementary educational services to students that are eligible for Title I services.
Extreme Learning hired local substitute teachers as tutors and provided after school
tutoring to students on the campus. Networks for Success was the name of the outside
consultant that the school was required to hire to lead the staff through an academic
program audit. The comments from staff regarding Networks for Success were
particularly disparaging, one staff member commented that the process was a
‘humiliating experience.” The rankings for these two interventions was < 2 on a scale
with 1 = no impact and 2 = very little impact.
80
The second group of interventions, those perceived as district led, is the next
lowest ranked group with both interventions receiving a ranking less than three (3).
This group included summer school and the High Point ELD curriculum. The
comments regarding summer school were varied. Some saw the program as just a
mandated program that does little to help students. Others made specific
recommendations regarding the teacher quality and curriculum for summer school while
still other commended the program for having students ready to learn in the fall.
The third group of interventions that were perceived by the staff as site
developed interventions. The staff ownership of programs that they developed is
indicated by the very high ratings, >3 for all of these interventions. It was interesting
that the lowest ranked of these programs was the Scholastic Read 180 intervention
which achieved a ranking of 3.23 from the staff familiar with the program. The Read
180 intervention is the only one of the interventions in this group that is not school-wide
in nature. Many of the staff responded that they were unfamiliar with the program and
only those that worked with the program could identify the students that were involved
in the program.
Of the site based interventions the Homework Club received the most favorable
comments and was appreciated by all of the staff. The Homework Club that received
the highest ranking 3.67 of all of the interventions is an after school tutoring program
that uses high school students as tutors. The homework club is an intervention that is
open to all students everyday after school on a drop in basis. It is interesting that the
Homework Club (3.67) with its high school tutors was ranked more than twice as high
81
as Extreme Learning (1.40) which used adults as tutors. In addition the Homework Club
received a ranking from every staff member while Extreme Learning was rated by less
than one-third of the staff members. I believe this was due to the fact that homework
club focuses on homework completion which is an immediate result seen by teachers
while Extreme Learning focused on skills tutoring which did not produce an immediate
result. In addition Extreme Learning cost four times what the Homework Club cost to
operate and the money to pay for Extreme Learning came out of the school’s Title I
budget which the staff felt could have been better spent on other programs or materials
to support teachers. The fact that the Extreme Learning program was a result of the
school’s PI status was also a very negative influence on the view of the program.
The Principal’s walk-throughs (3.41) were also ranked very high by the staff
which based on the comments is a reflection that the staff appreciated feeling that the
administrators are part of the team and the high esteem that the staff holds for the
principal.
The qualitative data from the survey and discussions with staff helped to provide
a clear picture of the issues that face the school as it works to improve achievement for
EL students.
82
Chapter 5
Summary, Discussion and Recommendations
Summary
Quantitative Summary
One look at the 2007 growth API for the EL subgroup gives testimony that the
2006-2007 school year was not a successful year for raising the achievement of the El
students at MFC Intermediate School. The matched results showed that the impact
from the targeted interventions was small and the school-wide efforts to address EL
achievement as measured by the growth of the control groups were not successful at
raising EL CST ELA scores.
The targeted intervention Scholastic Read 180 did not demonstrate a statistical
impact on either the measure of the scale scores or the performance band scores. The
Read 180 did not demonstrate practical impact on the CST ELA scale score. The Read
180 program did however demonstrate practical significance of a gain greater than 10%
with a gain of 12%, on the CST ELA performance band score. Even with the increase,
the post test mean for the Read 180 sub-group is still in the lowest performance band,
Far Below Basic. The Read 180 sub-group did out perform the EL students with no
intervention sub-group in all four measures and outperformed the school-wide non EL
subgroup in two of the four measures.
The targeted intervention of summer school had similar results to the first
experimental group. The summer school intervention groups demonstrated statistical
83
significance on the CST ELA scale score. The summer school group had an observed p
of .099 is less than .150 which meets the accepted statistical standard for significance.
The scale score growth for the summer school group of 6.21 points was more than triple
the growth of any of the other three sub-groups. The summer school intervention group
outperformed both control groups, the EL students that received no targeted
intervention and the school-wide non-EL students in each of the four measures.
The control group of EL students that received no targeted intervention had the
worst growth performance on the two of the four measures. This group’s growth
performance was worse than both of the experimental groups on every measure. The no
intervention EL group did outperform the school-wide non-EL group on two of the
measures. This control group did not meet the standard for statistical or practical
significance in any of the four measures.
The school-wide non-EL control group had the lowest performance in two of the
measures. This group did slightly outperform the Read 180 subgroup in the two scale
score measures. This subgroup did not meet the standard for statistical of practical
significance in any of the four measures.
The two experimental groups and two control groups provided interesting
comparisons. The summer school experimental group was the best performing group.
Only the two experimental groups achieved either the standard for statistical
significance or practical significance on any of the measures. The summer school
intervention group could be considered successful as compared to the control groups as
it outperformed the other three groups in all four measures. The Read 180 intervention
84
also outperformed the EL students that did not receive any intervention on each of the
four measures so could also be considered a success compared to the control group of
similar students. While in the comparison to similar EL students both experimental
groups demonstrated measurable improvement, caution must be exercised in claiming
success as in three of the four measures the pre X post difference could be due to
chance.
The comparison to the statewide API subgroup growth and the school-wide and
EL sub-group growth of similar schools was not favorable to the school. The negative
API growth school-wide (-8) and for the EL sub-group (-23) demonstrates that school-
wide efforts did not positively affect student achievement, particularly the EL student
achievement. The two comparison schools and the statewide results all showed positive
growth with the EL subgroup in contrast to MFC’s results. The state showed a growth
of five (+5) points on the API for the EL subgroup, Similar School 1 showed a growth
of ninety five (+95) points on the API for the EL subgroup and Similar School 1
demonstrated growth of twelve (12) points for the EL subgroup.
An indicator of the significance of this growth is that with the 2006 EL subgroup
API, MFC Intermediate School had the highest score of the four groups with an API
score of 625 for the EL subgroup. With the 2007 results, MFC School had the lowest
API score of 602 for the four comparison groups. The shift was most significant
between MFC School and Similar School 1. For 2006 MFC School had an EL API of
625 while Similar School 1 had an EL API of 508 by far the lowest of the four groups.
85
For 2007, MFC School had an EL API of 602 while Similar School 1 had an API of 603
for the EL subgroup.
The school-wide API for MFC School also declined by eight (-8) points.
Interestingly Similar School 1 also had an API school-wide decline of ten (-10) points
even with the significant EL subgroup growth. Similar School 2 and California both
had positive overall API growth. Even with the negative growth MFC School still had
the highest overall API for both years with an API score of 780 in 2006 and of 772 in
2007. The score is high for a middle school as shown by the statewide rank of an eight
and an overall score close to the statewide API target of 800.
The API scores for MFC School do illustrate a problem. While the experimental
intervention groups showed some improvement the school-wide achievement scores
dropped. The staff development and training teachers received and the use of identified
strategies to increase student achievement did not have the desired effect.
Qualitative Summary
The discussions and feedback from staff provide some insight into what
happened during the 2006-2007 school year. Looking at the rankings and comments for
the identified interventions there are some trends that become evident.
The first trend that is linked by comments and the staff rankings of the
interventions is resentment over Title I and the school’s Program Improvement (PI)
status. There are two interventions that are a part of Title I requirements and both
received the lowest scores from the staff. The first is the required supplementary
educational service provider, Extreme Learning. Supplementary educational service is
86
a tutoring program beyond the regular school day that is contracted with an outside
agency using Title I funds. Many of these private tutoring companies are growing due
to the Title I regulations requiring PI schools to contract with SES providers. Extreme
Learning employed local substitute teachers as tutors and operated a homework center
on the campus after school. The cost of the program was approximately forty thousand
dollars. Comments about the program were all similar to this statement, “The nicest
thing I can say about Extreme Learning is that it was a waste of money.” Another
comment was that Extreme Learning was “a waste of resources with no positive impact
on student achievement.” The second intervention was the Network’s for Success
School Audit Team. As a school in the third year of PI status it was required that the
school contract with an outside agency to lead the staff through a self review of the
academic program at the school. The purpose is to identify areas of strength and
weakness so that the staff can focus the resources on addressing the weaknesses. There
Comments about this program were negative. One comment was particularly harsh,
calling the audit process, “The most awful and ridiculous experience of my teaching
career. It was ridiculous to have entire programs assessed on less than ten minutes of
classroom observation per teacher.” Another comment said, “This was a long tedious
process that resulted in feedback that we [the staff] explained to the audit team at the
beginning.” The rankings of these two programs based on the perceived impact on
student achievement 1=low impact and 4=high impact was below 2 for both. Extreme
Learning’s average ranking was 1.40 and the Network for Success Audit Team’s
87
average ranking was 1.80. Only the two outside providers were universally perceived
negatively.
The second trend is the grouping of the interventions that are perceived as
district interventions. Two interventions, the selection of High Point as the ELD
curriculum and the operation of summer school are perceived as district programs and
received the next lowest rankings from the staff. The High Point curriculum received
an average rank from the staff of 2.34 and summer school received an average rank of
2.76. The comments on High Pont were mixed. One comment described the material
as “adequate but not excellent material” while another said that “…this program does
not best serve our students. I feel that the materials lack connection to the students.”
Summer school had mixed comments. One very simply stated that there was
“minimum improvement in written language and math.” Another comment said that
summer school helped students that were making a transition for special education
classes to regular education classes. Another commented that the teachers of summer
school are “…not necessarily asked to teach a subject they’re competent in.” On the
positive side, another comment was that “most of my students participate and are better
prepared to return each year because of it.” One staff member commented that it is
needed but should be restructured so that classes are offered to address specific skill
such as reading comprehension, writing conventions, or responding to literature.
Finally the third grouping of the interventions is the group of interventions that
were generated by the school. Each of the interventions generated at the school level
88
had an average rating above 3.00 in fact the lowest average ranking was for the Read
180 intervention which received an average rating of 3.23 from the staff that offered an
opinion. The highest ranking 3.67 was for the after school homework club which has
been a program that has operated for several years. The homework club also had a
response from every staff member as did the principal monitoring walk-throughs which
received an average ranking of 3.41 and which was the second highest average rank.
Staff development on academic language also achieved a high average rank of 3.32
while standards based teaching received a rating of 3.47. The high rankings for the staff
generated interventions speak to an overall attitude from the staff that the PI status is
unfair, that the staff is doing a great job and that it isn’t the staff’s fault that the EL
students are disadvantaged and aren’t ready for the material or don’t have the support at
home to complete homework.
The praise for the homework club is also an indicator of the importance of
homework in the schools culture. Some teachers use homework as fifty percent of a
student’s grade. Comments such as “Homework Club is significant for students whom
otherwise wouldn’t have the support,” and “This [homework club] allows some students
who might have a difficult time completing homework at home the opportunity to
complete work and get additional help” demonstrate the value of the intervention but
also indicate that the beliefs and attitudes about the students. The homework club has
been a feature of the school for more that a twelve years. The emphasis on homework
has served the school as a part of the grade as served the school well for many years as
indicated by the popularity. However reliance on the homework club with its high
89
school student tutors to help students succeed may have to give was to an examination
by the staff on the emphasis of homework as a part of the grade or teachers making sure
that the students can perform the homework before leaving the classroom. It was the
difficult discussions involving an analysis of practice that made the Network for
Success Program Audit such an “awful” and “ridiculous” process for the staff.
Discussion
An additional factor that was not controlled for in this study was teacher
efficacy. It must be noted that during the 2006-2007 school year a new teacher was
hired and was assigned to teach English Language Development (ELD). As the ELD
teacher many of the EL students were exposed to the teacher. Unfortunately as a new
and inexperienced teacher there were significant instructional performance issues that
resulted in the release of the teacher at the end of the year. The practice of assigning the
most inexperienced teachers to the students of greatest need is not a practice unique to
this school. Beginning teachers are disproportionately assigned to teach low-income,
high minority students in lower track classes who most need skilled teachers (National
Commission on Teaching and America’s Future, 1996). A new teacher isn’t always a
problem as some new teachers are excellent however in this situation the use of a new
teacher was a problem as evidenced by the teacher’s release. Teaching will be further
addressed in the recommendations.
Richard Rothstein a research associate at the Economic Policy Institute and an
adjunct lecturer at the Harvard School of Education spoke at length about the racial
achievement gap and the use of criterion referenced tests during a roundtable discussion
90
with Hoover Institute Fellow, Chester Finn during the California Achievement Gap
Summit in November 2007. Rothstein described underlying fallacies associated with
the achievement gap that were strictly associated with the measurement tools. He
criticized the movement to criterion referenced tests and described how the gap on a
criterion referenced test is directly related to where proficient is defined, and that the cut
score can move. To highlight the arbitrary nature inherent in a criterion referenced test
he provided two extreme examples, at one extreme end a fourth grade test that only
tested single digit addition would have no achievement gap as everyone would do well
or a fourth grade test that tested differential calculus would have no achievement gap as
no one would do well. He spoke of states that are playing with the gap simply by
changing the cut scores required. Rothstein said that before we can really discuss the
achievement gap we must first accurately measure the gap using a norm referenced test.
Then to have meaningful discussions to evaluate differences then the disaggregated
groups need to be very discreetly organized. For example rather than looking at
economically disadvantaged students as all students that receive free and reduced lunch
that this group needed further breakdowns within the group as the range of that group is
from 185% of the poverty line to literally having no income. He suggested that the
recent announcements that race not income status was a more determinant factor in test
scores would not be supported by a closer analysis as it is likely that many of the white
students receiving free and reduced lunch were very likely grouped near the 185% level
and many of the African American and Latino students were probably grouped near the
bottom of the scale. Again Rothstein’s point is that the well publicized analysis of
91
educational results by the federal government, by the states and by schools and school
districts is flawed because we are not accurate with our measures (Rothstein, 2007).
In my study I found that there was great accuracy in Rothstein’s statements, and
I have to agree that a norm-referenced test would be a valuable tool. Finding out that
the percentage proficient had changed for each grade level between 2006 and 2007
created significant validity issues for the study. The California Department of
Education (CDE) recognizes the measurement issues in their system and on the paper
printout of test score results sent out annually CDE states that the scores are not suitable
for the examination of cohort groups and are meant to compare the same grade level in
different years, for example the seventh grade of 2006 compared to the seventh grade of
2007 (California Department of Education, 2007). However with the percentages
changing between the two years even this type of comparison is flawed. Also cohort
analysis is a critical tool of measurement as the change over time of a cohort group is an
important evaluative tool. Analysis of cohort groups is also important as additional
information when looking at the same grade in different years as one year’s grade could
have different demographic characteristics that would also affect any comparison
between the grades.
While understanding there are limitations in the measurement CST ELA it is
still an expectation of my position to analyze cohort groups as part of my annual reports
to the governing board and to use this analysis to help to determine the effectiveness of
our programs. Using matched scores provided some mitigation to the problems created
by using the CST ELA because although the students were grouped each student’s
92
result was individualized for analysis. Additionally while the CST ELA percentages for
proficient changes could have an effect on the performance band scores the scale scores
are not affected by those changes by CDE.
The research related to the two targeted interventions, summer school and the
Scholastic Read 180 Program is consistent with the results of this study. The literature
regarding best practices for schools has significant implications for planning the next
steps to address the EL subgroup’s learning needs.
Research on Read 180 tended to focus on the difference of the experimental
group from a control group. The results by Papelewis (2003) were interesting in that it
used a norm referenced test which as previously discussed is a superior measure for the
purposes of statistical analysis. Her analysis demonstrated a significant gain three
Normal Curve Equivalents (NCEs) for reading and two NCEs in language arts. For
both subject tests the observed gain p<.05 was significant. This was a more significant
result than was found in the study of MFC School.
While summer school did not achieve the standard of statistical or practical
significance on three of the four measures used, summer school as an intervention did
have a measurable difference as compared to the EL students that did not receive any
intervention. The following comment about summer school illuminates one of the
positive attributes of summer school for the EL students “It [summer school] supports
English acquisition. The teachers reported a positive effect…” Both the Basic
Interpersonal Communications Skills (BICS) and the Cognitive Academic Language
Proficiency (CALP) are reinforced by the added practice of English usage during
93
summer school (Cummins 1994). This additional practice during the summer may not
lead to academic growth during the summer session but it may stem the regression that
researchers Allington and McGinn (2003) claim has a large effect on the achievement
gap. The data from summer school did indicate that the students in the summer school
experimental group made one year’s growth or more on the CST ELA. Using the CST
ELA as the measure the students that attended summer school did show slight
improvement in their scale score and their performance band score which indicates at
least one year of academic growth. According to the research, if summer school
prevents regression it has accomplished an important mission in closing the
achievement gap and in this case the data seemed to indicate that the MFC summer
school succeeded in that endeavor.
The work done by Robert Marzano and others on identifying the best classroom
practices to improve learning has significant implications for MFC. While many of the
practices that are described in the book Classroom Practices that Work have been
implemented at the school, the best practices are not school wide practices but are rather
islands of effective teaching. The principal’s walk-throughs that were identified by the
staff as effective for students begin to address the issue of creating school-wide
practices but have not gone far enough to address the system wide implementation of
effective practices. Currently the walkthroughs are focused on one element of
teaching; is the standard being taught in the lesson clearly stated and written for on the
board for students. This is a fairly easy task to monitor for the principal and to comply
with by the teacher and identifying what is to be learned is also effective for students. It
94
would be interesting if the staff would rank the walk-throughs as highly if there was
additional accountability attached to them.
Recommendations
While the Scholastic Read 180 program has demonstrated success in other
settings it had limited success among the target population at MFC School. It is a very
expensive program and it is not a replacement for direct instruction. To be effective it
must be used in conjunction with effective teacher led literacy instruction as a
supplementary program. My recommendation for the school regarding Read 180 is to
continue to use the program but to provide the staff using it with the training in Whole-
Class Instruction, Instructional Reading; Small-Group Instruction; Modeled and
Independent Reading and Whole Class Wrap-Up as recommended by the publishers
(Papelewis, 2003).
Reading the research on the disproportionate effect of the summer vacation on
socio-economically disadvantaged students (SED) brought to mind the need for a
further investigation: Are single track year round schools more successful with
improving the achievement of SED students than schools on the traditional calendar?
Research seems to indicate that students on the year round track do perform better.
While this is not an issue that can be addressed at the school site maybe it is time if the
state is serious about the achievement gap than it maybe necessary to make some
system wide changes in our school system. At the school level summer school must
expand to reach more students that would benefit from additional instruction. In
addition summer school and academic summer programs for grades Kindergarten
95
through eighth grade must insure that there is a volume of high success reading
activities throughout the school day. In addition children must have easy access to
books that are engaging and provide successful reading experiences (Allington, McGill-
Franzen 2003).
“Education is increasingly important to the success of both individuals and
nations, and growing evidence demonstrates that-among all educational resources-
teachers abilities are especially crucial contributors to students’ learning” (Darling-
Hammond & Bransford, 2005). It is not possible to quantify the effect of a single
ineffective teacher however the sharp drop in the API for the EL subgroup seems to be a
strong indicator of the power of a teacher. The release of a teacher that was evaluated
as ineffective was done and additional steps were taken this year to address the EL
learning needs. A zero period ELD class was created for students that had received a
score of BASIC on the CST ELA. By offering the ELD at zero period the students have
a mainstream language arts placement during the regular school day. These students
were grouped into four classes and four experienced, positively evaluated teachers were
given the assignment for these students. A new curriculum was purchased and the
teachers received on-going coaching throughout the year in the use of the curriculum
and effective pedagogy for English Learners. Additionally the teachers that were
assigned to teach the EL students for their language arts core were experienced teachers
that have received very positive evaluations.
I believe two key elements in raising EL student achievement are to put into
96
practice the strategies identified in Classroom Instruction that Works (Marzano, et.al,
2001) and to add a greater sense of accountability to the implementation of the
strategies during the principal walk-thorughs. One of the best things about the
strategies is that they are free. There is no cost to the implementation of strategies such
as: similarities and differences; note taking and summarizing; reinforcing effort and
providing recognition; homework and practice; nonlinguistic representations;
cooperative learning; setting objectives and providing feedback; generating and testing
hypothesis; questions, cues and advance organizers. Many of these strategies have been
components of the professional development that the staff has received. In an interview
with the principal however he admitted that in a discussion with his staff during a
meeting that approximately fifty percent of the teachers said that they had implemented
the strategies that they had been taught. This is clearly unacceptable if the school is
going to make the changes necessary to improve EL student achievement.
While it would not be recommended to begin the implementation of all nine
instructional strategies identified by Marzano et. al (2001) all at once, an analysis of
which practices are already institutionalized such as homework, reinforcing effort and
providing recognition then refine those current practices to be more effective based on
the research of what is effective about those practices. For example homework is a
significant feature about MFC; its importance is reinforced at almost every parent and
student meeting. But the process of giving homework assignments is not consistent
across the curriculum. Some teachers grade of give feedback on homework while
97
others simply check that it is completed. The meta-analysis in Classroom Instruction
That Works (2001) shows that graded homework or homework with feedback has nearly
three times the percentile gain when compared to homework that is merely assigned.
As homework is a prominent feature of the school it is important to make sure that it is
utilized well.
In addition to refining strategies already in place the staff needs to be involved
in choosing additional strategies such as identifying similarities and differences and
summarizing and note taking and provide professional development in the use of these
strategies across the curriculum. Both of these strategies are currently employed
sporadically across the curriculum and should be structured so there is consistent
implementation.
A key element of change is to use the principal’s walk-throughs to actively
monitor the implementation of the strategies that are effective. The principal must be
able to hold the teachers accountable for the implementation of best practices. The
walk-throughs can also be a coaching tool to provide constructive feedback to teachers
to improve their practice. Part of my role will be to provide a tool to measure the
implementation of key instructional strategies and concepts. As the staff becomes
proficient in a strategy additional best practices are implemented. Change will begin
when teacher performance is monitored to the same degree as the student data.
98
References
Allington, R. & McGill-Franzen, A. (2003) The impact of summer setback on
the reading achievement gap. Phi Delta Kappan. p. 68-75 Vol. 85, No. 1.
Anderson, J. R. (1995). Learning and memory: an integrated approach. New York:
Wiley and Sons.
Anderson, L. W. & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and
assessing a revision of bloom’s taxonomy of educational objectives.
New York, NY: Addison Wesley Longman Publishing Company Inc.
Borman, G. D. & D’Agostino, J. V. (1996). Title I and student achievement: a
meta-analysis of federal results. Educational evaluation and
policy analysis vol. 18.
Brennen, T., Leuer, M., Boyer, D. & Dalessi, M. (2006). Rhetoric to reality:
addressing reading achievement in secondary education. Anaheim
Union High School District. http://teacher.scholastic.com/products/read180
Calderon, M. (2007). Teaching reading to english language learners grades 6-12
Thousand Oaks, CA: Corwin Press.
Caine, R. N. & Caine, G. (1991). Making connections: teaching and the human brain.
Menlo Park, CA: Addison-Wesley.
California Department of Education Accountability Progress Reporting. (2006).
cde.ca.gov/ta/ac/ar/
California Department of Education Accountability Progress Reporting. (2006).
cde.ca.gov/ta/ac/ar/
Chamot, A. U. & O’Malley, M. J. (1994) The calla handbook:. Implementing the
cognitive academic language learning approach. New York, NY: Addison
Wesley Longman Publishing Company Inc.
Clark, R. & Estes, F. (2002). Turning research into results. Atlanta, Georgia:
Center for Effective Practice, CEP Press.
Covington, M.V. (1983). Motivation cognitions. In S. G. Paris, G. M. Olsen, & H. W.
Stevenson (Eds.), Learning and motivation in the classroom. Hillsdale, NJ:
Lawrence Erlbaum
99
References
Creswell, J. (2003). Research design: qualitative, quantitative, and mixed methods
approaches 2
nd
edition. Thousand Oaks, CA: Sage Publications Inc.
Cruncher School Academic Data Base. (2007). Clovis, CA: Lighthouse Software
Publications.
Cummins, J. (1979) Cognitive / academic language proficiency, linguistic
interdependence, the optimal age question and some other matters. Papers on
bilingualism, No. 19. Toronto, Canada: Ontario Institute for Studies in
Education.
Cummins, J. (1994) Knowledge, power, and identity in teaching english as a
second language. Educating second language children (pp 33-58). New York,
NY: Cambridge University Press.
Darling-Hammond, L. & Bransford, J. (Eds). (2005). Preparing teachers for a
changing world: what teachers should learn and be able to do.
San Francisco, CA: Josey Bass, John Wiley and Sons.
Denbo, M. & Eaton, Martin. (2000) Self regulation of academic learning in middle
level schools. The elementary school journal. vol. 100, No. 3.
Dong, Y. R. Learning to think in english. Best of educational leadership
2006-2007. Alexandria, VA: Association for Curriculum and Development.
Freeman, Y. S., Freeman, D. E. & Mercuri, S. (2002). Closing the achievement gap
how to reach limited-formal-schooling and long-term
english learners. Portsmouth, NH: Heineman Publishing.
Hayes, D. P. & Grether, J. (1983). The school year and vacations: when do
students learn? Cornell Journal of Social Relations. Vol. 17
Hill, J. D. & Flynn, K. M. (2006). Classroom instruction that works with english
language learners. Alexandria, VA: Association for Curriculum and
Development.
Kohn, A. (1993). Punished by rewards: the trouble with gold stars, incentive
plans, a’s, praise, and other bribes. New York, NY: Houghton Mifflin.
Krashen, S. & Terrell, T. (1983). The natural approach: language
acquisition in the classroom. Hayward, CA: Alemany Press.
100
References
Lachet, M. A. (1999). What policymakers and school administrators need to
know about assessment reform for english language learners. Providence,
RI: Northeast and Islands Regional Educational Laboratory at Brown
University.
Marzano, R. J. (2003). What works in schools: translating research into action.
Alexandria, VA: Association for Curriculum and Development.
Marzano, R. J., Pickering, D. J. & Pollock, J. E. (2001) Classroom instruction
that works research based strategies for increasing student achievement.
Alexandria, VA: Association for Curriculum and Development.
Networks for Success. (2006). MFC intermediate school audit. San Mateo County
Office of Education Program Improvement Intervention Team.
Newell, A. & Rosenbloom, P.S. (1981). Mechanisms of skill acquisition and the law of
practice. In J. R. Anderson (Ed.) Cognitive skills and their acquisition.
Hillsdale NJ: Erlbaum and Associates.
Olsen, L. & Jaramillo, A. (1999). Turning the tides of exclusion: A guide for
educators and advocates for immigrant students. Oakland CA: California
Tomorrow.
Olsen, L. & Romero, A. (2006). California tomorrow. Oakland, CA: Los Angeles
County Office of Education.
Parrish, T.B., Merickel, A., Perez, M., Linquanti, R., Socias, M., Spain, A. (2006).
Effects of the implementation of proposition 227 on the education of english
learners, K-12: findings from a five year evaluation. Palo Alto, CA: American
Institutes for Research.
Papaleweis, Richard. (2003). Final Report: A study of read 180 in middle schools in
clark county school district las vegas nevada. Scholastic professional paper.
New York, NY: Scholastic, Inc.
Papaleweis, Rosemary. (2004). Strugggling middle school readers: successful,
accelerating intervention. Scholastic professional paper. New York, NY:
Scholastic Inc.
Rothestein, R. (2007). Closing the achievement gap. California achievement gap
summit. http://www.closingtheachievementgap.org/cs/ctag/print/htdocs/
summit_2007_keynote.sp.htm
101
References
Waits, M.J., Campbell, H.E., Gau, R., Jacobs, E., Rex, T. and Hess, R.K. (2006). Why
some schools beat the odds and others don’t. Arizona State University:
Morrison Institute for Public Policy, School of Public Affairs, College of Public
Programs.
Williams, T., Hakuta, K., Haertel, E. (2007). Similar english learner students,
different results: why do some schools do better? Mountain View, CA:
Ed Source.
Wright, Wayne E. A catch-22 for language learners. Best of educational leadership
2006-2007. Alexandria, VA: Association for Curriculum and Development.
Zadina, Janet. (2001). Using brain research to enhance & energize instruction a
workbook for educators. New Orleans, LA: Tulane University School of
Medicine.
Abstract (if available)
Abstract
This study reviews the effect of two targeted intervention programs used in a California middle school to improve English Learner achievement. The study uses an O pre X O post design to compare the effects of the interventions for the English Learner students that received the interventions to English Learner students, and native English speaking students that did not receive the targeted intervention. The study uses matched scores on the 2006 and 2007 California Standards Test in English Language Arts (CST ELA) to compare the disaggregated subgroups described, (dependent groups) and also to compare the schools results for English Learners to 2 similar schools and the state results. The study measures for statistical significance and practical significance. The literature review examines learning issues and best instructional practices for English Learners as well as the two targeted interventions. The study concludes that the CST ELA criterion referenced system creates internal validity issues that make it difficult to determine significance in a study comparing two different years. Comparisons of the groups show that there was a greater increase for the students that received the targeted intervention although the difference was small. The study also concludes that while the school attempted to implement many of the research based, best instructional practices school-wide, the monitoring of the implementation must be increased to improve the use and implementation of the effective instructional practices by the staff.
Linked assets
University of Southern California Dissertations and Theses
Conceptually similar
PDF
Evaluation of the progress of elementary English learners at Daisyville Unified School District
PDF
The comparison of hybrid intervention and traditional intervention in increasing student achievement in middle school mathematics
PDF
The effect of a language arts intervention on the academic achievement of English language learners
PDF
English learners' performance on the California Standards Test at Aviles Elementary
PDF
The effectiveness of the cycle of inquiry on middle school English-learners in English-language arts
PDF
The effectiveness of the literacy for success intervention at Wilson Middle School
PDF
The effects of open enrollment, curriculum alignment, and data-driven instruction on the test performance of English language learners (ELLS) and re-designated fluent English proficient students ...
PDF
An evaluation of the impact of direct instruction intervention on the academic achievement of English language learners
PDF
Alternatives for achievement: a mathematics intervention for English learners
PDF
Teacher perceptions of instructional practices for long-term English learners
PDF
The impact of restructuring the language arts intervention program and its effect on the academic achievement of English language learners
PDF
Narrowing the achievement gap: Factors that support English learner and Hispanic student academic achievement in an urban intermediate school
PDF
Effective reading instruction for English learners
PDF
English language learners utilizing the accelerated reader program
PDF
An analysis of the impact of the total educational support system direct-instruction model on the California standards test performance of English language learners at experimental elementary school
PDF
The impact of professional learning communities and block scheduling on English language learner students at Oak Point Middle School
PDF
The implementation of strategies to minimize the achievement gap for African-American, Latino, English learners, and socio-economically disadvantaged students
PDF
Raising student achievement at Eberman Elementary School with effective teaching strategies
PDF
An evaluation of the impact of a standards-based intervention on the academic achievement of English language learners
PDF
The implementation of studio intervention at a medical magnet high school
Asset Metadata
Creator
Bachicha, Michael C.
(author)
Core Title
Examining the effectiveness of the intervention programs for English learners at MFC intermediate school
School
Rossier School of Education
Degree
Doctor of Education
Degree Program
Education (Leadership)
Publication Date
07/30/2008
Defense Date
06/17/2008
Publisher
University of Southern California
(original),
University of Southern California. Libraries
(digital)
Tag
middle school English learners,OAI-PMH Harvest
Place Name
California
(states)
Language
English
Advisor
Hocevar, Dennis (
committee chair
), Hentschke, Guilbert C. (
committee member
), Stewart, John (
committee member
)
Creator Email
bachicham@cabrillo.k12.ca.us
Permanent Link (DOI)
https://doi.org/10.25549/usctheses-m1461
Unique identifier
UC1127631
Identifier
etd-Bachicha-20080730 (filename),usctheses-m40 (legacy collection record id),usctheses-c127-90324 (legacy record id),usctheses-m1461 (legacy record id)
Legacy Identifier
etd-Bachicha-20080730.pdf
Dmrecord
90324
Document Type
Dissertation
Rights
Bachicha, Michael C.
Type
texts
Source
University of Southern California
(contributing entity),
University of Southern California Dissertations and Theses
(collection)
Repository Name
Libraries, University of Southern California
Repository Location
Los Angeles, California
Repository Email
cisadmin@lib.usc.edu
Tags
middle school English learners