University of Southern California Dissertations and Theses

A Paradigm For The Implementation Of Accountability Measures In Bilingualeducation

(USC Thesis Other)

A Paradigm For The Implementation Of Accountability Measures In Bilingualeducation

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content INFORMATION TO USERS
This material was produced from a microfilm copy of the original document. While
the most advanced technological means to photograph and reproduce this document
have been used, the quality is heavily dependent upon the quality of the original
submitted.
The following explanation of techniques is provided to help you understand
markings or patterns which may appear on this reproduction.
1. The sign or "target" for pages apparently lacking from the document
photographed is "Missing Page(s)". If it was possible to obtain the missing
page(s) or section, they are spliced into die film along with adjacent pages.
This may have necessitated cutting thru an image and duplicating adjacent
pages to insure you complete continuity.
2. When an image on the film is obliterated with a large round black mark, it
is an indication that the photographer suspected that the copy may have
moved during exposure and thus cause a blurred image. You will find a
good image of the page in the adjacent frame.
3. When a map, drawing or chart, etc., was part of the material being
photographed the photographer followed a definite method in
"sectioning" the material. It is customary to begin photoing at the upper
left hand corner of a large sheet and to continue photoing from left to
right in equal sections with a small overlap. If necessary, sectioning is
continued again — beginning below the first row and continuing on until
complete.
4. The majority of users indicate that the textual content is of greatest value,
however, a somewhat higher quality reproduction could be made from
"photographs" if essential to the understanding of the dissertation. Silver
prints of "photographs" may be ordered at additional charge by writing
the Order Department, giving the catalog number, title, author and
specific pages you wish reproduced.
5. PLEASE NOTE: Some pages may have indistinct print. Filmed as
received.
Xerox University Microfilms
300 North Z eeb Road
Ann Arbor, M ichigan 48106
ff
74-21,513 f
TRDY, Bernard Allan, 1934- !
A PARADIGM FOR THE IMPLEMENTATION OF
ACCOUNTABILITY MEASURES IN BILINGUAL
EDUCATION.
University of Southern California, Ph.D., 1974
Education, psychology
University Microfilms, A X ER O X Com pany, Ann Arbor, Michigan
THIS DISSERTATION HAS BEEN MICROFILMED EXACTLY AS RECEIVED.
A PARADIGM FOR THE IMPLEMENTATION OP
ACCOUNTABILITY MEASURES IN BILINGUAL EDUCATION
by
Bernard Allan Troy
A Dissertation Presented to the
FACULTY OP THE GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
(Educational Psychology)
June 1974
UNIVERSITY OF SOUTHERN CALIFORNIA
THE GRADUATE SCHOOL
UNIVERSITY PARK
LOS ANGELES. CALI FORM IA 3 0 0 0 7
This dissertation, written by
under the direction of h.is... Dissertation Com
mittee, and approved by all its members, has
been presented to and accepted by The Graduate
School, in partial fulfillment of requirements of
D O C TO R OF P H IL O S O P H Y
Bernard Allan Troy
the degree of
o
Dean
DISSERTATION COMMITTEE
Chairman
DEDICATION
This study is dedicated,
with love and affection,
to Maria, Danny, Brian and Michael,
in partial compensation for
the days and weekends in which
they were virtually orphaned or widowed,
and for the patience, tolerance and
understanding they showed.
TABLE OP CONTENTS
CHAPTER ONE
Introduction........................................... 1
What This Study Intends to Accomplish................2
Background: Bilingual Education...................... 4
Background: Accountability.......................... 30
Need for the Study................................... 37
Justification of the Study.......... 40
Importance of the Study..............................42
Procedures to-Be Employed in the Study..............43
Scope and Limitation of the Study................... A/ * -
Summary of Chapter One and Preview of the
Remainder of the Study............................46
Definitions...........................................50
CHAPTER TWO
Purpose of the Present Chapter................ 53
Organization of the Present Chapter........... .....53
Literature Concerning Accountability............... 54
Literature Concerning Systems Approaches...........60
Literature Concerning Program Evaluation...........62
Literature Concerning Approaches to the
Evaluation of Bilingual Programs.................69
Summary of the Review of the Literature............ 71
CHAPTER THREE
Method Employed in the Study........................ 73
Introduction to the Paradigm for the Implementation
of Accountability Measures in Bilingual
Education..........................................74
Assumptions Implicit in the Construction of the
Paradigm............................................ 74
Involvement...........................................78
Commitment......................................... .. .84
Implementation........ 98
Summative Evaluation................................ 102
Accounting........................................... 106
Selection of "Expert" Evaluators................... 107
Duties of the Evaluators............................109
Method Employed in Contacting the Evaluators......110
iii
CHAPTER FOUR
Replies of Evaluators .........................112
Review of Replies: General Impressions.............113
Review of Replies: Specific Comments...............127
Summary of Revisions or Clarifications of
the Paradigm......................................147
Summary of the Chapter..............................154
CHAPTER FIVE
Summary of the Study................................ 155
Discussion........................................... 159
Recommendations......................................163
APPENDICES AND REFERENCES
Appendix A: Paradigm for the Implementation of
Accountability Measures in Bilingual Education.164
Appendix B: The EPIC Model..........................166
Appendix C: Stake’s Evaluation Model...............168
Appendix D: List of Evaluators of Title VII
Programs to Whom the Paradigm Was Sent for
Evaluation........................................170
Appendix E: Cover Letters and Materials Sent to
Evaluators with the Paradigm.................... 173
Appendix F: Replies of Evaluators.......... 182
References...........................................215
iv
CHAPTER ONE
Introduction
On January 2, 1968, the Congress of the United States
enacted Public Law 90-247, Title VII, sec. 702, 81 Statute
816. The bill was passed as an amendment to the Elementary
and Secondary Education Act of 1965 and is referred to as
"Title VII, The Bilingual Education Act." With respect to
its potential impact on education, the bill is notable in
several respects. First, it is the first time in history
that the Congress approved the notion of bilingual educa
tion. Whatever the relative validity of arguments about
the advisability of educating all students in English
versus educating them in a dominant non-English language,
the value of bilingual education is legally established.
We can infer that the Congress felt that bilingual education
was an acceptable form of "American" education.
Second, in implementing the bill, the United States Office
of Education officially committed itself to a program of
accountability. (Lessinger, 1969a). This was the first
instance where the government adopted the approach that
would spread through all the states with such rapidity.
In December, 1972, the Legislature of the State of
California enacted Assembly Bill No. 2284, The Bilingual
1
Education Act of 1972. The purpose of the law was to allow
public schools of the State to establish bilingual programs.
The goals of the program are "to develop competence in two
languages for all participating students, to provide
positive reinforcement of the self-image of participating
children, and to develop intergroup and intercultural
awareness among pupils, parents and staff..." (California
State Assembly, 1972). Section 5761.6 of the Act mandates
specific accountability procedures: identification of
goals and objectives for each program, description of
student achievement and criteria of acceptability, specific
management plan, teacher and aide pre-service training,
in-service training, link with nearby institution of higher
education to upgrade the program, parent-teacher communi
cation plan. The California law mirrors the federal law in
linking the authorization of programs with the demand for
accountability.
What This Study Intends To Accomplish
This study is an attempt to construct a paradigm or
model for accountability to be used by teachers, adminis
trators or interested citizens who wish to implement
bilingual education. It is not geared toward obtaining or
monitoring federal grants or state funds for program
implementation. The procedures mandated for submitting
3
proposals for grants afford a "how to do it" approach for
those interested and insure a modicum of accountability.
See, for example, Project Manual for Applicants and
Grantees, (United States Office of Education, 1971).
This study is intended to supplement such procedural
guidelines with more concrete suggestions. It is also
intended as an aid for small districts with single school
or single classroom programs or even for individual
teachers who may wish, without formally undertaking a
program as such, to evaluate their effectiveness in
teaching children whose primary language is not English or
who do not share in the dominant culture. While it is
hoped that the model can be generalized to other groups and
other areas, the author's experience and interests are
directed toward the situation of Mexican-American students
resident in the State of California. For this reason, the
study focuses on the idea of accountability with this
specific population in mind.
A number of factors suggest themselves as reasons why
such a paradigm may be useful. In spite of national
interest in accountability, there are still schools
relatively untouched by the trend. Bilingual education is
also very new as a widespread trend. The skills demanded
of teachers in bilingual education go well beyond those
required for traditional classroom teaching. Implicit is
4
a need for greater cultural-linguistic sensitization on
the part of school personnel. Expertise in teaching
English as a Second Language must be developed. If the
teacher is not bilingual, team teaching is involved, or
mastery of a new language. A great amount of new
curriculum must be developed locally due to scarcity of
materials. With all the demands, cultural, linguistic
and professional, placed on teachers, administrators,
aides and advisors undertaking such a program, there may
be little psychic energy left to be addressed to the
concrete, specific elements involved in a program of
accountability. It is hoped that the present program will
be suggestive enough and helpful enough to obviate such
problems.
Background; Bilingual Education
Prior to World War I, the languages of non-English-
speaking residents were generally tolerated, except in the
case of the Blacks and Native American Indians. Fishman
(1966, p. 22) claims that the ambivalence of the Bureau of
Indian Affairs over the policy of detribalization versus
tribal autonomy has weakened the ability and interest of
Indian tribes to maintain their languages. A case in
point is that of the Cherokees. Walker (1968) estimates
that they had a literacy rate of 90$ in 1830. They had
developed a written form of their language, published
bilingual newspapers and maintained schools where academic
subjects were taught in Cherokee. This was despite
government seizure of their press in Georgia and forced
repatriation to Oklahoma. In 1906, by Congressional
decree, they were detribalized, their publications ceased
and their academies closed. The Cherokees today are a
virtually illiterate nation.
The history of Blacks and their language is even more
astounding. Black slave families were broken up for a
number of reasons. To maintain their value as salable
commodities, they were discouraged from developing or
maintaining strong ties of friendship or kinship that
might lead to traumatic scenes or violence at the prospect
of separation by sale or trade. To prevent rebellion,
escapes, etc., slave-holders systematically broke up
family units and separated members of tribes speaking the
same language. The effect was the suppression of African
language and its total disappearance in this country.
(John and Horner, 1971).
Speakers of other languages, however, were tolerated
to a greater degree. German was the language of instruc
tion in some public schools. Spanish was one of the two
official languages of the State of New Mexico, and all
public school teachers were expected to be bilingual in
6
English and Spanish. Numerous private and parochial
schools offered instruction in French, Polish, Hungarian,
etc. This tolerance toward speakers of other languages
changed sharply after World War I due to a widespread fear
of the growth of European nationalism. (John and Horner,
1971).
There are, however, indicators of a reversal of this
trend and a return to greater tolerance since World War II.
America's increased involvement overseas has created the
need for speakers of foreign languages in the Department
of Defense, State Department, Peace Corps, and numerous
other public and private agencies. Strengthening of
teaching of foreign languages was one of the goals of the
N.D.E.A. legislation of the late 1950's.
The spread of Head Start programs and the careful
monitoring of compensatory education programs have
highlighted the educational problems of students whose
dominant language is not English. The influx of Cuban
refugees led to great interest in programs to teach
English as a Second Language (E.S.L.) or for Teachers of
English to Speakers of Other Languages (T.E.S.O.L.)
Growing militancy on the part of minority groups has led
to resistance to constant correction by teachers for
speaking "non-standard" English, to complaints that rich
ethnic history was distorted or ignored in mainstream
textbooks, and to resentment over the subtle assumption
7
that "different" from middle class American culture is a
euphemism for "inferior." General social awareness is
indicated in the language used in the popular literature.
We have progressed from the study of "deprived" children
(Riesman, 1962) to the study of "disadvantaged" children
(Bereiter and Engelmann, 1966; Deutsch, et al., 1967)
to the study of "different" children (LaBelle, 1971).
Acceptance of bilingual education appears to parallel
our growing awareness of our own ethnocentrism. Until the
mid-1960's bilingual education was frowned upon for a
number of reasons. In the popular mind, English is the
official language of the country. To foster other
languages, to allow children to learn in other languages
was seen as a lessening of the incentive to learn English
and to assimilate, leading to retarded acculturation,
reduced academic and occupational opportunities, and
lessened probability of social ascendancy. (Huntsman,
1972). Psychologists dnd linguists also voiced concern
over possible adverse effects of bilingual education.
Weinreich (1953) suggested the probability of linguistic
interference, the intermixing of sounds, vocabulary,
grammar and word meanings in the two languages, as a
result of bilingualism. Jensen (1962) felt that teaching
a second language too early would lead to distortions of
speech and rhythms of the dominant language. Darcy (1963)
8
suggested that learning two languages may hinder a child's
intellectual functioning and development, lead to a
smaller vocabulary, simpler sentences, confused word order,
misuse of negatives and idiomatic expressions and the use
of literal translations. White (1965) states that a child
between ages 5 and 7 uses language for problem solving.
When forced to express ideas in a second, weaker language,
the child may be inhibited and lose his curiosity or
interest. The possibility of emotional trauma is suggested
by the results of a study on stuttering among bilinguals.
(Travers, Johnson and Shaver, 1937). Bilingual children
were found to have a significantly higher incidence of
stuttering than monolingual children. Of those bilingual
children who stuttered, 26$ acquired the stutter at the
time of the introduction of the second language. Prior to
the middle 1960's, it was generally assumed by linguists
that bilingual children had lower IQ's. However, Darcy
(1963) states that socio-economic factors and cultural-
linguistic bias of tests were rarely controlled in
earlier research projects.
Results of a survey of educational leaders from
around the world (opinion sruvey, not based on research)
were reported at a seminar on bilingualism in 1965
(U.N.E.S.C.0., 1965). The respondent from Luxembourg felt
that bilingualism places emphasis on the cultivation of
linguistic memory and thus discourages rational
functioning. Facile and superficial mental attitudes
are encouraged, and the building up and consolidation of
ideas and concepts is inhibited. In neither of their two
languages do children find a firm and solid basis for
reference or a sufficiently stable frame within which to
think. (U.N.E.S.C.O., 1965, Pg. 145). Evidence from
Czechoslovakia suggests that children growing up in a
bilingual setting do not command the exactitude and
precision of expression of monoglot children. Evidence
from Ireland appears to indicate that retardation is
aggravated if the medium of instruction is the language
with which the child is less familiar. Furthermore,
because of the need to teach two languages, the curriculum
of the school is sometimes restricted and this restriction
aggravates academic retardation. (U.N.E.S.C.O., 1965,
Pg. 146).
While virtually all of the above arguments cause
scepticism about the concept of bilingual education in se,
they can be taken as reasons for beginning instruction of
young children in their mother tongue. If at some time we
wish these young children to master the national language
(in the case of children whose primary language is not the
national language), we are faced with the prospect of
bilingual education or of teaching the national language
10
as a foreign language. Consensus seems to be swinging
toward the acceptance of bilingual education.
The first step toward this consensus was the spread
of E.S.L. programs. Saville and Troike (1970) point
out that most teachers saw no need to differentiate
the language arts instruction of native-English-speakers
from that of non-English speakers.
We often think of language learning as
simply learning new words to express our
concepts. Learning a language is actually
a far more complex task, for a language is
composed of more than just words and their
meanings. Language is an integrated system
of phonology, grammar and lexicon. We acquired
control over the phonology and grammar of our
native language quite unconsciously, and
almost entirely within our own preschool years.
It is understandable, then, for most people
to remember only the most conscious and most
recent aspect of their language development—
vocabulary building— and to be largely unaware
of other aspects of language learning.
We must be aware that each language is a
total system for transmitting meaning and
beware the subtle fallacy that direct translation
between languages is possible. Such a view is
fostered by our traditional textbooks, with their
seemingly clear-cut English/Spanish (Navajo, etc.)
glossaries, but this view is simplistic and
misleading. Each language characterizes the world
of experience differently (pp. 13-14).
The E.S.L. approach gave evidence of concern over the
academic progress of non-English speaking students and
seemed to be a more logical approach. However, even
though there were never large groups of teachers with
strong preparation in E.S.L. methods (suggesting weak or
11
ersatz programs), results were not encouraging. Carol
Phillips (1970) reported that only one-third of high
school aged Mexican-Americans in New Mexico were enrolled
in high school. Of 60,000 Spanish-speaking students in
school, over one-third were in first grade (stimulating
curiosity over how long they stay in first grade).
Fifty-five per cent of those students above first grade
are more than two years over-age for their level (and this
does not take dropouts into account). In Texas, 20$ of
Mexican-Americans entering first grade are not promoted.
For older Mexican-Americans in the Southwest, average
schooling is 8 years compared with 12 years for Anglo-
Americans. The dropout rate for Mexican-Americans is
twice the national average.
In the early 1970's, the United States Civil Rights
Commission undertook a systematic study of the equality
of educational opportunity enjoyed by the Mexican-Americans
of the Southwestern United States. The Commission found
that Mexican-Americans tend to be segregated by district
and, within districts, by schools. Mexican-Americans tend
to be under-represented on faculties and school boards
relative to their population at large. The majority of
Mexican-American faculty and school board members are
found in districts with populations that are predominantly
Mexican-American. (U.S. Commission on Civil Rights,
April, 1971).
12
School holding power is weaker for Mexican-American
students. They have a 40$ dropout rate compared to 14$
for Anglo-American students. Twenty-five per cent of the
Mexican-Americans go on to college compared to a rate of
about 50$ for Anglo-Americans. Reading achievement of
Mexican-Americans is disproportionately poor. The ratio
of Mexican-American students with reading scores below
grade level is 51$ in 4th grade, 64$ in 8th grade, and
63$ in 12th grade compared to 25$ of the Anglo-Americans
in 4th grade, 27$ in 8th grade, and 34$ in 12th grade.
Sixteen per cent of the Mexican-Americans in the Southwest
repeat first grade. In Texas, more than one out of every
five repeat first grade. The average for Anglo-American
students is 6$. There are nine times as many over-age
Mexican-American students in Junior High School as there
are over-age Anglo-Americans. The ratio decreases in
twelfth grade, but probably due to the dropout rate.
Whether enrolled in schools where Mexican-American
students are in the minority or are a majority, they are
under-represented in extra-curricular activities.
(United States Commission on Civil Rights, October, 1971).
Investigators for the Civil Rights Commission found
little evidence that the culture and heritage of Mexican-
American students was recognized in the schools. One-third
of the schools in the Southwest have policies of actively
discouraging the use of the Spanish language. In many
instances, offenders are physically punished. The irony
is that, though principals estimate that 47$ of the
Mexican-American children entering first grade do not
speak English as well as their Anglo-American peers,
speaking Spanish is discouraged and systematic efforts to
teach English are not widespread. Three programs that
might be implemented to deal with the fact that students
do not speak English with facility are Bilingual Education,
English as a Second Language, and Remedial Reading.
In 1969, 2.7$ of the Mexican-American students in the
Southwest were enrolled in Bilingual Education programs,
5.5$ in E.S.L. programs, and, in spite of the fact that
63$ of the 12th graders placed below grade level in
reading scores, only 10.7$ were enrolled in Remedial
Reading courses.
Investigators noted that enrollment figures were in
inverse proportion to the degree of curriculum change and
teacher training required for program implementation.
(U.S. Commission on Civil Rights, May, 1972).
Mexican-American history, heritage and folklore are
absent from the curriculum. Textbooks were judged to be
biased and selective in their historical presentations.
The Anti-Defamation League of B'nai B'rith conducted
a national survey of social science testbooks used in
Junior and Senior High School. While the study failed to
14
find a single textbook that presented a reasonably
complete and undistorted picture of America's minorities,
it characterized the Mexican-American as having replaced
the Black as the nation's "invisible man" (U.S.
Commission on Civil Rights, May, 1972, Pg. 31). Only 7.3$
of the secondary schools and 4.3$ of the elementary
schools included courses on Mexican-American history.
The Commission investigators surveyed school
activities for evidence of appreciation and awareness of
Mexican-American cultural heritage. While hoping to find
things like P.T.A. brochures printed in Spanish and
English, parent education groups in Spanish, numerous
books in the library on Mexican-American heritage and
culture, history of Mexico in song and dance, Mexican-
American youth organizations on campus, etc., they found
instead a preponderant celebration of a "fantasy heritage"
(giving the impression that the country was settled by men
who spent all their time polishing silver spurs or women
who always wore camellias behind their ears). (U.S.
Commission on Civil Rights, May, 1972).
The Commission also reported evidence that the
Mexican-American community is excluded from the schools.
Seventy-five per cent of the elementary schools and 89$
of the secondary schools do not send notices in Spanish
to Spanish-speaking parents. Ninety-one per cent of the
15
elementary schools and 98.5$ of the secondary schools do
not use Spanish at P.T.A. meetings. Seventy-five per cent
of the districts did not have community advisory boards.
Of those districts that did have advisory boards, 75$ did
not meet five times during the year. (Members of advisory
boards were surveyed. They listed in-service education of
teachers in Mexican-American culture as their primary
concern.) Eighty-four per cent of the districts did not
have community relations specialists. Most community
relations specialists who were employed worked in large
districts, and, of these, only 31$ were Mexican-American.
Only 4.7$ of the districts employed consultants on Mexican-
American affairs for 10 days or more. (U.S. Commission on
Civil Rights, May, 1972).
Finally, Commission investigators undertook a
systematic analysis of student-teacher interaction in a
representative sample of 52 schools, using Flanders'
Interaction-Analysis techniques. They found that teachers
praised or encouraged Anglo-American children 36$ more
often than Mexican-American children; they built upon
their contributions 40$ more often; they questioned them
21$ more often; they responded positively to them 21$ more
often; and spent 23$ more time in noncriticizing talk with
them. Anglo-American students spent 27$ more time
speaking than did Mexican-American students. (U.S.
Commission on Civil Rights, March, 1973).
Growing sensitivity to defects in our educational
system (as opposed to deficiencies, deprivations, and
disadvantages of our students) led to increased scepticism
of linguists over the assumptions of compensatory
education. The validity of evaluation results came to be
increasingly questioned (Deutsch, Fishman, Kogan, North
and Whitehead, 1964). There was a growing suspicion of
cultural bias in testing where simple linguistic
translation was employed without psychometric standardi
zation on the new population (Saville and Troike, 1971,
Pg. 14). While it seemed very improbable, it was
established that substantial numbers of students
classified as mentally retarded were not retarded at all,
but unable to understand English or express themselves in
that language (Chandler and Plakos, 1969). And there is
growing scepticism about the diagnostic validity of many
of the remaining placements in such programs, even on the
basis of tests administered in Spanish (Mercer, 1972).
The young child who enters school without a grasp of
English perceives a number of social realities, consciously
or not. People in authority do not speak his language.
There is a lack of common ground between family and friends
and the school system. Language is a membership badge in
a linguistic community. If his language is not deemed
important, he may develop the impression that his
community is not important. Where use of his mother
tongue is seen by teachers as a type of secrecy or
rejection of majority culture, he may feel himself stigma
tized as an outsider or a rebel. Where it is presumed
that mastery of English is a correlate of average
intelligence, he may come to feel that he is less than
intelligent. The reality is that, because of the status
associated with English and the repression of Spanish,
many children come to deny that they know the language.
Up until recent years, it was not uncommon for children
of Mexican-American descent to refer to themselves as
"Spanish," consciously negating their Mexican roots.
(Saville and Troike, 1971, pp. 19-22).
Gaarder (1967, p. 51) stated that forcing non-
English-speaking children to ignore their mother tongue
and be instructed in English leads to their cognitive
retardation and the development of a poor concept of
themselves, their parents, and their homes. Monolingual
schools ignore the fact that a child begins to learn long
before he comes to school and has spent much time
mastering his native language, the values, and belief
system that attach to it. Later, Gaarder (1969, p. 33)
hypothesized that conceptual development and acquisition
of experiences and information would continue at a normal
rate if the mother tongue were used in the early years of
18
school. Apart from the fact that native competence in
foreign language and the cultures that attach to it
constitute a national resource, use of the mother tongue
by the school staff would establish a strong reinforcing
relationship between the home and the school.
As minority group members began to reflect on their
situation, they came to the realization that, while there
is one national language in this country, there is not a
legal culture. Ignoring this fact probably contributes
heavily to the academic problems of minority students.
Lezama (1971) details a number of reasons why language
programs for minority students may be weak. He notes a
tendency to translate the English curriculum into second
languages with no modifications whatsoever. Anglo-American
teachers may tend to stereotype Mexican-Americans and
segregate them. E.S.L. programs imply an assimilation
thrust that is distasteful to most Mexican-Americans and
may be resisted. Students frequently complain of lack of
empathy on the part of the staff. Program development is
often weak and unsystematic. (At times, the hiring of a
bilingual aide is characterized as undertaking a bilingual
education program.) Teachers who have credentials to
teach foreign languages do not necessarily make good
bilingual-bicultural teachers. Plans of instruction
rarely mirror the needs of the community. High priced
19
hardware and highly developed program materials cannot
take the place of efficient, warm teachers. Self-identity
is often overlooked.
Mexican-Americans began to articulate what they felt
were some cultural differences that were not taken into
account in their education. Jaramillo (1973) points out
that in general students from Latin culture, in contrast
with Anglo-American students, conceive of time to be used
for the development of human relationships rather than for
more impersonal achievements. They tend to be more
motivated by group rewards than individual rewards.
Boys and girls do not compete, sex roles are not equated,
and there is little sexual egalitarianism. Teachers and
other persons in authority are expected to be formal and
correct. Instructors are expected to know their material
from memory and are not expected to admit to uncertainty.
Informality in the classroom is often perceived as an
indication that teachers are poorly prepared and do not
care about anything. They tend to express emotions openly
and do not expect this to be such a private affair. They
like to be part of the action and participate actively.
Conversely, they dislike sitting still, standing in line,
keeping still, etc. They tend to be more free and expect
more latitude in their use of space compared to the Anglo-
American, who is more private and whose use of space is
20
more measured. They are accustomed to a great deal more
body contact, and their body language, gestures, and
exclamations are different.
Other authors are sensitive about such comparisons,
fearing that they reinforce stereotyped perceptions of the
Mexican-American, especially where the examination focuses
on the "disadvantaged" Mexican (Casavantes, 1969). They
urge that care be taken in assuming that there is a
"typical" Mexican-American culture (Arciniega, 1971).
But such examinations, even where there is danger of
stereotyping, serve the purpose of sensitizing Anglo-
American teachers to the danger of ethno-centric
approaches that may be offensive to their students.
Saville and Troike (1971, p. 15) and Olguin (n.d.)
document a number of classic gaffes. Teachers are
frequently observed asking Navajo students to speak
more loudly. (This is boorish in their culture). It is
common to see teachers encouraging Hopi children to
compete. (When running a foot race, they will be careful
to see that they all cross the finish-line together).
They note a tendency to interpret diverted eye contact
on the part of Navajo children and many rural Mexican
children as symptoms of deviousness. (This is a sign of
respect and expected behavior in their culture.
The command, "Look me in the eye!," often given during
21
scoldings, is a request for behavior that the child has
been taught is brazen and rebellious.)
Anglo-Americans usually attribute to irresponsibility
the failure of some Mexican-American mothers to keep
appointments for parent conferences. Conducting that kind
of business is, in many places, the role of the father and
not the mother. It would be very unseemly and out of
keeping with her sex role to take charge. When confronted
with the discomforting insistence of the teacher that she
attend, it would be too harshly confrontative and offensive
to say that she will not. Instead, she will signify demure
compliance and say what she thinks the speaker wants to
hear. She does not intend to deceive but to be careful
about the sensibilities of another. A person aware of the
subtleties of communication in that culture would know from
cues, such as intonation, body language, enthusiasm, etc.,
whether the "Si!" did, in effect, imply assent. Generally,
however, such behavior is not noted, and these mothers
acquire reputations as deceitful, irresponsible, and
disinterested in the education of their children.
On the part of Anglo-American teachers, these authors
note an insensitivity to amenities in other cultures,
displayed in such things as indirectness of communication.
The Spanish language reserves direct address in the second
persons form of the verb to a closely prescribed circle of
intimates and to very small children. All others are
22
addressed in the third person as "your reverence." The
overfamiliarity and overdirectness of English speech can
sound boorish and offensive to Latins. For example,
direct requests such as "Give me that book!" sound harsh
to the Spanish ear. Much of what we Anglo-Americans like
to think of as frankness and directness is perceived by
others as boorishness and lack of taste.
The cultural awakening of the minorities has led to
a call for cultural pluralism and cultural democracy in
the classroom (Ramirez, 1970a and 1973; Castaneda, 1973).
Zintz (1969, p. 4) cites values with which the school
child in America is confronted. He is expected to try to
be successful and to prize achievement. He must be precise
in temporal orientation and future oriented. He must
accept the teacher’s reiteration that there is a scientific
explanation for all phenomena. He must become accustomed
to anticipate change and is expected to esteem change as a
good thing in itself. He must display socially approved,
aggressive, competitive behavior. He must internalize the
belief that with some independence he can shape his own
destiny, as opposed to remaining an anonymous member of
his society. It may seem difficult for Anglo-American
educators to conceptualize alternatives, but the evidence
is that minorities will resist the imposition of these
values as a matter of course.
23
Furthermore, the awareness of what language conflict
can do to young students and what cultural conflict can
lead to has caused some linguists to question the place of
standard English in the curriculum. Saville and Troike
(1971, Pp. 10-12) point out that each of us speaks an
idiolect, a personal language derived from our unique
linguistic experiences. A collection of similar idiolects,
based on common language, experience, and peer influence,
becomes a dialect. Similar dialects form a language.
For a variety of social, economic, and political reasons,
some groups of people in a country will enjoy more social
prestige and power than others. Their dialect comes to be
considered better than others. The language of the upper
class in a society thus becomes the de facto standard
dialect. In teaching standard dialect, any approach that
stigmatizes the student's own speech should be avoided,
since this will simply humiliate him and create an
environment that is not conducive to learning. Huntsman
(1972) rejects the "corrective approach" of E.S.L.
teaching because of what she feels are faulty assumptions:
that standard English actually exists, that it is necessary
for improved intellectual performance, that it is best
acquired through constant correction by the teacher.
These trends have also caused investigators to re
examine the effects of bilingual education. Richardson
(1968) evaluated the Coral Way Program (a bilingual program
for Cuban immigrants in Dade County, Florida) after three
years of operation. He found that children in the
bilingual program learned equally well in either language
and that bilingual education was as effective as the
standard curriculum in the teaching of academic subjects.
Tucker, et al., (1971) studied a French Canadian bilingual
education program where English speaking children were
schooled in French and control groups were maintained.
After five years of instruction (k-4), the students in the
bilingual education program showed no native language
deficit, no subject matter deficit, and no indication of
cognitive retardation. Furthermore, they demonstrated a
more firm grasp of French than those children who studied
French as a second language. (However, they note for
comparison that English is a high status language in
Canada, that the children had no feeling of inferiority in
the school, that their teachers did not hold low academic
expectations, that their socio-economic status gave them
considerable power in the community, and that they were
not expected to compete with native French-speakers in the
bilingual classroom.) Participants at the United Nations
seminar on bilingualism (U.N.E.S.C.O., 1965> p. 146) note
that while there is reasonable fear that bilingualism may
have adverse effects on the functioning of a child's
intelligence, the learning of a new language may, of
25
itself, stimulate the operation of a child's intelligence,
afford him with a wider frame of reference for comparative
investigations and judgments, and enhance opportunities for
broader social and cultural contact.
This leads to the consideration of what bilingual
education might do to Anglo-American children enrolled in
the program where a two-way bilingual model is employed
(i.e., where the idea is for students, Anglo-American and
non-English-speakers, to learn two languages). John and
Horner (1971, p. xxiii) point out that the real potential
of these programs is unknown. But the hope is for greater
appreciation and awareness of foreign cultures and
languages, reduced ethnocentrism, greater tolerance in the
area of human relations, and mastery of a second language.
Jones (1969, p. 13) suggests five reasons for the early
introduction of young children to the learning of a second
language. Because a young child lacks inhibition, he can
spontaneously indulge himself in second language learning
and acquire the second language much the way he does the
first. He hypothesizes that the young brain has greater
plasticity and is therefore more capable of acquiring
speech. There is a general and popular impression that
younger children learn second languages with much greater
facility than do older persons. As they grow older, they
will have to take their place in a world that is
26
increasingly international minded. And finally, young
children will accept a school’s linguistic milieu without
special motivation; later, however, they may be more
resistant to linguistic changes.
While Tucker et al., (1971, pp. 47-48) found that
students educated in the bilingual education program did
not become more favorably disposed to French Canadians
than did control groups, most investigators find a
connection between language learning and attitude change.
Peal and Lambert (1962, pp. 12-13) found that bilingual
children had markedly more favorable attitudes toward
other language communities than did monolingual children.
Malherbe (1946), studying bilingual education in South
Africa, reported clear academic advantage on the part of
bilingual students as well as a lower degree of inter-
cultural antagonism. Walsh (1969) found children in
bilingual education programs to be more tolerant and
receptive of their own and other cultures than comparable
monolingual children. Lambert and Klineberg (1967)
propose bilingual education programs for the purpose of
better human relations. They state that six year olds are
more conscious of differences than similarities. Early
multicultural education can offset the roots of
stereotyping behavior that begins at this age.
There are numerous models for bilingual education
(Andersson and Boyer, 1970, Vol. II. pp. 63-82; Valencia
1969a; Mackey, 1969). They may plan to serve just the
non-English-speaking children (one-way programs) or
English-speaking children as well (two-way programs).
They may be single medium programs (only the mother tongue
used) or dual medium programs (both languages used).
They may be transfer programs (planned to teach students
to function in English and then cease instruction in the
native tongue) or maintenance programs (planned to improve
mastery of the mother tongue even after English has been
learned). They may be geared toward the acculturation
of students in Anglo-American culture, or they may be
irredentist programs (geared toward the maintenance of the
minority culture). Language usage in instruction may be
different or equal. That is, some subjects may be taught
in one language and others in another, or all subjects may
be taught in both languages. And program changes may be
complete and abrupt or gradual. The federal manual
(U.S.O.E., 1971) defines bilingual education as follows:
Bilingual education is the use of two
languages, one of which is English, as
mediums of instruction for the same pupil
population in a well-organized program which
encompasses part or all of the curriculum and
includes the study of history and culture
associated with the mother tongue. A complete
program develops and maintains the children's
self-esteem and a legitimate pride in both
cultures (p. 1).
There are numerous rationales propounded in support
of bilingual education, some implying different and even
contrasting models of bilingual education. Valencia
(1969b) states that learning theorists are questioning the
idea that IQ is a constant. As a result, early, formal
exposure to cognitive experiences (such as Head Start) is
increasing. Bilingual-bicultural education emerges as a
logical result for non-English-speaking children. The
delay of concept formation until the acquisition of
English language skills is not a necessity. And finally,
the affective aspects in early childhood education can be
given attention if teachers are aware of the culture the
child brings to the classroom. (He notes the goal of
perpetuation of Spanish language and culture as another
question apart from pedagogival practice.) Modiano (1968),
as a result of studies on the language training of Indian
populations in Mexico, states that the mother tongue is
the best medium for learning, especially in the early
years.
Saville and Troike (1971, p. x) acknowledge
unresolved ambiguities and a lack of consensus on
bilingual education as a form of compensatory education,
as a bridge to the more efficient learning of English or
as a good in itself. They feel that most proponents are
in agreement with each other in expecting higher self
29
concept and academic achievement from students, a bettering
of socio-economic levels for minority groups members, the
preservation of our collective and diverse national
heritage, and greater understanding between groups.
Rosen and Ortego (1969) caution that progress will be
slower in bilingual education programs, but they feel that
reading in English will improve and achievement in other
subjects will be enhanced due to greater transfer and to
the greater reinforcing value of the mother tongue.
Andersson and Boyer (1970) point out the anomaly of
spending millions of dollars in this country to teach
foreign language while neglecting bilingualism. In
surveying proposals for bilingual education, they conclude:
1. American schooling has not met the
needs of children coming from homes where
non-English languages are spoken; a radical
improvement is therefore urgently needed.
2. Such improvement must first of all
maintain and strengthen the sense of identity
of children entering school from such homes.
3. The self-image and sense of dignity of
families that speak other languages must also
be preserved and. strengthened.
4. The child’s mother tongue is not only an
essential part of his sense of identity; it is
also his best instrument for learning,
especially in the early stages.
5. Preliminary evidence indicates that
initial learning through a child's non-English
home language does not hinder learning in English
or other school subjects.
6. Differences between first, second and
foreign languages need to be understood if
learning through them is to be sequenced effectively.
30
7. The best order of learning basic skills in
a language— whether first or second— needs to be
understood and respected if best results are to
be obtained; this order is normally, especially
for children: listening comprehension, speaking,
reading and writing.
7. Young children have an impressive learning
capacity; especially in the case of language
learning, the young child learns more easily and
better than adolescents or adults the sound system,
the basic structure, and vocabulary of a language.
9. Closely related to bilingualism is
biculturalism, which should be an integral part
of bilingual instruction.
10. Bilingual education holds the promise of
helping to harmonize various ethnic elements in
a community into a mutually respectuful and
creative pluralistic society (Vol. I, p. 49).
John and Horner, (1971, p. xxiii) see bilingual
education as a more humane and enriched experience for the
non-English-speaking child. And the U.N.E.S.C.O. study
(1965, pp. 152-153) states that, in the case of persons
who do not speak the national language, the value of the
uniqueness of their own language is ultimately an ethical
rather than a utilitarian matter.
Background: Accountability
Leon Lessinger (1969a) acknowledges that he is often
credited with being the "father of accountability."
Appearing as associate Commissioner for Elementary and
Secondary Education of the United States Office of
Education, he addressed a subcommittee of the Committee
31
on Appropriations of the House of Representatives on May 6,
1969 in connection with Title VII and Title VIII of the
Elementary and Secondary Education Act. His testimony,
in part, went as follows:
In order to make Federal dollars more
effective and more efficient, we have tried
to introduce some educational management
concepts in both the bilingual and the dropout
programs...
The programs of dropout prevention and
bilingual education are employing several
management tools which should improve results...
Among the tools are accountability, technical
assistance, and independent educational audits...
Basically, accountability means that the
grantee will be held responsible at any time
during the project for accomplishing the objectives
of the project which the grantee himself proposed
within the time periods specified, within the
budget limitations and according to the standards
established. Thus at any point in the life of the
project there will be a benchmark against which
to measure performance (pp. 16-17).
While this statement might be taken as the formal,
historical beginning of the movement for accountability,
it would seem more appropriate to view it simply as an
event that crystallized a long-growing trend.
Accountability is an analogous term with connotations
of "calling somebody to account." In the late 1960’s
there were more than a few people ready to call educators
to account.
Beginning in the 1950's, the nation began to
question the institution of education. We were still in
the process of berating "progressive education"
(Lynd, 1953; Bestor, 1953; Smith, 1954). With the end
of the war in Korea came the revelation that many American
soldiers, imprisoned by the North Koreans, had collaborated
with their captors and become "turncoats." In some
instances, the North Korean captors employed physical and
psychological torture, "brainwashing." Nevertheless, it
came as a shock to most Americans that members of their
own society could turn against thorn. Confidence that
society was adequately socializing its youth was shaken.
Seeds of doubt about the success of the educational system
were sown.
Several years later, the Russians launched Sputnik.
Public complacency was again shaken to find that we had
fallen behind the Russians in cold-war competition.
There was disillusionment that schools were not meeting
implicit national goals. It has become commonplace to
refer to "Sputnik" as a landmark in American educational
history.
In the middle 1950's, Jack Kerouac and Allen Ginsberg
became popular as prototypes of the "beat generation."
They were forerunners of the beats, the beatniks, the
hippies, the yippies, the crazies: a substantial
proportion of young people who quite consciously rejected,
to a mild or marked degree, traditional values held by the
33
majority of society (the beginning of the decline of the
work ethic) and who criticized or strove to undermine the
institutions that reflected or promoted such values.
The spread of this sub-culture led to the rise of
another pehnomenon, alluded to by historians of the last
decade as the "generation gap." Pretzel (1970) claims
that there has been a massive shift in the values held by
young people. He claims that there is now a tendency for
the traditional values of productivity, achievement, and
aggressiveness to be subordinated to the concern for
heightened awareness, desire for pleasurable experience,
and passivity. To the degree that his perceptions are
accurate, we can have parents criticizing schools for
their children’s lack of achievement at the same time the
children are criticizing the schools for the poor quality
of experience. The portrayal of schools as dreary places
and teachers as pedantic bores is as traditional as the
very literature of education itself. It is only in the
last few years, however, that the presence of bored
students has been taken as the primary indictment of the
schools (Silberman, 1970).
The last decade has also seen the rise of a number of
"young Turks" in the field of education: Edgar
Friedenberg?! Paul Goodman, John Holt, Herbert Kohl,
Jonathan Kozol, James Herndon, Neil Postman, Charles
Weingartner, Jerry Farber, et al. David Riesman
characterizes them as affected with ' ’aristocratic
insouciance," a reflection of the "general snobbery of
the educated upper middle-class toward the white-collar,
lower middle-class world of teachers, social workers,
civil servants, and policemen...the need to demonstrate
their moral and cultural superiority to the lower middle-
class from which they escaped" (Silberman, 1970, p. 90).
Mostly graduates of Ivy League schools, with tenuous
commitment to the institution of public education, they
have produced a body of writings that have had tremendous
impact on younger teachers and their students...and they
do not look with favor on the schools.
The fifties and sixties also saw the growth of the
civil rights movement. The abolition of "separate but
equal" schools by the Supreme Court was a beginning.
But activism on the part of individuals influenced
education to a much greater degree than legal moves
(which were, in the main, in response to demands of civil
rights activists). The Blacks, especially, demanded equal
opportunity which meant equal education. There was strong
public demand for education to address itself to the
situation of poverty and the condition of the disadvantaged
(Clark, 1970). Clearly, education was not for the poor
and the minorities the same means to succeed that it was
35
for the affluent majority. As the civil rights movement
expanded, it produced widespread demands for self-
determination. The demand grew for local control
("black power" or "brown power") and the acceptance of
minority culture and language. Implicit in the rhetoric
of the movement was the feeling that not only had schools
not helped the poor, but that they were a large part of
the problem of the poor and minorities.
Another current of educational events that took on
major proportions during this era was the Free Speech
movement and the growth of student activism. Coupled with
civil rights protests and anti-war protests, it led to the
bombing and burning of banks and buildings, to violence on
campuses and in schools, to public desecration of the
American flag, and to marked backlash on the part of the
"silent majority." Public reaction to these events
coincided with the beginnings of an economic recession
coupled with a continued rise in taxes. The result was
the "taxpayers' revolt," the first mass refusal of the
American public to support its educational system.
These trends had, by the late 1960's, created marked
and widespread discontent with American education. Those
with a conservative bent faulted the schools with failure
to inculcate traditional values and virtues, while those
with liberal inclinations tended to characterize the
36
schools as authoritarian bastions, dedicated to the
preservation of the status quo. Critics of more
traditional outlook clearly felt that schools were not
demanding that students achieve as much as they could.
The more progressive faulted the schools for being
drearily obsessed with achievement and characterized them
as stifling, boring and irrelevant. Minorities were
critical of schools for not meeting their aspirations.
Taxpayers were sceptical about supporting a system that
did not seem to be satisfying anybody. That did not
leave the institution of public education with a great
number of highly committed advocates.
It was in this ambience that Lessinger made his
remarks to the Congress and the term "accountability”
caught on in the popular literature. Prior to 1971 the
Annual Index of Research in Education (ERIC) did not carry
a subject heading for the topic of accountability. In
1971, the heading "Educational Accountability" was added
and there were 38 entries. By 1972 there were 102 entires.
Similarly, the Education Index did not carry the heading
"accountability" before it appeared in Volume 20 (July,
1969-June, 1970). However, there were no entries that
year. Volume 21 carried 38 entries, and Volume 22 carried
75 entries. Many of the articles bore titles that offered
promise of great things. Terms like "Hallmark of the
37
1970's” (Austin, 1971), "Watchword for the '70's"
(Norris, 1971) and "Age of Accountability" (Lessinger,
1971b) convey the impression that a new era had begun
in education.
In Chapter Two, we shall review some of the specific
meanings attached to "accountability." Whatever the
specific meanings, however, the term "accountability"
implies that schools will be more responsive to the will
of the people, more effective and more efficient.
This brings us to the intersection of the areas of
concern addressed by this study: bilingual education and
accountability. In a sense, bilingual education is itself
a form of accountability in that it represents an attempt
to respond more appropriately to the needs of non-English-
speaking children in particular and the needs of society
in general. However, the idea of widespread bilingual
education is novel, curriculum and methods for bilingual
education are largely undetermined and the prospects are
uncertain. To implement good intentions, a systematic
approach is called for.
Need For The Study
Provus (1971, p. 8) reflects on the observation of
a group of school administrators that Title I of the 1965
Elementary and Secondary Education Act may be forgotten as
a poverty act, but long remembered as the source of
38
systematic self-appraisal in American schools.
He concludes:
The statement was not prophetic. Today,
useful evaluation theory and practice are no
better established in public schools than
they ever were (p. 8).
Stake (1967, p. 524) states his conviction that
educators have never demanded the development of an
evaluation methodology that reflects the fullness,
complexity and importance of their programs.
Andersson and Boyer (1970), in surveying the
operation of bilingual education programs and their
evaluation, note that:
...evaluation is an indispensable part of a
bilingual, or any other program. Without testing,
a teacher cannot determine whether he and the children
have achieved the stated aims of the program.
Educators are pretty well agreed, however, that at
no point in this basic educational process have we
reached total agreement as to procedure. Aims are
frequently not stated in measurable terms. In
teaching, teachers tend to lose sight of their aims.
And rare indeed is the teacher whose testing really
measures to what extent he has taught what needed
to be learned...
The great need is for the most advanced thinking
about evaluation to be communicated to the bilingual
staff so that statement of aims, the teaching and
the testing may all be correlated (Vol. 1, p. 67).
Despite the existence of a number of bilingual
programs, most authors feel that there has been little
systematic description and evaluation (Bishop, 1965;
39
John and Homer, 1971; Pacheco, 1971; Tucker and
d'Anglejean, 1971; Tucker, et al., 1971). Fishman (1970,
p. 1) notes that bilingual education in the United States
suffers from the lack of evaluated programs.
Saville and Troike (1971, p. 65) point out that the
expanding scope of bilingual education programs in the
United States will have a wealth of information to answer
questions regarding the most successful techniques, best-
suited instructional materials, appropriate interaction
between culturally oriented activities, and even the
validity of our evaluation instruments themselves.
We will have this information, however, only
if innovative bilingual projects report on their
procedures and progress in a way that allows
objective evaluation and comparative assessment
among programs.
Many roadblocks lie in the way of reliable
analysis, including the lack of completely
appropriate evaluation measures, but we can make
the best use of what is available now and
participate in the development of better
instruments (p. 65).
Most authors (Andersson and Boyer, 1970, Vol. 1,
p. 137; Saville and Troike, 1971; Jakobovits, 1970;
Spolsky, 1968; Upshur, 1968) concede that existing
instruments and procedures for evaluating bilingual
programs are inadequate. John and Horner (1971) report the
approach to evaluation taken by the directors of a bilin
gual education program in the Ocean Hill-Brownsville School
40
District in Brooklyn :
The organizers of the program Believe that
large-scale evaluations of its effectiveness are
inappropriate until procedures and materials are
relatively fixed, approaches are consistent, and
something is known about what works in bilingual
education programs in large urban schools. The
focus is now on "self-evaluation" to help teachers
to grow and to develop their own skills. Each
teacher prepares a bi-weekly statement of his
teaching goals in behavioral terms, and at the
end of each two week period assesses his effective
ness on the basis of his daily log and the children's
performance in relation to these stated goals
Such an approach gives evidence of both common sense
and sincerity. We must recognize that bilingual education
is new to us and that we are unsure of the best way to
proceed. However, to use the words attributed to the late
G. K. Chesterton, "If a thing is worth doing, it is worth
doing badly!" We will not meet the needs of minority
students unless we attempt to do so, even when we
recognize the defects and deficiencies of our attempts.
Our hope is that by trial and error, by building on the
experience and insight of others, we will experience
incremental advances toward our goal.
p. 56).
Justification Of The Study
In his critical comments on the CIPP model, Michael
(Stufflebeam, 1971) notes:
41
It is important to point out that the
heuristic properties of the CIPP model for
doctoral dissertation research are great
indeed, even if its use displaces the
overworked classical experimental-control
group models. Doctoral committees should
encourage students to do developmental
studies even though they represent a break
with tradition.
This study is not an extrapolation of the CIPP
program evaluation model. Nevertheless, it is udertaken,
as a developmental study, much in the vein alluded to
above.
Though much has been written on accountability,
and numerous designs have been drawn up for the evaluation
of specific bilingual education programs, we feel that
there is value in attempting to construct a paradigm for
accountability in bilingual education. It will attempt
to acknowledge what has been done to date, and as a
corollary it will highlight what remains to be done.
And hopefully, it will serve as a guide for those who
are novices at bilingual education, or accountability
procedures, or both. Finally, it is hoped that it will
serve as a means of overcoming what California State
Assemblyman Leon D. Ralph (1970, pp. 31-32) refers to as
"the schools' penchant for resisting evaluation" and
their "lack of responsiveness to parents and other members
of the community."
42
Importance Of The Study
The schools have been the object of harsh criticism
in recent years. Some of this criticism may be the result
of democratic institutions in a pluralistic society trying
to be responsive to too broad a base of constituents.
It will be impossible to satisfy completely the desires of
political liberals and conservatives, of proponents of
progressive education and of"traditional education, etc.
This problem can be obviated, somewhat, if schools stop
pretending to be able to do everything and if their goals
and objectives (arrived at through democratic processes)
are made explicit and if they are objectively monitored.
This is, in effect, what an accountability approach to
education implies.
In the case of Mexican-American students, the need
for such an approach is more cogent. The evidence
presented above suggests probable neglect of the needs of
Mexican-American students in the schools of the Southwest.
At the very least, there is lack of conclusive evidence
that schools are marshalling their resources and doing
their best to meet the needs and develop the potential
of such students. In some instances, the treatment of
Mexican-American students would appear to constitute
infringement of their civil rights.
It appears that schools would do well to attempt
43
to be more responsive and accountable to the public in
general and especially to minorities. This study was
undertaken with the intention of aiding those involved
with schools serving large numbers of Mexican-American
students to become more responsive and accountable.
Procedures To Be Employed In The Study
This work is undertaken as a developmental study
rather than experimental research. An attempt will be
made to develop a paradigm for the implementation of
accountability measures in bilingual education programs.
First, bibliographic research will be undertaken.
All references to "accountability*', "educational
accountability" or "bilingual education" appearing in
the Education Index or the Annual Index to Research in
Education (ERIC) will be checked and consulted where
deemed appropriate. Second, a paradigm for the
implementation of accountability in bilingual education
will be devised, based on suggestions encountered in the
bibliographic research. Third, the paradigm will be sent
to a committee of experts (all the persons who have been
designated to evaluate federally funded bilingual programs
in California) for their critical reaction. Fourth,
reactions of experts will be summarized and, where
appropriate, responded to. Finally revisions in the
44
paradigm, suggested by expert evaluators, will be detailed.
Scope And Limitations Of The Study
The purpose of this study is the production of a
paradigm or model for accountability in bilingual-
bicultural education ( with particular emphasis on Mexican-
American students in the public schools of the State of
California). Essentially it will be a description of
the systems approach to bilingual education. It is not
intended as a specific design for the evaluation of a
single project. At each step of the paradigm,
recommendations or suggestions regarding possible
implementation will be made. Though an attempt will
be made to be reasonably thorough in these suggestions,
there is no pretense made at being exhaustive. The
paradigm is not written with a specified body of users
in mind. However, the model will be constructed so that
it may be used in smaller school districts without
necessitating recourse to outside specialists simply for
consultation on procedural matters.
In devising the paradigm, two major sources were
consulted: the Education Index and the Annual Index to
Research in Education (ERIC). All references to
"accountability,” "educational accountability," or
"bilingual education" were noted and consulted where
45
deemed appropriate. References to evaluations of specific
bilingual programs were checked for specific practices or
procedures that might be included in the paradigm.
A survey of approaches to program evaluation was also made
in search of promising insights and suggestions.
The product of this study is an evaluation model.
In an attempt to establish its value, the suggestions of
several writers on evaluation will be followed. At the
suggestion of Scriven (1969), a "meta-evaluation" will be
attempted. That means, simply, that knowledgeable people
will be asked to evaluate the paradigm.
The original plan of procedure was to submit the
completed paradigm to a committee or jury of experts for
critical review. However, instead of arbitrarily selecting
such a committee, an attempt was made to solicit critical
reaction to the paradigm from all those persons designated
as evaluators of federally funded Title VII Bilingual
Education Programs in California. These evaluators were
asked to evaluate the paradigm according to the criteria
suggested by Stufflebeam, Foley, Gephart, Guba, Hammond,
Merriman, and Provus (1971): internal validity, external
validity, reliability, relevance, importance, scope,
credibility, timeliness, pervasiveness, and efficiency.
These evaluations are summarized below and, where called
for, a reply is made.
The original plan of procedure also called for
reaction to the paradigm by an "adversary evaluator."
While evaluators are asked to render balanced critical
judgments, an adversary evaluator (Levine, 1973) is asked
to question assumptions, design, completeness, relevance,
etc., with the specific purpose of finding faults or
possible defects and to establish a contradictory report.
The purpose of such a procedure is to enhance the validity
of an evaluation model at that stage of evaluation where
scientific criteria are most vague. Nevertheless, the use
of an adversary evaluation was not adopted for two reasons.
First, the attempts (documented below) to enlist the aid
of a suitable expert to perform the adversary evaluation
met with frustration. Second, it was decided that the
members of the dissertation committee, in effect, served
the function of adversary evaluators.
Summary Of Chapter One And Preview
Of The Remainder Of The Study
In Chapter One, the background of the increased
demand for accountability in education and the rationale
for the legal authorization and spread of bilingual
education programs have been examined.
It was noted that there were numerous reservations
expressed concerning the value of bilingual education.
These generally arose from fear that bilingual education
would diminish the intellectual or academic functioning
of young children; that instruction in non-English
languages would lessen the necessity, the opportunity and
the desire to learn English; and that, as a consequence,
acculturation and assimilation would be diminished. More
recent investigators who have examined the literature
have expressed the opinion that it is not bilingualism,
per se, which causes lowered intellectual functioning or
academic growth, but rather the socio-economic conditions
associated with being a member of a minority, the trauma
involved in beginning childhood education in a strange
language, and the methods and circumstances involved in
teaching the language.
It now seems generally accepted that early school
instruction should be in a child's stronger language or
in his mother tongue if he is monolingual. If handled
properly, it is felt that a second language may be
introduced at this time without adverse effects.
Recent research seems to bear this out. As far as the
weakening of assimilation is concerned, many proponents
of bilingual education feel that there is not one American
culture but many cultures, each as respectable as the next.
A goal of bilingual education is the preservation and
enhancement of minority language and culture. It is
hypothesized that the inclusion of Anglo-American children
48
in these programs will result in increased understanding,
better human relations and the greater acceptance of
multi-culturalism as central to the American way of life.
The minorities were not the only people in recent
years who felt that schools were not responsive to their
needs and desires. Great masses of Americans were
disenchanted with various aspects of education. This
led to the demand in the late 1960's and early 1970's
for greater effectiveness and efficiency.
The present study proposes to review pertinent
literature and to construct a paradigm or model for
accountability in the implementation of bilingual
education. Upon completion, the model will be submitted
to evaluators in an attempt to establish and increase
its validity and relevance.
Chapter Two of this study will involve a review of
the literature. First, pertinent literature on
accountability, systems approaches, and program evaluation
will be summarized. Then, review will be made of
literature in related areas deemed to be of importance
to this study.
Chapter Three will detail procedures followed in the
study and will contain the paradigm for accountability
in bilingual education.
Chapter Four will contain summaries of the responses
49
of persons designated to evaluate the paradigm. Response
will be made to comments of evaluators where appropriate
and possible.
Chapter Five will contain any revisions in the
paradigm called for by observations of the evaluators.
50
Definitions
The following terms are used with special meaning
in this study:
1. Accountability. The assumption of responsibility
for the accomplishment of goals and objectives. In this
study, the term is equated with a systematic process
including needs assessment, establishment and accomplish
ment of goals and objectives, and public reporting of
achievements.
2. Anglo-American. In this study, used loosely to
refer to those people whose mother tongue is English and
who would be associated with mainstream culture in this
country.
3. Biculturalism. Awareness of differences in
value and role patterns in two cultures, and the ability
to adapt to such patterns and to function in each culture.
4. Bicultural Education. Educational program which
attempts to incorporate the value and role patterns
characteristic of the minority culture, so that the
minority culture is legitimized in the school and students
develop the ability to function comfortably in both
cultures.
5. Bilingualism. Strictly speaking, the ability to
speak two languages with facility. (The term is used
loosely at times to refer to pupils entering American
51
schools who do not speak English).
6. Bilingual Education. The use of two languages,
one of which is English, as media of instruction for the
same student population in a well-organized program which
encompasses part or all of the curriculum and includes the
study of history and culture associated with the mother
tongue (U.S.O.E., 1971, p. 1).
7. E.S.L. English as a Second Language. Program
or course designed to teach English as a foreign or second
language in contrast with most English courses which
presume prior mastery of oral English.
8. Meta-evaluation. Evaluation of evaluation
theories, methods and designs.
9. Process evaluation. Periodic monitoring of the
program to see if the goals, objectives and strategies
that were planned are indeed being implemented in the
classroom and other areas of the program.
10. Program evaluation. Evaluation procedures
directed toward determining the effectiveness, efficiency,
relevance, etc., of an entire program in contrast with
evaluation procedures that focus on the progress and
achievement of individual students.
11. Progress evaluation. Periodic checks during the
course of program implementation to verify that progress
is being made toward planned goals and objectives.
52
12. T.E.S.O.L. Teaching (or Teachers of) English
to Speakers of Other Languages. Essentially, the same
as E.S.L. (q.v.)
CHAPTER TWO
Purpose Of The Present Chapter
In Chapter One, the historical development of
bilingual education in the United States was traced and
the backgrounds for the current demands for accountability
in education were alluded to. In this chapter, review
will be made of literature dealing with specific approaches
to accountability, systems approaches to program imple
mentation, and specific suggestions concerning bilingual
programs. This review was undertaken with a view to
synthesizing the most promising elements and approaches
to accountability in bilingual education. The resulting
synthesis is the paradigm for the implementation of
accountability measures which appear in Chapter Three.
Organization Of The Present Chapter
Following is a review of literature relating to
four areas of concern: Definitions and descriptions of
accountability, systems approaches to evaluation, program
evaluation, and specific suggestions found in the
literature concerning evaluation of bilingual education
programs.
53
54
First, the term "accountability" was investigated in
the literature. All references to "accountability" found
in the Annual Index to Research in Education (ERIC) and
the Education Index were consulted and an attempt was made
to reconcile the various definitions and descriptions of
"accountability."
The most comprehensive operational approach to
accountability appeared to be the systems approach. Hence,
a review of systems approaches to program implementation
and evaluation was undertaken.
The term "accountability" appears to have been coined
by Leon Lessinger. It was the result of attempts to carry
out more valid program evaluations as mandated by Congress.
Since "accountability approaches" were specific aspects
of the more general area of program evaluation, the broader
area, program evaluation, was also investigated.
Finally, in the course of reviewing the literature,
several specific suggestions concerning the evaluation
of bilingual programs came to light. Summaries of these
were included because of their relation to the purpose
of this study.
Literature Concerning Accountability
As mentioned in Chapter One, the beginning of the
"accountability" movement is often traced to Lessinger's
use of the terra before a subcommittee of the Congress
(Lessinger, 1969a). Though the use of the term may have
come to the attention of the public at this time, the
demand for accountability and the development of measures
to insure it were the culmination and crystallization of
trends that had been developing over long periods rather
than inventions de novo. Most authors agree that the
greatest stimulus to the development of accountability
approaches was the demand for program evaluations in
connection with E.S.E.A. legislation of 1965 and 1967
(Grobman, 1968; Provus, 1971). Whatever the catalytic
factor that initiated the movement, a scrutiny of the
literature suggests that proponents of the term see it as
promise that their particular concerns will be addressed.
The tem is explained and highlighted from almost as many
perspectives as there are writers.
Briner (1969) sees accountability as holding teachers
or administrators accountable for failure. Bain (1970,
1971) and Darland (1970) see it as the occasion for the
growth of professionalism and an increased voice of
classroom teachers in the governance of the schools.
One Florida project, recognizing the relationship between
autonomy and accountability, has focused on the building
principals of the schools. They are allowed complete
autonomy as long as they specify goals and meet them
56
(Nation’s Schools, 1970). Costa (1970) sees accountability
as the occasion for greater autonomy and responsibility
on the part of all involved in schools, students, teachers,
and administrators. Some see it as merit pay for more
effective teachers or as incentive pay (Beavan, 1970;
Education USA, 1970).
Others see accountability as an accountant might.
Nyquist (1970) sees accountability as the assurance that
funds are spent for purposes indicated and that the
expenditure is efficient. U.S. News and World Report
(1970) envisioned accountability as principally a more
careful cost-effectiveness control. Lessinger and Allen
(1969) refer to the allocation of resources to meet
performance criteria, and Ehrle (1970) sees accountability
as the allocation of resources in conformity with
priorities. The Dorsett project in Texarkana is frequently
referred to as the beginning of accountability in action,
and many authors seem to equate accountability with
performance contracting (Lessinger, 1970b; Bratten, et al.,
1970; Cass, 1970; Elam, 1970).
Others perceive accountability as a demand that
schools be responsive to the public will (McLenon, Caperton
& Nilson, 1970; Harrison, 1970; Glass, 1972). A logical
vehicle for engineering responsiveness and accountability
is the voucher system (Lieberman, 1970). In this vein,
some see accountability as the duty to account, to report
57
to constituents on the true state of affairs (Cook, 1970;
Harrison, 1970). Bahr (1969) and Kruger (1970) imply
that it entails scrutiny by outside evaluators (the
educational audit). Others see National Assessment as the
way to account (Pierce, et al., 1970; Grieder, 1969).
Phillips (1970) sees accountability as means to
offset possible teacher mediocrity as a result of tenure
laws, an assurance that they will take seriously the task
of teaching all students, especially the poor and
minorities. Others (Geller, 1971; Clark, 1970) see it as
a way toward truly educational opportunity for the poor.
Harlacher and Roberts (1971) envision it as a procedure
for the continuous and systematic upgrading of instruction
by stating goals and objectives and checking effectiveness
of accomplishment.
Kennedy (1970) equates accountability with management
by objectives, being freed from routine procedures and
regulated practice on the stipulation that objectives are
met. Tyler (1970) feels that the accountability movement
will be characterized by the use of criterion-referenced
tests rather than norm-referenced tests. Glass (1972)
suggests that accountability would include the use of
quasi-experimental designs to compare alternatives.
Lessinger (1969) and Peterson (1970) state that
accountability is merely evaluation, but evaluation in
58
terms of student learning or "output" rather than in
terms of the more traditional measures that are used to
evaluate programs. As a result, Lessinger (1970a) feels
that education will become less teacher-centered, more
learner-centered and relevant. Others highlight the
aspect of stating behavioral objectives and meeting them
(Deterline, 1971) or evaluation by objectives as opposed
to the "cult of the gut response" (Premack, 1971).
Finally, and most commonly, accountability is seen
as the implementation of P.P.B.S. (Duncan, 1971) or as
the application of the systems approach to education
(Kaufman, 1971; Deck, 1971).
The accountability movement, of course, is not
without its critics. Robert Bhaerman (1971), head of
the national A.F.L.-C.1.0. teachers' union, expressed fear
that the movement would lead to dehumanization, an
education monopoly, and teacher mustrust. Helen Bain
(1970 and 1971), head of the N.E.A., and D. D. Darland
(1970) caution that teachers cannot be responsible if they
have no power. Garvue (1971) reiterates the fear that
accountability might come to mean the unilateral
establishment of unrealistic performance objectives for
classroom teachers. He feels that all sectors of
society should be accountable and that schools cannot be
truly responsible for results if they operate in a vacuum.
59
Jordano (1971) expresses concern that the use of
standardized tests as accountability measures would be
demoralizing since they do not differentiate goals and
objectives and do not promote variety. He feels the
tendency might be to teach to the test and the net result
may be mediocrity in the form of compliance with minimal
objectives. Cunningham (1969) notes resistance to
accountability in the concern over lack of valid
information on which to make accountability judgments,
the fear of appraisals and judgments, disinclination
toward change, self-satisfaction, ingrained bureaucracy,
and concern for professionalism versus community action.
Glass (1972) feels that virtually all present approaches
to accountability have serious defects. Dolmatch (1970,
pp. 19-20) states that in its present form, accountability
is a marketing device for vendors, a selling device for
school administrators, and a slogan for practitioners.
In summary, critics of accountability appear to voice
concern over possible misapplications of accountability
approaches or over inadequacy of presently developed
techniques to realize the goal of accountability rather
than over the concept itself. In general, there seems
to be agreement that accountability means responsiveness,
responsibility for results and efficiency. The most
inclusive approach to accountability, subsuming other
60
elements or approaches (as performance contracting, use
of behavioral objectives, etc.) appears to be P.P.B.S.
or the systems approach to educational management
and evaluation.
Literature Concerning Systems Approaches
Hill (1972) traces systems theory to the 1920's when
two trends began. First, General Systems Theory developed
in Physics and the Biological Sciences. Boulding (1956)
sees General Systems Theory as an attempt to reach a broad
view of those commonalities that might exist between
elements and relationships within a scientific body of
knowledge, and of those elements and their interconnections
common to all sciences. Second, applied Systems Theory was
developed at the same time at Bell Laboratories. Here the
attempt was to reduce the system to a mathematical model,
enabling managers to arrive at sophisticated analytical
techniques in arriving at decisions (Hill, 1972).
This approach received wide attention and support following
publication of Cybernetics (Weiner, 1948). Shortly
thereafter Rand Corporation formalized the Planning,
Programming, Budgeting approach to management (Alioto
and Jungherr, 1971, p* 8).
Hill (1972, pp. 16-21) lists six essential components
of the systems model: (1) mission statement; (2) design
61
criteria; (3) performance goals; (4) inputs; (5) outputs;
(6) human feedback circuitry (the decision-making
associated with the system).
Carss (1969* pp.43-44) lists a number of programs
in education that have grown out of systems theories:
instructional systems, such as the I.P.I. Program;
project management systems, such as P.E.R.T., C.P.M., etc.;
management information systems, the establishment of a .
data-base to meet the "need to know" of decision makers
and those concerned with the operational control of
schools on a day to day basis; operations research or
analysis, like simulation, the attempt to study the effect
on output of changed inputs without direct intervention
in operation of the school; planning-programming-
budgeting systems.
Alioto and Jungherr (1971, p. 10) state that the
basic components of P.P.B.S. are the generation of a.
series of objectives that assist the school to meet its
responsibility to society, study of alternative approaches
to objectives and selection of best alternative, allocation
of resources, and review of actual accomplishments leading
to revision of objectives, priorities or programs.
The basic steps one follows in the implementation
of P.P.B.S. are: (1) the development of broad goals and
objectives; (2) design of program structure; (3) the
62
definition of objectives and means for measuring their
accomplishment; (4) identification or design of alternative
approaches to attain objectives; (5) analysis of cost-
effectiveness of alternative approaches; (6) selection of
best approach and allocation of resources; (7) evaluation
of results and feedback to the planning process.
(Koerner, 1972; Alioto and Jungherr, 1971; Hartley, 1968).
Literature Concerning Program Evaluation
The difference between systems approaches and program
evaluation models is mainly one of emphasis. Prior to
E.S.E.A. legislation, educational administrators were
trained in management strategies that did not generally
include extensive consideration of research or evaluation
methodologies (Hartley, 1968, pp. 47-63). Evaluation
specialists, at the same time, tended to be trained in
techniques that were more appropriate for the aims of
science, the generalizability of results, than for aiding
managers and decision makers (Grobman, 1968; Stufflebeam,
et_al., 1971).
In surveying literature on program evaluation, most
authors seem to concur that very little was done prior to
1965 other than the work of Ralph Tyler (Metfessel and
Michael, 1967, p. 931; Stufflebeam, et al.. 1971, p. 8;
Provus, 1971, p. 10). Tyler (1942) claimed that the
63
purposes for which evaluation was undertaken were usually-
presented as too narrow in scope (for grading, reporting,
promotions). He felt that other important purposes of
evaluation were to check on the effectiveness of the
institution, to help teachers and pupils clarify their
goals and move toward them more concretely, and to provide
a sound basis for constructive and cooperative community
relations. (Thus, he foreshadowed the movement for
accountability by a generation.)
Following is an outline of the procedure that Tyler
(1942, pp. 498-500) suggests be followed in evaluation:
1. A. Formulate a statement of educational
objectives.
B. Classify statements of objectives into
major types and indicate the kinds of
evaluation procedures essential to an
adequate appraisal.
2. A. Define each of these objectives in
terms of behavior.
B. Identify situations in which students
may be expected to display these types
of behavior.
3. Select and try promising methods of obtaining
evidence regarding each type of objective.
(This may involve use of existing instruments
or construction of new instruments).
4. Select, on the basis of preliminary trial,
the more promising appraisal methods for
further development and improvement.
5. Devise means for reporting and using the
results of various instruments of evaluation.
64
Approaches to program evaluation following Tyler
appear to have several things in common: they are
increasingly influenced by systems theory, they are
increasingly complex, and, basically, they are not greatly
different from the model originally proposed by Tyler.
This has led Popham (Provus, 1971) to remark:
...if you look at evaluation models and
structural models in a continuum from the
simple to the esoteric, you find that the
esoteric is more conceptually satisfying,
but it may be promulgationally less
effective.
In other words, I have always been
pleased with Professor Tyler's model,
because although it is fairly simple
and it's just got a few empty boxes,
I find I can transmit that empty box
model to people and they can tolerate
it (p. 132).
Following E.S.E.A. legislation, models for program
evaluation appear with greater frequency. Metfessel and
Michael (1967) offer what appears to be an expansion of
Tyler's model. They have embodied, as an explicit
procedural step, Tyler's assumption that teachers, parents,
and students (and, presumably, anybody else concerned)
be involved in evaluation. They have emphasized periodic
observation of behavior (implying that process is
monitored as well as product), interpretation of data
(implying that outcomes are explained as well as judged
significant or not), and recommendations for further
65
implementation, modification or revision of goals and
objectives (implying that evaluation is formative as
well as summative). A further notable point is their
observation that judgmental decisions are involved
throughout all phases of the evaluation process. They
seem to imply that evaluators are clinicians or
"decision-makers."
Another model used for program evaluation is that
developed by a committee commissioned by the E.esearch
Advisory Committee of Phi Delta Kappa and headed by
Daniel Stufflebeam (Stufflebeam, et al., 1971). They
proposed the C.I.P.P. Model (Context, Input, Process, and
Product) for evaluating programs. In their view, the
purpose of evaluation is "not to prove but to improve"
(p. v). They see evaluation as "the process of
delineating, obtaining, and providing useful information
for judging decision alternatives (p. xxv). They
oppose evaluation to research methodology and state that
it should facilitate decision making and management.
For this reason, evaluation design must take into
account the decision making setting. This setting may
be homeostatic (to maintain the system), incremental
(to implement small changes), or neo-mobilistic
(to implement large scale innovations). Decisions may be
synoptic (based on consideration of all possible
consequences for all possible alternatives in terms of all
relevant criteria), disjointed-incremental (based on little
information and geared to small changes) or geared to
planned change (for complex change over a long time span
based on research, diffusion, and adaptive procedures).
Finally, evaluation procedures must consider context
(purposes, needs, problems, opportunities as related to
values systems), input (resources and procedures required),
process (actual processes and their congruence with
intended processes), and product (actual results and their
congruence with intended results).
Provus (1971, Pp. 46-58) provides a model that he
calls "discrepancy evaluation" for educational program
improvement and assessment. He posits five stages in
program evaluation: (1) design; (2) installation;
(3) process; (4) product; and (5) cost. At each stage,
a judgment is made about the congruence or discrepancy
between the actual situation and some standard. In the
first stage, the design is derived from consensus of
those involved. The standard for judgment of the design
is expert opinion regarding theoretical and structural
soundness. Upon revision, the design becomes the standard
of judgment for steps (2) through (5). At each stage,
discrepancy between standard and reality leads to
restructuring of that stage, revision of the design,
67
or both.
The Center for the Study of Evaluation at U.C.L.A.
proposes an approach that emphasizes needs assessment
for the best statement of goals and objectives, program
planning (systematic consideration of alternatives),
implementation evaluation (to insure that proposed program
is actually being followed), progress evaluation (to check
on attainment of mediating objectives), and outcome
evaluation (Klein, et al., 1971).
The foregoing evaluation models have a linear
character to them. The perspective is the logical-
temporal sequence. Implicit is the idea that these plans
can be placed on a time-line in orderly sequence (though
there may be overlapping between steps). Other models
have been proposed that are more three-dimensional than
linear. The focus is not centered on the evaluation task
as performed in time, but on the logical totality of the
evaluation process.
Robert Hammond (1967) proposed the E.P.I.C. model
(See Appendix B). This approach entails systematic
consideration of variables considered to be operating in
a three-dimensional structure. The three dimensions are
instructional, institutional, and behavioral.
Implementation of the evaluation involves: (1) establish
ment of baseline data and definition of what is to be
68
evaluated; (2) definition of descriptive variables in the
instructional and institutional dimensions; (3) statement
of behavioral objectives; (4) assessment; and (5) interpre
tation of relationship between factors, determination of
effectiveness of the program, and feedback.
Robert Stake (1967) suggests a pair of matrices to
aid the evaluator (See Appendix C). The purpose of the
matrices is to highlight the logical contingency between
antecedents, transactions, and outcomes, and the congruence
between intents, observations, standards, and judgments.
In summary, it appears that since the early 1950's
approaches to management and approaches to evaluation have
tended to converge. The influence of systems approaches
on management and the essential position of operations
analysis in the systems approach have enhanced the concern
of educational administrators over evaluation methodology.
The development of evaluation methodology, springing
originally from experimental research methodology, has
come closer to the systems approach in concern over goal
statements, process evaluation, and continuous and
systematic feedback. Though some models for program
evaluation are characterized by a logical-structural
approach, the majority focus on procedural steps similar
to the systems approach and there appear to be few
essential differences between them.
69
Literature Pertaining To Approaches To Evaluation
Of Bilingual Programs
In the Manual for Project Applicants and Grantees,
the U.S. Office of Education (U.S.O.E., 1971) suggests
a systems approach to implementation and evaluation,
supplemented by certain accountability procedures.
The basic components are assessment of needs, program
planning and design, derivation of goals and objectives,
component design (i.e., planning of instructional program,
acquisition and development of materials, staff develop
ment, parent and community involvement), statement of
performance objectives, evaluation, and coordination with
other programs. Also suggested as possibilities are the
use of the independent educational accomplishment audit
and the letting of performance contracts for technical
assistance.
Saville and Troike (1971, pp. 65-71) merely allude
to the need for evaluation and emphasize the desirability
of establishing baseline data. They suggest an evaluation
program that gives heavy emphasis to the assessment of
competence in both languages, that surveys home conditions,
and that takes intellectual potential into account.
John and Horner (1971, pp. 142-163) suggest that any
evaluation procedure must: (1) carefully specify the
population under study (i.e., note the nature and extent
of the bilingualism involved as well as related socio
economic factors); (2) specify the general and specific
goals of the program (and the selection of instruments
to measure the specific components of desired goals);
(3) specify the educational procedures (i.e., careful
description of the "independent variables"); and
(4) specify the choice of assessment techniques.
Expanding on the last component, they suggest three areas
for assessment: general language competence (including
language usage and language proficiency), intelligence
or general ability, and special skill achievement.
Suggestions are made under each of these headings for
standardized or systematic assessment.
Andersson and Boyer (1970, Vol. 1, pp. 59-68) also
offer a series of suggestions for program implementation.
They highlight formation of a study committee and of an
advisory committee, appointment of a program coordinator,
planning of a publicity program and parental involvement,
preparation of teachers and aides, statement of objectives,
planning of curriculum, preparation of materials,
experimentation, and testing.
Ulibarri (1970, p. 41) suggests an approach to
implementation of bilingual education programs that
includes: (1) conceptualization of ideas for change;
(2) community survey; (3) determination of type of program;
71
(4) setting objectives and first definition of program;
(5) piloting; (6) implementation; and (7) evaluation.
Systematic suggestions regarding evaluation of
bilingual programs tend to fall in one of two major
categories. Writers with strong linguistic concerns tend
to conceptualize evaluation procedures as similar to
experimental procedures, with the purpose of clarifying
points of interest to linguists (Saville and Troike, 1971;
John and Horner, 1971). Others (Andersson and Boyer, 1970;
U.S.O.E., 1971; Ulibarri, 1970) offer suggestions that
imply a systems approach to evaluation.
Summary Of The Review Of The Literature
The introduction of the term "accountability” into
the literature generated a wave of articles on the topic
before there was general consensus on the exact meaning
of the word. In general, there seems to be agreement that
accountability implies greater responsiveness to public
will, evaluation in terms of output rather than by more
traditional means, and the clarification of responsibility
for results and efficiency. The most inclusive approach
to educational accountability is the systems approach to
educational management and evaluation or P.P.B.S.
This approach is an outgrowth of management concerns, but
has become very widespread since the federal government
72
demanded systematic evaluations of programs funded under
the E.S.E.A. Act of 1965 and 1967. Prior to that time,
evaluation methodology had been considered to be an
extrapolation of the methodology employed in experimental
research. Since 1967, numerous models for program
evaluation have been suggested. The majority of these
are variations of the systems approach to evaluation.
Handbooks on bilingual education tend to be concerned
with the general rationale for and background of such
programs. Emphasis tends to be on start-up and procedural
concerns. Where the question of evaluation is addressed,
writers with linguistic concerns tend to suggest
experimental type evaluation procedures. Others suggest
a systems approach to implementation and evaluation.
CHAPTER THREE
Method Employed In The Study
This is not an experimental study, but an attempt to
develop a model for the implementation of accountability
measures in bilingual programs. Following is a summary
of the activities undertaken in the course of the study.
(1) Background research was undertaken in the areas of
accountability, program evaluation, and bilingual
education. (This research is documented in Chapters I and
II.) (2) A paradigm for the implementation of
accountability measures was constructed. (The paradigm
is contained in Appendix A. The explication of the para
digm follows in this chapter.) (3) The paradigm was sub
mitted to a committee of expert evaluators for a ' ’ meta-
evaluation", i.e., their critical reactions to the paradigm
(Scriven, 1969). (4) The reactions of the evaluators
were summarized. In response to their comments, the
paradigm is revised, defended or further explained.
(Summaries of the comments of the evaluators and response
to them is contained in Chapter IV.)
73
74
Introduction To The Paradigm For The Implementation
Of Accountability Measures In Bilingual
Education Programs
Following is a model or paradigm to aid in the
implementation of accountability measures in a bilingual
education program. (See Appendix A*)
The paradigm was devised with Mexican-American
students in mind, and more specifically those attending
the public schools of the State of California.
The paradigm is not intended to be exhaustive, but
only suggestive. Nevertheless, it is hoped that it will be
sufficiently complete and clear to serve as a guide to the
implementation of accountability measures, particularly in
small school districts or single schools. The emphasis is
on accountability measures, not bilingual education
methodology. It is not an explanation of how to implement
bilingual education but a systematic approach to the
evaluation of such program implementation.
Assumptions Implicit In The Construction
Of The Paradigm
Implicit in any approach to evaluation are certain
assumptions. The essential components to an accounta
bility approach to evaluation are responsiveness to public
will, effectiveness, and efficiency in the attainment of
goals and objectives. Of these, the effectiveness of the
75
specific program is the central concern. The most
practical criterion for judging the effectiveness of a
program is the observation of results in comparison with
stated goals and objectives. The more clear, precise, and
objective such statements of goals and objectives, the
greater will be the probability of consensus regarding
the effectiveness of the program. Clarity, precision, and
objectivity of objectives are intended to enhance their
validity as indicators of quality and effectiveness.
Trivial objectives or objectives that do not bear a clear
and cogent relevance to program goals are no guarantee of
real accountability. Thus, accountability implies that
personal judgments will be made. Such judgments, however,
will be informed by most careful attention to objective,
reliable, and valid data.
Accountability also implies personal responsibility
on the part of persons involved and the autonomy that
such responsibility entails. It implies a high degree
of public involvement in setting goals and objectives
and in assessing program effectiveness. Finally,
accountability implies sincere and open public "accounting"
regarding programs.
Appendix A contains the paradigm for accountability
in bilingual education. The model contains two columns,
one detailing program implementation procedures and one
detailing process evaluation procedures. In the
implementation of any program, information will have to be
obtained in order to modify the program where advisable
(formative evaluation), and outcomes will have to be
examined in order to judge program effectiveness
(summative evaluation). All these factors related to
evaluation are an integral part of a systems approach to
program implementation. In this paradigm, process
evaluation procedures have been set off as distinct
because they are more reflexive. While the distinction
is somewhat tenuous, process evaluation is directed more
toward the establishment of the validity of program
procedures than toward the effectiveness of the program
in meeting its goals. Of course, process evaluation is
also a means toward the establishment of program
effectiveness. In essence, process evaluation is to
establish that procedures planned for meeting goals and
objectives were executed faithfully as planned.
There are five major steps listed in program
implementation and evaluation: involvement, commitment,
implementation, summative evaluation, and accounting.
The designation of five steps rather than four or six,
etc., is somewhat arbitrary. Each step might have been
broken into smaller components. Likewise, steps might
be grouped (for example: involvement and commitment;
77
summative evaluation and accounting). The intent was to
include as few complicating details as practical without
omitting essential components of a program of
accountability. The stages listed here as commitment,
implementation, and summative evaluation are part of
any systematic approach to program implementation and
evaluation. They subsume elements such as mission
statements, consideration of alternatives, selection of
the best alternative on the basis of cost-effectiveness,
implementation, feedback, evaluation, etc. The remaining
two stages, involvement and accounting, were highlighted
because of the major concerns associated with implementing
a program of accountability in bilingual education.
Systematic involvement is designated as a distinct stage
because of the widely expressed concern over whether the
Mexican-American community has been adequately taken into
account in educational planning. Formal accounting is
designated as a distinct stage because it appears to be
a relatively uncommon practice (at least in the case of
accounting to the public) in spite of the apparent
widespread support for the idea.
Following is an explication of the five stages we
have designated as components of a program committed to
accountability: Involvement, Commitment, Implementation,
Evaluation, and Accounting. (Vd. Paradigm for the
Implementation of Accountability Measures in Bilingual
78
Education Programs, Appendix A, p. 155).
Involvement
The emphasis on community involvement in the
implementation of bilingual education programs derives
from several underlying principles. First, public
institutions should serve all the public. This implies
a periodic examination to assure that schools are indeed
serving their constituents. Consultation with the public
is an integral part of such an examination. Furthermore,
unless a systematic and objective examination of the
schools (or any other institution) is made from time to
time, there is danger that the concept of working
towards ends (the idea of service) will fade, to be
replaced with preoccupation over procedural concerns.
Almost unconsciously intent can come to be taken for
realization. The priorities that communities may attach
to elements of the curriculum can change over time.
Unless we continually re-examine such priorities in the
light of real community needs and wishes, our curriculum
will tend to become irrelevant.
There are two facets to community involvement.
First, all the members of the community should be
consulted whether directly or through spokesmen. Since
community sentiment is such a dynamic thing, this
79
consultation should not be static but continuous and
periodic. Second, along with the development of
communications networks, involvement implies
responsiveness in meeting community needs and desires.
California's Joint Committee on Goals and Evaluation
(1972) suggests techniques for systematic involvement of
community members. Of special interest are the principles
of involvement suggested by Wilcox (1972). The United
States Office of Education (1972, pp. 67-72) suggests
systematic involvement of parents, advisory groups, and
technical experts. Such involvement would extend to all
aspects of the program, including planning, implementation,
and evaluation. Some functions of a community advisory
group (composed of parent and community representatives)
are to help develop and implement the project, to
represent parental and community interests, to assist in
evaluating the project, to help establish procedures for
consideration and resolution of grievances, to promote
the project in the community, and to help mobilize
community resources in support of the project (California
State Department of Education, 1968).
As noted above, community consultation and
participation are not static events, but should occur
continuously throughout the program. Systematic means
to consult with and report to the community at large
should be developed. With regard to community
consultation. Stufflebeam, et al.. (1971, p. 155) suggest
the use of questionnaires, public hearings, the Delphi
technique, neighborhood seminars, survey committees,
P.T.A. meetings, Board meetings, Faculty advisory groups.
To this might be added frequent meetings of community
advisory groups, studeht and alumni groups (where
feasible), consultation with cultural or ethnic groups,
and consultation with technical experts. In developing
open communications and reporting to the community members,
problems may arise because of language differences.
Where significant numbers of parents or community members
are weak in English, attempts should be made to communi
cate in Spanish. This entails entails at least report
cards and bulletins in Spanish as well as English, and
the presence in the schools of bilingual personnel to deal
with Spanish speaking parents. Ideally, receptionists
in schools with Mexican-American students will be
bilingual. If this is not the case, translators might
be designated at each site and a telephone exchange
designated specifically for use by Spanish speaking
parents. Periodic newsletters in two languages might be
sent to parents. Where numbers warrant, Spanish speaking
chapters of the P.T.A. could be formed. Translators could
be present at school board meetings, if not all the time,
at least at designated meetings. Relations could be
established with Spanish language periodicals and radio
or television stations in the area. Where none exist,
provision might be made for press releases containing
school news in English and Spanish to be printed in the
local paper on a specific day each week. Where practical,
community relations specialists might be employed with the
specific participation of community members.
The consultation and participation described above
would be focused toward making schools more responsive
to the desires and needs of the community. In theory,
the wishes of a community for its schools are expressed
periodically in the results of the school board elections.
In practice, however, the community is rarely aware of
the position of various board members on specific matters.
Elections cannot be taken as blanket endorsement of school
philosophy, goals, objectives, and curriculum. To insure
responsiveness, the schools can undertake extensive and
detailed community surveys. This can be accomplished
through questionnaires, opinionnaires, solicitation of
reports from local groups (such as the Chamber of Commerce,
churches, civic and social organizations, ethnic groups,
youth groups, Welfare Department, Human Resources
Development Agency, etc.), P.T.A. reports, "town hall"
meetings conducted by the school board, etc.
In assessing needs prior to implementation, the
United States Office of Education (1971, pp. 13-19)
suggests consideration of numbers and location of children
from environments in which the dominant language is not
English; evidence concerning the linguistic competence
of the children; evidence that the educational needs of
children are not being met; and evidence concerning the
socio-economic level of their families. It must be kept
in mind, however, that these suggestions were made to
assess needs that might be met within the legal guidelines
of Title VII. They are by no means open-ended
considerations. California's Joint Committee on Goals
and Evaluation (1972) suggests several approaches to the
assessment of needs. In general, such approaches employ
questionnaires or the use of structured group meetings.
Ulibarri (1970, pp. 43-44) suggests the use of survey
techniques to provide data on socio-cultural patterns
(religion, family, education, health, recreational
activities, economic attitudes, and politics). Such data
could be used to discern characteristic value patterns of
ethnic group members by sex, age, occupation, education,
socio-economic status, etc. These characteristic value
patterns could then be compared with characteristic value
patterns of school personnel. Significant differences
might indicate the presence of cultural conflict in the
83
community and in the school that would have to be
investigated before the program could be planned
effectively. Of course, central to the planning of a
bilingual education program is the establishment of the
linguistic needs of the community. Saville and Troike
(1971, p. 20) suggest a questionnaire that can be adapted
to determine the extent and degree of bilingualism in the
community.
With regard to process evaluation at this stage,
systematic checks on involvement are not highly developed.
The U.S. Commission on Civil Eights used demographic data
and systematic observation techniques in evaluating the
schools* response to the needs of Mexican-American
students. Attention was given to the following data:
segregation by district or by school; relative
representation of Mexican-Americans on faculty and school
boards; ratio of Mexican-American college entrants, drop
outs, jobless between age18-25; under-achievers; over-age
for grade placement, repeaters, and participants in extra
curricular activities; estimate by teachers and principals
of linguistic competence of students and relative number
of programs to enhance such competence where judged
deficient; evidence of consideration of Mexican-American
culture in the formal curriculum of the school; use of
Spanish on report cards and other communications in cases
84
where parents do not speak English; formation of Spanish
speaking P.T.A. groups; existence of advisory committees;
employment of community relations specialists or technical
consultants; objective examination of differential
treatment of students (U.S. Commission on Civil Rights,
April, 1971; October, 1971; 1972; 1973).
Apart from such demographic considerations, there
are few standards against which to evaluate involvement.
However, accountability might be enhanced if specific
persons (v. gr. an ad-hoc committee appointed by the school
board, the advisory committee or the program evaluator)
reviewed systematic attempts to enhance community
involvement and school district responsiveness and rendered
a judgment regarding their effectiveness. Initially such
reviews might verse about descriptive statements of what
had been done to date. Such reviews could include
suggestions for incremental improvements in the future.
Such suggestions would eventually form the components of
a series of goals and objectives regarding community
involvement.
Commitment
The commitment stage of program implementation
involves clarification of philosophy and goals, formal
commitment to the program, pre-assessment, adoption of
85
implementation model and major curriculum elements,
formulation of program objectives, and development of
instrumentation for summative evaluation.
Generally, instructional programs will be enhanced
by general consensus regarding program philosophy.
In the case of bilingual education, however, the approach
is so new and the issues are so complex that consensus
may prove elusive. For example, some may be inclined to
foster the permanence of Spanish-speaking barrios while
others may see this as separtism. Some may feel that
formal courses in Spanish and in Mexican-American culture
should be offered as separate courses adjunct to the
academic curriculum while others feel they should be
integrated into the entire curriculum. Some may see the
use of Spanish in early grades as a more effective means
of introducing children to English later on, while others
support the preservation of Spanish throughout the school
years. Some will see the schools as "melting pots,"
socializing agencies whose function is to feed children
into the mainstream culture. Others will see schools
dedicated to training in the basics of literacy with
no call to weaken minority heritage even with the prospect
of greater economic rewards for graduates.
Given the fact that the issues in bilingual education
are so complex, an attempt at the formulation of a broad
and exhaustive philosophy is likely to prove frustrating.
A preferable approach would seem to be to establish a
loose knit set of working principles. Such principles
might include the rationale for the existence of the
program, and a statement of the pedagogical, ethical,
cultural, and political principles that people feel are
involved. A number of such working principles can be
found in the literature on bilingual education.
See Andersson and Boyer (1970, pp. 41-58); John and
Horner (1971» pp. xxii-xxxv); Saville and Troike (1971,
pp. 10-23). As statements are formalized, community
members can react to them and the total statement can be
revised accordingly. Most people are reactive rather
than proactive thinkers. We tend to clarify our personal
philosophies not so much by logical cogitation, but in
situations demanding decisions where we must reflect on
the meaning of our actions. The same can be true of
statements of program philosophies. They can be modified
and enlarged over a period of time. The use of logical
analysis takes place after the statement of principles,
when the community, faculty and expert opinion are
consulted regarding validity of the principles.
While philosophy statements may be loose knit and
relatively vague, goal statements, by theri nature, must
be clear. An attempt to be accountable implies a clear
87
statement of what is to be accomplished.
The United States Office of Education (1971,
pp. 29-30) suggests a number of possible program goals:
adult mastery of both languages by children who were
initially English-speaking and by children who were non-
English-speaking; progression through school, graduation
and entrance into college by non-English-speaking
children; enhanced pride of non-English-speaking students
in their language and heritage; development by all who do
not choose to pursue higher education of the skills
necessary for employment; participation of non-English
speaking parents in school programs to the same degree as
English-speaking parents; greater esteem of participants
for a multicultural society; and the development of
attitudes and skills on the part of the faculty necessary
for the maintenance of an effective bilingual program.
The California law authorizing bilingual education
also suggests several program goals: the development of
competence in two languages for all participating pupils;
positive reinforcement of the self-image of all
participating children; and the development of intergroup
and intercultural awareness among pupils, parents, and
staff in participating school districts (Assembly of the
State of California, 1972). Again it should be noted
88
that such goal statements are suggestive and, in the
above cases, are limited by the specific intent of the
laws in question.
At some time furing the implementation of any
educational program, a formal commitment to the program
must be made. This means simply that those in authority,
school board and administrators, must authorize the work
to be done. As mentioned above, the ends of accounta
bility are better served if assignments specify the exact
nature of the work to be performed, the expected outcome,
the person responsible for each component task, the
expected time of completion, and the person responsible
for juding successful task completion.
Prior to implementation of a specific model of
bilingual education, pre-assessment is called for.
This is, in a sense, an extension of needs assessment.
Once community needs have been clarified and translated
into educational goals, assessment indicates how far
the school is from the realization of those goals.
Pre-assessment in bilingual education should include
examination of two major factors: the status of students
in relation to stated goals, and adequacy of school
district resources to facilitate goal attainment.
In assessing the status of students, specific
evaluation will derive from program goals. It would seem
logical, however, to give systematic consideration to the
evaluation of student command of English and Spanish to
determine the need for special training in English,
Spanish, or both. In programs with a bicultural focus,
evidence might be sought regarding the possible existence
of cultural trauma in students, the level of self-esteem,
and the awareness and appreciation of their culture.
In assessing the status of students, it might be well to
use the same measures that are developed for summative
evaluation of the program. The results of the pre
assessment can then be used as pre-test results, serving
as a base-line for the specification of particular
program objectives, particularly where appropriate
criterion-referenced evaluation measures can be employed.
In assessing resources, the primary concern is the
ability of teachers to manage bilingual programs. Though
teachers might manage by teaching in teams or by working
with bilingual aides, the ideal would be for teachers to
be bilingual themselves. Another essential characteristic
for successful teachers is expertise in methods of
Teaching English as a Second Language. The training in
Language Arts usually given to teacher candidates is
intended to prepare them to teach young children who have
already mastered oral English. However, in dealing with
children whose primary language is not English, an
altogether different approach and methodology must be
employed in the early years of school. Where teachers
have not been trained for such an approach, a special
training program would be called for. Along with
bilingualism and mastery of second-language teaching
techniques, many authors point to requisite cultural
and attitudinal factors of teachers. Mention was made
above of Ulibarri's suggestion that a value survey be
undertaken. Where there is a discrepancy between the
values held by community members and by the faculty, a
seminar might be offered to foster awareness of such
differences and to assess the ability of teachers to
accept such differences. Simon, Howe, and Kirschenbaum
(1972) offer a number of suggestions for implementing
such seminars. Jaramillo (1973) also offers suggestions
for the evaluation of cultural differences.
Ulibarri (1970, pp. 19-21) suggests that prospective
teachers in a bilingual education program be checked for
adequate credetnial training, strong foundations in
psychological principles, ability to analyze cultural
differences, knowledge of the impact of social class
membership, personal empathy, cultural sensitivity, and
bilingualism. (He states that since so few teachers will
attain this level of development, the primary concern
should be with the elimination of those who exhibit
91
extreme defensiveness, anxiety, or militancy.)
Andersson and Boyer (1970, Vol. 2, p. 89) suggest that
teachers take periodic examinations in their second
language and that they be rated on the scale of
Qualifications for Teachers of Modem Foreign Languages.
It is unlikely that many schools will have textbooks
and other materials required for bilingual education
programs unless they have undertaken such programs in the
past. Most handbooks on bilingual education contain
extensive bibliographies that can be consulted. Materials
should always be evaluated to determine their effectiveness
in fostering academic objectives. Suggestions for such
evaluation of materials can be found in the manual of the
United States Office of Education (1971, p. 36). It is
also suggested that such materials be evaluated regarding
their treatment of minority groups. The Detroit Public
Schools Staff (1969) offers practical suggestions in this
regard.
Along with faculty and materials, of course, the
availability of other resources will have to be checked,
such as necessary funds, locale of operation, sufficient
time for completion of essential tasks, necessary support
personnel, etc.
Following the pre-assessment, decision is made
regarding the specific program model to be implemented.
92
Mackey (1969) suggests a typology of bilingual education
that can be of great aid in the consideration of
alternative approaches that might be adopted. At this
stage, decisions will have to be made regarding the nature
of the bilingual education program. This will derive from
the needs assessment and goal statements adopted by the
district. Among specific issues that will have to be
considered are the purpose of second language training (to
facilitate later transfer to English or to be maintained
throughout the educational career), the cultural intent (to
facilitate assimilation into mainstream culture or to
foster preservation of different minority cultures), the
relative use of the language in instruction (some classes
in one language and others in another, or all classes in
both languages), student distribution (segregation
according to stronger language or integration), desired
rate of change to new methods (immediate or gradual), the
relative status of the languages in the school environment
(equal or subsidiary to English), etc. As the community is
consulted regarding needs assessment, as philosophy and
goals are clarified, these issues should be determined at
least implicitly. The type of program adopted will be a
logical consequence of such determination and curriculum
selection can proceed as an extension of program adoption.
As curricular components are developed, program
93
objectives should be clarified and specified. The work
of Mager (1962 and 1972) is frequently cited in discussion
of educational objectives. Another frequent suggestion
is that objectives be arranged in hierarchical order.
In this respect, the taxonomies of Bloom, Englehart, Purst,
Hill, and Krathwohl (1956) and of Krathwohl, Bloom, and
Masia (1964) are most frequently cited as models.
Numerous other taxonomies of objectives exist and may
serve as models. Of special interest in bilingual
education are the taxonomies of second language behaviors
of Lee (1972) and Norris (1970). Though not specifically
concerned with behavioral objectives, Sako (1969) offers
a very fine overview of procedures that may be used in
evaluating language proficiency. Such procedures may be
adapted for the statement of behavioral objectives or for
the construction of criterion-referenced tests. Prances
Sussna (1970) of San Francisco's Multi-Cultural Institute
offers a taxonomy of objectives related to cultural
appreciation and improved human relations that may be very
helpful as a model in establishing local objectives.
In conjunction with the formulation of specific
objectives and the development of curriculum components,
plans should be made for the evaluation of program
outcomes. As mentioned above, the principle evaluation
measures can be used both for pre-assessment and for
94
sumraative evaluation of the program.
Because of the novelty of bilingual education
programs, it is unlikely that evaluation needs will be
well served by the use of existing tests. This means that
most districts will have to develop their own evaluation
measures. While normative tests can be developed based
on local norms, the most valid and reliable approach
would seem to be the development of behavioral objectives
related specifically to program goals. (To the extent that
such objectives are clear and objective, they can also
be normed for comparison with other schools where the
situation warrants such comparison). Behavioral
objectives foster accountability because they afford clear
indication of the degree to which operational statements
of district goals have been attained.
Though virtually all the authors who treat the
subject urge caution regarding the use of standardized
tests with non-English-speaking children, such tests can
at times serve a valuable purpose, depending on specific
program goals. Where stated goals imply true and thorough
bilingualism and biculturalism, or achievement of non-
English-speaking children on a par with that of English-
speaking children, such tests afford an indication of the
attainment of such goals (though probably not as reliable
an indication as that obtained from the use of behavioral
95
objectives and the use of criterion-referenced tests).
The State of California mandates the use of certain
standardized tests in the schools and requires that
school districts report on the result of such tests.
Where bilingual education is seen as a means to more~
effective functioning in Anglo-American society, it can
be expected that deficits in test scores attributable to
language or cultural difference will gradually diminish.
Most authors, however, urge great caution in the
use of currently available standardized tests to assess
Mexican-American children or to evaluate the effectiveness
of programs containing large numbers of minority students.
(Mercer, 1973; Chomsky, 1973; Bane and Jencks, 1973;
Laosa, 1973; DeAvila, 1973). Through the years, there
has been contention regarding the content validity and
construct validity of standardized tests, particularly
when they were used with minority students or non-English-
speaking students. Such discussions tended to be purely
academic exercises in light of the excellent predictive
power of the tests in question. In recent years, however,
the legitimacy of this approach has been called into
question. Though standardized tests may predict
achievement in the schools as they exist now, it is
precisely the character of school programs as they relate
to minorities that is being questioned. Hence, academic
96
achievement of minority students is now a questionable
criterion for the establishment of predictive validity.
This means that where currently available standardized
tests are used, a close and careful examination of the
content and construct validity of the test for the
population in question must be undertaken. Where such
tests are used, results should be tallied to establish
local norms for comparison purposes.
Along with behavioral objectives and standardized
tests, a number of other measures can be used as
indicators of the attainment of program objectives.
The National Consortia for Bilingual Education (1971)
has compiled a list of all the measures employed in
the evaluation of Title VII programs. Metfessel and
Michael (1967) offer an extensive list of indicators
of program status and progress.
Other measures that may be used to determine program
effectiveness are surveys, questionnaires, and
opinionnaires. These can solicit impressions and
judgments of students, faculty, and community members
regarding the program. In implementing a program of
accountability, numerous judgments must be made regarding
the accomplishment of component objectives and of program
goals. Clear and specific judgments at each stage can
be facilitated by the use of structured questionnaires.
97
Process evaluation at this stage is not highly
distinct from the implementation activities already
suggested. In checking the development of philosophy and
goal statements, comparison can be made with other
programs and tentative statements can be referred to
community members, school personnel, and experts for
reaction. The same holds true for the adoption of an
implementation model, program objectives, and curriculum
elements.
Where large scale programs are envisioned, it is
advisable to implement small pilot programs in order to
check the adequacy of objectives and curricular elements
before undertaking the program on a wide scale.
In pre-assessment procedures and in the development
of evaluation procedures involving the use of standardized
tests, certain cautions should be kept in mind, especially
those articulated by Deutsch, et al., (1964).
Standardized tests should be checked and an estimate made
of their probable validity, reliability, relevance, and
practicality. Buros* Mental Measurement Yearbooks
(Buros, 1938; 1941; 1949; 1953; 1959; 1965) are a valuable
aid for this purpose. Program objectives and curricular
objectives can be submitted to an independent examination
to estimate their relevance, objectivity, and practicality.
Collections of statements of objectives should be examined
98
to see if they effectively embody goal statements.
Where the concept of accountability is perceived as
threatening, or where concern with objectives is directed
toward rigid, mechanical procedures, then statements of
objectives may become trivial, irrelevant, or ritualistic,
and, taken as a whole, they may be inadequate as
operational statements of goals.
Implementation
The implementation stage of bilingual education
programs involves four steps: recruitment and training
of teachers, aides and other personnel, assignment of
supplementary tasks, acquisition or development of
materials, and the formal implementation of the program.
The pre-assessment procedures should afford a clear
picture of the composite skills, strengths and weaknesses
of school personnel. The specific skills and areas of
competence needed will depend on the goals adopted and the
program model chosen for implementation. Where general
deficiencies are noted on the part of many of the school
personnel, in-service training programs may be undertaken,
or arrangements made for special classes to be offered by
nearby colleges and universities. Where a few teachers
are weak in desired skills or background knowledge, or
where the skills and knowledge in question are specific
99
and relate to just one grade level or curriculum area,
individualized instruction methodology (learning programs
or learning activity packages) may be employed.
The assignment of supplementary tasks is an extension
of the formal commitment to the program. Each person
involved in the program should have a clear statement
of his responsibilities. As the program develops,
unforeseen tasks will arise. Unless continuous attempts
are made to check progress toward goals, there is a
tendency for people to operate within their job descrip
tions and concentrate on earlier concerns rather than to
address themselves to the tasks that were not foreseen
at the outset. Systematically and periodically, feedback
should be considered, participants should be consulted,
and judgments should be made about work remaining to be
done. As this done, specific tasks can be clarified
and responsibility for their completion assumed by
participants.
The acquisition of material was alluded to above in
the assessment of resources. Bibliographies on bilingual
materials can be found in the handbooks on bilingual
education. Where materials suitable for students in the
given district and adequate for the program objectives
cannot be found, they may have to be developed locally.
(This is a case in point where tasks unforeseen at the
100
outset of the program may become obvious. In implementing
bilingual programs, however, it should be presumed at the
outset that the acquisition and development of materials
will be more demanding than is normally the case. Where
possible, extra staff time should be alioted for this
purpose.
The formal implementation of the program will most
likely be the occasion for a number of revisions.
Observations of progress may show that program goals were
too ambitious or too conservative. Textbooks, supple
mentary materials, evaluation instruments, etc., may prove
to be inadequate or impractical. The training of personnel
may prove to have been inadequate. Student or community
reaction may shift and entail the necessity for revision
of program goals or implementation strategies.
Throughout the program, and especially in the early stages
of implementation, careful observation should be undertaken
by advisory groups, school personnel, consultants, etc.,
to determine whether revision is called for in any
elements of the program.
With regard to process evaluation at this stage,
training of teachers can be checked in a number of ways.
Opinionnaires can solicit reaction of teachers and aides,
self-assessment scales based on desired component skills
can be employed, opinions of community members can be
101
solicited. The results of process evaluation and progress
evaluation during the formal implementation of the program
and the results of the summative evaluation can also be
examined for indications of the adequacy or inadequacy of
the preparation of school personnel.
In checking to see that essential tasks, originally
unforeseen, are clarified and responsibility assumed for
their completion, periodic meetings might be held where
participants "brainstormed" concerning deficiencies noted
in program implementation. The program director could
use results of such meetings to clarify remaining tasks.
Suggestions were offered above concerning the
acquisition of material. The same principles would be
applied to material development.
As the program is implemented, periodic observations
should be made by qualified observers to determine that
the program that was planned is, in effect, the program
that is being implemented. An attempt would be made to
establish that teachers are committed to program goals
and objectives, and that they are indeed implementing
language training and cultural training strategies
as planned.
Throughout the program, progress evaluation should
also be undertaken. A timeline should be established
for the attainment of mediating objectives. In cases
102
where the program is not progressing toward desired goals,
this periodic evaluation of progress allows for early
detection of the situation and for program modifications
and offsets the need to wait for program completion to
determine if the program was indeed successful.
Summative Evaluation
At the completion of the program, data is gathered
to determine program accomplishments. Two kinds of
judgments can be made. First, results of the program
can be examined to identify those elements of the program
that contributed to success and those that were ineffec
tive. This type of information can be used for planning
future programs and may be generalized to other types of
programs. In a program emphasizing accountability,
however, the desire for results that can be generalized
to other situations is subordinate to the collection of
data that allow for judgment on the effectiveness of
the specific program in question.
In a program emphasizing accountability, results of
established evaluation procedures are summarized and
interpreted and a formal judgment is made regarding
program effectiveness. Along with such a judgment,
however, supplementary observations and generalizations
regarding results may be made.
The decision about what data to examine is made prior
to the implementation of the program, and expected
outcomes are explicitly stated. At this stage, the data
are examined to determine discrepancy between expectations
and actual results. Procedures and assumptions for the
interpretation of data should also have been clarified
prior to program implementation, so that all that is
required at this stage is the application of procedures
for interpretation and statement of results. While
progress evaluation during the implementation stage may
have led to program revision, such revisions should also
have entailed revised statements of objectives and, where
appropriate, a revision of evaluation procedures.
The essence of the evaluation is the judgment that
program goals and objectives have or have not been met.
Where such goals and objectives have been stated in terms
of behavioral objectives or where expected levels of
attainment on standardized tests and other measures have
been specified, such a judgment is a reiteration of
evaluation results. Nevertheless, it is suggested that
formalized judgments be made at each level. That is,
teachers and aides will determine the degree to which
students have met each of the curricular objectives of the
program. Principals and supervisors will determine the
extent to which teachers and aides have met the objectives
104
they set for themselves. This would involve assessment of
the progress of their students as individuals and their
classes taken as a whole as well as the accomplishment of
other objectives that they had set for themselves (for
example, relating to professional improvement). The
performance of support personnel, building principals,
and all others engaged in the program should be reviewed
in light of the objectives that had been set for them.
Finally, a judgment should be made by the school board or
advisory board or by a committee of persons appointed for
the task, that the program taken as a whole was or was
not effective.
At each level in the process of evaluation,
observations can be made to supplement the judgment of
effectiveness. Such observations might be reservations
about the judgment itself or about the program. (For
example, objectives were met because of extraneous factors
not because of the curriculum adopted, or objectives
proved to be too conservative, etc.).
Recommendations should be solicited regarding possible
revisions of the program and decisions to continue the
program or drop it, etc.
Process evaluation at this stage entails a review of
the history of the program for variables that might
influence the reliable collection or the valid
105
interpretarion of data. Stufflebeam, et al.t (1971,
Pp. 171-172) summarize the threats to the validity of
experiments suggested by Campbell and Stanley (1963)
and by Bracht and Glass (1968). These summaries may
be suggestive or factors that might weaken the validity
of interpretation of data at the completion of a program.
The decision that such variables are indeed present
and that originally planned interpretation measures
should be modified, however, should be undertaken prior
to the interpretation of evaluation results.
A double check on the validity of the evaluations and
judgments of program effectiveness can be obtained by
consultation with an outside evaluator. This might be
either a consultant who specializes in evaluation and
contracts for such work or might be some knowledgeable
person or group of persons, possibly school district
employees, community members, or personnel in nearby
districts. The primary consideration would be whether
they possessed the necessary competence to review
evaluation procedures used (dependent on the complexity
of such procedures) and were in a position to view
outcomes objectively (i.e. were not themselves involved
directly in the program).
106
Ac covin ting
Accounting is formalized as a distinct stage to
highlight the desirability of informing constituents
as openly and clearly as possible regarding the true
status of programs. "Accounting" means simply a clear
and unembellished report on how well program goals have
been attained.
It is suggested that teachers send a report to the
parents of each of the children they teach, detailing the
extent to which their children have met program objectives
and the extent to which the classes they teach, taken as
a whole, have met program objectives. (Where practical,
aides and other support personnel might report to the
parents of the children they have served on attainment of
objectives.) Such reports should state the judgment of
the evaluator or evaluators regarding the effectiveness
of implementation of program goals. Principals could
report to parents on the achievement of objectives by
classes and by the school as a whole. The superintendent,
where more than one school is involved, would report to
the school board on the accomplishment of objectives by
schools and by the district as a whole. All of these
reports might go to the members of the advisory board.
The superintendent's report might be forwarded to local
media.
107
Such formal statements would serve several purposes.
They would educate the community regarding what the
schools were attempting to accomplish and what they were
actually achieving. In subsequent years, such statements
could serve as stimuli for the re-evaluation of community
feeling regarding philosophy and goals of the district.
Most importantly, they would afford concrete evidence that
school personnel were totally committed to accountability,
to responsiveness to community needs and desires, to the
most effective accomplishment of established goals, and to
open and honest reporting to the public.
Selection Of "Expert" Evaluators
Upon completion, the foregoing paradigm, along with
explanatory material, was sent to evaluators. Originally,
it was planned that the paradigm would be submitted to
a committee of experts for a "meta-evaluation" (Scriven,
1969). A number of experts were to be approached and
asked to react critically to the paradigm. In addition,
an "adversary evaluator" was to be appointed, following
the suggestion of Levine (1973).
Dr. Michael Scriven, of the University of California
at Berkeley, was contacted and asked if he would serve
as "adversary evaluator." He was very gracious, but he
declined because of lack of time. He suggested that
Dr. James Popham, of the University of California at
Los Angeles, be contacted. Dr. Popham also declined.
He suggested that Dr. Marvin C. Alkin, of the Center for
the Study of Evaluation at U.C.L.A., be contacted.
Dr. Alkin also declined but suggested that one of his
graduate students, who had done research work on the
evaluation of bilingual programs in California might
be contacted. At this juncture, it was decided that
the plan of employing an "adversary evaluator" would be
discarded. Ordinarily, when original evaluation models
are involved, the procedures suggested by Levine (1973)
offer an added safeguard to the validity of evaluation
procedures. However, in the present instance, it was
decided that the dissertation committee served the function
envisioned for the "adversary evaluator."
A second revision was made in the planned procedures
with respect to the committee of evaluators. Initially,
the plan had been to select, arbitrarily, a certain number
of experts in the field of evaluation or bilingual
education. A change in plan occurred, however, and it was
decided to send the paradigm to all those consultants who
had contracted to evaluate federally funded bilingual
programs in the State of California. One of the
conditions for reception of federal funds for the
implementation of bilingual programs under Title VII of
109
the Elementary and Secondary Education Act is the
performance of an evaluation of the program by a person
specifically designated for that task. It was felt that
"practitioners" of such evaluations would be in a very
good position to render judgment on the validity and
practicality of the paradigm.
The committee of experts designated to evaluate the
paradigm, then, is composed of those consultants who had
contracted to evaluate federally funded Title VII
bilingual programs in the State of California. At the
suggestion of the Dissertation Committee chairman,
Dr. Thomas Thomas of the Stanford Research Institute was
also asked to react to the paradigm because of his special
expertise in the area of evaluation of bilingual
education programs.
Duties Of The Evaluators
The evaluators were asked to evaluate the paradigm
according to the criteria suggested for the evaluation
of evaluation designs and models by Stufflebeam, et al.,
(1971) (v. gr., internal validity, external validity,
reliability, relevance, importance, scope, credibility,
timeliness, pervasiveness, and efficiency).
Approximately 30 pages of explication accompanied the
paradigm (pp. 74-107 of this study). It was anticipated
110
that those contacted would be busy professionals. Hence,
a qualifying statement was added to the effect that it
was not absolutely essential to follow the format of
Stufflebeam, et al., and that any critical reaction
would be appreciated. Copies of the cover materials
sent to evaluators are contained in Appendix D.
Method Employed In Contacting Evaluators
An attempt was made to obtain from the California
State Department of Education a list of consultants
responsible for the evaluation of Title VII, E.S.E.A.,
Bilingual Programs. At that time (Nov. 1973), there
was not available any comprehensive list of specific
persons responsible for evaluation of bilingual programs
in the State. However, a list of all federally funded
bilingual programs was obtained along with names and
addresses of program directors. Letters were sent to the
directors of bilingual programs for Spanish-speaking
students. Forty-two replies were received with the names
of forty-five evaluators or potential evaluators.
(Some projects had more than one person engaged in
evaluation. Some persons were designated as evaluators of
several projects, and some persons were designated as
"having bid" on evaluation contracts, though final
decision had not been made at the time of reply.) The
Ill
paradigm and cover letter was sent to all persons whose
names were received. (Names and addresses of project
evaluators are contained in Appendix E.) It was not sent
to corporations where no contact person was designated
(v. gr., Rand Corp., Santa Monica).
In the following chapter, replies are described
and summarized.
CHAPTER IV
Replies Of Evaluators
The paradigm for the implementation of accountability
measures developed in this study was sent to 46 persons
who had been designated as evaluators of federally funded
bilingual programs in California. Follow-up requests
for reaction were made late in March and early in April.
Replies were received from 19 persons. Of these,
however, six persons declined to evaluate the paradigm.
The replies of the remaining evaluators are reproduced
verbatim in Appendix F, and a summary of their remarks
is presented in this chapter.
It should be noted that none of the evaluators
employed the critical format suggested originally
(i.e., their estimate of the paradigm vis-a-vis the
criteria suggested by Stufflebeam, et al.: internal
validity, external validity, reliability, etc.). One
evaluators, Dr. Richard Seligman, pointed out that the
amount of time required for such a task would be excessive.
Under the impression that this might be so, the request
for evaluations of the paradigm had embodied the proviso
that any observations concerning the validity of the
paradigm or its potential effectiveness would be helpful.
112
113
All of the reactions received were addressed to these
more general considerations.
Review Of Replies: General Impressions
In general, the reaction of evaluators was mixed.
None accepted the paradigm without reservations. Several
appear to have found it totally irrelevant.
Mr. Clarence Cate (Sweetwater U.H.S.D.) observed:
The paradigm is appropriate for accountability
measures in bilingual education or the
implementation of any other educational
program.
Dr. Dene Lawson (New Haven U.S.D.) stated:
Your one-paper guide is useful as a check sheet....
My overall assessment of the narrative section
is "weak"....
I find your narrative usually unclear. It suffers
from big words and obtuse phrases....
If your paradigm includes the narrative for purposes
of clarification, it fails miserably. Stufflebeam
or Provus can provide the model for evaluation in
their book. Unless you write a book, your efforts
to explain will only confuse.
So your contribution really is in clarifying the
specifics of bilingual education and I for one
have found this area extremely difficult to define.
Apparently you are having some difficulties with
definitions also.
With respect to the observation that it would be
necessary to write a book, similar to those of Provus
and Stufflebeam, to explain evaluation, this was not the
intent of the paradigm. In the introductory material, it
is stated that the material is intended to be suggestive,
not exhaustive. While suggested procedures are not
114
explained at length, an attempt is made to refer the
reader to appropriate bibliographic resources for more
detailed description and explanation.
In regard to the observation that the function of the
paradigm is in clarifying specifics of bilingual education,
that was not the specific intent of the paradigm either.
The paradigm was directed towards measures that increase
accountability. (Program evaluation is a means to this
end, but accountability implies a responsiveness to the
community not necessarily present in all program goals.)
The specific definition of bilingual education originally
intended in this paradigm was whatever form the community
agreed upon. This position has been modified in reaction
to the response of Dr. Thomas Thomas, listed below.
Dr. Charles Leyba (Director, Project Maestro,
California State University, Los Angeles) observes, in
a vein similar to Dr. Lawson:
A broad reaction to the paradigm is that it is a
good, general set of steps that can be used as a
checklist for the development of bilingual programs.
I don't think that beyond this point it is very
useful. It certainly is not unique in this respect.
There are other equally vague guides. Perhaps this
accounts for the poor development of bilingual
programs once funding has occurred.
Dr. Leyba's observation is similar to those of
Dr. Lawson. Again it is asserted that the intent of the
paradigm was to be suggestive. In reaction to the
observation that it would not be useful other than as a
set of general steps, the paradigm was intended to
describe general steps in program implementation and
evaluation. However, the narrative goes beyond steps to
list bibliographical references to material that would
serve the purpose of clarifying specific points. Such
references are made relative to suggestions on: community
involvement, especially in relation to Mexican-Americans
(Wilcox, 1972; U.S.O.E., 1972; California State Department
of Education, 1968); needs assessment (U.S.O.E., 1971;
California Joint Committee on Goals and Evaluation, 1972);
examination of differences in value systems (Ulibarri,
1970; Jaramillo, 1973); surveys on the extent of and
need for bilingualism in the community (Saville and
Troike, 1971); systematic checks on the character of
community involvement and the extent to which the schools
are serving the needs of Mexican-American students
(United States Commission on Civil Rights, April, 1971;
October, 1971; 1972; 1973); possible models for philosophy
or goal statements (Andersson and Boyer, 1970; John and
Horner, 1971; Saville and Troike, 1971; U.S.O.E., 1971;
Assembly of the State of California, 1972); criteria for
selection of teachers (Ulibarri, 1970; Andersson and
Boyer, 1970); evaluation of materials (U.S.O.E., 1971;
Detroit Public Schools Staff, 1969); statements of
objectives which may serve as models for the statement
116
of local objectives (Lee, 1972; Norris, 1970; Sako,
1969; Sussna, 1970); cautions to be observed in the use
of standardized tests in these programs (Deutsch, et al.,
1964; Mercer, 1973; De Avila, 1973); a list of all the
in-house and standardized tests used in the evaluation of
all Title VII programs (National Consortia for Bilingual
Education, 1971); a list of non-test measures of program
status and progress (Metfessel and Michael, 1967); and
possible threats to the validity of evaluation results
(Stufflebeam, et al., 1971; Campbell and Stanley, 1963;
Bracht and Glass, 1968). Where the nature of the
procedures alluded to was not clear, it was understood
that these references would have to be consulted for
elaboration.
Dr. Fred C. Niedermeyer (Southwestern Regional
Laboratory for Educational research and Development)
observed that: "Criteria are sensible and OK." (It is
not clear from the remark whether he was reacting to the
paradigm itself or to the criteria suggested by
Stufflebeam, et al., for the evaluation of evaluation
models. A copy of these criteria was enclosed with the
paradigm for the convenience of those evaluators who
wished to evaluate the paradigm against the standards
suggested by the criteria.)
Dr. Niedermeyer noted that the term "accountability"
117
was not operationally defined. He enclosed an article
of his own in which he stated that four conditions for
operationalizing teacher accountability were outcome
statements, means of assessment, validated programs, and
performance reports.
Appended was the further observation that the article
was "typical of what those in evaluation (particularly
Stufflebeam) have been doing the past several years.
i.e., take any new term or process in education (e.g.,
product development, accountability) and redefine it as
'evaluation'. To them, everything is evaluation."
There are two points to be addressed in response
to these observations: the scope of accountability and
the relation between evaluation and accountability.
Prom his article, "Conditions for Successful Teacher
Accountability," it would appear that he would judge
successful realization of objectives as the essence of
accountability.
Leon Lessinger claims to have coined the term
"accountability" as it is currently used in educational
circles. The meaning he attached to the term appears
to be exactly what Niedermeyer says it is. Nevertheless,
in the blizzard of articles that have appeared in recent
years, the term has been employed with dozens of
connotations, many contradictory. Among the remarks
118
or assumptions implicit in an accountability approach,
it was stated that accountability implied effectiveness,
(successful realization of objectives). To this,
however, were added the notions of responsiveness to the
will of the community (to avoid escape from accountability
to constituents into secure, but trivial or irrelevant
objectives) and efficiency.
Complaints of minority spokesmen about the schools
have versed about absence of minorities from positions
of influence (presence as school board members,
administrators, teachers, college bound students, student
officers, etc.), lack of sensitivity to minority culture
on the part of school authorities, lack of appreciation
of the difficulties encountered by non-English-speaking
students, etc. And while accountability approaches focus
on "outcomes" rather than "input," it is precisely the
area of educational input that appears to bother most
spokesmen for the Mexican-American community.
They appear to be concerned with how their children are
educated, with the self-esteem and dignity of their
children, with the attitudes of teachers and administra
tors, etc. In this sense, then, "input" becomes the
variable to be assessed. The educational and human climate
of the school, to the degree that they can be measured,
must be measurably improved to satisfy the demands of
119
community members. And if they are not improved, then
somebody should be hald "accountable."
For this reason, the entire paradigm is presented
as an avenue to greater accountability. If community
members are concerned about their representation, then
procedures suggested in the involvement stage may be
employed to realize the objectives of more representative
involvement and greater responsiveness of the school to
community needs and desires. The commitment stage suggests
procedures for cooperative determination of objectives
and assignment of responsibility for their realization.
Checks are suggested at the implementation stage because,
again, how children are taught may be viewed as more
important than what they are taught. (So the methodology
becomes the objective— rather than simply an alternative
means to an end.) The summative evaluation stage and the
accounting stage would encompass the essential judgment
of program effectiveness.
In summary, the paradigm is intended to encompass a
range of concerns (many social and political) much more
broad than simply the improvement of cognitive skills
of students. Intended outcomes may be stated in many
ways other than by student performance (v. gr., increased
participation of Mexican-American parents on school
advisory committees, increased percentage of minority
120
employees, more favorable survey ratings, greater
indications of tolerance on the part of administrators,
teachers and staff, increase in the number of "soft signs"
of appreciation of minority culture, etc.) The total
paradigm may be employed in an attempt to make a total
program more accountable, or parts of the paradigm may
be employed to increase accountability with respect
to one or several specific objectives. Returning to the
original point, the operational definition of accountabi
lity would be the successful realization of whatever
objectives had been agreed upon.
With respect to the second objection, that everything
(accountability included) is seen as evaluation, it would
appear that the order should be reversed. Evaluation is
an essential part of accountability; there cannot be
accountability without valid evaluation. And while many
descriptions of accountability (v. gr., responsibility for
outcomes stated in terms of student performance) would
seem to obviate the necessity for extensive evaluation
measures, this is not true (at least in the present state
of the art) if a program addresses itself to the broad
concerns alluded to above.
Dale Russell (Coordinator for Research, Testing, and
Evaluation, Coachella Valley Unified School District)
stated:
121
I "believe that you are right on target and will
provide a useful tool for all evaluators—
including for bilingual project evaluators.
Dr. Richard Seligman (Associate Director, Center for
the Study of Evaluation, U.C.L.A.) replied, in part:
As I read your paper, I find that it deals more
with the implementation of bilingual programs
than it does with the implementation of
accountability measures for those programs.
To be sure, you included material on
accountability in bilingual education, but
it was not sufficiently set apart so that a
"paradigm" emerged clearly...
Your paper would be much improved if there
were a more clear cut statement of specific
evaluation activities which must be undertaken,
the sequence in which these activities should
occur, and the relationship between the outcomes
of these activities and the management or
operation of the program.
Earlier in his reply Dr. Seligman noted:
My own notion of accountability is that it
involves nothing more than a statement of
what will be attempted in a particular
program and how, at the conclusion of a
program, it will be possible to show that
what was attempted was achieved or not
achieved.
This concept is very similar to that suggested by
Niedemeyer. The notion of accountability implicit in
the paradigm differs in several respects. First, it is
important to note who arrives at the statements of what
will be attempted in a particular program, and how they
arrive at such statements. To the extent that concern
of those involved in the program was directed more at
the atmosphere of the school, at the treatment of minority
122
children, at responsiveness to needs idiosyncratic to
minorities, etc., the way a program was implemented might
be the basis of program objectives.
Ordinarily, a person assuming the role of program
evaluator would be concerned with steps three, five, and
six of the commitment stage (pre-assessment, development
of instructional objectives, and development of instrumen
tation for summative evaluation), with all the activities
of the summative evaluation stage and with all the
activities designated as process evaluation procedures save
the summative evaluation by an outside evaluator. However,
the paradigm was designated with the intent of enhancing
"accountability in bilingual education programs." The
goal was to assist districts become more responsive to
needs and desires of community members, particularly in
districts with a large population of Mexican-American
students. For this reason, the differentiation between
program implementation and program evaluation has been
vague. The specific role of a person designated as
evaluator has also been downplayed, since it is assumed
that many persons might fulfil the function of evaluation
without any one specific person being designated as
responsible for the evaluation of the entire program.
The designation of responsibility for evaluation would be
idiosyncratic to each program, much as would the adoption
123
of specific program objectives.
One must agree with Dr. Seligman that the paradigm
would be more helpful if there were a more clear-cut
statement of specific evaluation activities, the sequence
in which they should occur, etc. Nevertheless, it is
difficult to conceptualize how that would be valid when
the paradigm addresses itself at this stage, to purely
hypothetical programs, goals, and objectives.
Dr. Robert Senour (Evaluator: Bilingual Instruction
to Improve Educational Opportunities, Cucamonga)
Dr. Senour declined to evaluate the paradigm because
of confusion. (The paradigm, referred to as "Appendix A"
in the explicatory material, was attached as a cover sheet
in the materials sent to evaluators. Prom his remarks,
it appears that the paradigm was not included in the
packet of materials that was sent to him). However, he
went on to offer some observations. His is, perhaps, the
most disappointing reaction of all!
Some of the concepts you have in your writing
is "garbage" - it isn't applicable because the
premise is based on an Anglo identification of
schools. When you are constructing a flow
diagram for school children whose schooling
and whose parents' schooling is not nor has
not paid off in terms of getting part of the
action then the program must start from where
"they" are and not where an Anglo thinks they
should be.
This reaction is disappointing because it represents
a point of view that was specifically intended to be
124
accomodated in the paradigm. In conferences and
conventions dealing with the education of Mexican-American
children, in first-person articles, in the remarks of
spokesmen for Mexican-American students, a common theme
emerges: the sense of alienation and of dispossession
to a greater or lesser degree. To address this sentiment,
the paradigm attempted to highlight the need to involve
all the people of the community, to be responsive to the
needs and desires of the people of the community and to
formally account to the community regarding realization
of objectives adopted cooperatively. If the paradigm is
not helpful in suggesting ways to "start where 1 they'
are, not where an Anglo thinks they should be" then this
would be a major failing.
Dr. Patricia Caldwell Simmons (Evaluator: Double
Bilingual-Bicultural Program, Hayward; Bilingual
Bicultural Education, Los Nietos; Esperanza Program,
Oxnard)
Dr. Simmons began by suggesting the examination of
Rabbi Gerald Teller’s article, "What Are the Myths of
Accountability?"
Teller (1974) lists seven myths underpinning the
accountability movement: (1) the concern of Americans for
efficiency; (2) the idea that business is efficient and
other disciplines rather slovenly; (3) the belief that
125
behavior is more important than underlying motives;
(4) the idea that the accountability movement will be a
cheaper way to educate; (5) the idea that the accountabi
lity movement will help solve our political problems;
(6) the idea that technology will solve our educational
problems; (7) the belief that the accountability movement
will produce educated men. Rabbi Teller questions each of
the above "myths" and expresses serious concern over the
accountability movement. It may be presumed that
Dr. Simmons suggested the article as a caution against
the naive belief that implementation of accountability
measures would be a magical means to the improvement of
education.
Some of Dr. Simmons specific criticisms will be
detailed below. However, it should be mentioned here
that she points out,
My experience as an evaluator and a designer
of research is if your audience is school adm.
the paradigm cannot be too specific and clear.
If your audience is fellow researchers, okay,
broad statements are acceptable, i.e., analysis
of covariance for product.
It can be inferred from Dr. Simmons' remarks and Dr.
Seligman's remarks, that the paradigm is lacking somewhat
in clarity, (especially for the person lacking expertise
in evaluation, and the paradigm is intended for such
persons). This again leads to the problem of how to make
the paradigm more specific and clear when discussing
126
hypothetical issues. An evaluation or design would have
to be specific and clear, but a paradigm or model, by its
nature, would seem to have to be more general and abstract.
In the absence of recommendations about how to make it
more specific, it is difficult to respond to these obser
vations. (An attempt will be made below to respond to
criticisms of specific aspects of the paradigm).
Dr. Leslie W. Six (Coordinator: Research and
Evaluation, Chula Vista City School District) replied,
"I concur with your paradigm, the basic elements of which
are essential for accountability in all educational
programs.” He went on to make several comments regarding
the feasability of implementing several of the suggested
measures. These suggestions are listed below.
Dr. Ivor J. Thomas (Professional Associate,
Educational Testing Service) replied,
The paradigm you presented is comprehensive
and provides a fairly accurate description
of essential procedures for the implementation
of accountability measures in bilingual
education...the paradigm...provides a good
outline of effective procedures in the
evaluation of bilingual programs.
Dr. Thomas also appended several caveats and comments
reinforcing points made in the paradigm. These
observations are also noted below.
Mr. John Vatsula (Evaluator: Valley Intercultural
Project and Upper Valley Intercultural Project, Sacramento)
127
replied,
The paradigm, in my opinion, is well contrived.
Somehow the Community references seem less
congruent than the remainder.
Mr. Vatsula also appended comments that went to specific
elements of the paradigm and these are noted below.
In summary, none of the evaluators replied with a
blanket endorsement of the paradigm, though that was not
really expected. Dr. Niedermeyer's reply implies that
there is a clear-cut difference between accountability and
evaluation and indicated that the paradigm was closer to
evaluation than accountability. Dr. Seligraan suggested
that evaluation/accountability procedures were confounded
with implementation/management procedures. Dr. Senour
felt that the paradigm was irrelevant because of the Anglo
mentality it embodies. Dr. Simmons, Dr. Lawson, and
Dr. Leyba suggested that the paradigm be more clear and
specific, especially if intended for an audience other
than professional evaluators. Mr. Cate, Mr. Russell,
Dr. Six, Dr. Ivor Thomas, and Mr. Vatsula generally
concurred with the paradigm, though most had specific
reservations which are noted below.
Review Of Replies: Specific Comments
Mr. Edwin Garvin (San Francisco U.S.D.) pointed out
that to have true accountability, a program should have
128
freedom from restrictions, ability to innovate.
It is difficult to depict, in the abstract, a balance
between the autonomy necessary for creativity and
innovation and the constraints inherent in accountability
measures. An attempt was made in explaining the paradigm
to highlight the aspect of flexibility needed for effect
iveness and innovations. (This may have contributed to
the reaction of several evaluators that the program was
too vague or unclear.) Dr. Thomas Thomas, in observations
summarized below, notes that one method of fostering
accountability and autonomy is to run alternative programs.
Dr. Dene Lawson states:
You need to clarify the reason for including
initial selection of personnel and basic
curriculum in evaluation procedures.
These remarks were made in elaborating on pre
assessment procedures. Part of the formative evaluation
of a program entails checking to see that competent
teachers and adequate resources are available for the
implementation of the program as originally planned or
envisioned.
Your section on writing goals and/or objectives
does not help. Who is your audience? A novice
needs more help than you give; a professional
will probably find it too trivial.
The audience intended was not professional evaluators,
but persons who, in this context, would probably be novices
or amateurs. However, as noted above, it was not the
129
intent of the paradigm to explain procedures completely.
It was felt that such explanations could be found in the
reference works.
The section on testing and selecting instruments
is amateurish. Criterion-referenced testing
for example is an entirely new, sophisticated
approach. Validity and reliability are concepts
not yet solved with CRT, so we can hardly suggest
that local tests can be "normed" and compared
to local districts, etc.
Validity and reliability of criterion-referenced
tests are certainly a potential problem. It should be
noted, however, that validity is a relative thing, not an
absolute. Furthermore, at least technically, we speak of
validity of results rather than of validity of tests
themselves. Results are valid if they serve the purpose
intended. Norm-referenced tests are constructed with the
intent of ranking subjects vis-a-vis specific attributes.
Standardized achievement tests tend to be insensitive to
results of instruction. A question arises whether it is
really "achievement" that is being measured or some
general aptitude. These factors suggest serious defects
in construct validity of norm-referenced tests. For these
reasons, results of norm-referenced tests are probably
not highly valid indicators of achievement of specific
program objectives.
A more logical (though, in practice, not necessarily
more valid) approach is to attempt to develop tests that
130
will measure specific achievement of local objectives.
It is possible, perhaps probable, that such local tests
will lack the sophistication of nationally standardized
commercial tests. In spite of lack of sophistication,
these local tests will probably have much greater face
validity than standardized tests.
With respect to the observations on ’ ’ noraiing,"
norming serves two purposes. In the case of standardized
tests, local norming helps eliminate the effect of
confounding variables (economic, cultural, linguistic
factors and, past educational experiences), which will tend
to be more homogeneous in the local district than
nationally. Statistical control of these confounding
variables (through local norming) will tend to make test
results more sensitive to program main effects. To the
degree that local norming, especially in districts with
predominantly low test scores, stretches out the curve of
test scores, the scores will also be more sensitive to
program effects. A second purpose is envisioned for
local norming, particularly in the case of criterion-
referenced tests devised to evaluate achievement of
local objectives. Such norming aids in evaluation of
alternative programs or approaches that attempt to
achieve the same objectives.
131
Your suggestions for evaluating the quality of
professional staff smacks of witch hunting.
Instead of opinionaires and self-asses3ments, etc.,
you should find out a) what a teacher or aide
needs to know to serve effectively, then b) find
out if she has those competencies. If she does
not, then c) in-service training for specific
reasons is administered, followed by a test on
the needed competencies which were taught
specifically.
The use of opinionnaires and questionnaires was
suggested as a means of asking teachers what they needed
to know, and, in the case where they had been given
in-service training, as a means for evaluating that
training with a view to future improvements. However,
even where a training program appears adequate, the extent
of real impact in the classroom remains. For this reason,
process and progress evaluation procedures are suggested.
In practice, the procedures suggested by Dr. Lawson
will probably be sufficient in most cases. But where
question arises (whether founded or not) concerning
competence, quality, or dedication of teachers to program
goals, then the suggested procedures are a means of more
objective examination of the situation.
Such procedures may smack of "witch-hunting."
However, it may well be that they would protect teachers
from unfounded charges in situations where "witch-hunts"
did occur.
Dr. Charles Le.yba reacts to a number of specific
points. He finds the references to community participation
132
vague, and in need of some highly specific guidelines.
Further, he notes that training of advisory committee
members is necessary. Otherwise, they tend to view
themselves as executive committee members, with the
possible result of conflict among persons involved in
the program.
An attempt was made in developing the paradigm to
emphasize that responsibilities be made clear and
explicit. It was not specified what responsibilities
should be allocated to various persons or groups involved
with programs, nor what procedures should be followed
in arriving at such decisions. Dr. Thomas Thomas, in his
remarks, makes much the same point, but with a slightly
different focus. His view is that, in general, what is
needed is a shift in control of programs away from
bureaucrats to concerned community members. But apart
from Dr. Thomas' suggestions, and the references cited
here, it is not clear what specific guidelines a district
would wish to adopt in specifying the nature of
community involvement.
Dr. Leyba also questions the nature of the step
labelled "Formal Commitment To The Program." In reaction
to observations by Dr. Six, noted below, this step was
revised to "Tentative Commitment To The Program." It was
suggested that formal commitment to the program be
133
treated elsewhere.
s
Another point that Dr. Leyba makes is that the
paradigm is silent on various kinds of models and the
variables of which they are composed. However, reference
is made to the work of Mackey (1969) where such
information can be found.
Dr. Leyba also finds vague the step labelled
"Assignment of Supplementary Tasks." The intent was that
specific tasks necessary for successful attainment of
goals should be clearly assigned to specific persons
and clearly revised when necessary.
Another observation is that use of criterion-
referenced tests be emphasized and that baseline and
midyear assessment be undertaken. Though the latter
point might be made more clearly, the paradigm suggests
pre-assessment and setting timelines for attainment of
mediating objectives.
Mr. Dale Russell notes:
The management and implementation procedures
seem complete and clear. The process evaluation
procedures seem more nebulous. Perhaps the
Metfessel, Michael paradigm for multiple
criterion measures in the evaluation of ed.
programs might assist to clarify some
process assessment.
In suggestions for the development of instrumentation
for summative evaluation, the Metfessel and Michael (1967)
article was alluded to. A comment was made further on
134
in the explanatory material that process evaluation at the
Commitment Stage is not highly distinct from the implemen
tation activities already suggested. The suggestions of
Metfessel and Michael are probably the best available in
such concise form. They were not repeated verbatim in
this paradigm for fear of redundancy. In the construction
of the paradigm it was considered sufficient to allude
to procedures, resource material, etc., that might be
employed without the need for complete reproduction or
exhaustive explanation of the material.
Mr. Russell also notes:
It would be helpful to me to see your paradigm
charted after a flow or PERT model to
visualize the interrelationships and see the
interplay between formative and summative
evaluation.
This reinforces Dr. Simmons’ concern about the lack of
clarity surrounding the exact role of the evaluator and
specific procedures to be followed in evaluation of the
program. Reference to PERT (Program Evaluation and
Review Techniques) were not included in the explanation
of the paradigm since this was considered to be mainly a
device for designing a work plan. However, this may have
been an oversight, and allusion to Cook's work should be
contained in revisions of the explicatory material on
implementation. As far as a PERT chart reflecting the
performance of functions or procedures suggested in this
135
paradigm, such a chart would seem to be idiosyncratic
to each program.
A further comment by Mr. Russell,
Have you considered the role of established
groups within the ed. system, i.e. District
site advisory groups, administrative and
curriculum councils, cabinet, etc.
Throughout the explication of the paradigm refernce is
made to possible functions of such groups. However, no
specific roles (specific in the sense of being character
ized by functions prescribed exactly and a priori) were
conceived of as appropriate to such groups. Accountability
would seem better served by ad hoc groups formed specifi
cally to serve a given function or meet a need than by
standing committees in search of a role. But then, that
is a rather general— and experimentally unfounded—
observation.
Dr. Patricia Caldwell Simmons notes that:
Some of your process evaluation, i.e., comparison
of patterns with demographic patterns is product
or done after the fact.
Process evaluation to be process is ongoing
happening that I would monitor. If you are
implying observing in a one tailed test fashion
on a matrix: former demographic patterns,
participation with what is now happening, ok.
Use of jury is also product to me, unless
observers.
I would like to see three columns as:
Procedure Process
Very Specific
Needs Assessment
procedures too
vague
Product
136
In the explication of the paradigm it was stated
that process evaluation is directed more toward the
establishment of the validity of program procedures than
towards the effectiveness of the program in meeting its
goals. The distinction is admittedly, tenuous, and it
was prompted by the observations of Provus (1971) to the
effect that, in education, the results of experimental
groups are often compared with the results of control
groups, while in the classrooms teachers have made no
modifications, no matter what the status of the group or
in intent of the investigator.
The procedures or steps suggested here in the column
designated Process Evaluation Procedures were developed
with a double function in mind: (1) formative evaluation
(i.e., the systematic review of the effectiveness of
implementation procedures with a view to revision of those
procedures where appropriate) and (2) process evaluation
to determine that the implementation procedures planned
for the program were faithfully executed so that the
summative evaluation of the program will embody a judgment
on the effectiveness of the program as planned, not on
an ersatz version.
The question would then seem to arise: are the
measures suggested here valid for the purposes intended
(as indicators of need for program revision or as
137
indicators of faithful execution of planned program
components?) The issue of whether the measures were
"product" or not would not seem to be as important as
whether they were valid. The examination of "products"
at each stage is suggested because these seem to be
potentially valid indicators that the implementation
procedures are leading to the realization of desired
objectives.
Dr. Leslie Six offered several comments regarding
the feasability of the paradigm.
In the Involvement Stage, there was nothing
said of the fact that there has to be a
strong commitment of both time and some
resources, on the part of the school district
to go through the laborious, though necessary,
steps of Community Involvement and the early
steps of Commitment. These activities alone,
might take the better part of a year and will
cost the school district both time and money.
Dr. Six* comments are well taken. However, rather
than make a specific statement to this effect as part
of the paradigm, it might be better to include such
comments in the list of assumptions implicit in the
adoption of an accountability approach.
Another observation:
In the Commitment Stage, it seems to me that
following the steps outlined in the paradigm,
a commitment is to be made before the program
has been adopted. Whether federally or state
funded, or through local resources, approximate
costs of a program should be determined to see
if there are sufficient resources available.
138
Lacking such, a district might have to modify
or abandon a plan. Perhaps there should be a
number 7 under II entitled "Formal Adoption
of the Program" since at this point all costs
would have been determined that could be
compared with financial resources available
to be expended.
The kind of commitment Dr. Six alludes to is what was
intended in Step two of the Commitment Stage: Formal
Commitment to the Program. However, in view of the fact
that this step precedes pre-assessment in the paradigm,
it would appear preferable to change Step two to Tentative
Commitment to the Program. Then formal adoption of the
program could be added later at Dr. Six' suggestion.
Reply was made to the above observations because it
was felt that they were directed specifically at the
paradigm. Following are further observations that appear
to be more general in nature.
Dr. Six goes on to note that adoption of some goals
precludes the realization of others. More specifically,
the concentration on bilingualism will necessitate
modification of other aspects of curriculum. And, while
he agrees with the scope and sequence outlined for
summative evaluation of programs, he points out that in
reality programs are hampered by the paucity of reliable
and valid tests to measure academic growth and progress
of Spanish-speaking pupils who are taught in Spanish.
(This latter point is alluded to by almost all the writers
139
on the subject.)
Dr. Ivor Thomas, like Dr. Six, made several comments
that were not directed specifically at the paradigm.
His observations are summarized here.
First, the importance of representative community
participation cannot be overemphasized because of the home
and community environment on student achievement.
Reluctance of community members to participate because of
weakness in English should be kept in mind. Surveys should
take language and reading difficulties into account.
Also, there is a need to define exactly what constitutes
Mexican-American culture if biculturalism is to be adopted
as a goal.
While criterion-referenced measures may have to be
developed because of the novelty of programs and their
unique objectives, a number of difficulties arise. First,
many programs lack sufficient definition of objectives and
explication of content and activities to develop such
measures. Second, because of lack of expertise in test
development, many measures developed lack sufficient
psychometric sophistication to provide accurate measurement
of attainment of objectives. And, though criterion-
referenced measures may be necessary to assess objectives
unique to the program, norm-referenced testing should not
be overlooked. School boards and funding agencies will
140
often require evidence of achievement of students in a
program compared with students in general. And many
programs are required to offer evidence of achievement
beyond the unique contents of the particular program.
Because of the developmental nature of the programs,
summative evaluation in the early phases becomes quite
minimal. Emphasis is on continuous evaluation of
materials and activities. It is usually necessary to
modify commercial instructional and assessment materials.
The evaluator can be very helpful by systematic
observation and gathering of data for feedback during
various phases of development. His observations on the
fidelity with which the program as described was
implemented will aid in the determination of the degree
to which results are related to program implementation.
Along with these observations were included comments
that related directly to the paradigm. Dr. Thomas notes:
The major criticism I have is in relation
to statements made in the section on
commitment. In particular, on page 69,
you suggested an approach involving
establishment of a loose-knit set of
principles rather than formulation of a
broad and exhaustive philosophy. While
I may not understand completely what you
are implying, I am convinced that a great
deal of the trouble experienced in bilingual-
bicultural programs has been due to lack
of sufficient agreement on the philosphy
on which the program is based. This tends
to confusion; therefore, regarding the
appropriate emphasis to be given to the
141
various contents of the program and
instructional strategies. My fear is that
your paradigm in this regard may provide
the rationalization some would welcome in
avoiding answering the tough questions that
are a prerequisite for resolution of critical
dilemmas frequently encountered.
In view of this observation, the statement
explicating the paradigm should be revised. There was
certainly no intent to provide rationalization for
avoiding the answer to tough questions that are a prerequi
site for resolution of critical dilemmas frequently
encountered. The position intended to be conveyed in
the paradigm was that objectives, goals,.and operational
principles should always be clear and agreed upon. As a
practical matter, it was felt that most people are
unconcerned with abstract philosophical principles until
there is clarification of the concrete implications of
such principles in their lives. Witness the fact that
many are for equal opportunity but not for busing; for
free enterprise but not for the importation of small
foreign cars; etc., etc. Hence, it was felt that
philosophical principles should be elaborated in terms
of the types of programs and procedures and obligations
people were really willing to live with. If the statement
in the explication of the paradigm appears to minimize
the need for this kind of clarity, then it should
certainly be revised.
142
Mr. John Vatsula details several critical
observations:
At this stage I'm not sure I agree with the
order of 1 and 2 under I, "Involvement
Stage." The reverse order appeals to me
on the basis of my experience, and nothing
can be found in the literature you provided
to convince me otherwise.
In defense of the paradigm, it would seem logical
to make sure one was dealing with all or a substantial
or representative part of a community before attempting
to survey their needs and aspirations. In practice,
however, the processes may be simultaneous, or reversed
as Mr. Vatsula suggests.
Our 40 classroom "assistants" are most
adamant in their insistance that they
not be called "aides." They consider
themselves co-teachers and not flunkies
as the word "aide" seems to connote
to them.
The term was used in the paradigm because it is found
so frequently in the literature. And, while in sympathy
with Mr. Vatsula's observations, the connotation of the
term "aide" (or any substitute term) would probably depend
more on local philosophy regarding differentiated staffing,
professionalism vs. lay participation, etc.. than on any
other factor.
See page 72, paragraph 2, line 3. Is the
word "idea" or "ideal"? I could't
disagree more with that sentence.
The sentence referred to states: "Though teachers
might manage hy teaching in teams or by working with
bilingual aides, the idea (sic) would be for teachers to
be bilingual themselves." The word "idea" is in error
and should read "ideal." It is difficult to understand
Mr. Vatsula's negative reaction to the statement in view
of the fact that he did not clarify the rationale for his
remarks. It might be hypothesized that Mr. Vatsula took
the suggestion that teachers be bilingual as a code-phrase
meaning that teachers should be Mexican-American. While
the section of the paradigm dealing with involvement
attempted to express sympathy with aspirations for greater
participation by Mexican-Americans at all levels of
education, such was not the intent here. The statement
means exactly what it says. In bilingual education
programs, the ideal would be for teachers to be bilingual
Mr. Vatsula goes on to object to a suggestion by
Andersson and Boyer that teachers be examined periodically
and rated on the scale of Qualifications for Teachers of
Modern Foreign Languages. This would suggest that his
statement means exactly what it says, that it would not be
ideal for teachers in bilingual education programs to be
bilingual. It is difficult to comprehend the basis for
such a statement. The suggestion that teachers be
bilingual is commonplace in the literature. Dr. Senour,
144
in his reply, was most emphatic in support of this view.
And it would seem logical in programs planned to develop
bilingualism, that teachers would be as competent as
the students.
Mr. Vatsula offers two further suggestions. First,
he recommends that roles and responsibilities be
cooperatively developed and clearly delineated. He also
recommends that feedback from the evaluator to project
personnel be continuous and allow for interaction.
In explaining the paradigm, an attempt was made to
establish these points, particularly in the section
dealing with program implementation. However, in view
of Mr. Vatsula's observations, it might be well to
emphasize these points or to state them more specifically.
Dr. Thomas Thomas. Besides those persons designated
as responsible for the evaluation of Title VII Bilingual
Programs in California, the paradigm was also sent to
Dr. Thomas Thomas of the Stanford Research Institute.
Dr. Thomas was involved in a review of the evaluations
of Title VII programs and was contacted because of his
special expertise in the area.
The paradigm was sent to Dr. Thomas along with a
cover letter. He did not respond in writing but by
telephone. However, his remarks were very enlighting and
helpful. In order to incorporate them in the study,
145
his observations were summarized and returned to him for
his signature. (The summary of Dr. Thomas' remarks is
contained in Appendix P along with the replies of the
evaluators).
In general, Dr. Thomas was favorable to the paradigm,
but he noted two major deficiencies. He stated that the
paradigm was "better than others I've seen, but marginally
better."
His first reservation or criticism verses about the
fact that the paradigm depicts an administrative control
system. It describes accountability from a school
administrator's point of view. This would be acceptable,
if the stake holders in the situation trusted one another.
But there is nothing in the paradigm to accomodate the
views of Chicanos or students. What is needed is the
establishment of independent control groups with money
and real power to implement their views. The situation
of most bilingual programs calls for change. But change
only comes through a change in the locus of power. What
is needed is a paradigm of political accountability.
Another point addressed by Dr. Thomas was the adoption
of an implementation model. Implicit in the paradigm is
the notion that an attempt will be made to arrive at
consensus in the adoption of programs. Dr. Thomas points
out that programs are rarely terminated as a result of
146
evaluation. Even where it is clear that programs are not
realizing their objectives, it is usually not politically
feasible to shut down programs. A more "accountable"
approach would be to set up a number of alternative
programs to be tested by "consumers." As long as a
sufficient number of parents supported a program it would
be maintained. But attempts should also be made to
maintain alternative programs and options. The context
is set up so that, if a program turns out to be poor,
there are easy places to move to. Options allow a program
to die by political choice. Programs should be set up
from the beginning to avoid confrontations and to allow
for needed change. That is best done by multiplication of
options and alternatives.
Dr. Thomas' suggestions regarding options seem very
sound and astute. And note should be made in the paradigm
of this added factor of accountability.
Dr. Thomas' observations regarding the weakness of
any evaluation program that constitutes an "administrative
device" is somewhat more complex. In the development of
the paradigm, suggestions were made, particularly at the
Involvement Stage, in an attempt to insure representative
community participation in programs. The employment of
an external evaluator at the Summative Evaluation Stage
is another suggestion intended to augment the validity of
147
program evaluation.
Admittedly, in an atmosphere of alienation or
confrontation, the suggestions made here are somewhat weak.
But the statement that a change of power structure is
needed appears to beg the question. Who are to constitute
the new power structure and what will be their powers and
responsibilities. To whom are they accountable? In view
of Dr. Thomas' suggestion regarding alternatives, one might
say that any group with a sufficiently large constituency
might be invested with autonomous central of a program.
The question then arises: how does one exercise
enlightened judgment in the use of this new power. If
accountability is to be equated with satisfaction of
constituents (parents, involved teachers, community
members, etc.) how is "satisfaction" arrived at. It would
seem that no matter who exercises power or control,
accountability to the public would more likely be attained
where systematic attempts to increase accountability
were undertaken.
Summary Of Revisions Or Clarifications
Of The Paradigm
In view of the reactions of evaluators to the
paradigm, a number of revisions are called for. Also,
a number of points should be expanded or made clear.
Following is a list of specific changes to be made.
148
Introduction To The Paradigm
1. The paradigm is not intended solely nor
specifically for program evaluators. It is not directed
toward the role of the evaluator but toward the function
of accountability, which is more diffuse and may be vested
in many people.
2. Accountability implies achieving specified
outcomes, and in education these outcomes are generally
understood to be stated in terms of student behavior.
Generally, it is not important which alternative means
are adopted in order to attain desired objectives. In
bilingual education, however, the concern of parents and
community members often verses about the way their children
are taught. Classroom climate, manner of teaching
students, respect for divergent culture and values, etc.,
can be more important than improvement in student achieve
ment measured in traditional ways. In this context,
accountability also connotes responsiveness to the desires
of the community and, hence, an attempt to improve the
quality of school atmosphere, to modify the way students
are taught. The "outcomes*' in these instances is a change
in the process or way we teach. Hence, there may not be
a sharp distinction between implementation measures and
evaluation measures. When the community is concerned with
the establishment of goals and objectives and the nature
149
of program implementation, then these factors become
outcome objectives.
3. The paradigm suggests steps for implementing
measures at all stages of a program to enhance or increase
accountability. However, to attempt to implement all
these measures may be well beyond the capacity or desire
of certain schools or districts. Total commitment to all
the stages and measures suggested here is not necessary
in order to increase accountability. Indeed, a more
limited or incremental approach might well be more
effective, in the long run, in attaining the goal of
greater accountability to the community. Rather than
making a half-hearted attempt to improve in all the areas
suggested here, a district or school might commit itself
to attempting substantial improvement in specific areas
year by year. For example, one year a program might
commit itself to upgrading skills and attitudes of
teachers and in developing curricular materials and
resources for a program; the following year it might
concentrate on developing communication with all the
members of the community, etc.
4. A commitment to accountability does not necessa
rily mean a cheaper program in terms of money, time, or
effort. In fact, the opposite is probably true. Persons
involved should be aware that commitment to community
150
involvement and to effective, high quality programs will
demand more time, money, and resources.
Involvement Stage
5. The need for community commitment to the program,
especially on the part of parents, should be emphasized
because of the effect of the home environment on the
performance of students in school. The possibility of
language difficulties on the part of parents (inability
to read or speak English) should be taken into account.
6. Existing groups should be involved in the program
where possible (P.T.A., advisory groups, etc.).
Commitment Stage
7. The explanatory material on development of
program philosophy suggests a "loose knit set of working
principles."
This statement appears to be misleading. Since it
could be taken to mean that it is preferable to proceed
without clarification of operational principles, it
should be deleted.
8. In discussing the adoption of objectives, it
should be emphasized that these objectives be coopera
tively developed with the participation of community
members, parents, teachers, and where feasible, students.
151
9. It should be noted that adoption of certain
objectives precludes realization of other objectives.
Program objectives should be clearly stated. It would
be worthwhile, also, to detail what will be deemphasized
in a program.
10. In adopting a program, it becomes necessary to
arrive at a specific, objective, and exact definition of
what constitutes Mexican-American culture or what should
be the nature of bilingual education. However, it is
unlikely that a consensus will be reached that will be
satisfactory to all involved.
There is an alternative approach to the problem of
setting specific objectives. This involves not seeking
consensus but rather, expansion of options and alterna
tives. If parents, students, or teachers are not content
with a program proposed, they might be given the option
of setting up a program that reflected their thinking and
desires. Then if they had enough constituents to warrant
implementation, their variation of a program would be put
into effect. Instead of several classrooms full of
students all following the same kind of program with the
same philosophy, goals, and objectives, there could be
several distinct programs in the same district or school
(in effect, a modified "voucher'* plan).
This approach would serve two ends. First,
accountability (in the sense of responsiveness to
community wishes) would appear better served by expansion
of options and alternatives and by increasing freedom of
choice. The second advantage relates to the area of
summative evaluation. Evaluators point out that inferior
programs are rarely terminated because of evaluations.
Often, no matter how a program is evaluated, it is simply
not politically expedient to terminate it. The existence
of alternative programs operating side by side, however,
gives people a place to go. Programs can be terminated
not by administrative decision but by defection of
constituents. The expansion of options, then serves a
purpose analagous to the maintenance of experimental and
control groups in more traditional and controlled attempts
at evaluation.
11. Step two of the Commitment Stage is called
"Formal Commitment To The Program." In view of the fact
that this precedes pre-assessment in the paradigm, it
should be labelled "Tentative Commitment To The Program."
Another step labelled "Formal Commitment To The Program"
should be added as Step six of the Commitment Stage.
Tentative commitment implies authorization to explore
costs, benefits, feasibility, etc., of potential programs
and to prepare plans for a specific program. Formal
commitment would imply authorization to implement the
153
program as planned.
12. In discussing the development of evaluation
measures, it is suggested that local behavioral objectives
be developed. A caveat should be added to the effect that
many programs begin operation without having clarified
objectives. Many persons involved in programs may not
know how to develop or state behavioral objectives. In
the absence of training for this purpose, accountability
would seem to indicate the use of traditional measures of
achievement. Though there are many threats to the
validity of results, it would appear better than no
evaluation at all.
Implementation Stage
13. At the Implementation Stage a step should be
added: "Planning Implementation." This would precede
the other steps.
In order to clarify responsibility for tasks, it
would help to state specifically what must be accomplished,
the person(s) responsible for accomplishment, and the
anticipated deadline for accomplishment.
Desmond Cook (1966) has a model, P.E.R.T. (Program
Evaluation and Review Techniques) that is helpful in
clarifying such steps.
154
Summary Of The Chapter
The paradigm was sent to forty-six persons who had
served as professional program auditors. Nineteen
responded. However, a number of these declined to react
to the paradigm. The replies of those who did address
themselves to the paradigm are contained in Appendix P.
General reactions to the paradigm were mostly
positive. However, some reservations of evaluators were
that: the paradigm was closer to evaluation that
accountability; accountability/evaluation procedures were
confounded with management implementation procedures;
the paradigm did not reflect the Chicano mentality; and
the paradigm should be more clear if intended for an
audience other than professional evaluators.
Observations directed toward specific aspects of the
paradigm were detailed and discussed. These led to
thirteen revisions or clarifications of the paradigm.
CHAPTER V
Summary Of The Study
This study was undertaken in an attempt to develop
a model that would aid community members, teachers, and
administrators to implement measures designed to increase
the element of accountability in bilingual education.
Prior to World War I, speaking languages other than
English tended to be taken for granted in the United
States because of the large number of immigrants. After
World War I, fear of foreign influence grew and languages
other than English were discouraged.
In educational circles, bilingual education was
viewed with scepticism. It was feared that bilingual
training would hamper the intellectual or academic
functioning of young children; that instruction in non-
English languages would lessen motivation to learn
English; and that as a consequence, acculturation and
assimilation would be diminished.
Research in recent years has gone counter to these
positions. It is generally accepted that early education
should be in a child's mother tongue. It is also felt
that a second language may be introduced at this time
without adverse intellectual effects. As far as weakening
155
156
motivation toward assimilation and acculturation, many
minority members see this as a positive factor, a remedy
to what they perceive to be the cultural chauvinism of
the schools.
The demand for bilingual education has been
paralleled by the demand for accountability. Originally,
as coined by Leon Lessinger, the term meant responsibility
for attainment of program objectives generally understood
to be stated in terras of student behavior. However, a
number of authors have used the term with a variety of
connotations.
In this study, accountability is defined as respon
siveness to public will, efficiency, and effectiveness in
realization of objectives.
In order to develop the paradigm, a survey of the
literature was undertaken. Special attention was given
to literature related to accountability, program
evaluation, systems approaches to program implementation,
and evaluation of bilingual programs. The paradigm was
developed as a synthesis of the literature. The position
is taken that the systems approach to program implemen
tation and evaluation is the best way to achieve
accountability.
The paradigm is composed of five stages, each with
several steps involved. The stages are: (1) Involvement
157
Stage; (2) Commitment Stage; (3) Implementation Stage;
(4) Summative Evaluation Stage; and (5) Accounting Stage.
Procedures described in the Commitment, Implementation,
and Summative Evaluation Stages are components of any
systems approach to program implementation and evaluation.
Procedures designated here as elements of the Involvement
Stage were highlighted because of complaints that members
of the Mexican-American community do not have adequate
voice concerning politics that affect the education of
their children. Procedures here designated as Accounting
Stage are suggested in order to formalize or ritualize the
idea that educators are responsible to the community and
should attempt to keep the community informed on the
actual status of programs.
At each stage, suggestions are made regarding
formative evaluation procedures to improve the ongoing
program. Likewise, measures are suggested to implement
process evaluation to insure that measures actually
implemented faithfully embody the philosophy and intent
of the program.
Following the development of the paradigm, an attempt
was made to have it evaluated. The paradigm was sent to
forty five persons who had been designated as responsible
for the evaluation of Title VII programs in California.
It was also sent to Dr. Thomas Thomas, of the Stanford
158
Research Institute. (Dr. Thomas was involved in the
review of all Title VII program audits). Nineteen
persons replied. However, a number of these declined
to react to the paradigm.
The reaction of the respondents to the paradigm was,
in general, favorable. Several negative reactions were
received. General criticisms of the paradigm were that it
was irrelevant because it did not reflect the thinking of
Mexican-Americans; that it did not clearly delineate the
role of the evaluator; that it confounded implementation/
management procedures with evaluation/accountability
procedures; that it should be more clear if intended for
persons other than professional evaluators; and that it
was defective because it was an administrative device
rather than a plan for shifting power.
In defense of the paradigm it is noted that an
attempt was made to suggest measures which would guatantee
that the concerns of Mexican-Americans were effectively
dealt with. The paradigm was not intended specifically
as a description of the role of a program evaluator.
Implementation was discussed at length, because concern of
the Mexican-American community is often directed more at
how schools operate than at what they achieve. The idea
that the paradigm will not effect accountability, because
it is an administrative device, seems to beg the question.
159
No matter who wields power, the ends of accountability
would seem better served if systematic attempts at
responsiveness, efficiency, and effectiveness were
undertaken. Complete autonomy of persons wielding power
does not seem a promising means to accountability
(in the sense of responsiveness to public will), whether
such persons be members of an entrenched bureaucracy or
newly constituted in power.
The evaluators also made a number of comments
regarding specific aspects or components of the paradigm.
This resulted in a number of revisions or clarifications
of the paradigm.
Discussion
In reflecting on the development of the paradigm
and reaction to it, a number of points come to mind.
It was inappropriate to ask evaluators to address
themselves to the criteria suggested by Stufflebeam,
et al., for the evaluation of evaluation models. The
plan to subject the paradigm to a "meta-evaluation" was
adopted before the paradigm was developed. In research
on accountability, the suggested criteria for the
evaluation of evaluation models was encountered and was
integrated into planned procedures. At that stage,
however, the intent was to have the paradigm evaluated
160
by a small panel of experts.
When the idea was conceived to consult all the
evaluators of Title VII programs in California, the
evaluation plan should have been changed. It was
unrealistic to ask so many strangers to undertake such a
systematic critique of so much material. One can only
surmise how data would have been summarized had all
forty-six persons contacted addressed themselves to
extensive consideration of the paradigm vis-a-vis the
nine criteria suggested. As a matter of fact, not a
single respondent addressed himself to the criteria
suggested by Stufflebeam, et al.
Apart from practical consideration involved in this
study, there is some question of the value of the above
mentioned criteria for the evaluation of any model.
While the criteria suggested appear worthwhile in the
critique of a specific design of an actual program, their
value becomes more questionable as the model becomes more
abstract and hypothetical. In reflecting on the matter,
it is interesting to note that the persons involved in
the seminar on the evaluation of the P.D.K. model, did
not address themselves explicitly to the criteria
seuggested by the model. (Stufflebeam, 1971).
Another weakness of the procedure that became apparent
in the course of the study was the request that evaluators
161
respond in writing. Several evaluators communicated by-
telephone. In reflecting on the insights gained from
such communication, it became evident that written
responses left much to be desired. Stufflebeam (1971)
and Provus (1971) both employed expert panels to discuss
their evaluation models. Such live discussion allows
for establishing a mutal frame of reference and
clarification of ambiguities. In the present case,
several evaluators made negative comments regarding the
paradigm and an attempt was made to defent the paradigm.
The evaluators, in this case, have no opportunity to
reply. Consequently, there is danger that the negative
comments of evaluators can be set up, like "paper tigers,"
to be rebutted too easily. The experience derived from
this study would seem to suggest that live panels of
evaluators should be employed in addition to written
surveys of reactions.
In general, most of the respondents were positive in
their reaction to the paradigm. A number of comments
made, however, raise questions regarding the utility of
the paradigm for specific audiences. It is not clear
whether the paradigm is too vague for the professional
evaluator or too complex for the layman. The intended
audience was teachers, administrators, parents, and
community members. It is recognized that the paradigm,
162
as explained here, does not contain enough specific detail
to serve by itself as a self-contained guide to program
evaluation and accountability. However, a paradigm by its
nature, is succinct. It was considered sufficient, for
the purpose of this study, to allude to specific proce
dures rather than to explain them exhaustively. To
satisfy practical operational needs of persons involved
in bilingual education programs, who may have not had
training in evaluation or accountability techniques, an
expanded version of the paradigm would probably be more
useful. It is hoped that the paradigm can be expanded
with a handbook explaining accountability measures in much
greater detail. The paradigm, however, was intended to be
suggestive rather than exhaustive, and the development of
a self-contained handbook of accountability measures was
beyond the scope and intent of the present study.
Another question regarding the paradigm concerns
the value of the general or global approach taken in its
development. The question of specific objectives, and
ways to measure them, was not highly developed. Almost
all writers on the topic agree that this is where
development is needed. However, because objectives tend
to be idiosyncratic and specific to particular programs,
it is difficult to mount large-scale attempts to develop
them. Nevertheless, such attempts should be encouraged.
163
The type of objectives derived by Prances Sussna (1970)
and by the Instructional Objectives Exchange of U.C.L.A.
should be encouraged in the area of bilingual education.
Finally, there is question whether the type of
activities suggested here will be of value unless
consideration is given to the political climate of the
program. Dr. Thomas Thomas suggests the need for a
paradigm of political accountability. The concerns to
which he alludes were not foreseen during the development
of the paradigm. Political accountability, to the extent
that it goes beyond systems approaches to program
evaluation and implementation, was not envisioned as
falling within the scope of this study. Nevertheless, it
is recognized that the topic is well worth studying
and developing.
Recommendations
It is suggested that further work in the area be
directed toward:
(1) Development of criterion-referenced measures
for bilingual education; and
(2) Development or refinement of measures that take
into account the social or political climate
concerned with academic achievement.
APPENDIX A
A Paradigm For The Implementation Of Accountability
Measures In Bilingual Education
164
A_PARADIQM_FOR THE IMPLEMENTATION OF .ACCOUNTABILITY MEASURES IN BILINGUAL EDUCATION
Management or Implementation Procedures
I. Involvement Stage.
1. Aeauranoe of community participation.
A. Syatematlo Involvement of community.
B. Ongoing participation and communication.
2. Meaeureo to enhance relevance.
A. Aeeeeement of oommunlty needa.
B. Survey of oommunlty aapiratione.
II. Commitment Stage.
1. Development of philoeophy and goal
atatementa.
2. Formal commitment to the program.
3. Pre-assessment.
A. Survey of etatua of atudenta.
B. Survey of reaourcea.
1. Teachera and aidea.
ii. Support personnel, consultants,
community resource people, etc.
iii. Materials,
iv. Facilities, time, funde, etc.
4. Adoption of implementation model and
major curriculum elements.
5. Development of instructional objectives.
6. Development of instrumentation for
summative evaluation.
A. Behavioral objectives.
B. Standardized teste.
C. Opinlonnairea, rating scales, etc.
D. Other measures.
•III. Implementation Stage.
1. Recruitment and training of personnel.
2. Assignment of supplementary tasks.
3. Acquieition or development of materials.
4. Program implementation.
IV. Summative Evaluation Stage.
1. Summary of data pertaining accomplishments.
A. Objectives.
B. Test results.
C. Opinionnaires.
D. Other measures.
2. Interpretation of data.
3. Judgment on effectiveness of program.
4. Statements of reservations, recommendations
for revisions, etc.
V. Accounting Stage.
1. Reports of teachers, aides, support
personnel.
2. Reports of building principals, local
directors, etc.
3. Report of superintendent.
4. District report.
Process Evaluation Procedures
Comparison of participation patterns with demographic
patterns.
Needa assessment procedures.
Reference to opinion of experts, community members
and school personnel. Comparison with other pro
grams.
Estimate of relevance, practicality, objectivity,
reliability, validity.
Training evaluated by opinionnaire and by reference
to process evaluation and progress evaluation during
implementation.
Evaluation of effectiveness of materials.
Process evaluation and progress evaluation procedures.
Review of factors that might influence reliable
collection or valid interpretation of data.
Independent judgment on effectiveness by outside
evaluator.
APPENDIX B
The EPIC Model
(Prom Hammond, 1967, p. 1)
166
BEHAVIOR
INSTRUCTION
° Rg4 N iza t
£ » n ! S ?
^ssr
C o s r
c O ^ INSTITUTION
APPENDIX C
Stake*s Evaluation Model
(Adapted from Stake, 1967, pp. 529 & 533)
168
-^-Logical Logical-►
169
Stake's Evaluation Model
Intenti
Congruence-^.
Observations Standards Judgments
! > »
o
G
0 )
t l f i
G
•H
-P
g
o
o
o
G
0 )
•G
•H
+>
g
o
o
Antec sdents
Transa Jtions
Outc )mes
I
« !
o
•H
u
•H
Ph-P
e g
M O
o
r-( O
r i G
o < u
• H to
U G
■ H - H
■ P . - P
E G
W o
O
Antec jdents
Transaiitions
Outc< mes
Description Matrix Judgment Matrix
(Adapted from Stake,, 1967, pp.529 & 533)
APPENDIX D
List Of Evaluators Of Title VII Programs To Whom
The Paradigm Was Sent For Evaluation
170
171
Name
Mr. David Aguirre
Mr. C. C. Cate
Mr. Joaquin Chavez
Ms. Juanita Cirilo
Mr. Reginald Cordes
Dr. Roberto Cruz
Mr. Raul De La Rosa
Mr. Juan Diaz
Dr. Ernest Garcia
Mr. Robert Garrow
Mr. Edwin Garvin
Ms. Barbara Gault
Dr. Lynette D. Glasman
Dr. Daniel Gomez
Dr. Wm. E. Hansen
Ms. Marienne Harrington
Mr. Barney Martinez
Dr. Dene Lawson
Dr. Charles Leyba
Dr. Keith Hartwig
Mr. Ruddie Irrizari
Mr. Mel Lopez
Ms. Barbara Merino
Mr. A1 Moreno
Dr. Steve Moreno
Mr. Noram Nicolson
Dr. Fred Niedermeyer
Mr. Virl Nuttal
Ms. Bertha Resendez
Mr. John Rich
Mr. Ed Rios
Dr. Monroe Rowland
Mr. Dale Russell
Mr. Robert Schaller
Dr. Richard Seligman
Dr. Robert Senour
Dr. Patricia Simmons
Dr. Leslie Six
Dr. Jerry K. Southard
Dr. Carl Stutzman
Dr. Ivor Thomas
Dr. Eleanor Thonis
Mr. John Vatsula
District
Salinas City School District
Sweetwater U.H.S.D.
King City Jt. U.H.S.D.
Hacienda La Puente U.S.D.
Mountain View Elem. S.D.
Jefferson City Elem. S.D.
Salinas City S.D.
Rowland U.S.D.
Barstow U.S.D.
Escondido U.S.D.
San Francisco U.S.D.
Sacramento City U.S.D.
Santa Barbara Co. Schools
San Bernardino City S.D.
Gilroy U.S.D., Gonzales H.S.D.,
Sta. Helena U.S.D.
Montebello U.S.D.
El Rancho U.S.D.
New Haven U.S.D.
El Monte Elem. S.D.
Sacramento City U.S.D.
Berkeley U.S.D.
Pajaro Valley U.S.D.
Redwood City Elem. S.D.
Santa Ana U.S.D.
San Ysidro U.S.D.
Santa Ana U.S.D.
Sta. Paula Elem. S.D.
Sweetwater U.H.S.D.
Los Angeles City U.S.D.
Healdsburg Union Elem. S.D.
Pajaro Valley U.S.D.
San Diego U.S.D.
Coachella Valley Jt. U.H.S.D.
Sweetwater U.H.S.D.
Sacramento U.S.D., San
Bernardino County Schools
Cucamonga U.S.D.
Hayward U.S.D., Los Nietos
Elem. S.D., Oxnard U.H.S.D.
Sweetwater U.H.S.D.
San Diego U.S.D.
Fresno City U.S.D., Sanger
U.S.D.
Fountain Valley U.S.D.
Marysville Jt. U.S.D.
Area III, Co. Supts. Review
and Control Board
172
Dr.
Mr.
Dr.
Name
Thomas Whalen
Bernando Worrell
Thomas Thomas
District
San Francisco U.S.D.
Riverside Go. Schools
(Stanford Research Institute)
APPENDIX E
Cover Letter And Materials Sent To
Evaluators With The Paradigm
173
174
January 5, 1974
Dear
I would like to request a favor of you.
I am working on ray dissertation for the Ph.d. in
Ed. Psych at U.S.C. and am presently engaged in a study on
accountability in Bilingual Education Programs. Hopefully,
the study will result in the production of a handbook of
accountability measures (things like descriptions of needs
assessment procedures, example survey forms, lists of
objectives, checklists for evaluation of materials, lists
of sample objectives and available tests, etc., etc.).
At present, however, my concern is with the notion of
"accountability" itself as an approach to evaluation.
There are probably as many ideas of what "accountability"
means as there are authors. And while virtually all the
approaches to accountability seem to make sense, at least
some of them (v. gr., accountability-performance
contracting) have not worked out empirically.
Your name has been given to me as one of the persons
involved in the evaluation of Title VII programs (you
are a hard group to locate!). The favor I would like to
request is to ask you to react to the paradigm for the
implementation of accountability measures attached, under
separate cover. The paradigm is accompanied by a number
of pages of explanatory material (which you may or may not
wish to examine carefully). Also attached is a list of
the criteria set down for the evaluation of an evaluation
model (Again, you may or may not wish to address yourself
to these criteria). At the very least, I would like to
solicit your impressions— as an expert practitioner—
to the paradigm, and to your feelings on accountability.
If you have any questions, please feel free to drop
me a line or call collect (805) 544-6841. If you would
like to be informed of results, just let me know. And
thanks for your time and help.
Sincerely,
(Signed)
175
Critique Of "A Paradigm For The Implementation Of
Accountability Measures In B.L. Education”
Attached are the criteria suggested by Stufflebeam,
et al. (the EDK National Study Committee on Evaluation)
in Educational Evaluation and Decision Making for the
evaluation of evaluation models or designs. It would
be appreciated if you could address yourself to these
criteria and could share your reactions and judgments
regarding the paradigm relative to each criterion.
Nevertheless, it is not absolutely necessary that you
follow this format. Furthermore, any observations that
you care to make about this model in particular or about
the validity, effectiveness, worth, etc., of an
accountability or systems approach in general will be
very helpful.
Name of evaluator:
Address:
Observations:
176
Criteria for the evaluation of evaluation models and
designs. Adapted from Daniel Stufflebeam, et al..
Educational Evaluation and Decision Making, Peacock
Publishers, Itasca, 1971.
1» Scientific Criteria
A. Internal Validity. (Are the information and the
processes used in the evaluation "true?" Do measures
and indicators used really give a true picture of the
program? Are outcomes considered truly results of
the program or of extraneous influences?).
B. External Validity. (Does the design allow for
extrapolation of results from this sample or program
in order to make generalizations concerning other
populations or programs?).
C. Reliability. (Does the design or model suggest that
resulting judgments and evaluations would be
consistent. Would different conclusions about a
program be reached if the evaluation process were
repeated?).
D. Objectivity. (Is the design or model such that
independent evaluators, following the procedures
proposed, would reach essentially the same
conclusions?).
2. Practical Criteria
A. Relevance. (How well does the design or model provide
for meeting the purposes of the evaluation? Are the
questions asked the right questions?).
B. Importance. (Are there elements of the design that
are relatively unnecessary or irrelevant?).
C. Scope. (Is the design sufficiently comprehensive?
Are there essential elements or considerations that
are ommitted?).
D. Credibility. (The quality of credibility refers to
the quality of the relationship between the evaluator
and his audience, especially when they are not in a
position to make an independent judgment concerning
the validity, reliability, and objectivity of results.
Are there elements of the model or design that, on
their face, would appear to inhibit credibility?).
177
B. Timeliness. (Timeliness is also a quality of a
particular evaluation, rather than of an evaluation
design or model. Are judgments and evaluations made
and reported in time to aid and inform decisions
(formative evaluation)? Again, the concern here is
whether this model contains elements that would appear
to impede the timeliness of an actual evaluation).
F. Pervasiveness. (Does the model suggest provisions for
the dissemination of evaluation findings to all
persons who need to know them?).
3. Prudential Criterion
A. Efficiency. (Efficiency refers to the cost/benefits
of a particular study. Have time, effort, resources,
etc., been wasted when this could have been avoided.
Would this model encourage inefficiency in the
evaluation process itself or in the educational
program in question?).
178
March 29, 1974
Dear
Several weeks ago, a copy of A Paradigm for the
Implementation of Accountability Measures in Bilingual
Education was forwarded to you. Your critical reaction
to the paradigm and your observations on the concept of
"accountability" in Bilingual Education were solicited.
By the very nature of your position, you will
probably be busy and occuplied with complex and detailed
work. Nevertheless, if you could find time to reply,
it would be very helpful to me, and, potentially, to
those who might be affected by the use of the paradigm
in the future.
Thanks for your kind attention.
Sincerely,
(Signed)
179
April 11, 1974
Dear
Several weeks ago, I sent you a copy of "A Paradigm
for the Implementation of Accountability Measures in
Bilingual Education*' and asked your critical reaction.
More recently, I sent a follow up request.
Originally, I had requested that you critique the
paradigm vis-a-vis the criteria suggested by Stufflebeam
and other members of the Phi Delta Kappa Study Committee
on Evaluation for the evaluation of evaluation models. A
number of respondents have stated that such a request was
too complex and demanding. They simply shared their
general impressions of the utility of the paradigm and,
in some cases, made observations regarding specific points
or details.
I would like to ask you again for your reactions
to the paradigm. In view of your experience in the
evaluation of bilingual education projects, I feel that
your remarks would be most helpful.
Sincerely,
(Signed)
180
February 9* 1974
Dr. Thomas Thomas
Stanford Research Institute
333 Ravenswood Avenue
Menlo Park, California 94025
Dear Dr. Thomas,
I am writing to you, at the suggestion of Dr. Newton
Metfessel of the University of Southern California, to
request a favor.
I am presently engaged in work on my dissertation for
the Ph.D. in Educational Psychology at the University of
Southern California. My area of concern is accountability
in bilingual education. It is my hope that the study will
result, eventually, in the production of a handbook of
accountability measures (things like descriptions of needs
assessment procedures, sample survey forms, lists of
sample program and instructional objectives, checklists
for the evaluation of materials, lists of existing crite
rion-referenced tests, etc., etc.).
At present, however, my concern is with the notion of
"accountability" itself as an approach to evaluation.
There are probably as many ideas of what "accountability"
means as there are authors on the subject. And while
virtually all approaches to accountability seem to make
some sense, at least some of them appear not to have
worked out empirically (v. gr., accountability-performance
contracting).
If I were to list what I feel are the essential
components of an accountability approach, I would
emphasize relevance, cost-effectiveness, and communication.
By relevance, I mean attempting to meet real needs and
desires of "clients"; not setting up ritualized or dummy
program objectives simply for "safe" evaluations or ease
of assessment. Cost-effectiveness would appear to be
essential to any accountability program (even though it
may not always be easy to determine, in a given instance,
what cost-effectiveness should mean). By communication,
I mean attempting to get the word out to the public in as
clear and objective a fashion possible concerning the
results of the assessment of a program, so that these
results can serve as baselines for attempts at incremental
improvement in all areas.
181
Dr. Thomas Thomas
Page 2
Attached is a paradigm for the implementation of
accountability measures in bilingual education. The favor
I would like to ask is that you react to the paradigm and
share your reactions with me. There are a number of
explanatory pages to the paradigm. You may or may not
wish to review these carefully. Also enclosed is a list
of the criteria suggested by the FDK National Study Group
for the evaluation of an evaluation model or study.
Again, you may or may not wish to address yourself to
these criteria. At the very least, I would like to ask
you to share your views on accountability with me. And
if you have any views on the status of evaluation of
bilingual programs, I would like to hear them.
I realize that this is quite a request, especially
from a total stranger. Though I would like to make it
clear that I would be happy to have any kind of a reaction
from you, no matter how brief, I appreciate the fact that
even this may be burdensome to a busy professional.
Should you wish to reply, if only to say that it is not
convenient for you at this time to comply with my requests,
a stamped self-addressed envelope is enclosed.
And whatever your decision, thanks for your kind attention.
Sincerely,
(Signed)
APPENDIX P
Replies Of Evaluators
182
183
Clarence C. Cate
"The paradigm is appropriate for accountability
measures in bilingual education or the implementation
of any other educational program.
"The key to evaluating the effectiveness of a
bilingual program in our district seems to depend more
upon examining data based upon criterion-referenced test
materials tied to specific behavioral objectives than to
data based upon standardized test results.
"Some of the students participating in our bilingual
education programs in our junior and senior high schools
have less than two years of formal education when they
enter our schools from Mexico while others have received
six or more years of formal education."
184
Edwin N. Garvin
"To have real accountability it is necessary that
a program functions without too many restrictions set
up by external factors, i.e. the program must have
the freedom to be innovative. Accountability of
measures in Bilingual Education might reflect the
School District's functioning as a whole rather than
just the program itself. (Please accept my apologies
for the long delay in replying, and for not being
able to devote more of my time to your project).
185
Dene R. Lawson
"I empathize with your desire for a doctorate.
It took me nine years to complete mine after acceptance
to the program at Stanford. My remarks truthfully, are
intended to assist, not hinder. However, if I understand
what you are trying to do, you are a long way from your
goal.
"In particular, I find your narrative usually
unclear. It suffers from big words and obtuse phrases.
I have tried to indicate a few examples. In education we
have a nomenclature that changes rapidly. New words are
added, and they have to be defined - unless your audience
consists only of you and your advisor.
"Incidently it is still not clear what is expected of
the individuals you asked for help. Instead of precise
criteria, I am asked only to react. For the most part,
my reactions have been negative, largely due in part to
the three intimate years I gave to evaluation of bilingual
education. If your paradigm includes the narrative for
purposes of clarification, it fails miserably.
Stufflebeam or Provus can provide the model for evaluation
in their books. Unless you write a book, your efforts to
explain will only confuse.
"So your contribution really is in clarifying the
186
Dene R. Lawson
(Cont.)
specifics of bilingual education - and I for one have
found this area extremely difficult to define. Apparently
you are having some difficulties also. My comments along
the way may be helpful - and at the same time discouraging.
"Specifically, the narrative makes a weak set of
suggestions for implementing (how to select teachers,
curriculum materials). You need to clarify the reason
for including initial selection of personnel and basic
curriculum in evaluation procedures. I speak of the
initial selections, not of subsequent need to evaluate
segments of the bilingual program.
"Your section on writing goals and/or objectives does
not help. Who is your audience? A novice needs more hlep
than you give; a professional will probably find it too
trivial.
"The section on testing and selecting instruments is
amateurish. Criterion-referenced testing for example is
an entirely new, sophisticated approach. Validity and
reliability are concepts not yet solved with CRT, so we
can hardly suggest local tests can be "normed" and
compared to local districts, etc.
"Your suggestions for evaluating the quality of
professional staff smacks of witch hunting. Instead of
187
Dene R. Lawson
(Cont.)
opinionaires and self-assessments, etc., you should find
out a) What a teacher or aide needs to know to serve
effectively, then b) find out if she has those
competencies. If she does not, then c) in-service
training for specific reasons is administered, followed
by d) a test on the needed competencies which were
taught specifically.
"I am not particularly anxious to continue this
critique. You are probably very conscientious and not
trying to do a minimal job for a dissertation.
Unfortunately my overall assessment of the narrative
section is "WEAK." Your one-page guide is useful as a
check sheet. Good luck."
188
Dr. Charles F. Leyba
"A broad reaction to the paradigm is that it is a
good, general set of steps that can be used as a checklist
for the development of a bilingual program. I don't think
that beyond this point it is very useful. It is certainly
not unique in this respect. There are other equally vague
guides. Perhaps this accounts for the poor development of
bilingual programs once funding has occurred.
Reactions by points:
I. 1. Assurance of community participation
A. This point is so vague as to admit a variety
of interpretations. If community participation
means anything special— and it must— then some
highly specific guidelines must be included.
In addition, as per our phone conversation,
pre-program training for advisory committee
members is a sine qua non, if guidance is to
be provided of a positive nature.
II. 2. Formal commitment to the program
What precisely does this mean, how indicated
or measured? Does it mean district/school
commitment? What are minimum indicators
for this?
III. 3. Pre-assessment
I feel that the time is ripe to have bilingual
program sites indicate the kind of model they
will install. Surely, enough sophistication
exists to determine the types of student needs,
relate these to staff and material resources
and come up with a distinct "model'' to serve
189
Dr. Charles F. Le.yba
(Cont.)
the community. As it is, project directors
know little about various models and the
variables of which they are composed.
The paradigm, presumably as a model, is
silent on this point and represensibly so.
To be sure, the word model is used (II, 4)
but, again, what does it mean?
III., V. Seem adequate except, perhaps, for III, 2
which is vague.
IV. General
No discrimation between the use of
standardized testing instruments and
criterion-referenced measures as evaluative
tools in determining program effectiveness.
While this observation is likely untrue with
respect to Item IV, 1, it is certainly true
that no reference is made to establishing
baseline data or to doing a midyear assessment.
Such normative data would obviously be
necessary in order to provide guidance for
teachers and the teaching strategies they
must employ. Since accountability is
generally accomplished through criterion-
referenced measures— and, my feelings are
that these measures alone should be employed
in bilingual programs, far more detailed
treatment should be given this point."
190
Fred C. Niedermeyer
"Criteria are sensible and OK. They don't
however, operationally define the term "accountability".
See enclosed article.
P.S.
"The article is typical of what those in evaluation
(particularly Stufflebeam) have been doing the past
several years, i.e., take any new term or process in
education (e.g. product development, accountability)
and redefine it as "evaluation". To them, everything
is evaluation."
191
Dale Russell
’ ’ The management and implementation procedures seem
complete and clear. The Process evaluation procedures
seem more nebulous. Perhaps The Metfessel, Michael
Paradigm for Multiple-Criterion measures in the evaluation
of educational programs might assist to clarify some
process assessment.
"It would be helpful to me to see your paradigm
charted after a flow or PERT model to visualize the
interrelationships and see the interplay between the
formative and summative evaluations.
"I believe that you are right on target and will
provide a useful tool for all evaluators - including
for bilingual project evaluators.
"At the accounting stage, have you considered the
role of establish groups within the educational system
i.e. District and site advisory groups, Administrative
and curriculum councils, Cabinet, etc.?
"Please let me know when you complete your study -
I'll drive by USC and read it. Best Wishes."
192
Dr. Richard Seligman
"I find it quite difficult to carry out the task
which you requested in your letter of January 5.
As I understand it, you would have liked me to critique
your paradigm in terms of Stufflebeam's criteria for
the evaluation of evaluation models and designs.
To do this, assuming it were possible, would require a
considerable investment of time and the preparation of
a written, lengthy essay. The amount of time required
for such a task would be excessive. Furthermore, in terms
of your dissertation, I would think that you would have
a tremendous problem in trying to systematically analyze
such an essay or essays that you might receive.
Nonetheless, I will offer some comments based on my
review of the materials which you sent me.
"My own notion of accountability is that it involves
nothing more than a statement of what will be attempted
in a particular program and how, at the conclusion of the
program, it will be possible to show that what was
attempted was achieved or not achieved. There are, of
course, many ways in which such a notion can be expanded
upon, but the basic notion of accountability is as I
have suggested.
193
Dr. Richard Seligman
(Cont.)
"I found it quite difficult to critique your paradigm.
My difficulty stems from the fact that the title of the
paper indicates that the paradigm deals with the
implementation of accountability measures in bilingual
education. As I read your paper I find that it deals
more with the implementation of bilingual programs than
it does with the implementation of accountability
measures for those programs. To be sure, you included
material on accountability in bilingual education, but
it was not sufficiently set apart so that a "paradigm"
emerged clearly.
"I believe that there is a tremendous need for
improving the accountability of bilingual education
programs in California and throughout the United States.
My criticisms of your work are not meant to discourage
you from pursuing this task, but rather are meant to
focus your attention to the accountability/evaluation
problem as distinguished from the program implementation
and management problem. Your paper would be much
improved if there were a more clear-cut statement of the
specific evaluation activities which must be undertaken,
the sequence in which those activities should occur, and
the relationship between the outcomes of those activities
194
Dr. Richard Seligman
(Cent.) ---------
and the management or operation of the program. I hope
these comments have been of some use to you and would be
interested in seeing the final results of your effort."
195
Dr. Robert Senour
"Mr. Troy, I've been trying for several hours to
figure out what you want. On the first page of your
paradigm appendix A is mentioned where I would find a
model for accountability in bilingual education.
I am unable to find appendix A. Consequently I am not
sure what I should respond to. However in the program I
evaluate the project is set up using a systems approach.
The model follows:
Needs Instructional Sequence of Evaluation
Assessment
-¥
Objectives Learning
-♦
Events
T Revision T Flow_______J______________
"Some of the concepts you have in your writing is
'garbage' - it isn't applicable because the premise is
based on an Anglo identification of schools. When you
are constructing a flow diagram for school children whose
schooling and parent's schooling is not nor has not
paid off in terms of getting part of the action then
the program must start from where 'they' are not where an
Anglo thinks they should be. To suggest, as is written
in the document, that receptionists should be Spanish
196
Dr. Robert Senour
(Cont.)
speaking is evidence of the problem. Teachers aides,
most administrators, custodians and all people associated
with any school where the language spoken by 40$ to 60$
of the population is different from English, should be
minimally conversant in that language plus culturally
aware.
"In summary, I can't do what you ask with what
you sent."
197
Patricia Caldwell Simmons
"1. Examine: Rabbi Gerald Teller. "What Are The Myths
of Accountability?" Educ Leadership. 31. (5)
455-456. Feb. 1974.
"2. Some of your process evaluation i.e. comparison of
patterns with demog. patterns is product or done
after the fact.
"Process to be process is on-going happening
that I would observe, monitor. If you are implying
observing in a one tailed test fashion on a matrix:
former demographic patterns, participation with what
is now happening - OK.
"3. Use of jury - ies. is also product to me unless
observers.
"4. I would like to see 3 columns as
Procedure Produe t Process
Very specific
Needs assessment
procedures - too
vague
My experiences as an evaluator/designer of research
is if your audience is school adm. the paradigm
cannot be too specific and clear. If your audience
is fellow researchers okay broad statements are
acceptable i.e. analysis of covariance for product
Patricia Caldwell Simmons
(Cont.)
My question is, I guess, do these procedures meet
the objective you have in mind - accountability
and on the basis of process I do not believe you
will know.
"I do not mean to be unduly critical rather
helpful. I wish you luck and if I can help
further - great."
199
Dr. Leslie W. Six
"I have read, with interest, the material you sent
to me. I agree that there is a need for a handbook of
accountability measures that you suggested might be an
outgrowth of your study and I would be interested in
acquiring such a handbook should it be completed.
"Generally, I concur with your paradigm, the basic
elements of which are essential for accountability in all
educational programs. It is perhaps in that area of
feasibility that I would like to make a few comments.
"In the Involvement Stage, there was nothing said
of the fact that there has to be a strong commitment of
both time and some resources, on the part of the school
district to go through the laborious, though necessary,
steps of Community Involvement and the early steps of
Commitment. These activities alone, might take the
better part of a year and will cost the school district
both time and money.
"In the Commitment Stage, it seems to me that
following the steps outlined in the paradigm, a commitment
is to be made before the program has been adopted.
Whether federally or state funded, or through local
resources, approximate costs of a program should be
determined to see if there are sufficient resources
200
Dr. Leslie W. Six
(Cont.)
available. Lacking such, a district might have to modify
or abandon a plan. Perhaps there should be a number 7
under II entitled 'Formal Adoption of the Program' since
at this point all costs would have been determined that
could be compared with financial resources available to
be expended.
"In the area of establishing goals again feasibility
rears its head and while the paradigm has self correcting
loops via the process evaluation, there should be a
realistic understanding of the fact that the establishment
of some goals may effectively reduce the possibility of
achieving others that have been established for other
purposes. As an example I cite a goal that seems to be
common in bilingual programs, that is, that all partici
pants will become bilingual in both languages. Since
Spanish as a second language will be added to the
curriculum, consideration need to be given to what other
subject areas, for which the school district already has
goals, will be deleted or slighted since there are only
so many minutes in the instructional day. Also, it has
been my experience that learning English as a second
language proceeds more rapidly for Spanish-speaking
students than does Spanish as a second language for
201
Dr. Leslie W. Six
(Cont.)
English-speaking pupils. So again, while the goal of all
students emerging bilingual by a certain period of time,
say be the end of grade 6, it may be difficult to achieve.
"In regards to the Summative Evaluation Stage, while
I agree with the scope and sequence outlined in the
paradigm, a very real stumbling block in our attempts to
evaluate a Title VII program has been the paucity of
reliable and valid tests to measure academic growth and
progress of Spanish-speaking pupils who are taught in
Spanish. This to me, is further evidence of the need for
a handbook of accountability measures.
"I hope these few brief comments will he of some
value to you. If I can be of further assistance,
please contact me."
202
Ivor J. Thomas
"I apologize for the lateness in responding to your
request for my reaction to the paradigm for implementation
of accountability measures you sent. I have been away
from the office a great deal of the time and just now
am beginning to catch up on some of the back
correspondence.
"The paradigm you presented is comprehensive and
provides a fairly accurate description of essential
procedures for the implementation of accountability
measures in bilingual education. Prom my experience as
an evaluator in bilingual-bicultural programs, I might
add a few caveats as well as underscore some of the
essential features of your paradigm.
"The major criticism I have is in relation to
statements made in the section on commitment. In
particular, on page 69, you suggested an approach
involving establishment of a loose-knit set of principles
rather than formulation of a broad and exhaustive
philosophy. While I may not understand completely what
you are implying, I am convinced that a great deal of
the trouble experienced in bilingual-bicultural programs
has been due to a lack of sufficient agreement on the
philosophy on which the program is based. This tends to
203
Ivor J. Thomas
(Cont.)
lead to confusion; therefore, regarding the appropriate
emphasis to be given to the various contents of the
program and instructional strategies. My fear is that
your paradigm in this regard may provide the rationali
zation some would welcome in avoiding answering the
tough questions that are a prerequisite for resolution
of critical dilemmas frequently encountered.
"With reference to the involvement stage, the
importance of obtaining representative community
participation cannot be overemphasized. There is
abundance of evidence to indicate the significant impact
of the home and community environment on student achieve
ment and attitude formation. It becomes imperative,
therefore, that programs must attempt to obtain maximum
involvement and participation of parents if they are to
succeed. Because of language difficulties, many parents
are reluctant to become involved in programs, and where
community members are weak in English, attempts should be
made to communicate in Spanish. One program I know of
has been successful in this regard by providing bilingual
members to sit by the Spanish-speaking parents to provide
a running account of the proceedings. Also, where survey
information is to be obtained, it is often assumed that
204
Ivor J. Thomas
(Cont.)
it is sufficient to have questionnaires in Spanish.
However, many of the Spanish-speaking parents may not be
able to read in Spanish, and therefore it is important
to attempt to sample opinions of this group by personal
contact by the community liaison person.
"One of the problems of evaluation of bicultural
programs with the Mexican-American has been the lack of
definition of what constitutes the unique elements of the
culture. Although most programs speak glowingly of the
^appreciation and awareness* of the culture, there is
little agreement beyond the description of a few holidays,
some clothing and foods of what constitutes ’the culture.*
More work needs to be done to obtain the necessary
agreement if appropriate measures are to be devised.
"You indicated that, because of the novelty of
bilingual education programs, it is unlikely that
evaluation needs will be well served by the use of
existing tests, and that most districts will have to
develop their own evaluation measures. It is true that
criterion-referenced measures will have to be developed
to assess achievement of the objectives unique to the
project or program. In this regard, however, experience
has shown that this area is fraught with difficulty.
205
Ivor J. Thomas
(Cont.)
In the first place, many programs lack sufficient
definition of the objectives and explication of the
content and activities which must be the base for
development of appropriate measures. Secondly, there is
often a lack of expertise in the area of test development,
so that many of the measures that are developed lack
sufficient psychometric sophistication to provide
accurate measurement of attainment of the objectives.
"I cannot stress too much the need for proper
orientation of the staff and proper procedures for test
administration and scoring, as well as analysis of the
data. Unless proper test administration procedures are
employed, the resulting data may not only be useless
but may provide a misleading or distorted picture of
project developments. Although, as has been indicated,
criterion-referenced measures relating to the unique
contents and activities of the program are helpful,
norm-referenced testing should not be completely over
looked. Boards of education and funding agencies
frequently require indications of the achievement of
students compared with students in general, for unless
programs can demonstrate achievement beyond the unique
contents of a particular program, they risk losing the
206
Ivor J. Thomas
(Oont.)
necessary support.
"I agree throughly that the acquisition and
development of materials for bilingual programs are
generally more demanding than is normally the case. This
highlights the developmental nature of most of these
programs and has several implications for evaluation.
Summative evaluation during this developmental phase of
the program, therefore, often becomes rather minimal in
nature, with greater stress on the continuous evaluation
of materials and activities. It has been found, for
example, that many of the commercially prepared materials,
although quite useful, may need to be modified consider
ably for use with these populations. This includes
assessment materials which often accompany the
commercially prepared instructional materials and
activities suggested. The evaluator, therefore, can be
especially helpful by observing and by gathering data
which may provide more objective feedback to the project
during various phases of development. The organized and
systematic observations of the evaluator also are
necessary to determine the fidelity with which the
program, as described, is implemented. Unless this is
properly carried out, it is not possible to determine
207
Ivor J. Thomas
(Cont.}
the degree of relationship of the results to program
implementation.
"I hope you will forgive this rather rambling
discourse. My comments, for the most part, are merely
to reinforce the essential elements you have depicted in
your paradigm, which I believe provides a good outline
of effective procedures for the evaluation of bilingual
programs. I wish you well in your study, as I feel it
will represent a significant contribution in an area of
great need.”
208
Summary Of Remarks Of Dr. Thomas Thomas
(Stanford Research InstituteT
"You have looked at accountability from a school
administrator’s point of view. Now, contrast that with
a political point of view. If all the stake holders
trusted each other, it would work beautifully. But there
is nothing in the paradigm to accomodate the views of
Chicanos and students.
"For example, you suggest a values survey. When the
values of school personnel differ from the values of
community members, you suggest sensitivity training as a
resolution. Let me suggest an analogy. Several years ago,
the peace movement caught on and substantial numbers of
people became upset with the war in Vietnam. Suppose a
values survey was taken at that time, and it was found
that the values of the generals at the Pentagon differed
markedly from the values of the public at large. Your
solution would be sensitivity training for the generals.
I am sceptical that that would bring about an effective
resolution of differences.
"You need to view evaluation as a political process.
Let me give you another example. We conducted the follow
through evaluation of Title VII programs here at Stanford
Research Institute. It was done professionally and was
209
Dr. Thomas Thomas
(Cont.)
very scientific and objective. But the bureaucracy
determined what we would look at. They controlled the
evaluation. Because of this situation, a lot of programs,
as we say in evaluation circles, are 'doomed to success'.
"Your evaluation model is nice, but it doesn't work
that way. If the model is to have impact, it has to be
useful to somebody. What I am really suggesting is not
that you change the paradigm. It is better than others I
have seen, but it is marginally better. Now talk to
participants and ask, 'How do we get a practical paradigm
so that, if decisions are not favorable to a group, they
have recourse to push something through'?.
"Your program should not be 'doomed to success'.
You should have an outside group, perhaps four Chicanos
and three teachers, who have no stake in making the
program succeed. They should be given power and money to
make the program work. It is this kind of thing I have in
mind when I say you need a model for political
accountability. You have to be able to ask the hard
questions. It is not the answers that are important, but
the questions. Control of the questions is where the
power is in a program. Tests on reading and writing
answer administratively 'neat' questions, but not the
210
Dr. Thomas Thomas
(Cont.)
questions that are central to the issues surrounding
bilingual education programs.
"We are talking about a change in the system. But
an administrative control system, such as your paradigm,
is not directed to change but to maintenance. Change only
comes through a shift in the locus of power. If you
relinquish power, it may wind up in the hands of a fiery
radical who wants to obstruct. That destroys the program.
But you have to take a chance. You have to build trust.
"A second point. You nicely talk about many different
views of what constitutes bilingual education and then
express a rather weak hope that consensus will be reached
and choice of a specific model will be made. But there is
really no need for that. If enough money is available,
alternative models or programs can be set up. This allows
free choice to the parents. And these ideas can be tested
by consumers. No one approach to bilingual education is
going to embody the type of education all the parents
desire. There must be more options, more options because
there is not unanimity in the Chicano commuity. That is
a way of building trust. Rhetoric is diffused. For
example, if you have a radical involved, and he has enough
supporters, you set up a class the way they want it.
211
Dr. Thomas Thomas
'(flontT)-----------
You evaluate the program at the end of the year and, if
the results are favorable, publicize the achievements.
The following year you may set up two classes.
"There is another reason for expanding options.
Evaluations never kill a program, decision makers do.
Expanded options allow a program to die by political
choice. The context should be set up so that there are
easy places to move to in the event of failure.
For example, suppose you set up a bilingual program and the
evaluation showed that your program caused brain-damage.
Are you going to kill a bilingual program? It is not
politically feasible. You should have alternative
programs where people can move to.
"I think your paradigm is really fine, but it should
be implemented under a political structure where the
proper questions are asked. Further, it should suggest
that programs be set up from the beginning to avoid
confrontations and to allow needed change. That is
best done by expansion of options, multiplication
of alternatives."
212
John Vatsula
"— At this stage I'm not sure I agree with the order
of 1 and 2 under I. 'Involvement Stage'. The reverse
order appeals to me on the basis of my experience, and
nothing can be found in the literature you provided
to convince me otherwise.
"— Our 40 classroom 'assistants' are most adament
in their insistance that they not be called 'aides'.
They consider themselves co-teachers and not flunkies
as the word 'aide' seems to connote to them.
"— See page 72, paragraph 2, line 3. Is the word
'idea' or "ideal'? I couldn't disagree more with that
sentence.
"— See page 73. I don't agree with Anderson and
Boyer (1970 Vol. II, p. 89). Implementation: Again,
in my experience and as a personal prejudice: before
anyone at any level starts doing anything, there should be
as specific a delineation of roles and responsibilities,
cooperatively developed, as is humanly possible.
In addition, the same applies to operational standards.
Timeliness of Reports: Schedules for mandated reports are
traditionally useless for proposal submission. Feedback
from evaluator to project personnel must be continuous,
preferably oral and written. The opportunity for
213
John Vatsula
(Cont.)
interaction must exist.
In Conclusion
"The paradigm, in my opinion, is well contrived.
Somehow the Community references (#1) seem less congruent
than the remainder.
"As an 'Internal Evaluator' I have responsibilities
outside the realm of evaluation, for example, community
and parent advisory group organization, in-services to
conduct, etc., I am one man with 1100 students, share a
secretary with two others, and by virtue of 'what is'
and 'what could be', the paradigm is 'lovely', almost as
idealistic as most proposals I've read. I'd love to see
'Stufflebeam, et al.' in action with the alligators in this
program. (Part of my problem is that I need more 'et al'.
"Please keep in mind that evaluators cannot, by virtue
of limited facilities, function in the same capacity as
researchers. I've been an evaluator of one of the larger
programs on the west ooast, and no 'researcher' has
knocked on my door and asked for my opinions as to those
areas in which they could make a contribution that could
be practical and relevant.
"Evaluators have the responsibility for gathering
214
John Vatsula
(Cont.)
information, organizing it, treating it, analyzing it,
arriving at conclusions, and submitting recommendations.
They have, however, no power to precipitate attention
to any of the above on the part of administrators.
(Let's assume that the recommendations are agreed upon
as practical and relevant).
"It was a pleasure dealing with you and unloading
on you, Mr. Troy. If you find a need for any further
pugnaciousness, please don't hesitate to contact me.
I have an intense interest in bilingual education and
evaluation."
REFERENCES
Alioto, R. & Jungherr, J. A. Operational P.P.B.S. for
education. New York: Harper and Row, 1971.
Andersson, T. & Boyer, Mildred. Bilingual schooling in
the United States. Austin: Southwest Educational
Development Laboratory, 1970. (2 volumes.)
Arciniega, T. The urban Mexican-American: a sociocultural
profile. University Park: New Mexico State
University, 1971.
Austin, G. R. Educational accountability: hallmark of the
1970's. Science Teacher. 1971, 38 (4), 26-28.
Bain, Helen. Self-governance must come first, then
accountability. Phi Delta Kappan. 1970, 51» 413.
Bain, Helen. Some misgivings about accountability.
Parent's Magazine. 1971, 46 (3), 40.
Bahr, J. Educational auditing: here to stay
School Management. 1969, 13 (11), 53-54.
Bane, Mary & Jencks, C. Five myths about your IQ.
Harper. 1973, 246 (1473), 28-40.
Beavan, K. Strike looms over ghetto merit pay plan.
Times Educational Supplement. 1970, 2880 til), 11.
Bereiter, C. & Engelmann, S. Teaching disadvantaged
children in the preschooll Englewood Cliffs:
Prentice Hall, 1966.
Bestor, A. Educational wastelands. Urbana: University of
Illinois Press, 195^.
Bhaerman, R. D. Great day of judgment. Education Forum.
1971, 35, 537-539.
Bishop, G. R., Jr. (Ed.). Foreign language teaching:
challenges to the profession. Princeton: Princeton
Press, 1965.
215
216
Bloom, B. S. (Ed.), Englehart, M. D., Furst, E. J., Hill,
W. H. & Krathwohl, D. R. A taxonomy of educational
objectives: handbook I, the cognitive domain.
New York: Longmans, Green, 1956.
Boulding, K. General systems theory: the skeleton of
a science. Management Science. 1956, 3, 197-208.
Bracht, G. H., & Glass, G. V. The external validity of
experiments. American Educational Research Journal.
1968, 5, 437-474":
Bratten, D., et al. Performance contracting and how it
works in Texarkana. School Management. 1970, 14
(8) , 8-10.
Briner, C. Administrators and accountability. Theory
Into Practice. 1969, 8, 203-206.
Buros, 0. K. The 1938 mental measurements yearbook.
New Brunswick: Rutgers University Press, 1938.
Buros, 0. K. The 1940 mental measurements yearbook.
Highland Park: Gryphon Press, 1941.
Buros, 0. K. The third mental measurements yearbook.
New Brunswick: Rutgers University Press, 1949.
Buros, 0. K. The fourth mental measurements yearbook.
Highland Park: Gryphon Press, 1953.
Buros, 0. K. The fifth mental measurements yearbook.
Highland Park: Gryphon Press, 1959.
Buros, 0. K. The sixth mental measurements yearbook.
Highland Park: Gryphon Press, 1965.
California State Assembly. Assembly Bill Number 2284.
The Bilingual Education Act o f 197^1 Sacramento:
State Legislature, 1972.
California State Department of Education. Handbook for
California school district advisory committees.
Sacramento: California State Department of Education,
1968.
Campbell, D. T. & Stanley, J. C. Experimental and
quasi-experimental designs for research. New York:
Rand McNally, 19*53.
217
Carss, B. W. Systems analysis in education: a statement.
Educational Products Report. 1969» 2 (5), 43-44
Casavantes, E. J. A new look at the attributes of the
Mexican-American. Albuquerque: Southwestern
Cooperative Educational Laboratory, 1969.
Cass, J. Profit and loss in education: Texarkana and
Gary, Indiana. Saturday Review. 1970, 53 (33),
39-40.
Castaneda, A. Cultural democracy in education. A.M.A.E.:
Journal of the Association of Mexican-American
Education: 1973. 1 (1) 39-45.
Chandler, J. T. & Plakos, J. Spanish speaking pupils
classified as educable mentally retarded. Sacramento:
California State Department of Education, 1969.
Chomsky, N. IQ tests: building blocks for the new class
system. Ramparts. 1972, 11 (l), 24-30.
Clark, K. Answer for the "disadvantaged": effective
teaching. New York Times. 1970, 119 (40,896), 50.
Cook, Constance. Accountability in higher education:
sharing the duty to account. Compact. 1970
4, 24-25.
Cook, D. Program evaluation and review techniques;
Applications in Education. Washington, D. C.;
U. S. Department of Health, Education, and
Welfare, 1966.
Costa, A. Who's accountable to whom? Educational
Leadership. 1970, 28, 15-19.
Cunningham. L. L. Our accountability problems. Theory
Into Practice. 1969, 8, 285-292.
Darcy, N. T. Bilingualism and the measurement of
intelligence: review of a decade of research.
Journal of Genetic Psychology. 1963, 103, 259-282.
Darland, D. D. The profession's quest for responsibility
and accountability. Phi Delta Kappan. 1970, 52,
41-44.
218
DeAvila, E. I.Q. and the minority child. A.M.A.E.:
Journal of the Association of Mexican-American
Educators. 1973, 1 (l), 34-^8.
Deck, L. L., Jr. Accountability and the organizational
properties of schools. Educational Technology.
1971, 11 (1), 36-37.
Deterline, W. Applied accountability. Educational
Technology. 1971, 11 (1), 15-20.
Detroit Public Schools Staff. What about us? Our
textbooks do not meet our needs. Educational
Products Report. 1969, 3 (2), 12-TST
Deutsch, M., Fishman, J. A., Logan, L., North, R. &
Whiteman, M. Guidelines for testing minority group
children. Journal of Social Issues, 1964, 20
(Supplement Number 2), 129-145.
Deutsch, M. & Associates. The disadvantaged child.
New York: Basic Books, 1967.
Duncan, M. G. An assessment of accountability: the state
of the art. Educational Technology. 1971, 11 (1),
27-30.
Education U.S.A. - Washington Monitor. Bonus pay for
Title I teachers. Nov. 2, 1970, 53.
Ehrle, R. A. National priorities and performance
contracting. Educational Technology. 1970, 10
(7), 27-28.
Elam, S. Age of accountability dawns in Texarkana.
Phi Delta Kappan. 1970, 51, 509.
Fishman, J. A., et al. Language loyalty in the United
States. The Hague: Mounton, 1966.
Fishman, J. A. Bilingual education in sociolinguistic
perspective. TESOL Quarterly. 1970, 4, 215-222.
Gaarder, B. A. Statement before the Special Subcommittee
on Bilingual Education of the Committee on Labor and
Public Welfare, U.S. Senate, May 18, 1967. Florida
F.L. Reporter. 1969, 7 (Spring/Summer), 33-34; 171.
219
Garvue, R. J. Accountability: comments and questions.
Educational Technology. 1971, 11 (1), 34-35.
Geller, Evelyn. Accountability: Right to Read program and
integration versus compensatory education.
Library Journal. 1970, 95, 1881.
Glass, G. V. The many faces of educational accountability.
Phi Delta Kannan. 1972, 48 (10), 636-639.
Grieder, C. P.P.B.S. and assessment. Where trouble could
erupt. Nation's Schools. 1969, 83, 8.
Grobman, Hilda G. Evaluation activities of curriculum
projects. Chicago: Rand McNally, 1968.
Hammond, R. A design for local evaluation. EPIC Forum.
1967, 1, 1-6.
Harlacher, E. L. & Roberts, Eleanor. Accountability for
student learning. Junior College Journal. 1971,
41 (6), 26-30.
Harrison, C. H. How to respond to public demands for
accountability. Nation's Schools. 1970, 86, 32-34.
Hartley, H. J. Educational planning-programming-budgeting.
Englewood Cliffs: Prentice Hall, 1968.
Hartley, H. J. Twelve hurdles to clear before you take on
systems analysis. American School Board Journal.
1968, 156, 17-18.
Hill, J. How school systems can apply systems analysis.
Bloomington: Phi Delta Kappa Educational Foundation,
1972.
Huntsman, Beverly S. Some sociological factors in
educating bilinguals. TESOL Quarterly, 1972, 6,
255-261.
Jakobovits, L. A. Foreign language learning:
a psycholinguistic analysis of the issues.
Rowley: Newbury House, 1970.
Jaramillo, Mari-Luci Cultural differences in the ESOL
classroom. TESOL Quarterly. 1973, 7, 51-60.
220
Jensen, J. V. Effects of childhood bilingualism
Elementary English. 1962. Part I: 39, 132-143;
Part II: 39» 356-366.
John, Vera & Homer, Virginia M. Early childhood
bilingual education. New York: The Modern Language
Association of America, 1971.
Joint Committee on Educational Goals and Evaluation.
The way to relevance and accountability in education.
Sacramento: Senate and Assembly of the State of
California, 1970.
Joint Committee on Educational Goals and Evaluation.
Education for the People. Sacramento. California
State Legislature and California State Department
of Education, 1972. (Two volumes.)
Jones, R. M. How and when do persons become bilingual?
In L. G. Kelley (Ed.) Description and measurement
of bilingualism. Toronto: University of Toronto
Press, 1969, Pp. 12-35.
Jordan, B. Educational accountability: a crucial question.
Junior College Journal. 1971, 41, 23-25.
Kaufman, R. A. Accountability, a systems approach and the
quantitative improvement of education: an attempted
integration. Educational Technology. 1971, 11,
21-26.
Kennedy, J. D. Planning for accountability via management
by objectives. Journal of Secondary Education.
1970, 45, 348-354.
Klein, S., Fenstermacher, G. & Alkin, M. C. The center's
changing evaluation model. Evaluation Comment.
1971, 2 (4), 9-12.
Koerner, T. P.P.B.S. and the schools. Education U.S.A.:
Special Report. Washington: National School Public
Relations Report, 1972.
Krathwohl, D. R., Bloom, B. S. & Masia, B. A taxonomy of
educational objectives: handbook II, the affective
domain. New York: David MacKay, 1964.
Kruger, S. Program auditor: new breed on the educational
scene. American Education. 1970, 6 (2), 56-57.
221
LaBelle, T. J. What's deprived about being different?
Elementary School Journal. 1971, 72 (1), 12-19.
Lambert, W. E. & Klineberg, 0. Children's views of
foreign peoples. New York: Meredith, 1967.
Laosa, L. M. Reforming educational testing and
psychological assessment A.M.A.E.: Journal of
the Association of Mexican-American Educators.
1973, 1 (1), 19-24.
Lee, R. R. A taxonomy of second language behaviors.
TESOL Quarterly. 1972, 6, 209-220.
Lessinger, L. A historical note on accountability in
education. Journal of Research and Development
in Education” 1969, 5 (l), 15-18.
Lessinger, L. M. Engineering accountability for results
in public education. Phi Delta Kappan. 1970, 52,
217-225.
Lessinger, L. M. Accountability in education.
Educational Technology. 1971, 11 (l), 11-31.
Lessinger, L. M. Teachers in an age of accountability.
Instrue tor. 1971, 80 (10), 19-20.
Lessinger, L. M. & Allen, D. Performance proposals for
educational funding: a new approach to federal
resource allocation. Phi Delta Kappan. 1969, 50,
136-137.
Levine, M. Scientific method and the adversary model:
some preliminary suggestions. Evaluation Comment.
1973, 4 (2), 1-3.
Lezama, J. S. Bilingualism and the Mexican-American
college student and his community. Paper presented
at the 5th Annual TESOL Convention, New Orleans,
March 4, 1971.
Lieberman, M. Overview of accountability. Phi Delta
Kappan. 1970, 52, 194-195.
Lynd, A. Quackery in the public schools. Boston: Little,
Brown, 1970.
222
Mackey, W. F. A typology of bilingual education. In
Andersson, T. & Boyer, Mildred. Bilingual education
in the United States. Austin: Southwest Educational
Development Laboratory, 1970. Volume II, Pp. 63-82.
Mager, R. F. Preparing instructional objectives.
Palo Alto: Fearon, 1962.
Mager, R. F. Goal Analysis. Belmond Fearon, 1972.
Malherbe, E. G. The bilingual school: a study of
bilingualism in South Africa. London: Longmans,
Green, 1946.
McIntyre, R. B. Evaluation of instructional materials and
programs: applications of a systems approach.
Exceptional Child. 1970, 37, 213-220.
McLenon, T. B., Caperton, M. & Nilson, T. G., Jr.
Can parents demand accountability?
Instructor. 1970, 80 (1), 47.
Mercer, Jane. IQ - the lethal label. Psychology Today
1972, 6 (4), 44-47, 95-97.
Mercer, Jane. Implications of current assessment
procedures for Mexican-American children. A.M.A.E.:
Journal of the Association of Mexican-American
Educators! 1973, 1 (1), 25-33.
Metfessel, N. S. & Michael, W. B. A paradigm involving
multiple criterion measures for the effectiveness
of school programs. Educational and Psychological
Measurement. 1967, 27 (Part II), 931-942.
Modiano, N. National or mother language in beginning
reading: a comparative study. Research in the
Teaching of English. 1968, 2, 32-43.
National Consortia for Bilingual Education. Tests in
use in Title VII bilingual education projects.
Fort Worth, 1971. (Research in Education, ERIC.
Ed 050 886.)
Nation's Schools. Florida accountability plan focuses
on the principal. 1970, 86 (5)> 54-55.
Niedermeyer, F. C. Conditions for successful teacher
accountability. Los Alamitos: Southwestern Regional
Laboratory for Educational Research and Development
223
Norris, J. E. Accountability: watchword for the 70's.
The Clearing House. 1971, 45, 323-328.
Norris, W. E. Teaching second language reading at the
advanced level:, goals, techniques and procedures.
TESOL Quarterly. 1970, 4 (1), 17-35.
Nyquist, E. B. Measuring purposes and effectiveness.
Compact. 1970, 4, 21-23.
Olguin, L. Unconscious cultural clashes. (Filmed Lecture
Series.) Santa Clara: Santa Clara County Office of
Education and Mexican-American Research Project. N.D.
Part I: Grand assumptions; Part II: Objective
acculturation; Part III: Outlook; Part IV: Look me in
the eye; Part V: Education versus educacion;
Part VI: Customs.
Pacheco, M. T. Approaches to bilingualism: recognition
of a multi-cultural society. The Britannica Review
of Foreign Language Education. 1971, 3, 71-124.
Peal, E. & Lambert, W. E. The relation of bilingualism to
intelligence. Psychological Monographs, General and
Applied. 1962, 76 (27), 1-33.
Peterson, R. W. Accountability in elementary and secondary
education. Compact. 1970, 4, 19-20.
Phillips, C. Spanish speakers in the U.S.: Mexican-
Americans, Puerto Ricans and Cubans. In Andersson,
T. & Boyer, Mildred. Bilingual schooling in the
United States. Austin: Southwest Educational
Development Laboratory, 1970. Volume II, Pp. 105-124.
Phillips, R. E. Whose children shall we teach?
Educational Leadership. 1970, 27, 471-474.
Pierce, W. H. & Hazlett, J. A. Accountability through
national assessment. Compact. 1970, 4, 4-8.
Premack, R. Accountability for the humanists.
Phi Delta Kappan. 1971, 52, 620-621.
Pretzel, P. W. Whales and Polar Bears. Research and Pupil
Personnel Services Newsletter. 1970, 8 (2), 5-7.
Provus, M. Discrepancy evaluation for educational program
improvement and assessment. Berkeley: McCutcheon,
1971.
224
Ralph, L. D. Report on the education of children in the
ghetto school; a legislative proposal for reform.
Sacramento: California Legislature, 1970.
Ramirez, M., III. Cultural democracy: a new philosophy
for educating the Mexican-American child.
National Elementary Principal. 1970, 50 (2), 45-46.
Ramirez, M., III. Cultural democracy, cognitive styles and
educational change through needs assessment.
A.M.A.E.: Journal of the Association of Mexican-
American Educators. 1973, 1 (1), 52-56.
Richardson, M. W. An evaluation of certain aspects of the
achievement of elementary pupils in a bilingual
program. Unpublished doctoral dissertation.
Coral Gables: The University of Miami, 1968.
Riesman, D. The culturally deprived child. New York:
Harper and Row, 1962.
Rosen, C. L. & Ortego, P. D. Problems and strategies in
teaching the language arts to Spanish speaking
Mexican-American children. Las Cruces: New Mexico
State University, 1969.
Sako, S. Writing proficiency and achievement tests.
TESOL Quarterly. 1969, 3, 237-249.
Saville, M. R. & Troike, R. C. A handbook for bilingual
education. Washington: Center for Applied Linguis
tics, 1970.
Scriven, M. The methodology of evaluation. In Stake,
R. E. (Ed.) Perspectives on curriculum evaluation:
A.E.R.A. monograph series on curriculum evaluation.
Number T ~ , Chicago: Rand McNally, 1967, Pp. 39-83.
Scriven, M. An introduction to meta-evaluation.
Educational Products Reports. 1969, 2, 36-38.
Silberman, C. E. How the public schools kill dreams and
mutilate minds. Atlantic. 1970, 225 (6), 83-95.
Simon, B. S., Howe, L. W. & Kirschenbaum, H. Values
clarification. New York: Hart, 1972.
Smith, M. The diminished mind. Chicago, Regnery, 1954.
Spolsky, B. Attitudinal aspects of second language
learning. TESOL Newsletter. 1971, 5 (1), 5-8, 14.
225
Stake, R. E. The countenance of educational evaluation.
Teachers College Record. 1967, 68 (7), 523-540.
Stufflebeam, D. A. Critique of the report of the Phi
Delta Kappa Study Committee on Evaluation. Feb 6.
1971. (Research in Education, ERIC. Ed. 056 074.)
Stufflebeam, D. A., Foley, W. J., Gephart, W. G., Guba, E.
G., Hammond, R. I., Merriman, H. 0., & Provus, M.
Educational evaluation and decision making.
Itasca: Peacock, 1971.
Sussna, Frances. Some concepts of a multi-cultural
classroom. San Francisco: Multi-cultural Institute,
1970. (Research in Education, ERIC. Ed 065 403.)
Teller, G. What are the myths of accountability?
Educational Leadership. 1974, 31 (5) 455-456.
Tucker, G. R., et al. Cognitive and attitudinal
consequences of following the curricula of the
first four grades in a second language.
Montreal: McGill University, 19717
Tucker, G. R. & d'Anglejean, A. Language learning
processes. Britannica Review of Foreign Language
Education. 1971, 163-182.
Tyler, R. W. General statements on evaluation. Journal
of Educational Research. 1942, 35, 492-501.
Tyler, R. W. Testing for Accountability.
Nation’s Schools. 1970, 86 (6), 37-39.
Ulibarri, H. Bilingual education: a handbook for
educators. Albuquerque: New Mexico University, 1970.
(Research in Education, ERIC. Ed 038 078.)
U.N.E.S.C.O. Re-port on an international seminar on
bilingualism in education at Aberystwith. Wales.
London: Her Majesty's Stationery Office, 1965.
United States Commission on Civil Rights: Mexican American
Education Study. Ethnic isolation of Mexican
Americans in the public schools of the Southwest.
Washington: U.S. Government Printing Office, April,
1971.
226
United States Commission on Civil Rights: Mexican American
Education Study. The unfinished education.
Washington: United States Government Printing Office,
October, 1971.
United States Commission on Civil Rights: Mexican American
Education Study. The excluded student: practices
affecting Mexican-Americans in the Southwest.
Washington: U.S. Government Printing Office,
May, 1972.
United States Commission on Civil Rights: Mexican American
Education Study. Teachers and students. Washington:
United States Government Printing Office, March, 1973.
U.S. News and World Report. A better way to spend the
education dollar? 1970, 69 (11), 46.
United States Office of Education. Programs under
Bilingual Education (Title VII, E.S.E.A.K Manual
for pro.ject applicants and grantees.
Washington: 1971.
Upshur, J. A. Problems and tests. Language Learning.
1968, 3 (3), 7-12.
Valencia, A. Bilingual-bicultural education:
a prospective model in multi-cultural America.
TESOL Quarterly. 1969, 3, 321-332. (a)
Valencia, A. Identification and assessment of ongoing
educational and community programs for Spanish
speaking people. Albuquerque: Southwestern
Cooperative Educational Laboratory, 1969. (b)
Walker, W. The design or writing systems for native
literacy programs. Paper presented at the American
Anthropological Association, November, 1968.
Walsh, D. D. Bilingualism and bilingual education:
a guest editorial. Foreign Language Annals.
1969, 2 (3), 298-303.
Weiner, N. Cybernetics. New York: Wiley & Sons, 1948.
Weinreich, U. Languages in contact: findings and problems.
New York: Linguistic Circle of New York, 1953.
227
White, S. H. Evidence for the hierarchical arrangement
of learning processes. In Lipsitt, L. P. & Spiker,
C. C. (Ed.) Advances in child development and
behavior. New York: Academic Press, 1965.
Wilcox, P. R. Selected principles for involving the poor.
In Joint Committee on Goals and Evaluation.
Education for the people. Sacramento: State
Legislature, 19?£• Vol. II, Pp. 246-253.
Zintz, M. Education across cultures. Dubuque: Wm. C.
Brown, 1969.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

A Factor Analytic Study Of Tests Designed To Measure Reading Ability

PDF

The Relation Of Sense Of Humor To Creativity, Intelligence, And Achievement

PDF

Teacher Assessment Of Creative Potential In Fifth-Grade Students

PDF

A Study Of Relationships Between Grades And Measures Of Scholastic Aptitude, Creativity, And Attitudes In Junior College Students

PDF

A Semantic Differential Investigation Of Significant Attitudinal Factors Related To Three Levels Of Academic Achievement Of Seventh Grade Students

PDF

A Comparison Of Two Instructional Programs For Severely Retarded Readers At The Junior High School Level

PDF

An Experimental Analysis Of The Relationship Between The Reliability Of Amultiple-Choice Examination And Various Test-Scoring Procedures

PDF

Attitudinal Variables Among Teachers Of Exceptional And Non- Exceptional Children

PDF

A Comparison Of Degrees Of Bilingualism And Measure Of School Achievementamong Mexican-American Pupils

PDF

A Study Of The Factorial Validity And Reliability Of The Individual Test Of Creativity

PDF

A Measure Of Cultural Deprivation

PDF

Delinquency As A Function Of Intrafamily Relationships

PDF

The Relationship Of Father-Absence, Socio-Economic Status, And Other Variables To Creative Abilities In Fifth-Grade Boys

PDF

A Study Of The Comparability Of The Wisc And The Wais And The Factors Contributing To Their Differences

PDF

An Analysis Of The Selection Criteria For Assignment Of Students To Advanced Placement Classes In The Los Angeles Unified School District

PDF

Personality Variables Associated With Narcotic Addiction As Measured By The Minnesota Multiphasic Personality Inventory

PDF

Prediction Of Therapeutic And Intellectual Potential In Mentally Retardedchildren

PDF

Selected Characteristics Of A Children'S Individual Test Of Creativity

PDF

The Effect Of Dissonance In Self-Esteem On Susceptibility To Social Influence

PDF

An Evaluation Of Levels Of Cognitive Learning In A Unit Of Fifth Grade Social Studies

Asset Metadata

Creator Troy, Bernard Allan (author)

Core Title A Paradigm For The Implementation Of Accountability Measures In Bilingualeducation

Degree Doctor of Philosophy

Degree Program Educational Psychology

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag education, educational psychology,OAI-PMH Harvest

Language English

Contributor Digitized by ProQuest (provenance)

Advisor Metfessel, Newton S. (committee chair), Lovell, Constance (committee member), Smith, Robert A. (committee member)

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c18-846152

Unique identifier UC11356443

Identifier 7421513.pdf (filename),usctheses-c18-846152 (legacy record id)

Legacy Identifier 7421513.pdf

Dmrecord 846152

Document Type Dissertation

Rights Troy, Bernard Allan

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the au...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus, Los Angeles, California 90089, USA