00001.tif

PDF

Download

Open document

Flip pages

Copy asset link

Request this asset

Transcript (if available)

Content ETHOS AND STUDENT TEXTS:
THE RHETORIC OF ANONYMOUS MASS ASSESSMENT
by
Randall Jay Adams
A Dissertation Presented to the
FACULTY OF THE GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
(English)
December 1985
UM I Number: DP23101
All rights reserved
INFORMATION TO ALL U SERS
The quality of this reproduction is d e p en d en t upon the quality of the copy subm itted.
In the unlikely event that the author did not se n d a com plete m anuscript
and th ere a re m issing p ag es, th e se will be noted. Also, if m aterial had to be rem oved,
a note will indicate the deletion.
Dissertation Publishing
UMI DP23101
Published by P roQ uest LLC (2014). Copyright in the D issertation held by the Author.
Microform Edition © P roQ uest LLC.
All rights reserved. This work is protected against
unauthorized copying under Title 17, United S ta te s C ode
P roQ uest LLC.
789 E ast E isenhow er Parkw ay
P.O . Box 1346
Ann Arbor, Ml 4 8 1 0 6 - 1346
UNIVERSITY OF SOUTHERN CALIFORNIA
THE GRADUATE SCHOOL
UNIVERSITY PARK
LOS ANGELES, CALIFORNIA 90089
Ph.D.
E
85
A2I&
*9 //
'• w * I ,> • .:> < »
This dissertation, w ritten by
Randall Jay Adams
under the direction of h~?.......... Dissertation
Committee, and approved by all its members,
has been presented to and accepted b y The
Graduate School, in partial fulfillm ent of re
quirem ents for the degree of
D O C TO R OF PH ILOSOPH Y
D e a U // Graduate Studies
Date ...I .?.?.?..
DISSERTATION COMMITTEE
j
Chairperson
ACKNOWLEDGMENTS
I thank the Educational Testing Service, the National
Assessment of Educational Progress, and the University of
Southern California Freshman Writing Program. Personnel
from each of these institutions were very generous in
answering my questions and providing me with needed
documentation describing their methods of student writing
assessment.
Special mention must be made of Louise Wetherbee
Phelp's contributions. Professor Phelps offered unflagging
encouragement and immense practical help. Too, the
day-to-day support of colleagues, friends, and family must
be acknowledged. I am indebted to the late Harvey Hammond,
Head Librarian at Childrens Hospital of Los Angeles, for
his continuing belief in my degree work. To Jaime
Hernandez Pliego, whose sacrifices for this project were
great, I offer the highest gratitude. Finally, I thank my
parents, Leroy and Donna Adams, for teaching me that the
pursuit of learning is the highest ideal.
ABSTRACT
The rhetorical situation of anonymous mass assessment
of student writing ability is discussed, with reference tJ
the practices of the Educational Testing Service and the
National Assessment of Educational Progress. Their
practices are summarized and their implicit rhetorical
"framework” is inferred from their practices. It is
claimed that these assessments are responsible for fully
accounting for authorial ethos, and that this is not
properly done. To. correct this, a contrasting rhetorical
framework is advanced against which assessment practice is
reinterpreted. This "modern-classicist" rhetorical
framework is within a "paradigm," or world view, called
"contextualism," which reveals heretofore unseen
inconsistencies and inadequacies in the practices of
anonymous assessment. "Ethos," "trust," "value," and
"choice" are used as "interpretive probes" for
reinterpreting assessment practices. It is argued that the
rhetoric of anonymous assessment attempts to replace trust
in discourse processes with skepticism and detachment,
value with value-lessness, and choice with determinism. It
is concluded that anonymous assessment treats the holistic,
complex event of discourse as an atomistic, ana finally
trivial, non-event. More broadly, the critique acts as an
exploration of the ways in which the assessors try to
depersonalize the situation--and assume incorrectly that
they have actually done so. As a "corrective" or
"restorative" to those notions, the responsibilities and
possibilities of a contextualist assessment of student
writing ability are explored.
CONTENTS
ACKNOWLEDGMENTS i i
ABSTRACT.....................................................iii
Chapter page
I. INTRODUCTION .......................................... 1
II. A MODERN-CLASSICIST CONTEXTUALIST RHETORICAL
FRAMEWORK: COHERING CONCERNS .............. 19
E t h o s ...............................................37
T r u s t ........... ................................ 51
V a l u e ...............................................61
Choice...............................................73
III. THE PRACTICES OF ANONYMOUS MASS ASSESSMENT . . . 82
The Assessing Institutions ..................... 89
Educational Testing Service ................ 89
National Assessment of Educational
Progress........................... . 108
Freshman Writing Program (University of
Southern California)............. . 121
Psychometric Assumptions of Writing
Assessment..................................126
IV. ANONYMOUS MASS ASSESSMENT REINTERPRETED FROM
THE PERSPECTIVE OF THE MODERN-CLASSICIST
CONTEXTUALIST RHETORICAL FRAMEWORK .... 146
Ethos and the Writer............................. 151
Ethos and the Text ...................... 165
Ethos and the Reader............................. 178
Conclusion.........................................190
v
Chapter I
INTRODUCTION
Since early in this century, the assessment of writing
ability has become an increasingly pervasive practice,
involving the gathering and subsequent mass, numerical
scoring of anonymous student essays. Such assessments are
performed by the Educational Testing Service (ETS) and the
National Assessment of Educational Progress (NAEP). Their
juanrornia wijli stana as an exemplar here: their Freshman
Writing Program [FWP], like many others, duplicates the
methods of the much-rlarger institutions.) In total, these
assessments involve hundreds of thousands of students
Lnnually, are costly and time-consuming, and are highly
Jconsequential. Based on the results, students are admitted
to college or denied admission and assigned course credit
and college grades. Moreover, claims about student writing
ability in general are based on these brief, anonymous
Istudent essays.
Briefly, what happens is that hundreds (or thousands)
of students write a short essay. Then the testers train
ethods are imitated widely (The University of Southern
1
readers by having them read and discuss sample papers.
This process indoctrinates the readers as to what kind of
paper should receive a score from a low of "1" to a high of
"4." This consensus can be maintained such that then any
reader, in isolation, will score a given paper virtually
the same as every other reader. This system is quick, and
thousands of anonymous essays are scored in a matter of
hours.
On the surface, it would appear that these writing
assessments are well-grounded and well-documented. The
test-makers employ a great deal of statistical and
psychometric methodology imported from the social sciences.
Even a brief look at basic statistics and measurement
textbooks indicates that the concepts and techniques of
social science measurement have been transferred quite
intact to the assessment of writing. (For example, see
Payne or Downie.) The ETS, NAEP, and FWP official
documents describe how the essay questions are chosen, how
the exams are administered, how the essays are scored, and
what the statistical findings of the assessments are. (For
ETS, see Kirrie and Conlan; for NAEP, see NAEP-authored
documents; for FWP, see FWP-authored documents.)
There is much additional commentary on these
assessments that lies on the borderline between official
2
institutional documents and disinterested professional
literature. That is, consultants for ETS and NAEP have
written widely giving descriptions of, and rationales for,
the efficacy of assessing student writing ability with
their methods. (For ETS, see Bishop; Diederich; Diederich,
French, and Carlton; Godshalk; and Anastasi. For NAEP, see
Lloyd-Jones.)
Even better evidence for well-groundedness of these
assessments should be found in the work of the composition
theorists who have explained the relationship between
assessment practices and rhetorical (discourse) theory.
Charles Cooper, Lee Odell, and Richard Lloyd-Jones are the
most prominent; the theoretical backdrop for writing
assessment has also been developed by Rexford Brown, E.D.
Hirsch, Jr., Alan C. Purves, and Stephen P. Witte and
Lester Faigley.
These composition theorists think of themselves and
are judged by the profession as having provided the
rhetorical basis for the testing. But I will argue that
both groups, the test-makers and the composition theorists,
are innocent of many of the implications of their
practices. On the one hand, the test-makers (arguing the
efficacy of such assessments) ignore the uniqueness of the
processes of writing and reading texts: they act as if
3
testing writing is like testing anything else (but, as a
natter of fact, it's not). In my opinion the assessments
are an attempt to lay a "scientific" super-structure over a
Lomplex of events that is highly resistant to scientific
quantification. The result is that even when such
(assessments are made to yield .high reliability and
predictive validity they may still lack reasonable
construct validity, this because they are not based on a
jcoherent, explicit, and justifiable set of assumptions
about the nature and purposes of written discourse. Such a
paradox is entirely possible when even a very sophisticated
statistical procedure is applied to any phenomenon without
a correspondingly sophisticated understanding of the real
nature of the phenomenon.
On the other hand, the theorizing of the
compositionists is fundamentally limited because it fails
to account for much that is important in the writing and
reading of texts. Their explanations do not have the
status of theory because their account of what happens
rhetorically in the assessment situation is vague and
incomplete. Perhaps they have been distracted from their
theoretical task by the resistance of the testing
institutions to dealing with texts at all. It was not so
long ago that multiple-choice tests were used to assess
4
student writing ability. Having just successfully argued
that a student must be assessed through an actual, full
text, these composition theorists may feel their task is
finished. I think they neglect the new, problematic task
before them: it may be harder, not easier, to fairly
assess a student's writing ability on the basis of a full
text than by isolated multiple-choice questions. They
often sound more hopeful than substantive in their efforts
to bring theoretical respectability to writing assessment.
Odell says that "procedures for evaluating writing must be
consistent with our best understanding of writing and the
teaching of writing" (113), and Cooper and Odell claim that
their scoring methods are "based on a carefully worked out
theory of discourse which can provide a valid and reliable
picture of students' writing" (32). But if they intend to
provide a philosophically congruent explanation of the
assessments, they must proceed philosophically, and they
don't. Assumptions about the epistemological status of
texts are left unexamined, as are assumptions about the
psychological and valuational dimensions of writing and
reading texts. Sometimes debts to discourse theorists
(especially to James Kinneavy and James Britton,
occasionally to James Moffett) are acknowledged, but
ill-served in a treatment that is sketchy, superficial, and
5
self-contradictory. The apologists for the assessments
claim the tests are founded on "current discourse
theory"— collapsing a controversial and vast field into a
simplistic uniformity it does not possess. From their
account of "current discourse theory" they extract little
systematic understanding with which to inform their testing
practices, too often offering only comfortable maxims like
"different writing tasks call for different approaches by
the writer." Crucial theoretical questions are thus either
trivialized or simply dropped as imponderables. They do
not treat what they call their philosophy as a philosophy.
This lapse is evident both in their perspectives on the
reading/writing process and in their perspectives on the
evaluation of those processes. I will argue that these
assessments are productive of little more than a set of
scores for student essays. The theorists have told us very
little about where .the numbers really come from or what
they really mean and. thus have offered little knowledge
that matters regarding the assessment of student writing.
I think we must newly recognize that these assessments
take place within a special rhetorical situation, one that
is important to understand— for several reasons:
1) because it is a rhetorical situation that is little
understood, yet has substantial social and educational'
consequences;
6
2) because test-makers and compositionists lacking a
thorough understanding of this rhetorical situation are
left to draw conclusions about student writing ability that
may be unfounded;
3) because with a better understanding of this rhetorical
situation we may be able to use such assessments more
wisely.
I feel that new light can best be shed on these
assessments by looking at the writer of these texts in a
new way, in a larger framework. After all, such
assessments are intended to assess a writer. The very
presence of a text, a reader, and the rest of the
assessment apparatus is just a means to that end.
The assessors claim they find out something important
about the writer— -his "writing ability"— but in my opinion
they do it in a way that pretends they can divorce that
"writing ability" from the person who supposedly has it.
The anonymity of writer for reader and vice versa, and the
"mass," impersonal quality of these assessments, are only
the most obvious clues that this is an unusual rhetorical
situation. It seems to me that if the assessors want to
find out anything important about the person writing, they
must look at the writing as something done b£ a person.
That involves acknowledging that a person wrote the text,
7
and the further complexity of acknowledging that the only
access to that writer’s writing ability is indirect,
through another person, the reader. Therefore, all we know
about the writing ability of the writer is second-hand:
what the readers are able to reconstruct of it from the
text. So what is really at issue— what makes these
activities assessments— is how and how well the readers
read. Reading here means a special kind of reading. The
readers are not reading normally. They do not read the
texts for meaning, but (as instructed) read only to infer
the writing ability of the student behind the text. This
means that the reader's response (as a special kind of
reading, and as our only access to the writer) is the key
to understanding these assessments.
Eventually, of course, the writer's own perspective
must be accounted for, too, as well as the rest of the
discourse scene. My approach does for now limit what we
can know about writers to the reader's "uptake." (For the
moment it is impertinent what the writer is putting into
his text, or who he "really” is, although it is just that
which the assessments claim to do: tell us what the
writer's "real" writing ability is through the "objective"
judgments of the readers.) Still, it is a beginning to
describe what the assessments claim to do, and to critique
that.
8
In rhetorical terms, the reader's judgment of the
student's writing ability is a construction of ethos. More
broadly, the functioning of both writer and reader as
persons, and how their "personal" nature affects their
roles in any discourse transaction, are likewise issues of
ethos. "Ethical" issues (meaning only the adjectival form
of ethos, not moral matters of right and wrong), are thus
the key to clarifying what is vague and incomplete in the
theory and practice of assessment in the very terms most
relevant to, yet most lacking, in their accounts: as
assessments o£ people by people.
Neither the official assessment documentation itself
nor the theoretical justifications for the assessments
account for the ethical issues raised by their practices.
They fail because their comments do not function as a
systematic rhetorical theory. They are inconsistent and
incomplete. An even more serious problem, which will be
evident once I make their largely implicit, or tacit,
framework explicit by inference, is that their rhetorical
theory is (paradoxically) an "a-rhetorical" theory. It is,
oddly, a rhetorical theory that ignores or denies
rhetoricality.
What is needed, therefore, and what I will offer, is a
genuinely rhetorical, systematic, and comprehensive
o
framework within which to describe the assessments. On the
way to accomplishing that, I will be fleshing out their
less comprehensive framework. A rhetorical framework (as
theory or theories used instrumentally) is practical in
that it enables one to describe the nature of, and the
possible relationships among, writers, texts, readers, and
the realities in which their behaviors occur and in which
their artifacts reside. More broadly, such a framework
gives a characterization of discourse, a view of language
that can be used to contrast it with other frameworks.
More broadly still, a framework will express an
epistemology— in this case, a ground of knowledge defining
the decision-making process, entailing judgments about
ethos. Most broadly of all, what this framework will do,
relative to the need here of explaining the rhetoric of the
assessment process, is to offer a view of human nature, a
view of life. This background is necessary because the
activities of writing and reading enact values about human
nature; to clearly illuminate the values of anonymous
assessment I need to deal with their implicit view of human
nature in contrast to the one I will offer. At this very
high level we will see the sources of their approach, which
differs greatly from the approach I will take.
10
The framework chosen must be sensitive to more of the
complexities of writing and reading these student texts
than have so far been addressed. It must be able to
describe the many processes and the multiple perspectives
generated by the participants and objects involved. I will
synthesize such a framework from certain themes in the work
of a number of important "modern-classicist" rhetoricians.
The modern-classicists' influence on modern
composition theory and other language disciplines is
important but diffuse. Including Kenneth Burke, I. A.
Richards, Chaim Perelman, Wayne Booth, and others, the
modern-classicists are united by a common era and
sufficiently cohering common interests to be compatible
here. Their work coheres in 1) its substantial basis in
classical rhetoric and 2) its genuine advancement on
classical thinking in light of twentieth-century
philosophy, psychology, and language theory.
At an even higher, more general level than these
similarities between their theories this group, I will
argue, shares a common "world view" within which their
theories can be discussed. This world view is what Stephen
Pepper called "contextualism," a paradigm that has become
increasingly influential and explicit in social science and
humanistic disciplines within the last decade. As we will
11
see, a contextualist framework has much in common with
Ihetoric, although it is not limited to a dramatistic
Lnderstanding of human behavior. The term contextualism
las other antecedents and usages which will not be
liscussed here— e.g., in New Critical literary theory. As
a humanist and philosopher, Pepper (and his contextualism)
allows dramatism as a subset of the contextualist view.
Contextualism itself is not an analytical tool; rather,
important terms in modern-classicist thought are the
analytical tools.
I will claim that contextualism ia particularly
revealing model within which to discuss these
modern-classicists. That is because contextualism
emphasizes the very elements of the modern-classicists'
theories most relevant to the assessment of student
writing, where event perception and prose comprehension in
a unique ecological setting are featured.
Not all work called "modern-classicist" is equally
useful for my purposes here. (Some, I will claim, is
misnamed "modern-classicist.") Neither is all rhetorical
work that could be called contextualist equally useful for
understanding ethos in writing assessment. I will draw
from this selected group of rhetorical theorists certain
congruences that characterize them as "modern-classicists"
12
specifically and "contextualists" generally, based on the
usefulness of these ideas for later critiquing the
assessment of writing. The ultimate purpose of
establishing a "modern-classicist’ ' contextualist rhetorical
framework is to use it to generate a "counter-description"
of the assessment practices. That is, having set out my
framework, and then theirs, I will "reinterpret" the
practices enabled by their framework in light of the
contextualist model. The contrasts will be between what
the assessment theorists say is happening in the assessment
practices and what I think is really happening, from a
contextualist point of view.
Chapter Two establishes my contextualist framework.
The framework is a set of four terms which are tools for
analysis. In expressing the values of this rhetorical
framework, these terms will serve as interpretive "probes"
for discourse. The terms are ethos, trust, value, and
choice. I chose these terms because they refer to
important recurring themes in modern-classicist thought
that indicate disjunctures ■ between the assessment
practices' treatment of writers, texts, and readers, and
what 1 will claim a modern-classicist interpretation says
about those discourse elements. Ethos is critical to the
framework because, as the projection of the author in the
13
text, it is the focus of our attention. The assessors
themselves claim this importance. Examining ethos will
tell us how the projected presence of the author is
reconstructed by the reader so that it becomes an essay
score. Second, trust in the discourse process is a crucial
attitude. Without trust in discourse (both as a general
subscription to the discourse form and as a practical
strategy when writing and reading) there can be no genuine
communication. An affirmative, trusting posture
distinguishes our modern-classicists from the rhetoric of
anonymous assessment which, I will argue, is skeptical and
detached. Third, there must be a concept of value in
discourse. From the microscopic level of cognitively
processing discourse to high-level attitudes about
language, there must be value for there to be meaning.
This contrasts with scientific views I will later ascribe
to anonymous assessment. I will ask how the assessors can
claim to be detached and value-free when "evaluating"
student texts— clearly a process about values. Fourth and
finally, the actions enabled in my rhetorical framework
have the character of choice. Committed choice describes
how we come to know discourse, and how we make and.receive
meaning. Entailing a humanistic view of "action" over mere
mechanical "motion," this term prompts us to ask how
14
purpose expresses itself through choice. We cannot
conclude anything about writers, texts, and readers if we
can't assume writers and readers are making choices.
Later, I will argue that the rhetoric of anonymous
assessment replaces a model of interpretive choice with one
that is mechanistic and determinate.
Chapter Three summarizes the actual practices of ETS,
tfAEP, and FWP (origins, purposes, methodologies, and
results). Their institutional documents allow, by
inference, a description of the assessors' implicit
rhetorical framework (what they think and say the readers
are doing).
Finally, Chapter Four takes the two competing
frameworks that have been set forth and reinterprets their
practices. I will offer a counter-description to their
description in light of what I feel is a more penetrating
perspective. The kinds of questions that
counter-description will address include the following:
what does the reader of an anonymous text really respond to
when he reads such a text? When he is purposely deprived
of certain kinds of knowledge, what is the nature of the
response? My contextualist framework will thus be used to
account for the reader's reconstruction of the writer's
ethos, and more broadly, to account for ethos as a function
15
of the person in the discourse transaction. I will
conclude that from the perspective of contextualism,
anonymous assessment treats the holistic, complex event of
discourse as an atomistic, and finally trivial, non-event.
I will conclude that anonymous assessment makes the futile
attempt to "de-contextualize" a text. More broadly, my
general critique of the assessment practice and theory
amounts to an exploration of all the ways in which they try
to depersonalize the situation--and assume incorrectly they
have actually done so.
I hope to accomplish several goals by this discussion.
First, I want to bring a needed, broader perspective to
these assessments. Second, I want to offer the
modern-classicist contextualist framework as an example of
rhetorical views that have gained theoretical legitimacy
but are still not part of the practices of writing
assessment. Third, I want to provide the beginning of a
principled basis for changes in writing assessment, as
appropriate. The logical implications of my analysis and
its values do not include suggestions for "improvements" on
their practices. The field is full of such commentary; the
institutions themselves are constantly implementing such
methdological "improvements." But by placing their
practices in a larger, more systematic, more explicit
16
framework we can uncover heretofore unnoticed implications
of these practices. That is to say, the very ways we have
of discussinq such practices are the real objects of my
concern, not the practices themselve, nor opposing,,
"better" ways to run the assessment. The danger in rushing
into "practical" suggestions is twofold: first, one risks
misrepresenting the very real pragmatic reaons why these
assessing institutions proceed as they do. (This is the
sociology of assessment, which is another subject.)
Second, tacking on ill-considered "improvements" to an
existing system may short-circuit the project of coming to
a thorough and genuinely new understanding of the system,
which is what is needed. This point can be seen,
analogously, in Chaim Perelman's claim in The New Rhetoric
regarding how we must proceed in examining rhetorical
theory.
The theory of argumentation which, with the aid
of discourse, aims at securing efficient action
on minds might have been treated as a branch of
psychology. Indeed, if arguments are not
compulsive, if they are not to be necessarily
convincing but only possessed of a certain force,
which may moreover vary with the audience, is it
not by their effect that we can judge of this
force? This would make the study of
argumentation one of the objects of experimental
psychology. . . .
We shall however proceed differently. We
seek, first of all, to characterize the different
argumentative structures, the analysis of which
must precede all experimental tests of their
effectiveness. And, on the other hand, we do not
17
I
think that laboratory methods can determine the
value of argumentation used in the human
sciences, law, and philosophy; and this for the
reason that the methodologies of psycholoqists is
itself an object of controversy and lies within
the scope of our study [emphasis added] (9).
I, like Perelman, will "proceed differently." I take
my task to be an examination of the very ways the assessors
constitute their methods of analysis. It may well be in
the end that we want to assess student writing differently,
but the prerequisite for this is a clearer understanding of
our present state. As a beginning Chapter Two takes up a
modern-classicist contextualist framework for discourse
that will provide the tools for an inquiry into the
practices of anonymous assessment.
18
Chapter II
A MODERN-CLASSICIST CONTEXTUALIST RHETORICAL
FRAMEWORK: COHERING CONCERNS
The final goal, as named in the Introduction, is to
understand the functioning of these mass, anonymous writing
assessments. The question that needs to be answered is
this: what happens rhetorically when the student's essay is
read by a trained reader? But that answer will not emerge
from a random, impressionistic description of the
assessment event such as the assessors themselve provide:
information on what topic the student writes on, what
qualities in the papers the readers are told to look for,
what the rating scale means, what uses the scores are put
to, and so on. Proceeding that way, there would be no way
of knowing if my subsequent analysis of these writing
assessments is consistent or thorough, nor if it is
contributing anything important to the already crowded
discussion of such writing assessments. What is needed is
a systematic and abstract rhetorical framework. A
"rhetorical framework" simply means here a sufficiently
coherent group of qualities, concepts and values regarding
discourse with which to pose questions of a certain kind
19
and to call for sustained probing of the issues grounded in
those questions. As a tool, the framework will allow us to
analyze discourse as having "force" and "effects" in
general, and more specifically address the way a given
value system makes a given text in a given scene
"rhetorical." My choice of this level of analysis is an
implicit claim that its comprehensiveness brings the
treadth necessary for analyzing anonymous assessment. I am
deliberately avoiding as too narrow the approaches of
cognitive science, process theory, and others. The broad
"dramatism" of my approach, I will claim, preempts them.
I will synthesize such a rhetorical framework from
certain cohering, important, modern rhetorical themes. It
is partly a "given" framework and partly "new." That is,
some of the correspondences, between Kenneth Burke, I. A.
Richards, Chaim Perelman, and Wayne' Booth have been noted
Dy others, and even by themselves: their common ground as
classicists and humanists, for example. On the other hand,
[ will develop new "comparabilities" among them: those will
te the terms of my rhetorical framework.
This group of theorists shares not only important
:heoretical and philosophical ground, but also an identity
as "modern-classicists." I must claim, and argue for,
their identity because some other commentators who "revive"
20
classical rhetoric revive it quite differently. Some go
back to classical sources and, in making connections with
modern rhetoric, ignore the influence of Burke, Richards,
and Perelman, or attribute their ideas to others. Some
miss the congruences between ancient and modern rhetoric
altogether. Andrea Lunsford and Lisa Ede argue that
Aristotelian rhetoric is misinterpreted by moderns such as
Brockreide and Fogarty; Yet Lunsford and Ede's
"corrections" (in particular, that Aristotle's rhetoric
encompasses both v"rationalistic" and "emotional" methods)
ignore similar— and more comprehensive on this
point— comments by Burke, Perelman, and others. Maxine
Hairston, trying to establish the modern "paradigm" for
composition studies, credits the revived rhetorical
traditions of audience, context, and intention only to
Corbett and Weaver, and then only in a subordinate role,
where it is central in Burke, Perelman, and Booth. C. H.
Knoblauch and Lil Brannon claim that classical rhetoric is
severely "culture-bound" and so has little application to a
modern composition pedagogy. Oddest of all, Erika
Lindemann simultaneously "revives" and renders neutral
classical rhetoric by making it quaintly impractical: "Of
course, many of us could teach writing without ever having
read Aristotle; knowing what he said won.'t necessarily make'
21
us better teachers. But we shouldn't feel reluctant to
study rhetoric for its own - sake" (33).
Obviously, my point Is otherwise: that 1) Burke,
Richards, Perelman, and Booth centrally (there are others
of some relevance) are a distinctive group of
modern-classicists; and 2) building on a thorough and
sympathetic reading of classical rhetoric (especially
Platonic, Aristotelian, and Ciceronian), they have found
its congruence with modern language disciplines. That I
would have to make such claims shows that the influences,
assumptions, and definitions of the discipline are not
self-evident. Indeed, as Janet Emig says, we are in a
pre-paradigmatic stage in writing research ("The Tacit
Tradition" 11). In a situation where nothing can be taken
for granted, my "reading" of these conceptual congruities
is as much an argument as it is a disinterested summary.
My topical approach to these theories is deliberately
broad and sweeping. This is because I intend to show that
similar (compatible) trends are available in many places in
the work of my group of theorists. The extent to which the
framework will be convincing is the extent to which the
framework does not seem to be a new creation, separate from
any individual theory or theories, but rather seems
"everywhere available."
22
This modern-classicist contextualist rhetorical
framework, then, will provide the tools necessary to later
analyze anonymous assessment. The four terms of the
framework are ethos, trust, value, and choice, and each
will be developed in turn. But it will clarify matters to
first acknowledge that even these high-level abstractions
are themselves a subset of an even more general
philosophical category, or "world view." The most general
cohering quality of my group of theorists is expressed in
their shared world view called "contextualism."
Philosopher Stephen Pepper has identified four such world
views or "relatively adequate hypotheses," one of which, he
says, is compatible with any theory. Contrasted with
"organicism," "mechanism," and "formism," "contextualism"
is one such general model within which theories, concepts,
and more specific models are generated. It is as
"contextualists" that I will advance the views of these
modern-classicists. (At this very general level,
contextualism could as well be used to describe
anything— physics as well as discourse, for example.)
Contextualism captures many aspects of the common world
view of the rhetorical theorists to be discussed, while of
course each theorist differs in specifics. In this way,
contextualism is a cohering force that will help focus the
23
compatabilities among them that I will develop as my
rhetorical framework. I will just briefly sketch the main
features of contextualism as they are relevant to my
modern-classicists and discourse analysis. I will return
to these themes as need be, in light of the rhetorical
framework itself.
Contextualism has as its root-metaphor "the world as
events," but not. Pepper says, "a past event; rather, . . .
the event alive in its present. . . . The real historic
event, the event in its actuality, is when it is going on
now, the dynamic, dramatic active event. . . . it is an act
in and with its setting, an act in its context" (232). We
will find that these modern-classicists feature certain
themes congruent with the contextualist sense of discourse
as "event." Among these are a holistic view of the event;
the relativistic nature of knowledge about the event; an
interactional (inherently participative) nature of the
agents in the event; the event’s dynamic quality; and its
reflexive quality.
As to the first, for these modern-classicists, the
"whole" is the complex of everything relevant to the
language scene. An example of the kind of argument this
holism supports is the indivisibility of one aspect of the
rhetorical situation from another (Perelman says that an
24
argument cannot exist without an audience and vice versa;
Weaver claims that logos and scene are inextricably
intertwined). The claim is that the elements of a
discourse scene are knowable only with the qualification
that, finally, each element is part of a whole. This puts
contextualism in contrast with systems' that divide a whole
(for our purposes the "whole" includes a writer, a text, a
reader, and a surrounding scene) into parts. "For
contextualism, element analysis is intrinsically
distortive," Pepper says (248). The only justification for
such atomism is some purpose we are pursuing— only a chosen
purpose, not inherent, determinate structures to which we
have some "objective" recourse. An event is a unified,
single occurrence. So no event is exactly like any other;
therefore, events are unique. The modern-classicists
feature the novelty of a given situation. People and
events are not replicable, interchangeable elements. Burke
claims that "language reflects the 'personal equations' by
which each person is different from any one else, a unique
combination of experiences and judgments" (Language as
Symbolic Act ion 28). The task, therefore, of discourse
analysis is to establish for each discourse act the very
specific set of values and constraints that can account for
that particular argument (the text), its source (the
25
rhetor), its response (the audience), and the larger
reality in which it occurs (the scene). Schemes that treat
writers, texts, readers, and scenes as generalized,
interchangable items are, of course, suspect from a
contextualist point of view.
Second, given an event that is holistic, the
contextualism within which these modern-classicists work
presupposes an epistemology of relativism. Pepper
emphasizes the relational nature of knowledge and Robert
Hoffman and James Nead say, "The claim of contextualism is
that the interpretation of 'basic units' at any one level
of description will necessarily rely upon contextual
factors at another level" (523). In language, as I. A.
Richards warns, "Stability in a word's meaning is not
something to be assumed, but always something to be
explained"? there are no meanings outside of contexts (11).
Perelman observes, about argumentation and discourse in
general, "One might think that the status of that which is
subject to opinion is impersonal and that opinions are not
relative to the minds which adhere to them. On the
contrary, . . ." (5). Even so-called "facts," Perelman
goes on, are only "agreements" operative in a given
rhetorical situation (67). ‘ It is only by defining those
agreements relative to their contexts that we can tell how
26
meanings "mean." In fact, it is by temporarily
foregrounding one relationship that meanings are explored.
For example, it is possible, even necessary, to look at
ethos in a discourse scene multi-perspectivally: from the
successive points of view of the writer, the text, and the
reader. Thus, an approach to texts that relies on one
"absolute" perspective on ethos is not part of a
sontextualist orientation.
A third aspect of contextualism our modern-classicists
exemplify regarding texts describes the agents in this
(holistic, relativistic scene: they "interact" in an
inherently "participative" way. Contextualism posits a
jcomplex and thorough view of human interaction in which the
agents, their actions, and the scene in which those actions
occur interpenetrate each other. . Social reality is
intersubjectively constituted. We constitute, and are
ohanged by, our experiences. As Pepper says,
When I perceive a table, there is, according to
the contextualist, an interlocking of two or more
continuous textures. . . . If I look away, that
perceptual texture is disintegrated and its
qualities, of course, disappear. If I look back,
it is reintegrated and the qualities emerge once
more. But the important point to notice is that
the qualities arise in the integration of the
texture and belong neither to me alone nor to the
table alone, but to the common texture. In
seeing a table I am interacting with my
environment and am so far out in it (265-66).
27
Our participation is a function of our intentions, and
our conclusions are inherently interpretive, never
"objective" or "neutral." "Rather, interpretation begins
from the postulate that the web of meaning constitutes
human existence to such an extent that it can never be
meaningfully reduced to constitutively prior speech acts,
dyadic relations, or any predefined elements" (Rabinow and
Sullivan 5). Applying this participative quality of
interpretation to language use, philosopher Michael Polanyi
says that we can never ' "eliminate this personal
coefficient" (253). Rhetorician Kenneth Burke believes
that humans, as symbol-using animals, participate in
language acts to the extent that those actions create,
determine, or choose the social hierarchy of which each
individual is a part. Hoffman and Nead give, in effect, a
contextualist account of reading in this summary of their
approach to studying cognition: "Rather than looking inward
at a person's hypothetical memory representations and
processes, the contextualist looks outward at what the
experimental tasks might be doing to the participants and
especially to what strategies they might be adopting"
(525). In such a view, it is important to look at the
experience of the reader (how and what he perceives, in
what ways he is changed by the reading process). For the
28
contextualist, perception goes out and actively selects
from the world. Therefore, a model of human behavior that
assumes that people can interact without being mutually
influenced by those actions is not part of a contextualist
view. A view of reading as mere "information processing,"
in which information comes ^n to the person, is not
contextualist. Such a model is typical of some scientific
work in which it is believed that the "personal
coefficient" (including the scientist as person) can be
accounted for and thus eliminated.
Fourth, in a contextualist view, the activities of the
agents--interacting in a holistic, relativistic scene--must
be described in dynamic terms. That is, no account of the
world as a static entity will ever capture a world which
consists of events, in Pepper's terms "dynamic, dramatic,
active" events (232). Even word meanings, Richards reminds
us, are not static. A word possesses a meaning only as
constant as the context that gives it that meaning (12).
Reading a whole text in the contextualist model likewise
exhibits that quality of dynamism. So there is no chance
of a text simply containing an inert "something" that the
reader simply perceives, as is. Rather, both text and
reader inevitably "enact" qualities that cause the other to
seem as it does. Because of this, rhetorical analysis
29
rests on finding out how the reader "creates" the text and
(perhaps this is confounding to the non-contextualist) how
the text "creates" the reader. So dynamism is critical to
a contextualist view. A model which denies the dynamic
"event" nature of discourse is not contextualist. There
are implications for the assessment of writing in both the
inherently participative quality of discourse and in its
necessarily dynamic aspect. From a contextualist
perspective, the reader must acknowledge that his judgments
of a student's writing ability (as a construction of ethos)
is partly a function of what he takes to the text. He must
also acknowledge that the ethos is an event rather than
some inert quality.
A fifth and final tenet of contextualism our
modern-classicists exemplify is a reflexive dimension.
This entails awareness of whatever system of analysis is
being used: the awareness that allows the system to know
itself. At a very general level, the contextualist
practice of any science requires the researcher's awareness
of himself relative to the experiment. Arguing against the
conventional image of the detached, objective scientist,
Hoffman describes how the researcher must undergo
perceptual learning in order to specify the event structure
of the domain. The point is, observations must include a
30
view of the observer. Bringing this insight to language
specifically, Polanyi notes that no strictly formal
(objective) account of language is complete because it must
include as well our perceptions £f the language use. The
act of reading or writing may itself involve a decision to
correct or modify the view of language that enabled the act
in the first place (95, 162). Of course, such a reflexive
dimension is critical to a dramatistic rhetoric. Kenneth
Burke calls this reflexiveness a necessary aspect of a
modern rhetoric, which must "concern itself with the
thought that, under the heading of appeal to audiences,
would also be included any ideas or images privately
addressed to the individual - self ..." (A Rhetoric of
Motives 38-39). The implications here for the the
assessment of writing in general and ethos in particular
include that a contextualist model allows a reader to
include in his response his awareness of that response. Of
course, that entails a view of human self-awareness that
many scientific models reject. Those models assume only
one point of view (the observer’s) and that a detached one.
A practitioner in that model denies the alternative,
reflexive point of view that ties him to the event.
These are five general tenets of any contextualist
model, and they are evident in any theoretical work that is-
31
contextualist. A number of composition and discourse
theorists are contextualist; some arrive at contextualism
through the influence of the modern-classicists, others
from strictly psychological sources, still others from a
combination of the two. Louise Wetherbee Phelps has
identified and described a contextualist viewpoint for
composition in her "The Dance of Discourse: A Dynamic,
Relativistic View of Structure." W. Ross Winterowd and
Donald H. Graves, without using the term, are contextualist
discourse theorists. Janet Emig and Ann E. Berthoff use
some of the contextualist terminology, and a number of
cognitive psychologists, anthropologists, and ethnographers
studying language are contextualists (for example, William
Labov, Michael Cole, Sylvia Scribner, and Clifford Geertz).
Using contextualist terminology here will make it easier to
develop the set of cohering, compatible features of the
modern-classicist, contextualist rhetorical framework.
Contextualism is concerned with a holistic, relativistic,
interactional, dynamic, and reflexive approach, and those
qualities distinguish those rhetorical theories that are
thoroughly contextual from those that are inconsistently
so. Likewise, remembering that the modern-classic ism I
will now develop relies on a definable sense of ethos,
trust, value, and choice will keep those rhetorical
32
theories that are thoroughly modern-classicist distinct
from those that are not. These distinctions will mark a
rhetorical framework that is highly directed and contains
practical tools for analyzing assessment.
The framework has, following the theories from which
it is synthesized, three notable strengths for later
analyzing assessment. First, it is strongly descriptive,
as opposed to prescriptive. This descriptiveness is
revived from many aspects of classical rhetoric. It is
especially notable since by the twentieth century most of
the rest of modern language philosophy had abandoned
rhetoric (and with it, of course, ethics), leaving rhetoric
as a body of decaying prescriptions. What developed as
twentieth-century philosophical ethics concentrates on
moralistic prescriptions that cannot account for ethos
descriptively (Wheelwright). Burke, Perelman, Richards,
Booth, and Weaver all remark on the "Cartesian split" which
culminated in modern positivism's removing rhetoric from
serious consideration as a theoretical explanation for
decision-making. The work of these modern-classicists is,
cumulatively, a thorough revival of descriptive rhetorical
theory. This descriptive quality brings a broader, less
judgmental view useful here for describing the assessment
practices, both because the assessments have been described
33
by critics in too-judgmental terms and because the
assessors themselves cast their own practices in
overly-prescriptive terms.
A second quality that marks this framework’s
theoretical sources and qualifies it for analyzing
assessment is its position that language has a necessarily
social dimension. For these theorists, language is neither
arbitrary nor objectively "there" (for Perelman, both
"nominalism" and "realism" are equally false) (513).
Instead, language is the very way we consititute ourselves,
others, and our realities. To Burke, rhetoric is the "use
of words by human agents to form attitudes or to induce
actions in other human agents" (A Rhetoric of Motives 41).
Our modern-classicists exemplify this newly social
epistemology of language in a range of ways: Burke,
dramatistically; Perelman, juridically and argumentatively;
and Booth, ethically.
Third, and finally, our modern-classicists are
distinguished by a newly "performative" view of rhetoric
and rhetorical theory. Performative means "dramatistic,"
in various ways, though Burke is particularly associated
with the term "dramatism." This performative quality
entails their claim that rhetoric is practical--in two
senses. One, they account for how people actually interact
34
through words to create changes, in attitudes and actions
(theory as description). Two, their accounts act as
information on how to make those effects occur (theory as
''textbook"- advice). As to the first, that sense of
"practical" distinguishes them from contemporary
deconstructionist rhetorics. Jacques Derrida and others
argue that texts and language are based on metaphors that
conceal their real natures. People are therefore- .kept
separate by and through language. But Burke says that,
given the generic divisiveness among people, rhetoric
describes how we come together: by "the use of language as
a symbolic means of inducing cooperation in beings that by
nature respond to symbols" (A Rhetoric of Motives 43).
Here, Burke at once acknowledges the original "break"
between a unified reality and the language which must
ignore that unity in order to attempt to induce new
harmony; and casts rhetoric in its practical role of
getting done the business of the world. The
deconstructionists claim that their network of "signifieds"
and "signifiers" demonstrate the idiosyncratic nature of
the communicative act, and they claim that explains why
communication cannot take place as our sense impressions
would lead us to believe it does. Our modern-classicists
dispute this. Richards' "context theorem of meaning," for
35
example, describes how a word, through successive contexts,
takes over , the duties of parts which can then be omitted
from the recurrence. This creates an abridgement in which
the word comes to mean "the missing parts of its context"
(34). For Richards, the tracking down Of successive
contexts is the basic research in his positive science of
knowledge. Each context traced is part of what we can
reliably take meaning to be. The modern-classicist
contextualists, therefore, proceed hopefully, and assume
that their efforts have immediate, practical results, while
some other contemporary rhetorics do not. That hopeful
posture and belief in practical results, common-sensically
arrived at, makes this rhetoric especially appropriate for
anlalyzing assessment, since those values echo the
assessors' own.
My framework, then, rests on the basis of the
distinctiveness and usefulness of the theories from which
it is derived: it is a descriptive, social, and
performative (dramatistic) rhetorical framwework.
36
2.1 ETHOS
Ethos is the first of the four elements in this
rhetorical framework, and its conceptual center. Ethos
will signify a set of relationships among the elements of
the communication triangle (writer-text-reader) which I
will use to explain actions enabled in a rhetorical
framework. As a cognitive experience while reading a text,
ethos is simply the author’s projected presence or
character, relevant to the aims and effects of the
discourse. But more broadly, ethos also indicates the
"personal” nature of the participants in discourse, and how
that selfhood affects their roles in any discourse
transaction. In this way ethos will be a term denoting not
just a logical dimension of rhetoric but an interpersonal
dimension as well. It is not enough to see ethos— or any
other rhetorical term for that matter--as merely an element
in a structure of ideas, separate from the people who have
those ideas. Rather we can only appraise ethos from a
perspective that allows that actual people--complex,
interacting, changing people— constitute that ethos. So
the choice of ethos, and that of the other rhetorical
elements, is not just a technical choice (i.e., what one
can do with the framework), but a choice of values as well
(i.e., what I believe a framework should be).
37
Classically, ethos is one-third of the
ethos-logos-pathos triad. The three form the artificial
proofs--those aspects of a discourse open to human
intervention and judgment. The modern-classicists develop
the congruences between ethos and modern philosophy,
psychology, and language disciplines. Ethos is essential
to understanding the character of the writer projected in a
text (Burke, Language as Symbolic Action 28). Perelman
believes that we cannot know a writer's intentions
directly. Readers can only take what they know about the
writer's permanent character (ethos in a broad sense), and
derive his probable intentions from that. That is how we
reconstruct the author as ethos.
The modern-classicists bring a new relativism to
classical ethos. Booth cautions that we can no longer
expect to find clear distinctions among the three kinds of
classical proof. He says, "Emotional and ethical proof
will often turn out to be 'substantive,' and logical proofs
will often turn out to be useless a,nd misleading" (145).
This is to say, we must locate ethos i_n relation to the
other discourse elements. We cannot isolate it, but must
instead seek it in a well-rounded way. (Burke's Pentad,
consisting of Act, Agent, Scene, Agency, and Purpose, is
certainly the finest example of the well-rounded,
38
multi-perspectival approach to ethos— or any other
rhetorical element.)
So we can analyze any rhetorical element in relation
to any other. But some relationships will be more
revealing that others. Here, obviously, the important
relationships are those of ethos and writer (a reflexive
relationship), ethos and text (how the text mediates
between the participants), and most important, ethos and
reader (the focus of mass, anonymous writing assessment).
That three-part approach to ethos is in fact the
organizational structure of my Chapter Four. There, I take
up those three relationships in turn to analyze in a
rounded way the assessment practices in light of this
rhetorical framework.
Ethos denotes particular ways of knowing or judging
the discourse. Depending on what the reader wants to know
about the writer (is the writer credible? does he write
correct Standard English?) the reader makes one kind of
conclusion over another. Perelman says that we construct
an image of the writer and by this image classify him into
categories to which certain qualities can be applied. The
writer's actions result in the reader's constant
"modification of our conception of the person, to whom we
shall explicitly or implicitly attribute certain new
39
tendencies, aptitudes, instincts, or feelings" (297).
Ethos, then, is critical to knowing discourse and
indicative of a necessarily "personal" interaction. Ethos
is part of a unified, balanced ethos-logos-pathos triad,
each element of which must be viewed in. terms of the
others.
The modern-classicists' "featuring" of ethos is not
casual. Their stance is an attempt to correct misguided
ideas about language and rhetoric that have held sway for
three hundred years. The scientific and philosophical
tradition they protest against had its beginning with the
1637 publication of Ren^ Descartes’ The Discourse on the
Method. Out of his delight with the neat certitudes of
mathematics, Descartes came to oppose the art of
argumentation fundamental to classical rhetoric. The only
acceptable communication model was one that adhered to the
principles of geometry requiring demonstration based on
clear definitions, axioms, and cause to effect relations.
In such a system there is no place for the rhetorical
syllogism, commonplaces which substitute verisimilitude for
reality, or emotional appeals. Since rhetoric cannot
produce "truth," so the argument became, it should be
relegated to communicating principles that logic and
experimentation can discover. This "Cartesian split"
40
separates "scientific" facts from "mere" opinions held by
inevitably subjective, interested humans. The idea gained
credence that the only scientifically pure knowledge (and
thus the only useful kind) was that not tainted by human
intervention. Scientific triumphs, especially in the
nineteenth century, coupled with corresponding movements in
literature and esthetics, tended to strengthen this view.
On the one hand, the Darwinian view pictured man as merely
the latest link in an evolutionary chain: he was not
qualitatively different from the animals, so observations
proceeded unfettered by any need to account for an inner
life of man. Further, one of the conclusions of Freudian
thinking was that man, driven by his neuroses (societal and
personal), was not dependably rational in his language use,
anyway. On the other hand, nineteenth-century Romanticism,
importantly including Coleridge's ideas of imagination,
reinforced the idea of ' a language act as idiosyncratic,
innate (as a skill), and ultimately unanalyzable. By the
early twentieth century it was taken for granted that there
were two kinds of knowledge: "real" facts (that somehow
existed by themselves "out there") and "mere" opinions
(held by subjective, inconsistent, unreliable humans).
Perelman names the opposition "positivist demonstration"
vs. "adherence"; Booth names it "fact" vs. "value"; Burke-
41
calls it "opinion in the 'scenic' order of truth" vs.
"opinion in the moral order of action" (A Rhetoric of
Motives 55). In the most rigid formulation of the
scientific method, logical empiricism, only those
descriptions of the world that can be made totally precise,
detached, and objective are considered valid statements.
This is what the modern-classicists dispute. To them,
discourse is necessarily personal and inevitably entails
issues that are ethical (pertaining to ethos). Anyway,
Booth argues, logical positivism applies standards of truth
so strict that it can't even take up most of what goes on
in everyday life. How, Perelman asks, can we account for
the huge area of human interaction wherein we convince
others of something or are convinced of something by
others, if not with a rhetoric that is personal? Perelman
claims that the most characteristic part of argumentation,
as opposed to demonstration, is that the hearer will
identify the speaker with his speech. A speaker's
assertions are not, Perelman goes on, part of a rigid,
formal system of demonstration that can be viewed outside
the person making the assertions. So "the person of the
speaker provides a context for the speech, [and] conversely
the speech determines the opinion one will form of the
person" (319).
42
What bothers the positivists most about this line of
thinking is that it seems we can never be objective: we
are always "implicated" in our discourses by our inevitable
"interests." To this, Perelman answers that impartiality
is not the same as objectivity: "being impartial is not
being objective, it consists of belonging to the same group
as those one is judging, without having previously decided
in favor of any one of them" (59-60). To be thoroughly
"objective" is to be so removed from the discourse scene as
to be uninterested in the occurrence. This confers no
ability on the person to judge better. Anyone close enough
to the discourse scene to have an "interest" can at most be
impartial, never objective.
The modern-classicists' reviving this discredited
territory of the personal is, of course, an attempt to
restore balance to the ethos-logos-pathos triad against a
too-logos-based rhetoric. We cannot, they argue, remove
ideas from the person who advances them. A behaviorist
approach, which seeks to study people as mere things, is of
course the central target of the modern-classicists. Burke
does allow that it is an unresolvable metaphysical debate
whether, as humans, we perform self-conscious "actions" or
are merely objects in "motion." The important point is
that what we do is treat each other as persons: "illusion
43
or not, the human race cannot possibly get along with
itself on the basis of any other intuition. . . . Even the
behaviorist, who studies man in terms of his laboratory
experiments, must treat his colleagues as persons"
(Language as Symbolic Action 53).
The modern-classicists argue for the restoration of
ethos in both its senses. First, ethos must be reinstated
as a form of proof. Our knowledge of a discourse is
incomplete without it. Second, texts cannot be made
impersonal. Ethos is an indicator of the personal
dimension of the participants in a discourse. "
The modern-classicists’ corrective regarding ethos is
not just directed at those conventionally . thought of as
"scientific." Their views contrast just as sharply with
those in composition studies who do not realize that the
prevailing school rhetoric itself is under the influence of
the Cartesian split. Richard Young and James Berlin have
separately advanced this idea. "Current-traditional”
rhetoric is the model that students, teachers, and (most)
scholars operate within, even now. This view forms an
epistemolpgy, largely tacit, of how we know writers, texts,
readers, and reality. Meaning exists independent of the
perceiving mind, reposing in external reality. The main
purpose of texts is to refer to a distinct, stable world.
44
The writer's task, therefore, is merely to arrange the
verbal form of meanings that exist objectively in the
external world. Meanwhile, the audience for a text is
conceived, in this rhetoric, to be observing the text
rationally and objectively in a nonproblematic world. The
modern-classicists would argue differently, of course. The
"current-traditional'’ rhetoric forecloses on the suasory
quality of discourse. Texts that are assumed to merely
"refer" to reality cannot partake of argumentation, which
seeks to change that reality. As Richard Weaver expresses
the distinction, "the listener is being asked not simply to
follow a valid reasoning form but to respond to some
presentation of reality" (205). In a world that is
objectively "there," disagreements are really only
differences about matters of information. "In an
epistemology in which meaning inheres in external reality
and not in a transaction between the observer and reality,
evaluation and judgment become meaningless or irrelevant
concepts" (Berlin 1). Further, the modern-classicists
argue that we must see the agents in discourse as
responding to or expressing a personal or social need,
problem, or goal. To look at a text as only an artifact,
divorced from the genuine, human impulses that are its
source, is to misrepresent a holistic event as an inert
45
object. Of more immediate import, ethos as the projected
author in a text becomes, in the "current-traditional"
rhetoric, a non-issue. In this rhetoric texts are supposed
to be transparent. A successful text has no author
"intruding" on the logos. Obviously, our
modern-classicists claim the impossibility of such a
rhetoric.
Their "corrective" rhetoric speaks not only to
assessors of student writing, but to many other
communications disciplines where the nature and identity of
the author is at issue. The framework I have been
developing becomes more persuasive yet when we note that,
for example, speech-act theory, reading theory, and
literary criticism are concerned, in their ways, with ethos
in texts. Ethos (under this or other names) is used as an
analytical probe in speech-act theory. From a
contextualist perspective, what a speech-act analysis hopes
to do is understand meaning inclusive of both its physical
features (a written text or an acoustic event) and its
accompanying assumptions or rules. Speech-act theory
proceeds from the view that language is a product of human
intentions, which is compatible with my definition of ethos
as an indicator of the necessarily human dimension of
discourse. John R. Searle, in Speech Acts notes, "In the
46
performance of an illocutionary act in the literal
utterance of a sentence, the speaker intends to produce a
certain effect by means of getting the hearer to recognize
his intention to produce that effect; . . ." (45). Ethos
(as the person projected in the discourse) is an
interpersonal event, a creation by hearer of speaker, and
vice versa. W. Ross Winterowd says he would amend
speech-act theorist Searle's definition in the following
way; "view the illocutionary act as the projection of the
perlocution and the perlocutionary act as the
reconstruction of the illocution" ("Dear Peter Elbow" 96).
Reading theory, too, at least the recent work, is
beginning to use ethos as an analytical probe for its
purposes. Reading theory is abandoning an early formalism
that tried to deal, -like the positivist rhetorics discussed
earlier, with just the words themselves. Typical
experiments in reading tested comprehension of, or memory
for, sentences in isolation. This formalism, including the
formalism of transformational linguistics, took meaning to
be a paraphrase or summary of the text. It is clear now
that the ability to summarize or abstract presupposes an
understanding of the text, rather than being identified
with it (Spiro). It is a context that enables
interpretation, not a text alone (Morgan and Sellner 197).
47
Replacing the "formalist" position is the "intentionalist"
position. Rejected is the attractive simplicity of
studying sentences in isolation in favor of the ecological
validity of studying language in its linguistic and
extra-linguistic context. "The study of text must take
place in a context of a writer who is attempting to convey
his or her intentions to a reader and of an active^reader
who is. attempting to form a representation of the
intentions of the author" (Spiro 84). So the perspectives
of writers and readers are being recognized as critical by
reading theorists.
Only a very rich sense of ethos could account for the
immense complexity of a single reading act. Readers do not
mechanically process single words or isolated meanings.
The act of reading is multi-leveled, interactive, and
hypothesis-based (Spiro). As to the first, knowledge
structures at several different levels are actively used in
the reading process. Traditionally-proposed levels.include
orthographic, phonological, lexical, syntactic, and
semantic. But additionally, higher level knowledge sources
such as inference rules and expectations about discourse
structure are also crucial components. Second, the
relationships between the levels must be seen ' in an
interactive way: "each knowledge source can contribute
48
input at various points in the complex process of
comprehending text . . ." (Spiro 7-8). Third, the
hypothesis-based nature of reading entails constraints on
what kinds of evidence will confirm or deny the hypothesis.
This acknowledges the "interest" on the part of the reader
that controls what is an allowable interpretation. So we
need a sense of ethos that is the reader's reconstruction
of the writer to describe the way the reader's "interests"
interact with the writer's intentions.
Finally, literary criticism is much concerned with
ethos, too. As a discipline of interpretation and
evaluation, criticism obviously depends on judging authors
through their texts. Modern criticism has focused
particularly on what the reader's reconstruction of that
author says about the reader himself. A whole tradition of
literary criticism, before and after the New Criticism, has
concerned itself with how the writer constructs his
audience, and conversely with how the reader constructs the
author. These theories look at the epistemological act of
the reader coming to know (the text, the author, and
himself). William Empsom, Roman Ingarden, Wolfgang Iser,
and Louise Rosenblatt are prominent here. Since the waning
of the New Criticism, many schools of thought have arisen
that treat, in literary texts, questions treated in my
49
rhetorical framework under the heading of ethos. While few
neat boundaries exist between them, these literary scholars
are often grouped as reader-response critics (Stanley Fish,
David Bleich, Norman Holland); adherents to the Yale school
(Harold Bloom, Paul de Man, Geoffrey Hartman, J. Hillis
Miller); and those examining "textual wholeness"-(Booth).
In any case, the issues are often these of ethos: the
"ideal" reader, the "implied" author, thez "projected"
audience, and so on. The necessarily interpersonal nature
of the reading (and writing) of literary texts, and more
specifically the way a self is projected in a text— and
reconstructed by a reader--are prominent questions in
literary criticism, and are questions of ethos.
Rhetoric relies on ethos. Ethos cannot be isolated or
removed from the ideas, arguments, or words of which it is
a part. Ethos is inevitably interrelated with the
processes by which we understand, or are "convinced," by
texts (we make, and continually revise, our "model" of the
author as we read). To the extent that we trivialize
ethical events in discourse, or try to analyze texts as if
ethos can be excised from our consideration, we
•misrepresent discourse. This element of the rhetorical
framework forces certain questions about reading and texts
that will be relevant later for the critique of writing.
50
assessment. For example, what happens when we take a text
for the writer? What is the effect of removing a text from
contextual information associated with it
("de-contextualizing" a text)? What is the effect of
assuming that reading can be neutral and disinterested? I
will take up these and other questions in Chapter Four.
2.2 TRUST
This rhetorical framework uses the term ethos to
denote the presence of, and the functions of, the people in
discourse. But those people don’t proceed without any
investment in, or attitudes toward, discourse as a whole.
Whatever the reasons people have for writing, they cannot
write at all without some attitude toward discourse. The
modern-classicists choose an attitude of trust in
discourse, both as an abstract faith in the discourse form
and as a practical strategy for dealing with texts. First,
they affirm that rhetoric as a form is inevitable, worthy,
and productive; second, that a given rhetorical transaction
is at least potentially valuable for the meanings and
knowledge it yields. (This position is a value and it is a
choice; these two concepts will be developed in turn as the
third and fourth elements of this rhetorical framework.)
An affirmation of trust denotes the qualities of both
51
yielding and of hopefulness. We cannot use language and
remain detached from it. Our participation in language is
a genuine interaction: we constitute our texts and. are
constituted by them. By this, we admit that selves and
language yield to one another; they combine, in an alliance.
Second, this rhetoric is hopeful, as opposed to certain.
It deals in just those conclusions that cannot be
demonstrably known, but must be arrived at by nonformal
means. This rhetoric is also hopeful as opposed to
skeptical. Our modern-classicists disdain the cynical
manipulation of language for mere effect in favor of a
genuine involvement in the possibilities of discourse to
produce real change. The source of this trust is neither
blind faith nor pure, analytical reasoning, but a new
synthesis from many ways of "knowing." Finally, all these
dimensions of trust rest on a positive view of human
nature. Rhetors, in this view, must show good will. They
must want to use the interaction to understand each other.
They must read (or listen) with conditional assent.
This rhetoric contrasts with rhetorics of "mistrust."
There, language is considered imprecise and avoided or
devalued as evidence. After all, the argument runs, if
texts and people are completely distinct from each other
and are, further, completely distinct from the reality to
52
which these texts and people refer, it is not the
interactions among these entities that are constitutive of
meanings and realities. If they are not so constitutive,
why trust the use of language as a vehicle for
understanding? When these views take on a "scientific"
cast, there is often a condition raised that the testimony
of (subjective/biased) language be corroborated by
"objective" facts. In this view, discourse is potentially
as misleading as it is enlightening. Of course, this is a
skeptical view, and is tied, finally, to a neutral or
negative view of human nature.
The modern-classicists affirm discourse as inevitable.
For Burke, the basis of rhetoric is inherent in the world
as we find it. Rhetoric lies in the nature-of things, a
"generic divisiveness which, being common to all men, .is a
universal fact about them. . . . Out of this emerge the
motives for linguistic persuasion" (A Rhetoric of Motives
1-46). This results in a "universal rhetorical situation."
Rhetoric is "rooted in an essential function of language, a
function that is wholly realistic, and is continually born
anew; the use of language as a symbolic means of inducing
cooperation in beings that by nature respond to symbols” (A
Rhetoric of Motives 43). So discourse grows out of the
very conditions of life, and is a basic function of
53
language. The classical roots of the position that
rhetoric is inevitable and pragmatic are in the distinction
between rhetoric and dialectic. Rhetoric is, for the
ancients, the real-world, language-based counterpart to the
abstractions of dialectic thought.
Just as discourse is inevitable and pragmatic, it is
also productive. We come to decisions,‘ form new attitudes,
or take actions as a result of rhetorical interaction.
Richards' "context theorem of meaning" rests on such an
affirmative position about language. The theorem calls for
tracing meanings back through their earlier uses
(linguistic contexts). Our knowledge of language, the
theorem argues, is contained in language use. So each
jsuccessive context traced yields aspects of what we can
reliably take meaning to be.
This contrasts with "mistrusting" rhetorics, for
example deconstructionist rhetorics. Derrida and others
Lrgue that texts are founded on convenient illusions
(metaphors) that make it difficult if not impossible to
make conclusions about texts. In their complex analysis,
writing exists only through what it excludes.
Deconstructionism is committed to demonstrating that
communication cannot take place as our common sense leads
as to believe it. does. But Burke preempts
54
deconstructionism here by focusing on the practical side of
rhetoric. He realizes that even if we are under that
(arguable) illusion, we (unarguably) do influence others
and we do come to decisions and attitudes. Burke
simultaneously embraces a critical attitude of the means by
which trust is sometimes gained in discourse and a
commitment to trust in discourse to expand our knowledge.
While deconstructionist contexts are ultimately (or at
least for a very long time) confounding, contexts for the
modern-classicists are not. Burke indicates that at the
outset of discourse there may be two contexts: the one the
writer is in .and the one the reader is in. But an
underlying faith in discourse tomake these two contexts
one by creating a new, shared context distinguishes the
modern-classicist position. In that single, shared context
we come to stable, "for now," decisions of what we know.
The element of trust in this rhetorical framework is
especially relevant to the readers of student texts. A
trusting rhetoric will shed more light on these assessments
than a mistrusting one, because of the expectations typical
of most readers. Most readers believe texts communicate,
and the testing procedures obviously exhibit the assumption
that texts communicate. It is unlikely that mistrusting
rhetorics, where skepticism is central and authorial
55
---------i_ ------------------------------------------------------------------------------; --------------------------------------------------------------- ------------
intention is unknowable in a common-sense way, would reveal
much about such readers.
Mistrusting rhetorics are a kind of sophistry, in this
view. In sophistic rhetorics a genuine involvement in
discourse is depreciated. Rather than starting with a
commitment to rhetoric as a means to more completely
understand our fellow rhetors, ourselves, and our world,
sophistry looks only at the superficial appearance and
effects of discourse, ignoring the necessarily human,
subjective, and genuine impulses which give it purpose in
the first place.
It is for lacking trust in discourse that Plato takes
the sophists so sternly to task in the Gorqias. There
Gorgias claims that "the main point about oratory [is] that
oratory is productive of conviction, and that this is the
be-all and end-all of its whole activity . . . producing
conviction in the souls of its hearers" (28). Gorgias
contends that a good speaker is one who makes a good case
(gains the conviction of his audience) no matter if he
tells the truth or not. Gorgias does not trust discourse
to lead to anything beyond the immediate adherence of his
audience. Plato, of course, disputes this. He says,
"reality rather than the [mere] appearance of goodness" is
the true goal of rhetoric. Self-knowledge and mutual
56
understanding of issues is the point, not verbal trickery.
To this end, we can and must trust discourse as a form, and
attend to discourse with an attitude of at least
conditional acceptance.
Modern rhetorics of mistrust, for example some
"scientific" approaches, seek objectivity by eliminating
the reliance on individual, "personal" people.
Unfortunately, the result is to invalidate all statements
"tainted" by human, subjective intervention. This creates
"insincerity," not the casual or incidental insincerity of
one misdirected person, but an institutionalized
insincerity, willfully chosen in hopes of purifying
evidence. It is the nature of language (with its shifting,
imprecise web of meaning) that is at fault, in this view.
But the attempts to remove its imprecision remove the
people, too. The modern-classicists often call the
denigration of this vast area of human communication a
"third sophistic" since it seems to elevate "appearance"
and "form" over substance in a way reminiscent of the first
and second sophistic ages.
The source of the modern-classicists' trust in
discourse is neither blind faith nor pure, analytical
reason, but a new epistemology which synthesizes many ways
of knowing. Philosopher Michael Polanyi says that what we
57
"know" is personal belief based on evidence from an array
of sources. Those beliefs are held together by, and
experienced within, a "fiduciary [trust] framework," so it
is trust that allows those beliefs to have their character
as "personal knowledge."
We must now recognize belief once more as the
source of all knowledge. Tacit assent and
intellectual passions, the sharing of an idiom
and of a cultural heritage, affiliation to a
like-minded community, such are the impulses
which shape our vision of the nature of things.
No intelligence, however critical or original,
can operate outside such a fiduciary framework
(265-66).
Of course, the shared "idiom," shared "cultural heritage,"
and "like-minded community" refer to the critical
importance of a (shared) context in establishing meaning.
Polanyi's "tacit assent" is, to Booth, a key to the
reunification of "value" and "fact," a unification
necessary so that we can use both as ways of knowing.
There are, Booth asserts, general kinds of knowledge that
we share, and ways of knowing that break down the hard
divisions between the subjective, personal world of feeling
and value, and the objective, impersonal world of knowledge
and truth or reality. These ways include 1) the strength
of one's own convictions; 2) agreement with other subjects;
3) coherence with other kinds of knowledge; and 4)
58
teachability or corrigibility. The final result is a
"rhetoric of assent," marked by a belief in a thing unless
we have cause to doubt it. This, according to Booth,
"depends on this fact which is a value: men ought to attend
to whatever good reasons are offered them by other men"
(142). Booth’s "rhetoric of assent" requires Burke's
concept of "identification," the means by which the speaker
and audience become "consubstantial" through a shared sense
of each other. Booth concludes, "when I assent to your
thought (or symphony or novel or account of your divorce)
the line between us grows dim; in the ideal sense it in a
sense disappears" (xvi).
Finally, and most broadly, trust in discourse in this
framework rests on a positive view of human nature.
Rhetors must have good will and good faith in order to have
discourse. Perelman notes that argumentation, unlike
demonstration, presupposes a meeting of minds: the will on
the part of the orator to persuade and not to compel or
command, and a disposition on the part of the audience to
listen. Such mutual goodwill must not only be general but
must also apply to the particular question at issue. (Such
"sincerity conditions" are similarly important to
speech-act theorists; see, e.g., H. P. Grice.)
59
Plato is an important classical source for this
positive view of human nature. Plato founds his view of
ethics on a positive, even idealistic view of human nature.
This emerges especially in the Phaedrus. The virtuous
rhetorician, a lover of truth, has a soul whose dialectical
perceptions are consonant with those of a divine mind.
Consequently the good soul will not urge a perversion of
justice ift: order to triumph rhetorically. The soul's
definitions will ideally agree with the true nature of
things. When people of good will use rhetoric well, it is
a tool for self-knowledge that can ^be turned to any
purpose. Plato says, "When we have adequately exercised
ourselves in this way in partnership with one another, we
can, if we think fit, set our hand to politics or to giving
our opinion about any other subject that attracts us"
<
(Gorgias 149).
While for the moderns truth is created rather than
found; still we act, they say, as ij_ there is shared truth
our discourses try to approach. By practical necessity, we
proceed with conditional assent; we must at least want to
communicate. In contrast, in a mistrusting rhetoric, the
institutionalized "insincerity conditions" are based on
neutral or negative views of human nature: such rhetorics
are thoroughly skeptical. They in a sense self-destruct,
60
since such rhetorics don't believe in the usefulness of
discourse.
The element of trust in this rhetorical framework,
like ethos before it, forces certain questions about texts
and reading that will be relevant later in the critique of
writing assessment. For example, how can readers claim to
not have attitudes about discourse when they read, or even
about human nature in general? How can readers be told to
deal with, and claim to deal with successfully, only the
superficial features and effects of texts, and not become
part of, and in some sense changed by, the reading act?
2.3 VALUE
Perhaps the most important form taken by trust in
discourse, where trust is a subscription to assumptions
(tacit or explicit) regarding discourse, is value. To the
modern-classicists, there can be no knowledge of discourse
or discoursers, and no meaning, without value. Value here
-means the significance or importance of one idea, attitude,
or position relative to another. Value is a quality of
discourse and discoursers, in the modern-classicist view
(see W. Ross Winterowd, "The Rhetoric of Beneficience,
Authority, Ethical Commitment, and the Negative"). Value
in language and language use- contrasts with positions that
61
seek to make language "transparent," and language use and
language analysis "objective," "disinterested," or
"mechanical." Briefly, the modern-classicists believe that
rhetoric cannot be neutral because language itself cannot
be neutral. Further, language cannot be neutral because
the people using language cannot be neutral. Those
discoursing inevitably enact values, and discourse must be
seen in this light.
The implications here for ethos in texts are 1) that
the reader reconstructing the writer (as ethos in a text)
is inevitably doing so from within a value system; 2) that
the writer he is reconstructing is also within a value
system; and 3) that the interpersonal nature of ethos for
both writer and reader is understandable only by recourse
to the values that enable and constrain them as people.
The question to be kept in mind for the later critique of
anonymous assessment is, how can an evaluation of
texts— clearly a process involving values— be claimed to be
"scientific," "objective," or "value-free."
The modern-classicists uniformly ground their theories
in a sense of value, in its many dimensions. Value is a
general quality of discourse. It informs, grants, and
constrains meaning, whether of individual words or of
entire texts. Second, a person's values describe how
62
attitudes are formed and changed through discourse; the
valuational nature of rhetoric describes rhetoric's purpose
for being and its goals, which are valuational. Third,
even as a practical necessity for cognitively processing
prose, value is necessary for meaning to be comprehended.
Finally, the starting point for commentary on discourse
(rhetorical analysis) is valuational, too. Those seeking
to understand the processes and effects of discourse must
start by acknowledging that the rhetor’s values (and the
analyst's as well) inform the analysis. I will briefly
develop the modern-classicists' sense of a valuational
rhetoric under three headings: 1) the inherently;
valuational nature of language; 2) the inherently
valuational character of people using language; and
finally, 3) the epistemologicaT and psychological nature of
values themselves.
As to the first, a rhetorical interaction invites
conclusive attitudes or actions. These language activities
are, by definition, valuational. Language is not neutral.
Meaning is not available "outside" complex language scenes.
This is true at all levels of analysis. At the concrete
level of a given linguistic context, a word's meaning is
not objectively available. Both Perelman and Richards
speak to this point. Perelman says that the belief that a
63
word simply has its meaning independently is an attempted
"realism" that belies the true nature of language.
Richards calls this belief the "Proper Meaning
Superstition": "that a word has a meaning of its own
(ideally, only one) independent of and controlling its use
and the purpose for which it should be uttered" (11).
Instead, contexts, some stable for a relatively long time,
control meanings. Burke adds that human scenes are too
complicated to be simplified into the "semantic ideal" of a
"neutral" vocabulary (The Philosophy of Literary Form 138).
The semantic ideal of an objective language has its place
in "physicalist terminologies" (those treating objects in
"motion," not people in "action") but in human scenes it is
an attempt to "make a totality out of a fragment, ’till
that which suits a part infects a whole" (PLF 138). In
complicated, human scenes, it is a mistake to chase "terms
that avoid ambiguity." Rather, we should seek "terms that
clearly reveal the strategic spots at which ambiguities
necessarily arise" (PLF 138). Perelman reminds us, "Values
enter, at some stage or other, in every argument
[discourse]" (75).
This is why scientific models of language are faulted
by the modern-classicists. The scientists attempt to
divorce the meaning or import of language from the
64
valuations that inhere in the language. It was long an
ideal of semanticists to make a "science" of language, one
that would create a purified, objective language. It would
communicate "the same" regardless of the participants or
the particulars of a given context. This view tries to
analyze meaning as directly available— it is supposedly not
mediated by anything. This view rests on a "man as
machine” metaphor. In this metaphor, writers and readers
are. reduced to mere brain cells, uniformly processing
information. Intention, point of view, and style are all
irrelevant and at most, leftovers from a view of language
too imprecise to become legitimized into a scientific
description of language. Semanticists using this
scientific view attempt to treat language as "transparent,"
or at least to limit their study to those examples of
language wherein only "neutral" language is allowed. Those
hoping to create a "scientific" language made little
headway toward dealing with the connotative qualities of
words or the affective qualities of language. But they
felt confident that they could at least begin by dealing
with bare, propositional content. A "scientific" approach
would extract the propositional content of the discourse
and treat that as the meaning. But, as it turns out, there
is no meaning without the (value-laden) surrounding
65
language scene. The scientists would try to look just at
the factual statements of a discourse. But at bottom there
are no facts? instead there are systems of value which
"enable" what shall count as a fact in a particular
context. As Perelman points out, "we are confronted with a
fact only if we can postulate uncont’ roverted, universal
agreement with respect to it. But it follows that no
statement can be assumed of definitively enjoying this
status, because the agreement can always be called into
question later" (67). So our choice to name an idea a
"fact" and not an "opinion" is based not on some extrinsic
status it supposedly enjoys, but rather on the strength and
duration of the agreement it possesses in its use, based on
the values of those using it. Meaning is neither an
obvious reflection of reality nor mere idiosyncracy. It is
the social use of language that communicates, influences,
and creates meaning.
Values, then, enter at the most concrete level of
linguistic processing. They also enter at higher-level,
whole-text comprehension. For example, even seemingly
"scientific," "objective" concepts like cause and effect
are not knowable independent of an inherently valuational
language. Richards points out that we assiqn cause and
effect as a result of the causal laws we happen to apply to
66
events. Such acts are thus interpretive, not determinate.
He notes that Aristotelian "final cause" reverses the
conventional time sequence of first cause, then effect.
Another example he uses of the interpretive nature of cause
and effect is that of the coroner’s decision that the cause
of a man’s death was a murderer's act, not the fact that
the victim was not wearing a bullet-proof waistcoat.
Richard Weaver corroborates the essentially valuational
nature of language: "Ever since I first heard the idea
mentioned seriously it impressed me as impossible and even
ridiculous that the utterances of man could be neutral. . .
expression purged of all tendency . . . rests upon an
initial misconception of the nature of language" (209).
By now, the second heading under which I will discuss
the modern-classicists' sense of a valuational rhetoric is
implicit: the inherently valuational nature of people using
language. The logical source of valuational language is
people who have values. For the modern-classicists, values
grow out of conditions fundamental to human nature.
Kenneth Burke says that people are subject to a "generic
divisiveness" that results in their inevitably having
"interests." If there were no separateness between people,
there would be no need for rhetoricians to proclaim their
unity, Burke says. If there were absolute separateness,
67
there could be no rhetoric, since opponents can only join
in battle through a mediatory ground that makes their
communication possible. For Burke, only in an "Edenic"
sense was there total unity, and now that unity has
"Babylonically" split, leaving to rhetoric the task of
dealing with this disunity as best it can. Out of this
divisiveness "emerge the motives for linguistic persuasion"
(£: Rhetoric of Motives 146). By valuing one position over
another, we represent our character, which in texts is
ethos. Weaver, too, speaks to the inherently valuational
nature of people discoursing. He locates it in man’s
"historicity." Born into a world of interests and points
of view, we always find some discrepancy, however slight,
between the situation we are in and the one we would like
to realize. "The fact that leadership is a human necessity
is proof that rhetoric as the attempt through language to
make one's point of view prevail grows out of the nature of
man" (209). Burke says that we "erect various communities
of interests and insights, social communities varying in
nature and scope" (A Rhetoric of Motives 146).
A person's language acts are not value-laden because
he is too biased or subjective to make them "value-less."
A scientific approach which tries to proceed from a
"strictly reasonable" model of human behavior misses the
68
point. Booth argues that an exclusively "rationalistic"
approach prevents us from understanding the nature of
discourse, and prevents us from using discourse
effectively. His goal is a "rhetoric of good reasons."
Discussing the philosophical split between "fact" and
"value," he faults two groups for the same extremism. On
the one hand, the fanatics who throw out logical thinking
because they find it too constraining have made specious
"reasonable" connections between abstract principles and
their conclusions "validated" only by an ill-considered
group of "significant others." On the other hand, the
"value-free scientismists" oftentimes choose, on abstract
principles, a validating group that ignores what the common
sense they share with their fellows would teach. We need,
Booth says, a "reasonableness" that acknowledges our and
others' values, not a "reason" that seeks to ignore them.
Human values do not force people into "hopeless
subjectivity" any more than a , supposedly value-less
language allows people to be totally objective. Values
simply and critically "enable" language. Still, Weaver
says, there are degrees of objectivity and, for example,
two people can of course look at the same clock face and
report the same time. The qualities of properties have to
be in the things, but they are not in the things in the
69
form in which they are framed by the mind. Weaver sums up,
"if there is anything that is going to keep on defying
positivist correlation, it is this subjectively born,
intimate, and value-laden vehicle we call language.
Language is a system of imputation, by which values and
precepts are first framed in the mind and then are imputed
to things" (209). Thus, the human mind is a tool for
thought and discourse, but we must not try to reduce its
functions to mere mentalism. The operations of the mind
are always subject to the values it holds.
This leaves the third heading under which I will
develop the modern-classicists' sense of an inherently
valuational rhetoric: the epistemological and psychological
nature of values themselves. Much of this has been
developed implicitly above. We come to what we "know"
through our faculties which perceive based on our
"interests." Burke speaks about "terminological screens"
which are, on the one hand, necessary for organizing
perception, yet on the other hand, necessarily limit what
we see. Philosopher Michael Polanyi would agree. His
epistemology of the "personal" claims that only persons
having beliefs make meanings. Only persons having beliefs
can know anything. Persuasive feelings, tacit assent and
intellectual passion enable knowledge. Our perceptions of
70
•even "brute facts" are shaped by an interpretive,
reflective process based on values.
The logical premises of factuality are not known
to us or believed by us before we start
establishing facts, but are recognized on the
contrary by reflecting on the way we establish
facts. Our acceptance of facts which make sense
of the clues offered by experience to our eyes
and ears must be presupposed first, and the
premises underlying this process of making sense
must be deduced from this afterwards • (162).
When we write (or read) we cannot predict the outcome from
previous language use (as a "scientific" theory of language
would claim), because the act of writing or reading itself
may involve a decision to correct or modify the view of
language that enabled the act in the first place.
Thus to speak a language is to commit ourselves
to the double indeterminacy due to our reliance
on both its formalism and on our own continued
reconsideration of this formalism in its bearing
on our experience. For just as, owing to the
ultimately tacit character of all knowledge, we
remain ever unable to say all that we know, so
also, in our view of the tacit character of
meaning, we can never quite know what is implied
in what we say (95).
Polanyi feels that we must accept this particular kind of
semantic indeterminacy since only words of indeterminate
meaning can have a bearing on reality. That is, the kind
of determinacy he rejects is that which would be formally
meaningful divorced from actual participants creating that
meaning. A strictly formal language operation would be
impersonal and would not include our participation in it.-
He says, "It may seem that we have saved the concept of
meaning from destruction only to expose it to being reduced
to the status of dogmatic subjectivity. . . . [but] Unless
an assertion of fact is accompanied by some heuristic or
persuasive feeling, it is a mere form of. words, saying
nothing" (253).
Booth addresses the same line of thinking. To him, a
valuational rhetoric reunifies the "fact-value split" that
plagues modern philosophy and rhetoric. Because that split
keeps "descrj.ptive" disciplines separate from "normative"
ones, we have no basis for asserting we know anything that
matters. He says, "Aristotle at least knew that practical
life required rhetoric and that rhetoric could not be
reduced to logic. But many modernists have moved in the
contrary direction, not only making logical proof prior
but, as we have seen, eliminating all other kinds entirely"
(144).
Value, then, is an indispensable part of discourse.
Ethos in texts necessarily proceeds from assumptions (tacit
or explicit) of value. To reconstruct ethos, the reader
must acknowledge the values of the-writer (and his own).
Like ethos and trust before it, the element of value in
72
this rhetorical framework forces certain questions
regarding the assessment of student texts. The central one
is, how or in what ways are the assessments valuational?
They clearly seek to evaluate a student, but what values do
the assessment theorists and the readers themselves hold,
openly or implicitly? To the extent that the assessment
rationales avoid or deny the inherently valuational nature
of discourse .and discoursers, there are disjunctures
between their rhetorical attitudes and those of the
modern-classicists I have been advancing.
2.4 CHOICE
I need one last "interpretive probe" to round out my
rhetorical framework. Within a trusting (affirmative)
valuational rhetoric, ethos must be analyzed as an active,
human, committed choice. To enact discourse is a choice.
To choose is to act ethically (actions relevant to ethos)
on one's values, within a trusting, affirmative rhetoric.
Choice validates ethos in texts as a personal proof, in
both senses of ethos. The concept of choice is relevant to
ethos as a cognitive experience for the reader of a text,
because choice enters both as a decision by the writer to
present himself in a certain light, and as a choice by the
reader to ascribe a certain presence or character to the
73
writer. Second, choice is relevant to ethos in texts in
the broader, sense in which ethos indicates the
"interpersonal" nature of active, human participants making
valuational choices that imply an epistemology in
discourse.
This contrasts with views of discourse that either
take people as unable to choose, or that see choice as
irrelevant. Because of the nature of language and of
humans, this argument runs, discoursers are unable to
choose, perceptually, valuationally, or epistemologically.
This is not simply the claim that we are not fully
conscious of our choices; rather the claim is that the
nature of language and the nature of humans prevents or
limits this choice, whether self-consciously or
unconsciously. The question to be kept in mind for the
later critique of anonymous assessment is, in what ways do
the assessment theorists foreclose on the freedom of
writers and readers to choose, relative to ethos, to
present and interpret selves in texts.
The range of actions that committed choice entails
will be discussed under three headings, from very concrete
to very abstract. Choice reveals itself in discourse oh 1)
a cognitive-psychological level; 2) on a valuational level;
and 3) on an epistemological level.
74
On a concrete, cognitive level, choice explains our
ability to "encode" and "decode" meanings in texts. These
choices are made by a necessarily real person— with
preexisting world knowledge and operative purposes of
understanding at a given time. This process of
knowledge-based, contextually-influenced, and purposeful
enrichment in comprehending language is what must be dealt
with as "constructing meaning." These structure choices
that are sometimes conscious, sometimes not.
Humans do: not process prose uniformly or mechanically.
The linguistic and extra-linguistic context in which a text
appears are important determinants of the psychological
structure and meaning of text. Knowledge, interest, and
perspective all contribute to the choices the reader makes
in deciding what a text means to him (including how the
reader takes the writer— ethos). The contextualist sees
reading not in the Cartesian sense of "making a mental copy
of the world," but as an evolutionary event, in which how
he constitutes his choices influences what he sees. The
choice we make while reading is a choice of conditional
assent, to "agree with the speaker's [writer's]
interpretation of the world that is" (Weaver 205).
Contemporary contextualist reading theory echoes this
insight, and recent interest involves "a shift in emphasis
from a traditional learning theory approach and from a
traditional process-modeling approach, to a description of
knowing in terms of the situational and task variables
which define experimental situations and which constrain
subjects into behaving 'as if* they possess a particular
form of knowledge" (Hoffman and Nead 552).
Second, the concept of choice has a valuational
dimension. Briefly, the modern-classicists claim that
beyond reductionist dichotomies that attempt to circumvent
man's relative free will by labeling phenomena with binary,
determinate categories, lies a new, unifying
"responsibility," literally an "ability to respond." For
example, Perelman speaks about two views of - language,
"realism" and "nominalism."' He says that both realism,
which treats language as objective, and detached, and
nominalism, which treats language as idiosyncratic- and
subjective, are equally’ false. Instead, we must
acknowledge that "adherence" to a position is humanly
chosen, self-consciously or not, on bases that neither of
these binary extremes can fully account for. Similarly,
Booth faults both the "hyperrationalists" and the
"fanatics" for trying to remove choice from language acts
by relying on either positivism or pure sensation.
Instead, he says, only humans making choices can approach
76
issues that count. In the same way, Polanyi points out
that neither religion nor science can substitute for
"personal belief." We must acknowledge, the
modern-classicists,maintain, the "choice" character of our
perceptions and interactions. This is what is entailed in
Polanyi’s comment quoted above about how language use
cannot be determinate because it may involve a choice to
modify the view of language that enabled the act in the
first place. The resulting choice, in Polanyi’s term, a
"semantic double indeterminacy," is necessary for our words
to have any bearing on an ever-constituting reality.
Even the so-called "hard" sciences, with their clear
distinction between objectivism and relativism, are
challenged by this view of choice. Richard J. Bernstein
posits a new synthesis between these two formerly discrete
positions in his Beyond Objectivism and Relativism:
Science, Hermeneutics, and Praxis. Indeed, he says.that
the underlying motive for the task of going beyond this
dichotomy is a "practical-moral concern"--a choice of
values. He notes the "growing awareness of the vital role
that interpretation plays at every stage of scientific
activity and a questioning of any permanent division
between theory and observation” (171). Left with the
"irreducibility of conflict grounded in human plurality . .
77
we seek to discover some common ground to reconcile
differences through debate, conversation, and dialogue"
(223).
This view of choice contrasts with the non-committal,
detached quality of sophistic rhetorics. It also contrasts
with prescriptive philosophical ethics, whichhas tried to
establish a "rational" basis for ethical force in language
acts. Charles L. Stevenson is representative here, and the
root problem is that his is a prescriptive ethics (in the
moral sense of right and wrong) and not a description of
ethos, which is the emphasis of the modern-classicists
discussed here. Stevenson belieyes that the basis for
moral judgments are pure, factual reasons which in turn
lead to ethical judgments (where "ethical" includes
knowledge of the rhetor’s character, but consists in
addition of moral judgments of right and wrong). In the
(many) instances he names where the validity of such
progressions are in question, he worries about what grounds
are left to account for the many ways and reasons why we
come to an ethical decision: "one may feel that no ground
for choice between them remains. Or if such a ground is
recognized, it may seem to involve only a crude, forensic
success . . ." (156). What Stevenson tries to dismiss as
"merely the empty rules of rhetoric" (156) is, for the
contextualist, the great question: what are the conditions
that guide our choices .in adherence to an argument in
scenes where scientific approaches cannot provide a
complete account?
Third and finally, choice in this rhetorical framework
has an epistemological' dimension. Choice to the
modern-classicists doesn’t rely on resolvings the
imponderable metaphysical debate over free will. Rather,
they simply note our sense that we act as j^f we have
relatively free choice, a choice that is enrhanced through
increased self-awareness. We exercise our choices
rhetorically" by deciding that we know or believe something;
this is a choice to interpret a text in a particular way.
Indeed, a scene or a proposition becomes rhetorical only
when there is choice in the matter. It must be, in
Perelman’s terms, "noncompulsive.” Burke agrees and says
that otherwise, a situation is not a rhetorical one.
Persuasion involves choice, will: it is directed
to a man only insofar as he is free. . . . Only
insofar as men are potentially free, must the
spellbinder seek to persuade them. Insofar as
they must do something, rhetoric is unnecessary,
:its work being done by the nature of things, .
•” (A Rhetoric of Motives 50).
At bottom, this epistemology rests on a view of human
nature where people are beings capable of choice. We are
■ __________________: ___ :_____. _________ 79.
people acting, not objects moving. Burke' expresses the
"pragmatic distinction between the 'actions' of 'persons'
and the sheer 'motion' of 'things"' (Language as Symbolic
Action 53).
Of course, this view contrasts with views of language
and language behavior that seek to foreclose on choices by
ascribing them to outward determinants, or by casting doubt
on the Certainty with which we hold to the results of our
choices. For example, a behaviorist view sees choice as
irrelevant. Second, deconstructionist rhetorics allow only
a tenuous confidence by which we can hold to rhetorical
choices. The play between signifier and signified is
analyzed, in deconstructionist rhetorics, on the hypothesis
that what would make a sign (e.g., contact between
signifier and signified) is really a gap, or an uncrossable
line. Signs do not really occur; they merely seem to.
Exact signification is impossible, delaying, if not
denying, choice. ^ I-
Without subscription to choice, there is no basis for
confidently concluding anything from texts. Unless we
ascribe a purpose to a text, the chosen purpose of a human
writer, we cannot reconstruct the ethos of a text as a
person with certain characteristics. As before, this
element of the rhetorical framework similarly poses certain
80
questions relative to the assessment of student texts. For
example, do the assessments allow for the dimension of
individual choice in their "psychology of reading"? Do the
assessments acknowledge the valuational context within
which these? choices are made? Finally, does the assessment
theory contain an- epistemol;©gy to account for the nature of
the choices by which the reeder - comes to. ..know the ethos of
the writer? - p
We now have a systematic and abstract rhetorical
framework. Synthesized from certain cohering, important,
modern . rhetorical • themes, this modern-classicist
contextualist rhetorical framework uses ethos, trust,
value, and choice as interpretive probes for analyzing
discourse, within a view that is holistic, relativistic,
interactional, dynamic, and reflexive. It is, further, a
descriptive, social, and performative rhetorical framework,
which makes it especially well-suited for approaching the
assessment of student texts. I will, in Chapter Four, take
up this framework to reinterpret the assessment practices,
put', first, we need to know what those practices are.
Zhapter Three gives a straightforward summary of them
(origins, purposes, methodologies, and results).
81
Chapter III
THE PRACTICES OF ANONYMOUS MASS ASSESSMENT
Mass, anonymous asse^s.merit of student writing ability
began on a national scale in 1916 with the College Entrance
Examination Board’s onerho'ur essay. Since then, the
methods of CEEB (which later became ETS) have been refined,
expanded, and duplicated by many other writing and
assessment programs. Today, hundreds ; of thousands of
students are assessed annually, at a cost of millions, and
with great consequence: college grades and college
admission itself. can depend on the results of a single,
brief essay written under highly prescribed conditions.
From the assessors' point of view, these assessments
are valid and reliable indicators of a student's writing
ability. I think this claim rests on two assumptions,
which will be taken up in turn. The first is that writing
assessments can and should proceed from a more or less
"scientific" (objective and fair) basis. The second is
logically imbedded in the first, that these practices as
executed are examples of "good science": that~ is,
psychometrically sound. The first assumption allows us to
' ,82_
describe their explicit practices (and, by inference, their
theory) as a rhetorical framework (a set of coherent ideas
about writers, texts, and readers)'.: The second assumption
allows us to compare their "science" with the psychometric
methodology of the social sciences from which it is
borrowed. r Therefore, I will look at these practices,, and
the implicit theory therein,- first relative to their
qualities as a rhetorical . framework--thoroughness,
justifiability, and consistency— and second, ' relative to
the theorists' obligation to practice "good science": to
be psychometrically sound.
We may suppose that student texts have been assessed
as long as there have been students, texts, and teachers.
Teachers and administrators have always wanted to know how
well their students can write, whether the students have
learned a given aspect of composition that term, of how to
place or rank students in a class or school system.
But such assessments, up to the beginning of- this
century, were different in two important ways. First, they
were not consciously "scientific" in the sense of being
consistent or systematic. Teachers naturally varied in
their approaches to assessment. Some used more or less
thoroughly elaborated methods of assessment of texts based
on the knowledge or values they held about discourse. Some
_.83__
used intuitive or frankly idiosyncratic approaches to
assessing student compositions. But certainly there was
nothing like a national assessment of student writing
ability, with consistent and systematic procedures.
Second, such assessments were "local" in that those who
were assessed knew the assessors and vice versa. In this
way, there was a lack of "scientific," "objective" distance
between writers and readers. There , was also a
corresponding presence of other kinds of "knowledge" of
assessor about those assessed, and vice versa. However,
that all changed at the beginning of the twentieth century.,
The change was based on the coincidence of two forces. On
the one hand, societal needs had changed, and on the other,
there became newly available the emerging science of
psychology (accompanied by its practical counterpart., the
science of testing). This confluence of forces is neatly
captured by Ralph W. Tyler and Richard M. Wolf in'their
Crucial Issues in Testing (1974).
Educational testing . . . began as a means for
selecting and sorting pupils, and the principles
and practices of testing that have been worked
out since 1918 are largely the refining of means
to serve these functions rather than other
educational purposes. They are based on the
psychology of individual differences rather than
upon the psychology of learning (4-5).
.84 J
Tyler’s statement points to both assumptions of mass,
anonymous assessment that I will examine. The first is
that such testing can and should be scientific— here, as a
sorting procedure based on the detection of "individual
differences." This assumption will be dealt with first,
taking each institution in turn (ETS, NAEP, and FWP). The
second assumption, that such testing is psychometrically
sound ("good science"), will be taken up in a second
section, immediately following. Throughout, I will point
out that these assumptions of mass, anonymous assessment
use scientific procedures to recreate, on a large scale,
activities formerly performed in "face-to-face"
environments. In the old (pre-scientific) days, an essay
was written and read in a local rhetorical context. A
specific teacher assigned a topic and a specific student
wrote an essay on it. The context consisted of the shared
knowledge of the conventions of writing, and more
critically, it consisted of mutual knowledge of writer for
reader, and vice versa. This mutual knowledge included
notions of value, taste, and identity and so on that made
the rhetorical situation particular. This knowledge varied
from participant to participant and is difficult to
describe scientifically, but we know it from its
consequences. For example, students writing on a topic
35
will adjust their discourse to take account of what they
think their reader Jcnows about the subject being discussed.
Conversely, the teacher reads the paper with some knowledge
(outside the text) ... of the student (perhaps his/her
performance on an earlier paper). This knowledge of the
context and of its participants allows— even requires— each
to adjust his discourse for maximum effectiveness. The
point is, modern assessments made by ETS and NAEP are
radically different rhetorical contexts from those earlier,
local, informal ones. Neither is a better or worse
rhetorical context, but they are very different ones. The
earlier, local ones contained a great deal of explicit and
tacit knowledge of the participants about one another; in
the modern ones, anonymity prevails on both sides. Ethos
here is deliberately limited to the text itself, in the
interest of fairness to all. The . important point here is
that we must be aware that the assessment theorists have
made a "leap of faith." They still seek to grade or to
give credit or to place a student just as that individual
teacher wanted to do in his classroom, but they have
radically altered the context. ■ - The most critical
alteration is that they have reduced the amount and kinds
of knowledge of the writer for reader, and vice versa. The
question to be kept in mind is whether they get the same
results when they proceed in such a different context.
86
By far the largest and oldest of these institutions is
the College Entrance Examination Board (with •' its
"independent" test-giving agency, the Educational Testing
Service). ETS administers hundreds of academic and
vocational tests to millions of people each year. Much
newer is .the National Assessment of Educational Progress,
but it is huge too, having to'date tested 900,000 students.
NAEP is different in structure from ETS (a non-profit
corporation) in that, NAEP is a government-supported,
recurring assessment of , the nation's students in many
academic and vocational areas. One measure of the
pervasive impact of the techniques developed by these
institutions is the way their methods have been duplicated
by local writing programs such as the Freshman Writing
Program at the University of Southern California.
While ETS and NAEP are giant institutions which assess
achievement in many academic and non-academic areas, I will
be concerned here only with their tests of writing ability.
The uses to which the results of such assessments are
put diverge widely among these institutions. Broadly
speaking, the writing samples gathered by ETS are used in
the decision-making process for college entrance. In
contrast, the writing samples taken by NAEP are used to
make descriptive statements about achievement levels in
87
writing for the nation's students as a whole. Again in
contrast, the writing samples gathered by FWP (and many
other writing programs across the nation) as used-as.part
of a student's grade in Freshman English courses (or in
some cases, to exempt the student from the course
requirement).
Even though the. uses to which these data are put
varies among the institutions, the actual practices of
gathering, assessing, and interpreting these data are
virtually identical, due in no small part to the fact that
ETS and NAEP are closely related by common rationales,
social science precepts, and the same personnel devising
and administering the assessments.
So it is possible to look at this gathering of writing
samples, and the subsequent assessment in a mass, anonymous
manner, as a coherent body of practices that .makes
particular implicit statements about what is meant by
composition, texts, and the acts of reading and
interpretation. That is, because ETS and.NAEP perform a
common, coherent body of practice (one that is imitated
widely on a smaller scale, by many other writing programs),
we are able to look at those practices as an .implicit
statement (an "informal theory") of what is meant by
composition, texts, and the acts of reading and
interpretation.
88
This chapter is in two sections. Section One gives a
brief history of these institutions' development and.use of
writing samples as assessment tools of student writing
ability. Section Two summarizes the psychometric
assumptions of'these assessments, showing how the terms and
concepts of educational testing have .beenrtaken up by the
testers to justify the efficacy, reliability, and validity
of their assessments.
3.1 THE ASSESSING INSTITUTIONS . /
3.1.1 Educational Testing Service "
The obvious reason the College Board gathers writing
samples is that they want to rank student writers in a
comparative fashion. Yet it has never been obvious that a
writing sample is the sole path by which to establish such
a comparative ranking. In fact, the very inclusion of a
writing sample has been problematic for the College Board
throughout its history. To understand the history of the
use of writing samples by the College Board, one must
temper the compositionist’s expectations with ETS’s
practical needs. That is, the primary need of the College
Board, regardless of subject area, . be it English
composition or algebra, was and is to derive numerical,
comparative scores of all those assessed. They were thus,
89
given their institutional, pragmatic needs, forced to see
English composition "the ,same" as any other subject area.
The uniqueness of composition as a performance or a skill
was never welcomed, since the point was to translate a
human behavior, somehow, into a number. It is true that
ETS has all along acknowledged the methodological
difficulties of translating the complexities of the writing
act into a single, numerical score. Still, one finds in
the history of writing assessment by ETS an approach which
necessarily reduces the complexity and uniqueness of
writing (and thus, later, reading texts) and, in terms of
the principles of test construction, tries to make writing
an essay more like (say) solving an albegra problem, rather
than less. The fact that a compositionist would consider
it obvious that an actual writing sample is critical
evidence in any assessment of writing ablity is not
operationally obvious to these test-makers. In search of
such numerical, comparative scores, then, the the use of a
writing sample was not so much a source of evidence, but an
operational problem— how to translate a piece of discourse
into a number, fairly.
This perspective is summarized in a 1966 conference
sponsored by ETS. Here we find ETS's ambivalence regarding
writing samples as indicators of writing skills.
90
Much the same sort of situation may exist in the
field of writing ability [as exists with speaking
ability] which has perennially been regarded as
one of the most elusive things to measure. That
is, writing ability may be a matter of
situational control, with achievement varying
according to the kind of task that is set.
Insofar -as good writing depends upon
knowledge of language responses—-morphological,
syntactical, and lexical--it will be found
related to the verbal knowledge-factor V; indeed,
in some studies of the College Entrance
Examination Board, it- appears that verbal
knowledge tests are more satisfactory measures of
writing ability than writing-tasks: themselves.
We have not yet been able to measure the
reasoning and organizational: aspects of good
expository and.creative writingaindependently of
writing itself, perhaps because the two are
inextricably bound'together (Anastasi 411).
Given this far different orientation--discourse not as a
rich source of evidence but as a recalcitrant thing, a set
of behaviors resistant to quantification—-we can view the
use of writing samples by the College Board with more
sympathy.
A much more recent ETS publication, this one from 1978
by Arthur Bishop, indicates that ETS feels that writing
samples still present practical problems.in assessment.
One of the most troublesome aspects of " any
program designed to improve writing abilities is
evaluating those abilities. By its very nature,
writing defies precise evaluation..
Multiple-choice, short-answer, and similar
forms of "objective" tests are relatively simple
to score, and they yield good assessments of
mechanical aspects of writing such as word usage,
grammar, and spelling. More sophisticated forms
developed in recent years deal with logical
91
relationships, categorizing, inferences, diction,
and relatively complex skills. To do well on
these kinds of questions, a person must have a
fairly well-developed verbal facility.
But many believe the best way to measure
such essentials as organizing ability, clarity of
expression, and other more intricate and subtle
factors is to have students write. Evaluating
writing samples is not, however, an; exact
process, since it depends heavily on human
judgments (8).
Still, ETS has for over half a century both used
writing samples and tried to validate their use.- The
latter was done in the early years by seeking correlations
between scores on human-scored essays and "more objective"
scores from multiple-choice tests. Those correlations have
been gotten in recent years through the use of the
well-regarded "holistic" scoring method, which uses
multiple readers who,, with proper training, achieve high
rates of "inter-rater reliability."
From 1916 to the mid-1970's, many test formats were
tried, used for a time, and rejected in favor of other
formats. At some points, a writing sample alone was used;
at others, a series of multiple-choice questions alone; at
still others, writing samples and multiple-choice questions
were used in tandem. A single one-hour essay, three
twenty-minute essays, four fifteen-minute paragraphs--all
were tried and eventually replaced by other combinations.
The main reason for the many format changes and the
92
on-again, off-again use of writing samples was that the
statistical correlations between readers' assessments arid
the student's score on; multiple-choice tests of verbal
skills weren't very.impressive." Again, from Bishop at ETS:
During the 1950’s and '60's, many research
scientists investigated the value of essay tests.
Some studies were quite elaborate. One involved
over 600 high school students who wrote five
essays, took six objective tests, and did two
other exercises. Each essay was evaluated by 25
readers and two of them by another 145 readers.
All these studies reached much the same
conclusion: Writing samples made slight
contributions to the predictive accuracy of
objective tests. The question then became: "Is
the increased accuracy worth the increased cost
of scoring?" The answer was most frequently "no"
(9).
Speaking in 1961, John W. French of the College Board
revealed the exasperation they were by now feeling.
At the present time, the College Board again has
an English Composition Test, called the "Writing
Sample," in its program, but this one doesn't get
scored at all. We just send carbon copies to the
colleges, and wonder what the colleges are going
to do with them and how much longer they are
going to want to receive all those almost
unreadable handwritten papers ("Schools of
Thought in Judging Excellence of English Themes,"
in Anastasi 587).
But this latest attempt with writing samples failed
too. Bishop describes that same 1961 innovation, its poor
reception, and the decision six years later to again drop
the essay.
93
The College Board and ETS tried one way around
the essay scoring problems in the early 1960's
when, in response to requests from colleges
concerned about the low writing ability of their
applicants, they introduced an optional writing
sample into their Admissions Testing Program.
ETS did not score the samples but sent copies to
colleges: where, presumably, English faculty
members evaluated them. The College Board
discontinued the writing sample in 1967 after an
ETS study found that few colleges--including some
that had requested it;— had much enthusiasm for it
or did anything with it (8-9).
In'the meantime, ETS devised something between a real
writing sample and an objective test: these were called the
"interlinear exercises." Passages of prose with deliberate
errors were presented to the student, who had to discover
and correct the errors.
Eventually, the test battery that emerged by the
mid-seventies (and is currently used) combines the
interlinear exercises, objective questions, and an optional
twenty-minute essay. Students choose to take either a
one-hour objective test or to take a forty-minute objective
test and write a twenty-minute essay. Whichever version of
the examination the student chooses to take, a final raw
score is then sent to the colleges with simply a notation
made as to which version the student took. That is, the
only score the colleges receive is a single, numerical
score, which may represent either one hour's worth of
objective questions or a mix of objective questions and an
essay.
94
As always, though, the rationale for the format is the
same: the correlations that are obtainable between the
writing sample scores and scores on objective items— either
between themselves, or taken together as predictors of
college grades, usually Freshman English.
This test battery is called the "English Composition
Test with Essay," an Achievement Test of the Admissions
Testing Program (ATP). It differs from all preceding
formats in two important ways. First, it is read by two
readers, not one; and second, it is no longer read
"analytically," but "holistically.” (The "analytical"
method breaks down an essay into components such as
spelling, diction, organization, and so on. The sum of
these separate scores becomes the final score of the essay.
But with holistic scoring, the readers are trained to. grade
impressionistically, quickly using an overall judgment to
assign it a score of 1 to 4.' In fact, the readers are
cautioned against looking too closely at any one element in
the text.) ETS's Bishop defines holistic scoring.
Called "holistic scoring," the method is based on
the belief that the overall impression that a
written passage makes is more important than
spelling, punctuation, organization, or any other
single aspect (9).
95
The quickness of the method then allows multiple readings
of thousands of essays in a short time. Marjorie Kirrie of
ETS describes the method and its assumptions.
The essays are read holistically, that is,
they are read for the total impression they make
on the reader. Readers are instructed to read
essays quickly and to score immediately while the
impression - the total essay creates remains fresh.
- Holistic scoring, assumes that each of the
factors that make up writing ^skill is related to
each of the other factors and that one factor
cannot easily be separated .from the others. It
also views a piece'of writing as a total work,
the whole of which is greater than the sum of its
parts; that is, a piece of writing is more than
diction, style, evidence, standard . idiom,
felicitous expression, and", a host of other
elements that make up a composition (4-5).
Since these are the procedures as they are currently
executed, I will make a fuller statement here of the
details of the assessment process. The discussion is in
three parts; selection of a topic for the essay, the
writing of the essay, and the training of raters and
reading the essays.
The selection of the topic is based largely on ETS's
sense of what unique contribution an essay can make, as
distinct from the accompanying multiple-choice questions.
Research shows that the combination of
multiple-choice questions and an essay yields
reliable and valid.scores. Tests made up solely
of essay normally do not yield scores that are as
reliable or valid as scores from tests that
combine an essay and multiple-choice questions.
96
The use of multiple-choice questions increases
the the sample of discrete knowledge that can be
measured and reduces some of the unreliability
that may result from the subjective judgments
that must be made when, as is the case with essay
writing, there is no single "correct" answer to a
test question and when even "correct" answers can
vary in quality (Kirrie2).
It is important here to remember the distinction
between the psychometric terms "reliable" and "valid."
Reliability is the consistency of the test: if the same
student takes the same test over and over again, he should
. . ' . 3 6 * Cl 5 ” : i
keep getting the same score. But since the student does
not and cannot do this, ETS looks to other measures to
establish reliability. High reliability suggests that
there is little random error in the assessment— error which
varies from one administration of the test to another.
However, validity is much different. That refers to
whether the test actually assesses the behavior in
question. So a multiple-choice test of writing skills;
might only show that a student could do well on such a
test, but not really be able to write.an actual essay.
What -ETS' wants, of course, is both reliability and
validity. This ETS tries to get. by. seeking high
correlations between the essay score and the
objective-questions score, or between the essay score and
later Freshman English grades. This would establish the
97
concurrent validity of the essay test. Kirrie makes this
point in the following in which she summarizes the results
of a CEEB study by Godshalk.
The results of the study confirm, among other
things, the following: that highly reliable
scoring of essays can. be attained by means of
tightly controlled procedures, that some types of
multiple-choice questions are valid as predictors
of students' writing ability, and that, given a
limited testing time,- . the combination of
multiple-choice questions and an essay will
provide the most valid measure of writing skill.
Although the correlations between the scores
on multiple-choice questions and scores on essays
are high, the inclusion of the 20-minute essay in
the English Composition Test adds an important
dimension to the writing assessment (3).
As.always, then, ETS views actual writing samples with
reluctance, but views multiple-choice questions as the
indisputable basis for reliability in such assessments. On
the other had, they claim that much greater reliability is
gained with holistic scoring over the old analytical
scoring method. This is because, the raters, before they
even begin to read "live" papers, read a series of sample
papers and compare their scores with each other after
reading each essay. These comparisons, in a fairly short
time, cause a consensus of standards to emerge, and quite
soon the whole group is scoring each paper almost
identically. This resulting consensus (which’ continues as
the raters go on to read "live" papers in isolation) is
98
called "inter-rater reliability." ETS explains in detail
the process of selecting each year's essay topic.
Each year, from the fifty or so topics
submitted by committee members for consideration, -
the Committee of Examiners prepares about twenty
essay topics for administration in pretest form
to students in College Board member colleges,
throughout the country (Conlan 2).
Before the committee of English teachers
selects essay topics, students write on them in
experimental administrations designed to provide,
information about the merit of the topics (Kirrie
3). '
A selected group of readers scores the essays and evaluates
the topics. Finally, only one topic is selected, and all.
students in a given examination are required to write on
that topic. :
ETS makes several comments on how that one topic.is
selected that relate first to student response to the toprc .
and second, to rater response to the essay.
To be useful, a topic must elicit expository
essays. If the topic invites other kinds of
responses or if it is one for which students have
difficulty finding examples, the topic is not
used in the test (Kirrie 3).
An. essay topic used in the College Board
Composition Test must meet specifications quite
different from those for a topic used only in a
classroom. It must be self-explanatory; it must
be defined and limited. It must be a topic that
every student in the country can be expected to
have some information about. It must be a topic
so stimulating that every candidate will have
something to say immediately. It must be a topic
99
that young and.inexperienced candidates can write
about intelligently when'they have only twenty
minutes in a tense situation to produce their
essays. . . . Does the topic allow the candidate
to call upon other than extremely limited
personal background to provide specific examples?
Is the topic limited enough? Is it structured
enough? Can the candidate put to good use the
short time he or she has for writing (Conlart
1- 2)? . • -
ETS has much to say, too, on the appropriateness of
the topic from the readers* perspective.
It must not be a topic . that calls for merely a
perfunctory presentation of cliches or a ' topic
that induces an emotional response on the part of
the candidate, even as it must not be a topic
that produces an emotional reaction on the part
of the reader of the essay. The ability to write
is what is to be evaluated, and everything that
interferes with that evaluation must be
eliminated or, at the very least, mitigated
(Conlan 1). _ :
A selected group of readers scores the essays and
evaluates the topic: Are there both good papers
and bad papers? . Will readers,be able to score
the responses reliably? Are the responses boring
to read? The committee deliberately tries to
avoid topic that invite readers to analyze the
candidates* psyches, for the primary purpose of
the topic is to give the candidates the best
opportunity to show how well they can write1 and
the readers the best opportunity to‘ judge that
ability fairly (Conlan 2).
Finally, then, this is exactly what the student was
presented with at a recent round of the examination.
Pi rect ions
You will have 20 minutes to plan and write the
essay assigned below. You are expected to
100
express your thoughts carefully, naturally, and
effectively. Be specific.. Remember that how
well you write is much more important than how
much you write.
DO NOT WRITE ON A TOPIC OTHER THAN THE ONE
ASSIGNED BELOW. AN ESSAY ON A TOPIC OF YOUR OWN
CHOICE WILL RECEIVE NO CREDIT.
. You must fit your essay on the answer sheet
- provided. You will.receive no other paper on
which to write. You will find that you have
enough space if you write on every line,, avoid
wide margins, and keep your handwriting to a
reasonable size.
. ’ DO NOT WRITE IN YOUR TEST BOOK.
You will receive credit only for what you write,
on your answer sheet.
THE TOPIC FOR DECEMBER 2, 1978.
First, consider carefully the following
quotation. Then, read and follow the directions
that are given in the assignment that follows.the
quotation.
"We have met the enemy and he is us."
Assignment:
What does this quotation imply about human
beings? Do you agree or disagree with its
implications? Support your position with
examples from your reading, observation, or
experience (Kirrie 5-6).
The process of training the readers has already begun
some days before the essays are written, and it continues
when the readers are finally presented with "live" papers.
The following report of their orientation sounds like the
actual speech the Chief Reader probably makes to the
readers.
It must be remembered that only 20 minutes are
allotted for the essay. To write a . polished
essay in 20 minutes can be expected of no one,,
least of all a student who is under the stress
that taking a nationally administered test is
likely to produce. The student does not have
101
access to a dictionary, has little time to ponder
the topic, and has little time to rewrite what
has been written. The 20-minute essay,
therefore, must not be regarded as the kind of
writing that a student would produce if time
permitted the preparing of a first and second or
final draft. Rather, it should be viewed as a
sample that allows experienced, trained readers
to judge the potential of the writer (Kirrie 4).
for the efficacy of holistic scoring is
readers, not in terms of its quickness
(though it certainly is that), but in terms of its
reliability. The argument is that because the readers are
in a kind of innate consensus, the scoring can quite easily
be made consistent. Again, Kirrie at ETS:
Holistic scoring relies on the experiences
of the English teachers who serve as readers.
Over the years these English teachers have read
and evaluated the essays their students have
written. They are aware of the quality of
writing they can expect from the students at this
level, and they are in general agreement about
what constitutes good writing.
Because holistic scoring essentially ranks
students' essays in relation to one another, care
must be taken to ensure that standards are
derived from the students' performance in
response to a topic— not imposed by ideal or
hypothetical criteria.: Establishing the
standards, therefore, becomes a process of
consensus (Kirrie 4).
The process of training raters is a hierarchical one.
The chief reader trains the assistant chief readers, who in
turn train the table leaders, who in turn train the actual
The rationale
then given to the
102
readers. Consequently, several days before the reading,
the chief reader and the assistant chief readers hold
meetings with Educational Testing Service consultants to
select the papers to be used as samples throughout the
reading to exemplify and maintain the standards. . This is
done by reading many papers to establish consensus on the
range of quality of the responses, and to select examples
of 4’s, 3's, 2's, and l's. Kirrie describes how the table
leaders are trained:
The : Chief Reader urges the readers to forget
their personal criteria for evaluating a paper
and, to adopt . the standards that the group
establishes.
The standards are set by the readers as they
read and score, one at a time, a series of sample
papers, reproduced in the students' own
handwriting, that represent the range of quality
found in the essays written on that topic at that
administration of the test. .
All readers score a particular sample essay
at the same time. The readers are instructed to
use a four-point scale with 4 as the high and 1
as the low score. Then by a show of hands they
indicate whether they gave the sample a score of
4, 3, 2, or 1. The tally of these scores shows
that most of the readers agree on the. score to be
given the sample essay. Readers who are not
scoring with the majority are asked to adjust
their scores with those of the majority.
The reading of the samples continues until
the Chief Reader decides that the standards have
been established firmly enough in the readers’
minds to permit the independent scoring of actual
papers (Kirrie 4).
103
After this training of the table leaders is accomplished in
the days before the actual reading, they are ready to
similarly train the readers themselves;
The chief reader presents the topic to, the
readers and defines the writing problem inherent
in the topic. He or she answers certain basic
questions about the scoring of the papers: How
much attention should be paid to misspelling?
What is considered off the topic? What do we
define as specific examples? . . . The scores for
a large population such as the one taking the
College Board Achievement Tests usually follow a
normal curve. . . . [l's and 4’s accounting for
15% each of the scores, and 2's and 3's
accounting for 35% each of the scores=100%]
After the discussion of the rating scale and
. of an ideal distribution of scores * the chief
reader presents a series of sample papers. This
group of papers has been judged previously by the
chief reader and the assistant chief readers as
representative of particular categories of
.competence. There is a- good paper, a merely
above-average paper, a just-below average paper,
a poor paper, and perhaps one or two others.
Readers are asked to rank the papers. When the
readers have completed that task, the chief
reader announces the score that the table
leaders, the assistant chief readers, and the
chief reader have given each paper. It is up to
each reader to compare the way he or she ranked
the papers with the way they were ranked by
experienced readers. . . . They do not score the
essays against an ideal standard, but against the
standard established by the actual performance of
students writing on this particular topic (Conlan
3-5) .
There is, then, pressure toward consistency of
scoring. ETS seeks to motivate that movement internally in
each scorer. Moreover, there are sanctions against anyone
who does not fall in with the group.
104
When there is a great difference between this
score and the score the readers have given, the
chief reader will explain the reason the
experienced group gave that particular score.
The chief reader does not prescribe the scores;
he or she,offers guidelines, in the light of
experience. It is the consensus of the readers
themselves that determines the standards. Always
during this process, you as an individual reader
are responsible for noting in what category the
particular score you awarded the paper fell. It
is necessary for all readers to adjust to the
standards of the majority. There can be no
exceptions if the papers are to be scored as
nearly as possible in the same way. An
individual reader who assumes that he or she has
the right to set individual standards is being
unfair to the candidates whose papers he or she
will read. . . .
The process of scoring samples, of reporting
scores by a show of hands, and of tallying these
scores continues until the.chief reader is sure
that the readers are in close agreement, that
they have a clear understanding of the bases for
judging papers, and that, 'insofar as is humanly
possible, all papers will be scored in the same
way. The chief reader then announces that the
actual scoring of papers is to begin, even though-
the table leaders may continue to work
individually with a few readers who are still not
scoring with the group. In order to ensure
reliable results, these readers' scores must be
brought into line with those of the other
readers.
From time to time throughout the reading,
readers will be given: other sample papers to read
and score. This continuous reading of samples,
as well as the numerous rest periods, helps
readers keep to the standards they have set
(Conlan 6-7).
The above describes two different kinds of
"reliability." The consistency from scorer to scorer on
the same essay is called "inter-rater reliability," while
105
the consistency of a single reader, over time is called
"intra-rater reliability." Actually, the single reader
does not re-read the same essay across time to check his
intra-rater reliability. Rather, this is checked by the
periodic group readings of given essays to ascertain that
all are still rating with the group. There still remain
several other important aspects of reliability and its
counterpart in testing, validity. These will be discussed
in Section Two of this chapter. They include "instrument
reliability" (how well various forms of the testing
instrument compare), and "content validity" and
"instructional validity" (how well the test measures what
was presumably taught).
While this process seems arduous, it is also in its
sheer numbers an amazing human feat: Bishop cites a mass
reading of 85,000 student essays, taking 6,000
reader-hours. With over two hundred readers participating
in the reading, the.whole process took only a few days.
For better or worse, that means each essay reading took
seventy seconds (Bishop 11).
The ETS practices go some distance toward implying a
rhetorical framework, that is, a description of the roles
of writers, texts, and readers. Ethos, as the projection
of the author (relevant to them only as the basis for
106
indicating writing ability) is limited to evidence from the
text itself. They only reluctantly use writing samples at
all, since they have not been able to "measure the
reasoning and organizational aspects of good expository and
creative writing independently of writing itself." Having
accepted their reliance on writing samples, though, they
look only at the sample itself: ETS "views a piece of
writing as a total work"--the text is the totality of their
interest. This position indicates their hopes for
objectivity through a "de-contextualized" text. The
requirements for texts sought also exhibit elements of a
rhetorical framework, in particular, values about what
texts should be like. They say that texts should be
"expository," which they essentially equate with "good
writing," though they end up with a definition that is both
more arid different from the innocuous and vague term "good
writing." They seek, through careful topic selection and
rater-training, topics that are unemotional, logical, and
essays expressed in "standard" idiom,, diction, tone, and
style. This "genre" sought is notable in two ways, first
because it is not very well defined and second, because it
invites the detection and rejection of responses (writers’
and readers' responses) that are at all unconventional.
' ' ■ -
ETS says about readers only that they "are in general
agreement about what constitutes good writing," which
forecloses inquiry on the subject. Evidence of "deviance"
in scoring simply means ,the reader is not reading
correctly, and he is retrained,until he begins to read
"correctly": score consistent with the others. Writers'
responses, too, are considered controllable within narrow
limits: "Certain essays, such as those that are off the
topic, illegible, or obscene, should be automatically given
to the table leader. So should essays that are really
poems, short stories, or plays" (Conlan 3). Conlan does
say that such papers "should be scored according to the
scoring policy set for them," but the policy is not named.
It seems that a genuinely "different" response is, in all
fairness to their system, genuinely ungradable. Thus,
their rhetorical framework is strongly restrictive, seeking
fairness through uniformity of writers, texts, and readers,
though relying on a tacit sense of what authorial, textual,
and reader-response possibilities are.
3.1.2 National Assessment of Educational Progress
As I move now to a brief summary of the, practices of
the National Assessment of Educational Progress, I will
note some differences between ETS and NAEP. Yet these
differences are in the main "institutional," having more to
108
do with their public-relations profile, than "essential,"
the actual methods of gathering and reading the writing
samples. The institutional differences (while they are
striking) serve mainly to. show that in fact writer
anonymity on the one hand, and the "leveled" reader on the
other that inter-rater reliability demands and creates, are
in fact practices that can be transferred quite intact.-.from,
one-institution to another.
To be sure, NAEP is quite a different institution. It
is more than a half-century younger, having been founded in
1967. Second, NAEP is immeasurably more "public" in its
willingness to shares its practices, rationales, and test
"results," including statistical findings, actual test
items, and actual essays. This certainly grows from its
different nature, which is
an education research project . mandated by
Congress to collect and report data, over time,
on the performance of young Americans in various
learning areas. . . .
After assessment data have been collected,
scored and analyzed, the National Assessment
publishes reports and disseminates the results as
widely as possible (NAEP, Procedural Handbook ii,
vii). ~~ ~ ~ ' ;
From its inception, NAEP has gathered information
about levels of educational achievement in art, career and
occupational development, citizenship, literature,
109
mathematics, music, reading, science, social studies, and
writing. In each achievement area, three age groups have
been periodically assessed: nine-year-olds,
thirteen-year-olds, and seventeen-year-olds. NAEP has
assessed writing three times, in 1969-1970, 1973-1974, and
1978-1979. NAEP is currently tabulating the results of a
fourth assessment of writing gathered in 1983-1984.- By
1981, 900,000 students had been interviewed and tested. In
the 1978-1979 writing assessment, 22,500 nine-year-olds,
30,500 thirteen-year-olds, and 27,500 seventeen-year-olds
were tested. The students who were chosen to participate
in each assessment were carefully selected to represent the
entire nation's student population. That is, census-like
criteria of age, sex, geographical, and socio-economic
distribution were taken into account such that the scores
that eventually emerged could be generalized to the nation
at large. As NAEP tells us in their Procedural Handbook,
"on the basis of the performance of about 2,500,9-year-olds
on a given exercise, we can make generalizations about the
probable performance of all 9-year-olds in the, nation"
(viii).
Immediately, we can see the character of NAEP
emerging. Far different from ETS, which seeks relative
scores on individual students so that admissions directors
110
can select candidates for college entrance, NAEP seeks to
assess general "attainment levels" for American students in
a variety of skill - areas. But the people who develop,
administer, and process the results of the assessments, are
either ETS or ETS-influenced staff. This has been the case
in the development of "writing objectives" for NAEP, all
the way through the process, in naming the principles of
item development, to developing the scoring rubrics, to the
rater-training, to the reading process which NAEP adopted.
There were two main writing objectives.developed for the
first round of assessment (1969-1970): ETS developed them
both. One objective was "correctness,," the other was that
the student reveal his sense of the importance of skill in
writing. By the second and third rounds, there were three
objectives: ETS developed two of them. The remaining one,
which related to creative or personal writing, came from
other consultants. As if this were not enough to establish
a great ETS influence on NAEP'from the very beginning, the
NAEP Newsletter recently carried this notice:
As of July 1, 1983 ETS assumed responsibility for
administering the National Assessment of
Educational Progress. It won the $19 million
grant award following competitive procurement
from the National Institute of Education, NAEP's
funding agency. This ended NAEP's 14-year
relationship with the Education Commission of the
States (ECS) ("NAEP Grant Awarded to ETS" 1).
So we can see that ETS, already the largest and oldest
institution using mass, anonymous assessment of student
texts, has taken an even more centralized role in such
assessments. We might expect, then, the methods of such
assessment to become ever more legitimized as their
monolithic control becomes more pronounced.
NAEP differs from ETS in an additional important
aspect. The ETS reluctance to use writing samples as an
indicator of writing ability is not found at NAEP. "Since
National Assessment staff and consultants feel strongly
that writing performance should be assessed on the basis of
writing samples rather than objective tests, most of the
writing exercises are open-ended, requiring students : to
produce a piece of writing" (NAEP, Procedural Handbook xi).
As an example, the seventeen-year-olds in each of the three
assessment periods were given the following item.
"Describe Something" Exercise
Everybody knows of • something that is worth
talking about. Maybe you know about a famous
building like the Empire State Building in New
York City or something like the Golden Gate
Bridge in San Francisco. Or maybe you know a lot
about the Mormon Tabernacle in Salt Lake City or
the new sports stadium in Atlanta or St. Louis.
Or you might be familiar with something from
nature, like Niagara Falls, a gigantic wheat
field, a grove of orange trees, or a part of a
wide, muddy river like the Mississippi. There is
probably something you can describe. Choose
something from around where you live, or
something you have seen while traveling, or
something you have studied in school. Think
112
about it for a while and then write a description
of what it looks like so that it could be
recognized by someone who has read your
description.
Name what you are describing and try to use your
best writing (NAEP, Writing Achievement 63).
Each student in the sample wrote on this topic for a total
of 15-25 minutes. (It should be noted that NAEP has
through the years assessed writing not only through essays
but through objective questions as well, and has also, put
those essays through a number of assessment modes, not just
holistic scoring. Some of those non-essay tasks have
included sentence-combining exercises and cloze tests.
Also, some of the other-than-holistic scoring methods which
have been applied to the essays have included ratings of
"T-units," coherence, mechanics, use of tense, point of
view, tone, and revision. It is still the case that actual
essays have always been their most valued source of
information on student writing, and that holistic scoring
of those essays have always been their major tool for
evaluating those essays.)
A summary of the training of raters and the holistic
reading of essays would be so similar to the earlier
summary of ETS as to be redundant. Suffice it to say that
NAEP took on ETS's methods virtually intact and that they
proceeded to use those methods with a confidence that
suggests that these methods are no longer possible objects
of controversy.. The very omission of any of those
ETS-style rationales in the voluminous NAEP publications is
significant in that these methods are now assumed to be
obviously reliable and valid.
However, NAEP did develop an interesting variation on
holistic scoring during the . more recent rounds of
assessment. This was to become known as Primary Trait
Scoring, credited to Richard Lloyd-Jones. NAEP was unhappy
that holistic scoring yields only simple rank-ordered
scores without regard to the kind of writing demanded by
the exercise (e.g., descriptive, persuasive, expressive).
NAEP felt that holistic scoring could be refined to address
the differences inherent in various kinds of writing tasks.
In the spring of 1972, NAEP staff held a
scoring conference of writing educators and
measutement specialists. . . .The result of this
conference was the beginning of the Primary Trait
System for evaluating responses to writing tasks.
. . They noted that a normal distribution
should not be imposed, that the definition of
quality should vary with the different tasks
rather than be uniform. , . . A dominant or
primary characteristic would be identified for
each writing task and papers for that exercise
would be rated according to how well they
fulfilled the purpose of that particular task.
It was noted that appropriateness to audiences
must be considered in any evaluation of writing.
• • •
The rationale underlying primary trait
scoring is that writing is done in terms of an
audience and can be judged in view of its effects
upon that audience. The approach used by the
writer to reach and affect his audience will be
the most important--the primary--trait of a piece
of writing (NAEP, Procedural Handbook 7-8).
For the first time in either the ETS or NAEP
documentation, then, is the rhetorical situation directly
confronted as an issue. This is the first consideration of
the implications of the fact that a writing act takes place
in a unique, real context in which writers and readers
communicate with specific purposes, knowledge, and
constraints.
The essence of the Primary Trait System is to
narrowly delineate the situation for the writer
by defining the variables. This means that for
each exercise three things must be specified: 1.
the identity of the writer (whether the
respondent is himself or is given a role to
play); 2. the audience (who the writer is writing
to); and, 3. the subject matter (what the writer
should communicate to the audience (Mullis
13-14).
Here, I will only summarize the NAEP guidelines for
creating exercises in line with Primary Trait Scoring. - It
will be later, in Chapter Four, that I will fully develop
the rhetorical situation implicit in their methods.
A carefully constructed testing situation
provides opportunities for respondents to
demonstrate their ability to choOse and
effectively carry out appropriate rhetorical
strategies, but the effects of the strategies
cannot be measured by actual success. For
example, it is not feasible to have respondents
write real job applications and then check to see
115
if they were hired. An evaluation of the writing
has to be based on its 1ikelihood of achieving
the desired effect. The features that would
contribute to this success must be identified and
defined in terms of their importance. These
definitions then become the scoring criteria. . .
questions and rhetorical situations should be
phrased in a way that all respondents would give
comparable responses (Mullis 9).
These, then, are the kinds of traits that can emerge and
become scoring criteria:
For example, the writer of a set of directions
must present facts in a logical and unambiguous
manner if he expects a reader to follow the
directions. Therefore, the primary trait of a
written set of directions would be unambiguous,
sequential and logical progression of
instructions. Successful papers will have that
trait, unsuccessful papers will not, regardless
of how clever or well written they may be in
other respects. On the other hand, the- purpose
of campaign literature is to persuade a reader to
vote for the candidate.• A successful campaign
paper will have certain persuasive traits that an
unsuccessful one will not have, and these traits
will differ from those necessary for a successful
set of directions (Mullis 8-9).
Thus, Primary Trait Scoring, a variation of holistic
scoring, likewise is made to yield a score of 1 to .4. It's
just that now the raters are trained to score holistically
with one criterion most prominent in their minds. Finally,
as at ETS, these mass readings are marvelously efficient:
"NAEP estimates that one reading for primary trait takes
about one minute for a paper written by a 9-year-old, one
116
and one-half minutes for a paper by a 13-year-old, and two
and one-half minutes for a paper by a 17-year-old" (Mullis
19).
The ETS and NAEP practices so far described may seem
to have little connection with recent (or even past)
academic, scholarly work in composition. Indeed,, this is
largely true. Lloyd-Jones’s Primary Trait Scoring is the
clearest example of the work of a noted composit ionist
beings directly incorporated into .the testing practices.
The other "theoretical" influences of composition . on
institutional testing are tenuous and indirect.' In fact,
more influence runs the other direction. ‘ Some scholarly
work on writing assessment explains and rationalizes ETS or
ETS-type tests, rather than laying the theoretical
groundwork on which practices are then based. This work
merely manipulates scores given to atomistic parts of a
given composition, comparing one element with another.
Typical empirical work matches lexical choices (e.g.,
simple vs. nominalization), against raters’ scores, or
"T-units" against age or sex. (See Bailey, Bamberg,
Crowhurst, Grobe, Nielsen, White, and Witte and Faigley for
representative work.) In this work, no general inquiry is
made into the nature, character, or description of the
process of the response itself, whether writer's or
117
reader's response. Scores are simply obtained and then
compared to yield correlations of various kinds. . More
relevant to my purpose here, little of this work has had a
direct influence on the testing practices of ETS and NAEP.
Test development, construction, and administration have
proceeded in relative isolation from the academic work that
(supposedly) would explain and justify it.
Others, especially Charles Cooper,' Lee Odell, and E.
D. Hirsch, Jr., speak from a more theoretically-grounded
position and have developed or explained actual assessment
techniques. Still, with the exception of holistic scoring
itself, which they embrace but did not invent, little of
their work guides ETS and NAEP practices. Instead, Cooper,
Odell, and Hirsch offer variations on standard assessment
techniques that could conceivably be incorporated into ETS
and NAEP practices at some point. They all emphasize the
composed product (the text) rather than the composing
process. Textual features are the focus, not the writers
or readers of these texts. Hirsch says, "In this
preliminary report we describe the theoretical basis for a
new method of measuring the quality of prose" (189). Note
that he looks not at the quality of a writer, nor at the
quality of a reader's response to the writer, but at the
quality of the prose itself. In Cooper and Odell's
118
Evaluating Writing, Cooper summarizes a number of holistic
evaluation techniques: the essay scale, the analytic
scale, the dichotomous scale, feature analysis, Primary
Trait Scoring, general impression marking, and the. "center
of gravity" technique. But in all cases, textual features
are the source of the score, arrived at by a group-guided,
rubric-based consensus. These compositionists: refer to,
rather than advance, rhetorical theory. For example,
Cooper calls Primary Trait Scoring a "practical application
of new rhetorical theories" (Evaluating Writing 11), but
doesn't say which ones. Similarly undefined is his phrase
a few pages later, "current discourse theory persuades us
that. . ." (14). Finally, he finds "theoretical reasons to
believe. . . (20) [that writing tasks should take into
account a speaker's role, an audience, and a purpose], but
he doesn't say which theories or what reasons he means.
The use of an actual text over multiple-choice questions is
what makes the situation "rhetorical" for Cooper--other
tests of writing skills are "a-rhetorical" (The Nature and
Measurement of Competency in English 12). Primary Trait
Scoring captures, for Cooper, a "full rhetorical context"
(Evaluating Writing 11), though the term is left undefined.
Whatever psychological, valuational, or epistemological
dimensions that context might have are ignored. When'
119
Lloyd-Jones, Cooper, and Odell do refer to rhetorical
theory, it is usually in terms of modes (narrative,
descriptive), functions (expressive, persuasive), or roles
(self, other). For this reason, their debt seems to be
mainly to Roman Jakobson, James Rinneavy, and James
Britton, but only in the use of some of these theorists'
categories and terms.
As a rhetorical framework, then, the work of Hirsch,
Odell, and Cooper is not antagonistic to the implicit
rhetorical framework of anonymous assessment. The emphasis
is on the text itself, and the mass, anonymous nature of
institutional assessment is not in conflict' with these
theorists. But their views, and the implied rhetorical
framework of anonymous assessment their .views are
compatible with, force certain questions relative to the
modern-classicist contextualist framework I developed
earlier. For example., how can we establish ethos as the
author's writing ability when we take the text for the
author? What can we say about the interpersonal nature of
a discourse transaction (ethos in the broad sense) when we
only assume the rhetorical context without defining it?
[How can we say the situation is rhetorical at all if we
don't say how?
120
3.1.3 Freshman Writinq Proqram (University of Southern
California! --- ------•-------
I will summarize the assessment methods of the
Freshman Writing Program mainly to demonstrate the
pervasive impact that ETS and NAEP have had on the way that
writing samples are gathered, interpreted, and.used. Any
number of other college writing programs would provide an
equally gOod example. (See Bishop's Focus 5: The Concern
l££ Writinq, which details the many school programs which
have been influenced by ETS.) Still, the presence of a
fair amount of documentation by FWP makes them a good
candidate for my purposes here. This summary will also
show by example how readily such methods can’ be adapted
from huge institutions like ETS and NAEP to a much smaller
scale.
ETS and NAEP methods have saturated the assessment of
student writing. What is most striking in the FWP
documentation is the complete absence of those rationales
for anonymous holistic scoring that were so prominent in
the ETS literature. The elaborate discussions of, for
example, correlations between holistic scores and
objective-question scores, or of the merits of holistic vs.
analytic scoring methods, are not to be found here.
Likewise absent is the defensive posture regarding the use
of writing samples as measures of writing ability. While
121
ETS and NAEP felt the need to justify the validity and
reliability of writing samples, it is as if for FWP, the
argument is now moot* The main document, the Guide to the
Freshman Writinq Proqram, deals with the procedures for
administering and processing the writing samples. It is
nowhere said why a single, time-limited writing sample is
to be preferred over any other way of determining the
writing ability of the student. Again, the point seems to
be that the choice-of holistic scoring over analytical
scoring, and essays over multiple-choice questions, have
become the "industry standard," What the Guide does take
up falls into two areas: the considerations for choosing
the.test items, and the specific procedures for rating the
essays. This last importantly includes training the
readers toward inter-rater reliability, in order that they
can holistically score the essays. The holistic scoring
method and the concept of inter-rater reliability are by
now standard practice. In fact the very absence of
explanations or rationales suggests both that such
practices are well-known and that they are no longer
possible objects of controversy. * :
Of course, the use of writing samples by FWP does
differ from those of ETS and NAEP in that the FWP essay is
one component of a student's graded performance in the
122
writing program. Students take'the examination for various
purposes. Those enrolled in regular sections of the course
take the examination as part of their final course grade.
Their numerical score is converted into a letter grade and
becomes one-third of their course grade (their instructor
assigns the remaining two-thirds). Other students, wishing
to use the writing sample as a "challenge" exam or a "skill
level" exam (those wishing to be exempted from USC’s
writing requirement), take the exam for those purposes.
Otherwise, FWP draws on methods developed by ETS and NAEP
throughout the phases of item selection, item pretesting,
rater-training, and essay-reading. As with ETS and NAEP,
potential essay questions are sought from "experts in the
field." Once nominated, the possible topics are screened
in light of these guidelines:
1. Will all students, including non-traditional
students and international students, have
something to say in response to it?
2. Will all students be able to..: find specific
examples to use?
3. Will some responses be unduly emotional or
patriotic or religious or in some way difficult
for readers to judge fairly?
4. Will the topic allow readers to make quick
judgments between essays at different levels of
competence?
5. Will the topic produce mediocre responses?
6. Will the topic allow the best students to
demonstrate their ability?
7. Does the topic encourage cliche-ridden or
"canned" responses?
8. Does the topic invite the candidate to try to
guess at what attitude the readers want them to
take toward the topic?
123
9. Does the topic fail to imply methods of
organization?
10. Does the topic confuse students?
11. How will readers react to the whole spectrum
of responses?
12. Will the responses be just plain boring?
13. Can judgments be made quickly on reasonable
grounds?
14. Would you be willing to read essays on this
topic for two days (Guide 40-41)?
These guidelines virtually duplicate those that ETS uses in
the topic selection phase regarding the acceptability of
the topic for the writers and for the readers (Conlan 1-2).
Then, potential items are pretested. Next, the pretest
essays are read, and decision are reached based on how well
the foregoing guidelines have been met. #
FWP for some time gathered two essays from each
student during the examination,, The first was a shorter,
35-minute "personal" essay, the second a longer, 70-minute
"analytical" one. But in the fall of 1983, the "personal"
essay was dropped in favor of the "analytical" essay only.
The qualities sought in this second, "expository" essay are
those ETS seeks. MOst important are precision, logic,
clarity, formality. This is pertinent here because this
brings FWP practice even closer to that of ETS, which
likewise avoids topics that are at all narrative or
descriptive, staying instead with clearly expository
essays.
124
As we might expect, then, the raters are trained and
the essays read in the way that is quite familiar from the
practices of ETS and NAEP.
The chief readers look through the essays to find
essays typical of each point in the range of all
possible freshman essays. . . Copies are made
of the sample essays for the training of the
table leaders. The table leaders are given
groups of sample.essays and asked to assign them
scores. Scores are compared after each batch of
three or four essays. Differences in scoring are
discussed. Some of the sample essays are
rescored after this. Copies of the sample essays
are made for the training of the rest of the
instructors. . . . The morning is devoted to
training of the instructors by the table leaders.
. The reading of "live" books" (previously
unread examinations) begins after lunch. . . .
The table leaders, from time to time, pull sample
essays from the already read groups at their
table, re-read them, noting on special forms any
wide deviations in judgment between themselves
and their readers. These forms are collected by
the question leaders, who scan them for evidence
of excessive deviance. Tables drifting as a
whole into these deviances are retrained by the
question leaders. Individuals who • develop
deviance in isolation are watched carefully for
further lapses, and if they continue deviant they
are pulled off their tables for use as runners or
waiters (Guide 45-47).
Finally, the numerical scores are converted into letter
grades.
125
3.2 PSYCHOMETRIC ASSUMPTIONS OF WRITING ASSESSMENT
If we are to understand the rhetorical context of
mass, anonymous assessment of student texts, we must probe
deeper than a mere summary of the procedural methods of the
testing institutions will allow. The institutional
documents of ETS, NAEP, and FWP do of course describe what
it is they do: how the essay topics are chosen, how the
writing samples are gathered, how the raters are trained,
and what the consequences of the examinations are.
Additionally, those documents sometimes include objectives
for writing tests and may even name qualities of "good
writing."
These institutions make it plain that they wish to
proceed in scientifically principled ways. Yet very often
the tests’ psychometric principles, which have been derived
from theories of educational -testing and measurement,
remain implicit. They are either unacknowledged or stated
incompletely.' But since the rhetorical situation of the
writers being assessed is characterized just as much by the
implicit assumptions of the psychometric concepts as it is
by the explicitly stated conditions of the assessment
procedures, we must thoroughly understand and articulate
them as well. This part of the chapter on the methods of
assessment will, therefore, take up the second of the two
assumptions I mentioned at the beginning of this chapter:
that these assessments are "good science." Three critical
aspects of this assumption that I will take up in turn are
1) these practices as an application of the psychology of
individual differences, rather than the psychology of
learning; . 2) these practices as a form of psychometric
"counting," and the level of abstractness of that counting;
and 3) the "fit" between the science of educational testing
in general and its application to writing samples in
particular.
As a beginning, some textbook definitions will
distinguish among the terms "assessment," "testing,"
"measurement," and "evaluation," which I have so far used
interchangably, since ETS, NAEP, and FWP do so, David A.
Payne's The Assessment of Learning: Cognitive and
Affective, addressing as it does the whole of educational
testing and measurement, brings a. broader perspective
needed here.
The use of the term assessment is similar to
Bloom's (1970) description of systematic
approaches to the description of relationships
between selected task requirements, criterion
behaviors, and the environment. The inclusion of
the environmental element distinguishes
assessment from other tasks and activities, such
as measurement and testing, associated with
evaluating teaching-learning situations.
Assessment concerns itself with the totality of
the educational setting, and it, is the more
inclusive term, i.e., it subsumes measurement and
evaluation. It focuses not only on the nature of
the learner, but also on what is to be learned
and how (5) .
This definition significantly limits the appropriate
terminology for the practices of ETS, NAEP, and FWP. They
are not actually "assessing” students, since they are
ignoring the environmental factors. These tests do not
take into account the student's environment in which he or
she supposedly learned the skills being tested (in the
past) nor do they consciously inquire into the environment
in which he or she . writes the essay (the immediate
rhetorical situation). As much as anything else* then,
these tests must be considered "measurement,” since they
are a method of assigning numbers to essays. This
disclaimer is important because it limits the
interpretations and conclusions that may be made from such
tests.
The first critical psychometric assumption is that
such "measurements" (like all similar educational tests)
are based on a psychology of individual differences, not a
psychology of learning. In that such "measurements" yield
a ranked, numerical score for each essay, they clearly seek
to differentiate one student from another based on
perceived differences. This is the sorting function Tyler
referred to in Section One of this chapter. However, while
128
the scores of all such educational achievement tests reveal
their basis in a psychology of individual differences, the
act of scoring the essays in the ETS, NAEP, and FWP
"assessments’ * is contaminated by the psychology of.
learning. We can see this in the following way. The
distinction between a psychology of individual differences
and a psychology of learning, when applied to modes of
testing, emerges as "norm-referenced" vs.
"criterion-referenced" tests. In their chapter in Tyler's
book, Peter W. Airasian and George F. Madaus contrast the
two modes.
For many years student performance hais been
graded on a norm-referenced, relative basis. A
student’s grade is assigned on the basis of how
he stands in comparison to his peers, not on the
basis of any absolute criterion of what his
performance is worth. - . . . [in
criterion-referenced measurement] emphasis is
placed upon the question "What has the student
achieved?" rather than upon the question "How
much has the student achieved?" . . . Both norm-
and criterion-referenced systems sort students,
but there is an essential difference. In
criterion-referenced measurement, interpretation
of a student's performance is in no way dependent
on the performance of his classmates"
("Criterion-Referenced Testing in the Classroom"
73-74).
On close inspection we can see that’ while the ultimate
score of writing samples such as those gathered by ETS,
NAEP, and FWP appear to be norm-referenced and relative to
129
the group, the scoring process itself contains elements of
criterion-referenced tests. These elements originate in
two characteristics unique to holistic scoring, as
distinguished from (say) machine scoring.
First, readers are indoctrinated, sometimes subtly,
sometimes explicitly, regarding qualities * of "good
writing.". That means that they are not just "reacting" to
the text (as a machine "reacts" to the correct answer in a
multiple-choice test). Rather,, they are interacting with
the text relative to their ideas of "good writing." While
much is done to "level" the readers to the extent that each
will give more or less the same score, each reader still
varies somewhat from every other reader in those
interactions with the text. They are told, in effect, to
look for a "genre," insofar as a genre is a set of
expectations. This particular genre has definite
boundaries: the essays are constrained by type of topic,
accepted idiom, tone, style— in short, the genre is as
specified as any other literary genre, such as the epic
poem or the mystery novel. For example, as we have seen,
the readers are told to expect writing that is "expository"
in mode, and specifically told not to expect (nor, we would
suspect, welcome) writing that deals in the "depiction of
character, or the establishment of mood or atmosphere."
. 130
They are told to expect cool, rational writing that does
not reflect "an emotional response on the part of the
candidate." Moreover, the student writer is placed by the
essay topic in an imaginary rhetorical situation which has .
been stipulated in advance. There is thus a purported
rhetorical situation (e.g., ".Imagine you are writing to the -
president of a company-to complain about a faulty product")
and a dominating, "real" rhetorical situation--the test.
The point is that the rhetorical situation that has been
stipulated in advance of the writing, task is the one which
the raters imagine themselves to be evaluating. In fact,
thg writing will arise from the rhetorical situation the
writer is engaged in when he actually writes, not
necessarily the one predicted by the rater. Whether there
are important disjunctures between the context predicted by
the rater, and the context as actually experienced by the
writer is a strictly empirical question. No matter how
clever the test-makers become at characterizing the
rhetorical situation, they are always doing so from a
predictive, reader-oriented point of view, when the
rhetorical situation of the writer is supposedly the object
of inquiry. The reader, then, is interacting with the
essays through a network of expectations. These
expectations range from tone to mood to diction to length
131
of the essay to supposed qualities of the rhetorical
situation. Beyond genre constraints explicitly named by
the chief reader, the would-be readers are indoctrinated by
example in all the "practice" papers they read: they come
to expect certain organizational patterns in the ,essays,,
even appropriate kinds of examples that the student will
bring to bear in support of his arguments.
The second characteristic unique to holistic scoring
is that each scoring session creates standards of quality
relative to the group of papers read. We have heard over
and over again the assessing institutions caution that the
scores have meaning relative only to the group being
assessed. But the norms in this case are ad hoc criteria
acted upon uniquely relative to each scoring session.
Still, just because the assessments are in these two ways
criterion-referenced,’ and thus allied more to the
psychology of learning than to the psychology of individual
differences, we should not think that the "learning" is on
the part of the writer. It is just the opposite. Assessed
anonymously and given no feedback from the raters, the
writer is not the object of learning. It is the readers
who learn. They respond to what they have learned about
"good writing," especially during the training period.
This argument will be picked up and developed later, in
132
Chapter Four, as one aspect of what creates the ethos of a
text. Ideally, we think of the ethos of a text as the
quality of the- projected author of a discourse. But the
foregoing suggests that something quite different is going
on in these anonymous assessments. The readers, purposely
deprived of the ability to build a sense of the writer are
forced, having only the text to go on, to respond to only a
truncated mental picture of the author. The result is that
these tests are thought to be an application of the
psychology of individual differences, while in fact they
contain elements of the psychology of learning. The
consequence, especially because this mixed mode is
unacknowledged, is that the assessments are not as purely
mechanical as they believe them to be.
The second psychometric assumption of mass, anonymous
writing assessment is one that is common to all educational
measurement. However, when applied to holistic scoring of
essays, the assumption takes on a complexity that imperils
interpretations made from essay scores. Payne, in The
Assessment of Learning, distills this assumption of
educational measurement. "all definitions [of educational
measurement] involve the systematic assignment of numerals
to objects and events. . * . Measurement is . . , the
comparison of something with a unit or standard amount.or
quality of that same thing, in order to represent the
magnitude of the variable being measured" (6-7). This
psychometric assumption, when applied to writing samples,
means that a single number is made to represent the whole
essay. It is one thing for a numerical score to correspond
to the number of objectively correct answers, in an
examination. It is quite another for a single number to be
made to stand for an entire essay. ' Still, this is a
practical necessity of such testing. It would- be wise to
keep in mind, though, the radically higher level of
abstraction that a numerical score on an essay represents
over a numerical score of correct objective questions. We
can see the degree to which this is so by comparing the
current holistic scoring method with the older, analytical
method. There, attributes such as spelling, grammar,
organization, and phrasing were each given subtotal scores,
and the score for the whole text was simply the total of
those scores. Compared to analytical scoring, in which the
components of the score stand for more or less precise,
consistently-recognizable attributes, the single numerical
score of a holistically-graded essay must bear the whole
burden of representing the quality of the essay, contained
in a less precise form. Just as seriously, the educational
testing and measurement, literature slights those occasions
134
when the writing itself is the object of the test. Typical
treatment of "the essay" takes "the essay" to mean only a
prose answer to what is essentially an "objective" question
(Payne 145-46).
This dearth of literature on the essay as an essay
leads to the third psychometric assumption that grows from
the assessors overall belief that they are practicing "good
science." That assumption is that the principles of good
test construction acknowledged by the social sciences is a
description of what they are doing in their assessments.
But Payne's list of Ebel's ten qualities of a good test
reveals a rather poor "fit" with the practices we have so
far described, leaving the conclusion that there is not as
much "hard theory" to support these practices as has been
assumed. Here, then, is Payne’s list.
1. Relevance
Relevance is the correspondence between the
behavior required to respond correctly to a test
item and the purpose or objective in writing the
test item. The test item should be directly
related to the course objectives and actual
instruction. When used in conjunction with
educational measurement, relevance must be
considered the major contributor to validity.
2. Balance
Balance In a test is the degree to which the
proportion of items testing particular outcomes
corresponds to the "ideal" test. The framework
of the test is outlined by a table of
spec i f ications.
135
3. Efficiency
Efficiency is defined in terms of the number of
responses per unit of time. Some compromise must
be made among available time for testing,
scoring, and relevance.
4. Objectivity
For a test question to be considered objective,
experts must be able to agree on the "right" or
"best" answer. Objectivity, then, is a
characteristic of the scoring of the test, not of
the form (e.g., multiple-choice, true-false) of
the test.
5. Specificity
If subject-matter experts should receive perfect
scores, test-wise but course-naive students
should receive near-chance scores, indicating
that course-specific learnings are being
measured.
6. Difficulty
The test items should be appropriate in
difficulty level to the group being tested. In
general, a maximally reliable test is one in
which each test item is passed by half the
students.
7. Discrimination
The ability of an item to discriminate is
generally indexed by the difference between the
proportion of good (or more knowledgeable
students) and poor (or less able) students who
respond correctly.
8. Reliability
Reliability is a complex characteristic, but
generally involves consistency of measurement.
Consistency of measurement might be judged in
terms of time, items, scorers, examinees, or
examiners.
9. Fairness
To insure fairness, an instructor should
construct and and administer the test in a manner
that allows each student an equal chance to
demonstrate his knowledge.
136
10. Speededness
To what degree are scores on the test influenced
by speed of response? For achievement tests,
speed should generally not be allowed to play a
significant role in determining a score, and
sufficient time should generally be allowed for
all or most of the examinees to finish the test
(19-20).
The assumption of these writing assessments is that
they proceed from principles of good test-making. But many
of the above guidelines are simply not applicable to
holistic scoring of writing samples. The idea of
correctness referred to several times by Payne cannot be
made to apply to writing samples which are not right or
wrong, but are judged qualitatively by the readers. The
binary quality of right vs. wrong is inapplicable to
writing samples, and it is problematic what effects this
will have on interpretations made from holistically scored
essays. This puts the qualities of "relevance" and
"efficiency," which rely on definite right and wrong
answers, into question. Second, the concept of an ideal
answer is likewise not a practical possibility in these
open-ended writing samples. The demand for an ultimate or
perfect answer is impertinent in a scoring system that is
relative. So "balance" and "objectivity" become
questionable. As we have just seen, testing theorists feel
that it is not only possible but necessary to construct
137
test that uses the essay form as merely a student-generated
prose answer to'.objective questions. Little comment comes
from the testing theorists about handling essays when the
text itself is the object of interest. Third, the quality
of "specificity” depends on a knowable, nameable quantity
of knowledge which is present to be tested. Since, as we
have already seen, the past environment is quite
uncontrolled in such tests, "specificity” too is irrelevant
to these tests. Brought into question along with
specificity are "difficulty" and "discrimination," since it
is unknown how much or how little the student knows going
into the test. Finally, "speededness" is at issue because
of the rigid and arbitrary time constraints of the tests,
which range from only fifteen to seventy minutes. Of the
remaining qualities, writing samples are in a sense "fair"
since all students are allowed an equal opportunity to.
demonstrate their knowledge, at least insofar as each
student takes the same .test. But it is by now quite cloudy
what is being tested, and how confidently interpretations
can be made about what the student writes.
So the final quality is "reliability," which will be
discussed now at length with "validity," a quality which
grows out of all ten rules taken together. These two must
be treated simultaneously because of the unusual
138
relationship these psychometric terms have when applied to
writing samples. Validity describes the degree to which a
test actually measures the behavior in question. It has
always been a source of embarrassment for ETS that their
objective tests of ~ writing skills in fact predict college
grades better than writing samples. That is, the most
reliable (highly correlated) predictors of college grades
in English were non-writing sample tests. But we have
already noted in Section One of this chapter the unusual
definition of "validity" that ETS uses. It is true that
having a student produce a piece of text is a valid
indicator of producing a text, but-we; know that validity
means whether the test actually assesses the behavior in
question, and the behavior in question is the ability to
write. To establish that validity as ETS does by recourse
to other measures (later college grades or scores on the
objective portion of the test) is not sufficient. What is
necessary is to look beyond the text in isolation. Only
by establishing what the student did, given the possibility
to make choices other than he made, could comparisons be
drawn against which a definition of "ability" could be
established. Then, the validity of the test could be known
in relation to that ability. One way to see this has
already been mentioned. The test-makers are in . the
139
competence-performance bind. They claim to assess
competence (an idealized model of the student's ability to
write) on the basis of a single, highly limited
performance. Even ETS1 Paul Diederich, whose Measuring
Growth in English provided the original basis for these
assessments, said that more than one sample was needed.
While it is not possible to measure lack of competence
(because a given test may simply fail to properly access
something that is there), one can measure good-performance
and infer competence from it. Further, . one can draw
reasonable conclusions from multiple failures. about
inability to access (use) any possible competence. Either
way, definitions of competence would necessarily be
operational definitions chosen to define competence in a
given way. Then, disjuctures between that definition of
competence and the performance would suggest definitions of
writing ability and ways of proceeding to measure that
ability. The problem is that anonymous assessment ignores
1) the competence-performance distinction, 2)the
performance of the (unobserved) writing task, and 3) the
fruitful possibilities to be explored in the relationship
between an actual performance and an operational definition
of competence. These problems wouldn't be so damaging if
the assessors limited their conclusions to what they can
take from this single writing occasion, but the fact is,
they broadly generalize from this by concluding that they
have in fact assessed the student's total writing ability.
In any case, the continuing problem with reliability
in assessing writing samples is in the reading, not the
/
writinq of these pieces. This can be seen in the long,
problematic history of correlations that ETS strove to get
in the scoring of essays, already discussed in Section One
of this chapter.
Another testing theorist from outside ETS, N. M.
Downie, bluntly challenges the efficacy of writing samples
from a strictly psychometric position. From his
Fundamentals of Measurement:
If we want to assess style, quality and other
aspects of writing, it is obvious that the essay
test item has to.be used. Even here, though,
studies have shown that objective tests of
writing ability can predict achievement in
writing, the latter measured by both teacher’s
estimates and grades received, better than do
essay tests. Typical of these studies is that
reported by Huddleston in which she used an
objective verbal test, the English examination of
the College Boards, a paragraph revision test, an
English essay examination, and a total English
test score based on the last 3 of those. Scores
on these tests were correlated with teachers'
ratings taken over two years of English and with
course grades in English averaged over a similar
two-year period. Results showed the essay test
to be more valid than the paragraph revision
tests, but less valid than the verbal test, the
College Board English test, and the total English
test. The verbal test predicted most accurately
of any of the tests (202).
141
But as we have already heard, ETS has over the years shored
up. the reliability of such tests in a number.of ways:
holistic scoring over the old analytical scoring, multiple
readers over the old single-reader method, and finally by
carefully weighting the parts of the exam (part
multiple-choice, part essay). Still, the importance ETS
attaches to high correlations between essay scores and
objective scores seems at odds with the rather
undistinguished figures they themselves give: .50 for.the
December, 1977 test and .48 for the December, 1978 test
(Kirrie 10).
I have, in this section, explored three psychometric
assumptions with implications for writing assessment.
First, the assessments are based on a psychology of
individual differences, but influenced by an unacknowledged
psychology of learning. This means that the "sorting"
function, contaminated by the readers’ interaction with the
texts, is not as neatly mechanical as would be necessary if
the readings were to be truly "objective." Second, the
assessments are based on the assumption that they are
measurement, and measurement is counting. But it is a
counting in which the numbers . stand in a highly abstract
relationship to the thing counted (the quality of the
essay). This means that the confidence with which we can
142
say that a student answered "3" questions correctly far
outstrips the confidence with which we can say that an
essay is a "3." Third and finally, these assessments
assume that they proceed from theoretically respectable
tenets of good test construction. But the degree to which
testing theory cannot be made to "fit" with these writing
samples is the degree to which we cannot be very confident
in making conclusions about a student’s writing ability
from such writing samples. Therefore, several special
problems result for holistically-graded essays which emerge
from these psychometric assumptions. The purpose here has
been simply to point them out. Later, in Chapter Four, I
will develop the implications of those problems. The
argument here has never been that these test-givers perform
their statistical, counting functions haphazardly. Once
the writing samples have been gathered and scored, the
whole statistical superstructure that is laid over the
scores is dispatched with great methodological rigor. This
is not at issue. What is at » issue is what it is possible
to find out about student writing ability through the use
of such tests. Since to be "scientific" these tests must
follow rules of good test-making, the testers must be held
responsible for whatever that "science" fails to
illuminate. That is why it is the responsibility of such
tests to be grounded in psychometric principles that relate
to what we know about writing in ways that are thorough,
firm, contextual, and rhetorical. I will in Chapter Four
claim that, on the contrary, the-"fit" of psychometric
principles to mass, anonymous assessment of student texts
is incomplete,. tenuous, de-contextualized, and
a-rhetorical.
Among the questions forced by their psychometric
assumptions is, to the extent that the tests are
norm-referenced or criterion-referenced, how can the reader
be prevented from reflecting on those norms and criteria,
and from haying that affect the scoring? That is, even
with high reliability coefficients, do we know whether we
are assessing a writer's competence, his performance, or
some other quality? Given that the test is read in
isolation from the scene and the processes in which and by
which it is produced, what important information about the
context and the process-product relationship are we
ignoring? Do all the training efforts to "level” readers
actually make all readers respond "the same?” If getting
readers to respond uniformly is so difficult, is it because
something importantly individual is being suppressed?
There are many questions, too, that arise about the
thoroughness, justifiability, and consistency of the
144
testers' rhetorical framework. To be fair, the ETS, NAEP,
and FWP documentation does not claim to develop a
rhetorical framework. As we have seen, the documents are a
description of methods and practices, not a theory of
rhetoric. But in describing rhetorical practices, they
imply a rhetorical framework, if vague and incomplete.
Having done so, they bring up many unanswered questions
about texts, especially about ethos which, as the writing
ability of the student as perceived by a reader through a
text, the assessments are all about, by their own claim. I
think a greatly enlarged view of the practices will become
available by "reinterpreting" these practices in light of a
more explicit and comprehensive rhetorical framework, the
modern-classicist contextualist framework already developed
as Chapter Two here. The purpose next, in the following
chapter, is to take these assessments as an example, and
re-examine them in light of that framework.
Chapter IV
ANONYMOUS MASS ASSESSMENT REINTERPRETED FROM THE
PERSPECTIVE OF THE MODERN-CLASSICIST
CONTEXTUALIST RHETORICAL FRAMEWORK
A given rhetorical framework allows, or should allow,
us to analyze discourse undertaken within that framework.
Since the framework of anonymous assessment "enables" the
writing samples gathered under its practices (and within
its implicit rhetorical theory), we should be able to
analyze that discourse and its participants within that
framework. In particular, for this is the purpose of such
assessments, we should be able to describe the writing
ability of.the authors of these texts. In rhetorical
terms, what we should be able to describe . is authorial
ethos, conventionally considered as the projection of the
author's presence or character in the text, relevant to the
aims and effects of the discourse. In the case of testing,
that projection is constrained to "writing ability" defined
in their "instrumental" rhetoric as technology (the
performance of certain skills), although that presumably
depends on a prior success by the writer in projecting a
fictional "image" according to the task set for him or her.
Since we have in this situation no direct recourse to'the
writer, but only reader response to the writer, we must
rely mostly-on that reader's response as a reconstruction
of authorial ethos. But we have indirect recourse to the
writer through the practices' implicit definitions of
writers and.texts. So while all that anonymous assessment
finally uses is rank-ordered scores,. still we are able to
ask what the nature and range of ethos in writers, texts,
and readers are in their framework. Since the assessment
documentation claims the testers are interested in
discriminating differences (variations) in writing ability,
we should be able to tell what they attribute to . those
writers as those differences. We should be able to say
what they find out about the writer, whose (various)
choices, motivations, and intentions are what constitute
the ethos they seek to assess. Next, we should be able to
know what account they give of texts, which exhibit ethos
in its mediating relationship between writers and readers.
We should know how that mediation occurs (its nature and
its range). Third (and most important, since they are our
only link to the writer we seek to understand), we should
be able to use the framework of anonymous assessment to
understand the readers of these texts. We should be able
to tell how the readers respond as they do.
147
On the contrary, I think that the rhetorical framework
implicit in anonymous assessment is inadequate to all of
these tasks. Anonymous assessment yields, essentially,
only numerical scores. This means that of the student’s
total output, or performance, only a part is considered.
There is a great deal that these practices misrepresent,
about writers, texts, and readers in general, and about
assessing student writing ability in particular. I have
already shown that ethos occurs in relation to the writer,
text, and reader involved in a discourse scene. It is by
analyzing those relationships that we can see ethos-— the
purpose of such assessments— in a well-rounded, thorough
way. But assessment practices truncate or skew all these
relationships. First, a text from an anonymous writer is
missing a great deal of information about that writer's
writing ability. Second, a single text written in a single
instance on a fixed, uniform topic likewise excludes much
that is important in judging a student’s writing ability.
Third, a reading act, in which , readers do not read
"normally" (for meaning) but are only allowed to respond to
formal, classificatory features of a text disallows much
valuable information about student writing ability. So all
three relationships, ethos-writer, . ethos-text, and
ethos-reader are problematic in the rhetorical framework
148
implied by the practices of anonymous assessment.
Assessment apologists are left with a logical dilemma. If
readers are unique people and their made-uniform responses
suppress that uniqueness,, causing the reader's score (a
number) to stand for the student's writing ability, their
response is vague and generalized, making writers, texts,
and readers seem more or less the same.. But if reader
responses are only vague and generalized, these assessment
practices fail to bring the richer, necessarily human
responses to the test that allegedly gives it its advantage
over machine-scored, multiple-choice tests.
There are, then, disjunctures between what the
framework of anonymous assessment should be able to do
(describe rhetorical situations, particularly ethos), and
what it is able to do (only yield up a set of numerical
scores). These disjunctures consist of both logical
inconsistencies (as above) and gaps (places where their
framework-simply ignores relevant rhetorical issues). I am
assuming that flaws and limitations of the framework will
lead to flaws and limitations of assessment practices, that
what the framework wrongly allows or cannot explain are
what the assessment practices will describe wrongly or
inadequately about student writing. Therefore, this
chapter takes a needed, broader perspective, seeking ethos
149
in the well-rounded, multi-perspectival way it requires. I.
will do this by reinterpreting the practices of anonymous
-assessment from the perspective of the modern-classicist
contextualist rhetorical framework.
In the contextualist framework, ethos . occurs
relationally with all the relevant participant's and
artifacts in a rhetorical scene.' Thus it would be possible
to pursue ethos in relation to many elements and under
various organization schemes. For example, issues of ethos
fall, in the description of the practices of anonymous
assessment, under the heading of "writer," while in the the
contrasting . contextualist framework, ethos is more
complexly related to readers and texts as well. It is
therefore judicious to reinterpret assessment practices,
from the perspective of my framework, along lines where
their practices and my framework most fully converge.
Anonymous assessment describes the relationships between
writers, texts, and readers. So, by taking up ethos in
relationship to each of the three, in turn, the
contextualist framework can feature their practices through
three key relationships. This organizational plan has the
virtue of featuring the practices through three key
relationships. Of course, the whole thrust of the
contextualist framework is the ultimate indivisibility of
150
such a whole into parts; still, this approach allows me to
temporarily, foreground three key. relationships in terms as
close to the assessors' as possible.
4.1 ETHQS AND THE WRITER
Assessment theorists see no need to inquire into the
mental processes of the writer. But many composition
theorists base their understanding of the writing
process--and thus writing abilities— on the many cognitive
choices the writer makes as he composes, most of which
don't appear in the finished text. Likewise, and again in
the interest of fairness, anonymous assessment avoids
inquiry into the writer's feelings and beliefs except as
they are expressly presented in the text. , Yet educational
testing theorist David Payne says that "to ask all students
to write on the same theme will result in variability of
grades on these themes that is basically related to a
student's interest in the topic" (The Assessment of
/ ‘ .
Learning 206). Finally, the assignments in assessment
purport to solicit the writer's genuine engagement with a
topic and its issues, but he is in fact judged on how
closely his essay matches the "model" essays used to train
the raters. This means that the possibilities involved in
a writer coming to "know" (in the sense of constituting a
151
view of his subject through the text) is strangely limited,
since he is probably aware that "coming to know" is not the
point of the exercise: the point is "display," not
"discovery" (in the sense of invention). These are three
examples,. from the" concrete cognitive-psychological.
through the valuational, to the very abstract
epistemological, of levels on which anonymous assessment
deals inadequately with or ignores ethical (about ethos)
issues relative to the writer. A broader rhetorical
framework can explain the relevant rhetorical issues on all
three levels better.
On the cognitive-psychological level, ethos in the
contextualist framework from the writer's perspective
begins with an acknowledgment of the uniqueness of the
individual. Ethos in discourse is, in Perelman's terms, an
"emanation" of the writer’s permanent character. The
mental processes of writing involve a reflexive
relationship between the writer and his developing text,
consisting of his sense of how that text-event represents
himself and his thoughts to his readers, given the scene
and the relevant arguments. A contextualist framework
posits a rich inner language life of the writer in which he
is constantly (dynamically) making choices, consciously and
unconsciously, about that;presentation (the text) in.light
152
of a given, unique scene and writing task. A given text is
an instantiation of some of the possibilities, from an
infinite range.' The linguistic terms "competence" and
"performance" are applicable here in their general tenor.
A writer has a language competence (a broad, inner range of
linguistic possibilities) and a language performance (a
single instance of that competence). The point is, a text
within the contextualist framework is always taken with the
qualification that a given text represents a part of the
writer which relates in some way to his larger, hidden
competence.
There is a great body of literature, especially since
the 1960's, that concerns itself with the huge range of
activities that occur in the production of a text. The
numerous, prominent researchers include Janet Emig, D.
Gordon Rohmah and Albert 0. Wlecke,. James Britton, Linda
Flower and John Hayes, and many others. This research on
the "composing process" investigates the cognitive and
affective states of writers at various stages of composing,
rhetorical strategies of all kinds that the writer chooses,
the writer's sense of the stages of the text, prewriting,
revision, and the like. The researchers' assumption is
that the text we see is just one of many "virtual" and
actual texts. These texts include notes, verbal reports by
153
the writer as he works, successive drafts, and biographical
information on the writer. The point for assessment of
student writing is that all this is evidence, the sum total
of which is the basis for an assessment of. the writer's
abilities. But the practices of anonymous assessment
disallow all these processes and texts, accepting only the
text taken from the writer. Anonymous assessment cannot
deal with the cognitive-psychological dimensions of wtiting
ability because neither the uniqueness of a writer's self
as a -cognitive-psychological entity nor his activities
enacting that uniqueness as ethos in a text are recognized.
As a cognitive-psychological model of humans, the
rhetoric of anonymous assessment suppresses natural
variations among individuals, in two senses. First,
natural variations across the population are flattened out,
in the interest of "fair" and "equal" treatment of all. By
having all students write on one, identical topic, the
natural variations of individuals are effectively limited.
Second, the range that variation might entail for a single
person— how he might perform were he given a range of
behaviors to perform— is likewise suppressed. As Paul
Diederich of ETS says, "We are judging writing, not
students" (Measuring Growth in English 12-13). I agree
that such assessments clearly are not judging students in
154
the sense required by a contextualist framework. (In the
next section, on ethos and texts, I will suggest moreover
that they are appraising writing too in only a very limited
sense.)
If the assessors wanted to, speculate about the actual
cognitive-psychological processes of the writer, they would
do well to consider as a beginning some of the obvious
constraints inherent in the testing scene. For example, an
essay which must be written in twenty minutes (ETS and
NAEP) to little more than an hour (FWP) is probably
constraining in a very particular way. Several
commentators on the writing prociess have claimed that the
real-time constraints of a writing task create their own
problems. Charles Stallard’s study of twelfth grade
writers is a case in point. He compared the writing
behavior of a group of good twelfth grade writers with that
of a group of randomly selected twelfth graders. He found
that while students in the "good" group wrote more words on
the average (343 to 309), they took considerably longer to
complete their work (40.8 minutes compared to 22.6
minutes). In other words, the better writers wrote
considerably slower than the randomly selected writers
(8.74 words/minute compared to 13.47 words/minute). The
implication for writing assessment may be that the time
155
factor itself inhibits quality and thus obscures writing
ability for all or certain kinds of. writers.
The valuational dimension of ethos, relative to the
writer in the contextualist framework, exhibits itself as a
relationship between the writer and his representation of
himself-in the text. The writer has a complete, • unique
value system and evidence of it within and outside of the
text allows us to know this valuational dimension.. But
anonymous assessment, by its sampling techniques and by the
topic and its attendant instructions, impoverishes the
richness of the writer’s ethos to the point where the
writer is little able to represent himself. He is from the
outset anonymous to the reader, of course, but moreover he
isn't even, operationally, any one person. Instead it is a
statistical "population" that is assessed. Each of the
three assessing institutions replaces the valuationally
unique personhood of the writer with a profile of
generalized demographic descriptors, or "tags.” NAEP
actively seeks out cross-sectioned populations to
assess— by age, sex, size of home town, and so forth. ETS
and FWP do not need to actively create the pool to be
tested: the occasion of assessment creates its own
demographic descriptors. A given ETS assessment
necessarily attracts a group identifiable by a common age;
156
educational level (in college admissions testing, they are
obviously all high school graduates); socioeconomic status
(e.g., those able to afford college); even motivational
level (they are all assumed to want to do well enough to be
admitted). So just as we saw that assessment theorists
treat this pool as undifferentiated psychologically, so the
pool is taken to be valuationally homogeneous.
Likewise, the topic and its attendant- instructions
artificially constrain the valuational dimension of
authorial ethos. A strong, even rigid set of expectations
is posited by the assignment. An unacknowledged but
definite value system is embedded in the topic which tries
to be neutral and "the same" for all. The "institutional"
"good writing" that is required is of course not neutral at
all, but a chosen, value-laden idiom. It does not at all
contain everything we might think of as "good." Further,
those values extend beyond discourse conventions to more
general values— political, for example, in a topic that
calls for criticizing a given institution. - Even if the
students want to "invest" in the assignment, they have a
difficult time doing so. The assessment methodology must
just assume the writers are doing their best, but John C.
Mellon, in National Assessment and the Teaching of English,
suggests this might be naive. He points out that NAEP asks
157
students to write for examiners they do not know; the
students are told that their teachers will not see the
writing, that it will not influence their marks or their
academic futures, and that they will receive no feedback at
all on their efforts (34). We need not be surprised that
NAEP reports that some students simply leave the room
early, their essays unfinished (Procedural Handbook 10). I.
do not need to claim that these Students are probably
little motivated by the assignment to write for twenty
minutes on the phrase, "We have met the enemy and he is
us." Perhaps some of the writers are highly motivated by
this assignment, and anyway, their motivation is a strictly
empirical question. What is clear, though,, is that inquiry
into such questions is cut off by the practices of
anonymous assessment, which simply can't and don't take it
into account.
Also crucial to this rich inner language life of the
writer are his intentions. The writer's intentions, which
have their source in his or her motivations, are keys to
his or her ethos. Richard Weaver corroborates the pivotal
role of intention in his comment that linguistic expression
is a carrier of tendency because man is ever a creature
responding to purpose. Michael Polanyi says that we are
always within a system of convictions when we express
158
ourselves, . and so we must start with an assumption of
"interests" For Burke, all persuasion necessarily involves
"choice" and "will," and to inquire into these willed
choices is to try to understand the processes of rhetoric.
Like Burke, Polanyi believes that these "interests" are
associated only with intentional human, "action," not mere,
meaningless "motion." Therefore, we must acknowledge the
importance of the writer’s inner (language) life to ethos.
To understand authorial ethos, the contextualist framework
requires us to look at what actually happens to a writer as
he addresses his topic and actively constitutes himself in
a text.
Just as the assessment . procedures ignore the
cognitive-psychological evidence for ethos that composing
process theorists think important, we can see by this
example how the assessors ignore important valuational
evidence of ethos— student motivation (here as interest or
"investment"). While ethos in the contextualist view is an
indicator of the writer's self (for Perelman, his
"permanent character"), anonymous assessment doesn't
recognize the processes by which this unique self is
instantiated in a. text. Instead, a mechanical model of
humans is used in anonymous assessment. Turning away from
the rich complexity of the composing process and of the
159
inner language life that process refers to, they suppose
that the "output" of the writer is all that needs to be
looked at.. Still, anonymous assessments make many
assumptions about what the writer’s values and valuational
behaviors are: it's just that the test methodology pretends
those behaviors can be made uniform, and so made
irrelevant. What is : being ignored -is the embedded
rhetorical act— within the apparent genre, which is
uniform, is the author's real relation to the topic, which
varies from writer to writer.
The problem of this embedded rhetorical act is most
vividly, though not exclusively, seen in NAEP's highly
regarded Primary Trait Scoring. By specifying in advance a
tightly-drawn set of criteria under which the writing is
judged rhetorically successful or unsuccessful,, it was.
thought that the writer's rhetorical values could be
standardized by enclosing them completely within the
wording of a topic. For example, in one test item, the
student writer was presented with a picture of children
playing on an overturned boat. The directions suggested
the writer enter the picture and describe his involvement
with the scene. . The "primary trait" of the "Children on a
Boat" item was named "Elaborated expression of a point of
view through entry into an imaginative situation." The
160
raters were given a rubric assigning specific point values
for the appearance of various features. It' was thought
important that the writer reproduce the fantasy games of
children, such as pretending to be pirates, shipwrecked, or
riding whales. But no "point" was given for merely stating
the fantasy. The rubric warns, "Stating the fantasy is not
enough; two or more unelaborated fantasy situations, or a
fantasy situation with at least one elaboration must be
present" (Mullis 31). Similarly, points were earned by the
author for entering the "world" of the picture "thoroughly"
(3 points), "slightly" (2 points), or "not at all" (1
point). The problem is that Primary Trait Scoring
encourages, even requires, raters to respond to this
supposed set of writer's (valuational) choices rather than
the ones the writer actually makes as he writes. To be
sure, a good writer might choose elaborated fantasy over
unelaborated fantasy; he might feel that a paper in which
he enters the scene, of the picture is better than a paper
in which he does not. We just don't know, and neither do
the assessors. Such questions are strictly empirical ones,
and even if answered would only be an account of how good
the assessors' guesses were.
Primary • Trait Scoring assumes that the valuational
experience of the writer can be accounted for exclusively
161
within the wording of the text item. This assumption
ignores the valuational dimension of the ethos-writer
relationship in which the actual and unique valuational
experience of the writer takes place.
Not only might writers make different choices, but
good writers might make other effective choices with
different rhetorical purposes. For the "Children on a
Boat" item the writer might have entered the scene as an
observer— a parent, or a bystander, for example. Given the
rubric, these alternatives would not be credited. Anyway,
such a rubric doesn't address what is supposedly at
issue--the rhetorical purpose of the writer. The assessors
are focusing on the instrument, not the goal. The writers
are made to "play" or "fictionalize" their own role, the
reader's", and that of the purported rhetorical task. This
is a confusing pretense in which the writer, the situation,
and the reader are all unstable and inauthentic. It is
hard to see how students could respect themselves or the
task, given its almost total inauthenticity. The students
are unable to trust discourse in the contextualist Sense.
They cannot believe that their texts are productive of the
purported purposes of the task. The essay task only
appears to be about the topic. In fact, . the essay is a
"display" text. The writers can only use their texts as a
162
display of skills that may or may not relate’ to their
constituting a self relative to actual, felt values. The
writer is forced into detachment from the language he uses
(and the ideas it carries). This empty manipulation of
language for its own sake, and not for actual pragmatic
purposes, preempts choice. The writer .is not discovering
and constituting personal choice as a response to his
adherance to a point of view,' but instead must divine and
echo the preconceptions of his reader. The values, he
presents as his arguments need not be.genuinely’felt since,
while they are not rejected as such, their congruence with
a high score is coincidental and not causal. This is a
problem because this formulaic writing can actually obscure
the writing abilities the assessors want to see. Ken
Macrorie describes the language used by writers in response
to such tasks as "Engfish." This kind of "automatic"
writing features "pat" phrases, platitudes, and vague
generalizations. (Macrorie has discussed how, when
students are forced to write on topics they can. think or-
feel little about, the result is a text full of remotely
felt, empty abstract ions.) Between arbitrary, tight time
constraints and convoluted, inauthentic rhetorical demands,
there is neither time nor encouragement for an authentic
voice to emerge, as it might with more reflection and
163
development. Actually, this system probably rewards those
students whose writing strength is in awkward and
inauthentic rhetorical situations, and penalizes those
students who genuinely want to address the topic in a
thoroughly felt and thought-out manner.
Third and finally, the epistemological dimension of
authorical ethos is peculiarly constrained in anonymous
assessment. ; The modern-classicist contextualists base
their rhetoric on its-use as a tool for knowledge. Mutual
goodwill of the discoursers is a prerequisite. From this
basis, they claim that we come to know both our topic and
ourselves better through our discourse. It follows that a
writer must be taken to be a complete, complex individual,
and that ethos in his text is, ideally, the representation
of his choices from that self in his text. It also follows
that a writer needs to believe that one of the purposes of
his discourse is to gain self-knowledge and knowledge of
the issues of his topic. But the writing assignments of
anonymous assessment are only coincidentally, if at all,
open to such an epistemology. Technically, of course, the
writer is free to write whatever he wants, but I have shown
that the assignment is other than it appears to be. Of
course, the cunning student probably figures this out, and
turns in a piece that displays the expected phrases and
164
"buzz words.” The honest student and the simply naive one
have no such recourse. The rhetoric of anonymous
assessment self-destructs. Because it is a rhetoric that
doesn't believe in the genuine usefulness of discourse to
produce real change it is, oddly, a rhetoric that is
"a-rhetorical." Assessment . practice narrows the writer’s"
range-of full, human responses to mere observance of forms.
His full, complex ethos is truncated and so misrepresented
because the evidence of it that they allow is skewed and
idiosyncratic. Even if we., accept that this "display"
writing is itself a genre, and further, that no one piece
of writing will exhibit the whole of an individual, the
assessors should at least acknowledge that what they take
as overall "writing ability" is actually success in
performing in this display genre.
4*2 ETHOS and the text
Ethos in a text, in the contextualist framework,
presents itself as a perspective on the relationships of
writers and readers to that text. Ethos in texts focuses
on 1) how a person, or an aspect of a person (here, writing
ability) is "enacted" in an artifact (the text); and on 2)
how another person (here, the reader) responds to that
text-event. What each participant' takes the nature and
165
range of the text to be. will define and constrain ethos,
both in ethos' limited sense as the artifact-event
representing the writer's writing ability, and in the
broader sense in which ethos in a text mediates this
necessarily "personal" transaction. The contextualist
framework posits a very rich sense of ethos and texts in
which the text is an event as well as an artifact, and in
which the text is a mediator making available much that is
important about the nature of the discoursers and about
their relationships to the text.
But from the perspective of the contextualist
framework, the practices of anonymous assessment falsify
and impoverish the real nature of ethos in texts by 1)
denying the text's "event" nature (by treating the text as
only an artifact), and 2) by denying the text's relatedness
to rhetorical elements (here, especially writers and
readers) with which it is inextricably, intertwined and
without which it cannot be understood. This falsification
emerges from an inconsistency inherent in the
"current-traditional" paradigm, under which anonymous
assessment is practiced. Richard Young has commented on
this problem. The "current-traditional" rhetoric, he
argues, disparages ethical proof because invention is
- , '
(wrongly) thought to be "mystical." The idea is that the
/
166
writer's inventive ability is inexplicable, and as an
uncontrolled "visitation," it is neither teachable nor
learnable. Despairing of objectifying invention, Young
says, those in this paradigm betray ethos altogether,
retreating to the comfortable illusion that a logos-based
rhetoric will answer all questions. .This leads to 1)
treating the text as an artifact, not an event, and 2)
separating our study of texts from those who create and
recreate them (writers and readers). While that paradigm,
like the assessments performed within it, is text-based
(deemphasizing the contributions of writers and readers),
still the final, if not the functional, goal of assessment
is knowledge about writers. The assessors, being
"scientific," rely on "things" more than people, and so
inordinately value texts separated from people. The
problem is that such a view will only serve the functional
goal of assessment (gathering essays), not the final goal
(understanding the writing ability of people).
The practices of anonymous assessment dictate that the
single writing sample that is taken i_s the representation
of the student’s writing ability. This assumes that
language is a fixed phenomenon, and that denies the
necessary and unique relationships between writers and
readers observable through texts, but not-in texts alone.
167
taken out of.context. The serious implication of this
assumption (that language and meanings exist in a fixed,
external reality) is that the assessors are left pretending
that the evaluation of "writing ability" somehow takes
place outside any evaluative system--leaving "good writing"
a matter of merely observing conventions. I will explore
examples of these problems on, again,
cognitive-psychological, valuational, and epistemological
levels.
To the contextualist, it makes no sense to talk about
cognitive and psychological dimensions of texts without
reference to 1) those texts as events, and 2) the writers
and readers who have those cognitive-psychological
perceptions and experiences relative to those texts. But
anonymous assessment attempts to deal with what is "in" the
text; as a static entity, and in isolation. As we have
already seen, the assessors take a view of the composing
process at odds with that of many compositionists.
Wrenching the text out of context, and disallowing all but
one of the many texts actually involved in the composing
process, truncates and misrepresents ethos in texts.
Likewise, the test itself, as a psychometric instrument,
skews the mediating potential of ethos in texts in its
cognitive-psychological dimension. This skewing is
168
revealed in the challenges to the test's instrument
reliability and instrument validity', touched on in Chapter
Three. The challenge to instrument reliability is this: no
matter how high the reliability coefficients (and these
have been raised by increasing the number of readers,
replacing the old analytical scoring with holistic scoring,
and weighting the essay with multiple-choice portions of
the test), no one can say what part of the writer's writing
ability is being assessed. As educational testing theorist
David Payne says in connection with such assessments, "Even
if an acceptable level of scoring reliability is attained,
there is no guarantee that we are measuring consistently"
(143).
Once we accept that people are complex, we have the
burden of accounting for complexity. Anonymous assessment
oversimplifies complex people. Coupled with instrument
reliability is instrument validity. Payne cautions that
the use of a single essay question is problematic: "the
number of questions on the test influences validity, as
well as reliability. As commonly constructed, an essay
test contains a small number of items. . . . [therefore]
the test will suffer from lowered validity— specifically,
decreased content validity" (173). The test's questionable
instrument; reliability and instrument validity in turn
169
force several uncomfortable views of texts and the acts of
producing and interpreting them. The first of these is
that the text is isolatable and isolated from the processes
by which it is produced, and the testing scene in which it
is produced. In his discussion of the problems inherent in
essay tests, Payne reveals the inadequate view of the
composing process that assessment apologists proceed from:
"Theoretically, the essay test allows the examinee to
construct a creative, organized, unique and integrated
communication. Very frequently, however, he spends most of
his time simply recalling and assembling information,
rather than integrating it" Q43-144)'. This "model" of the
composing process, neatly dividing as it does (the
apparently significant) "integrating" from (the.apparently
unimportant) "recalling and assembling" obviously misses
the recursive, interrelated nature of all -the behaviors
that ultimately result in a finished text. Even if this
mysterious item "integrating" could be isolated from.other
actions which lead to the production of a finished text, it
seems that it is not very important from a psychometric
viewpoint. After all, the readers are supposedly
responding only to the completed text they see before them,
and much is done, in full accord with the test methodology,
to pretend that what they see has nothing to do with how it
170
was produced. Further, "recalling and assembling" seems to
refer to a determinate content and its arrangement, while
"integrating" seems to refer to personal choices. These
terms assume a clear division between form and content,
each having no important relation to the other.
To the extent that a scene or context is even referred
to the educational theorists betray that writing assessment
apologists have scant theoretical and psychometric back-up
for their attitudes and practices. ' Giving guidelines for
appropriate use of essay tests over objective tests, Payne
counsels: "Use an essay.question for the purposes it best
serves, i.e., organization, handling complicated ideas, and
writing" (147), As to the tautological "Use an essay
question [writing] for the purposes it best serves, . . .
writing," I can only conclude that to testing theorists
writing means the display of grammatical correctness or
some other recognizable, "structural" features. As to
context, it is apparently something that is just there,
vaguely troublesome,- but unavoidable for the time being,
since as ETS reluctantly admits, "we have not yet been able
to measure the reasoning and organizational aspects of good
expository and creative writing independently of .writing
itself" [emphasis added] (Anastasi 411). Anonymous
assessment is left with a "theory" of texts as
171
psychological constructs that is impoverished. The
mediating role of texts which, from the contextualist
perspective, is a rich repository and transmitter of the
complexities of people is, in anonymous assessment,
falsified, truncated, and so misrepresented.
Just as ethos reveals itself relative to texts in the
cognitive-psychological experiences of writers and readers
of the text, so do texts reveal ethos in the valuational
sense that writers and readers have of the text. The
contextualist framework posits that values are necessary in
order to understand and assign meanings to texts, that our.
understanding is always done from within a matrix of
values, and indeed, that what a text means is its values.
The contextualist framework further posits that neither
language nor people can be "neutral"; it follows that no
text can be "neutral" either. In contrast, the rhetoric of
anonymous assessment suggests that the people involved in
making and receiving meaning are not significantly
valuational beings, because the text's values have been
"scientifically" controlled ("neutralized") by the
assessment methodology,, and so made irrelevant as
indicators of the ethos of these people.
Their implicit view is that the valuational dimension
of texts can be predetermined. We saw in the last section
172
that Primary Trait Scoring is the test format that most
obviously displays this assumption, but the other essay
formats contain this assumption implicitly. Both writers'
actual choices (valuations) and readers' actual affects
(here, values as evaluations) are thought to be neutralized
by what is done to make the writing and reading scenes
uniform. They feel this clears the way for the emergence
of the only distinction they're interested in: the
appearance or non-appearance of "good writing." Anonymous
assessment tries to neutralize (make uniform) the readers'
values regarding texts in several ways. First, the
trainers tell the readers that they are already more or
less uniform: "Over the years these English teachers have
read and evaluated the essays their students have written.
and they are. in general agreement about what
constitutes good writing" ; (Kirrie 4). The assessors
suggest that "good writing" is both valuable and easily and
universally recognizable. . "Good writing" means
unemotional, logical prose observing a "standard" idiom,
diction, tone, and style. But the fact that they don’t
define it in much detail, and yet spend so much time
training the readers to agree when they've seen it suggests
to me that it might be less truly valued and less
universally recognizable that they assume. Finally, the
173
asssessors choose only topics thought to be open only to
the kind of writing they seek. "If the topic invites other
kinds of responses . . .the topic is not used in the test"
(Kirrie 3). "It must not be a topic . . . that induces an
emotional response on the part of the candidate, even as it
must not be a topic that produces an emotional reaction on
the part of.the reader of the essay. The ability to write
is what is to be evaluated, and everything that interferes
with that evaluation must be eliminated or, at the very
least, mitigated" (Conlan 1). So, since most kinds of
interaction with the topic are "bad" and, far from
revealing ethos, obscure it, it is thought important (and
possible) to get value-free, detached, objective texts for
writers to write and for readers to read. Cooper says that
"it [PTS] asks readers to determine whether a piece of
writing has certain characteristics or primary traits that
are. critical to success with a given rhetorical task"
(Evaluating Writing 32). It is the writing, not the
writer, that apparently has rhetorical motive. There is
thus a confidence that the rhetorical motive of the
situation is a discrete entity that can somehow be
contained entirely and exclusively within the text (the
carefully worded test item, and the determinate, because
controlled, student essay). This defines rhetorical force
174
in a way that isolates it from its relationship to a writer
and a reader who create and perceive it. The valuational
dimension of the ethos-text relationship is ignored. PTS
is, in their own words, concerned with "the primary
rhetorical trait contained in the topic" (Mellon 36). Note
that the claim is that the rhetorical trait is contained in
the topic, not in either the writer or the reader.- The
contextualist framework• rejects this determinacy, of
course, recognizing that, as Polanyi says, only words of
indeterminate meaning can have any bearing on reality. The
contextualist framework acknowledges that we cannot deal
with a text in isolation but only in relation to writers
and readers. Therefore, we must ask not what is "in" the
text, but what "happens" between writers and readers (with
the text as the mediating event-art ifact in the
transaction). If we pretend otherwise, we are limited in
our texts to merely stitching together words and phrases
that may appear to be texts, but do not function as texts:
connecting one real person to another with communication
that is unique and genuine. While ETS's Paul B. .Diederich
claims "We are judging [evaluating] writing, not students"
(Measuring Growth in English 12-13), it is still people
(the raters) who do the judging. But he means "judging"
done from within a value system that restricts "judging" to
175
merely checking for the presence or absence of the features
of "good writing." It is ironic and paradoxical that
assessment works so hard (and so futilely) to be
value-free, since ultimately the whole purpose of
assessment is valuational. The practices of anonymous
assessment rely on a concept of the "universal audience."
In usual cases, the authors writing to a "universal
audience" are necessarily and legitimately anonymous. But
ethos in texts meant for a universal audience refers less
to a real person and more to the message itself. The
problem with assessment is that ethos is extrapolated to
refer to the "writing ability" of a real person, which
implies a class of all possible texts of that person (all
of which but one aren't present).
Finally, ethos reveals itself relative to texts on_an
epistemological level. The contextualist framework offers
an epistemology of ethos in texts describing writers and
readers coming to "know" via the text. In the
contextualist framework, we "know" the writer's intentions
through a text only as people having interpretive,
constructive, interactions with other people. In the case
of assessment, the interactions take place in complex
language Scenes. Objects, ideas, and words are only
material to be shaped into knowledge. The shaping is based
176
on agreements--facts are agreements of relatively long
duration— -and disagreements (the issues at. hand in a given
discourse). This knowledge is called, variously, opinions,
beliefs, judgments, or "what we know." The contextualist
framework not only enfranchises inquiry into what is known
but, reflexively, about the knowing itself.
By contrast, the rhetoric of anonymous assessment does
not inquire, but rather presumes. Mostly, what is presumed
is that there is nothing important to know about how we
"know" through texts. Anonymous assessment’s sense of
ethos and texts was implicitly developed in Chapter Two
under its paradigmatic heading "current-traditional"
rhetoric. In this epistemology, the text is a verbal form
of materials that objectively exist "out there" in the
external world. For "normal" people, texts are simply
perceived, as is, the same to all. Epistemologically,
ethos in texts is, finally, a non-issue in a world in which
a single external reality would answer all such questions,
given enough information. Disallowed is the dynamic
interaction of the contextualist framework in which
writers, texts, and readers all.have relationships with,
and are changed by, each other. E. D. Hirsch, Jr.,
explaining "intrinsic evaluation" in anonymous assessment,
illustrates the epistemology of texts of the assessment
procedures, so far discussed.
177
Intrinsic evaluation means judging writing
according to how well it does that which we
believe it is trying to do. On this principle,
each assessor temporarily forgoes his own values
in favor of the writer's. In that way, all
assessors, though judging independently, could
reach similar results by applying non-arbitrary
standards that were intrinsic to the particular
piece of writing. . . . the writer is granted his
meanings and purposes, and is judged only on the
effectiveness with which those meanings, are
purposes are fulfilled (194, 196).
Here, writers are thought to be able to put meanings into
texts (and readers are thought to be able to take them out)
without invoking those aspects of the self that perceive
them as meanings or purposes, and without invoking, or
being influenced by, a self that has opinions and.beliefs,
jabout the qualitiy or merit of those meanings and purposes.
This, to. the contextualist, is a non-epistemology in which
we do not personally constitute what it is we know, but
uniformly and mechanically process what is already "known"
by a text.
4.3 ETHOS AND THE READER
Ethos in regard to readers concerns us here with the
way a reader's response to a text reflects the writer's!
ethos. We need to know how and why readers respond as they
.do, to a text and to the writer the text refers to.
Ethical issues for readers range, as before, from the
cognitive-psychological, to the valuational, to the
epistemological.
The contextualist framework posits a complex
cognitive-psychological model of the reader. In writing
assessment, the issue is doubly complex since the person
reading is the interpreter of the intended meanings of
another, also complex, person. The contextualist framework
characterizes the reader's act as "intentionalist" as
opposed to "formalist." This means that the reader
actively constructs a representation of the' intentions of
the author. This act of reading necessarily includes an
interpreting reader. Meaning is not objectively available,
not even as simple cognition. Instead meaning is available
to the reader only through whatever . cognitive and
psychological processes the reader employs that enable and
^constrain what are allowable interpretations of the text.
Of course, shared discourse conventions narrow the options
a great deal. Interpretations are not idiosyncratic;
jstill, a thorough account of interpretations must examine
the constraints that apply in a given situation. So if we
want to understand a reader's response, we must examine
those qualities and constraints, assuming unique and
:omplex people.
179
In contrast, the practices of anonymous assessment do
not ask such questions. Much is done to try to make the
cognitive-psychological aspects of reader response
determinate, objective, and the same for all. The
rater-trainers do this by fostering a certain kind of
reading that greatly alters "normal" reading and narrows
the fullness of the possibilities of reader response, and
goes beyond merely agreeing on conventions of ' the genre.
Primary Trait Scoring is an example of the kind of reading
that is fostered in anonymous assessment. As noted
earlier, PTS was invented to fill a void between two older,
problematic scoring methods. The "analytic" method, with
its separate, weighted scores for features such as
mechanics, organization, and style, was criticized for
generating much data and no meaningful commentary. The
"holistic" method, with its ..single, numerical score was
criticized for generating little data and, likewise, no
meaningful commentary. But Primary Trait Scoring) by
specifying in advance a tightly-drawn set of criteria under
which the writing was judged rhetorically successful or
unsuccessful, was thought to yield the needed meaningful
understanding of the student's text. A sense of audience
was supposedly newly important in PTS. The reader was to
respond to whether he perceived the writer fulfilling (or
180
not fulfilling) the specified rhetorical demands of the
test item. • But I think that. PTS is simply a more explicit
expression of the same rhetorical ideas that are in the old
formats. PTS assumes that we can know and specify in
advance the rheorical situation the writer will be in when
he writes. Additionally, and more to the point here, the
raters are trained with a set of "expectations." . The
expectations consist of a rubric assigning specific point
values for the appearance of various features. In other
words, the raters are told to equate the rhetorical
situation they are supposedly responding to with a set of
prearranged features. The problem is one of interpretation
at two. removes. First, there is a . supposed rhetorical
situation that is less complex than the actual one the
writer is engaged in as he writes. Second, the raters are
asked only to decide in a binary fashion whether or not a
trait is present, not whether they were actually affected
in any rhetorical sense by what they read. Primary Trait
Scoring, as a design tool for exercises that create texts,
is not about either writer or reader response, as such. It
is a method of assigning points for the presence or absence
of formal features (or, in the terms I have established,
the presence or absence of the genre known as "good
writing"). PTS only makes explicit what the other scoring
methods posit about reader response: that such response is
not importantly interactive and interpretive, but
determinate, objective, and the same for all. This
hoped-for uniformity can be accounted for not only in what
the rater-trainers do to get the expected responses, but of
course as well by what they do to exclude varying or unique
responses. Assessment practice is to make the readers as
uniform as possible. The training process rewards
conformity and discourages uniqueness. We saw in Chapter
Three how the rater-trainers pressure the readers toward
conformity. They reject variations in responses, going so
far as to remove "deviant" readers altogether. As a
cognitive-psychological model of reading, the practices of
anonymous assessment suggest that people read mechanically.
We can say that the practices suggest that people do not
read as individuals with varying intentions (as in the
contextualist view, where intentions are seen to vary from
person to person, and even for the same person from
occasion to occasion). Assessment practice does not
suggest that there are many kinds of.reading for complex
people. Or, it may be that they do think so, and are
trying to control which reading mode is selected. Then,
the problem is that such an unusual mode is taken to stand
for more usual ones. That is, it is not normal reading
182
practice to read in this mechanical way, though it is, to a
degree, possible.
On a valuational level, the contextualist framework
posits a reading act that is unique (individual) , and
constructive, constitutive of and constituted within a
valuational system. . But assessment' practices posit
"disinterested" reading. The source of the importance they
attach to "disinterested" reading is, of course, their fear
of "bias." The assessment theorists want the reading act
to be "objective" and "fair." Paul Diederich> in many ways
the "father" of these tests, expressed this worry in a book
that laid the groundwork for assessment practice.
Another danger in grading essays is that we must
try to avoid bias on the part of the readers. . .
.Bias appears most obviously when a teacher is
grading the papers of his own students. . . . I
have tried to counter the argument . of teachers
who believe it is almost immoral to grade any
paper without the full knowledge of the
student— his ability, background and
circumstances. . . . We are judging writing, not
students" (Measuring Growth in English 11, 23).
Emerging from this belief that if the essay "itself" is
made the focus, the scoring will be fair, is the attempt by
the topic-selection committee to try to avoid topics that
will unduly involve the readers: "The committee
deliberately tries to avoid topics that invite readers to
analyze the candidates* psyches, for the primary purpose of
183
the topic is.to give the candidates the best opportunity to
show how well they can write . . . " (Conlan 2). Then the
readers are told that they can and should remove their
"interests" from their reading experience: "The Chief
Reader urges the readers to forget their personal criteria,
for evaluating a paper and to adopt the standards that the
group establishes" (Kirrie 4). The broader, contextualist
view approaches the reading act in general as an
opportunity to maximize the range of reader responses by
asking all appropriate questions--about the text, the
writer it refers to, and about the reader. But in
anonymous assessment, instead of asking probing questions
that would maximize what we would take from the reading
experience, the Chief Reader forcably truncates and
trivializes such discussion. "The chief reader presents
the topic to the readers and defines the writing problem
inherent in the topic. . . .. How much attention should be
paid to misspelling? What is considered off the topic?
What do we define as specific examples?_" (Conlan 3). IThese
are not just suggestions. Rather, "It is necessary for all
to adjust to the standards of the majority. There can be
no exceptions [emphasis in original] if the papers are to
be scored as nearly as possible in the same way" (Conlan
6) .
184
What the contextualist framework takes as inevitable
variations in readers, the practices of anonymous
assessment take as bias. . What the contextualist framework
embraces as evidence of ethos in writers and readers, the
practitioners of anonymous assessment disparage and believe
can be eliminated. They mistake objectivity for
impartiality. Perelman says that impartiality is not the
same as objectivity: impartiality consists of belonging to
the same group as those one is judging, without haying
previously decided in favor of- any one of them (59-60). To
be thoroughly "objective" is to be so removed from the
discourse scene as to be uninterested in the occurrence.
This confers no ability on the person to judge better.
Anyone close enough to the discourse scene to have an
"interest" can at most be impartial, never objective.
Assessment methodology tries (illogically) to have readers
not be in the group they are necessarily in.
I think that the contextualist framework enfranchises
safeguards against what the assessors worry about as
"bias." We are always free to name our standards (values)
for a given discourse scene but 1) those values must be
named more honestly and fully than the assessors do and 2)
those values must be acknowledged by all concerned as
constitutive of the results obtained. This contrasting
185
breadth of the contextualist framework makes the assessors
efforts at once unnecessary and futile, Of course readers
always have valuational interests. There are never not
interests. To the extent that readers can be artifically
constrained from reading "normally" the assessment
theorists have simply succeeded in creating a "generalized"
reading response that is depressed and oversimplified,
insensitive to any number of the real complexities of a
given reading act. Moreover, to depend on high
correlations between readers' scores begs the question of
objectivity. Neglecting to name explicitly the values that
are actually operating .in the reading doesn't prove there
aren't any— it disallows the kind of questions that could
reveal where the given responses emerge from.
On. an epistemological level, the contextualist
framework posits readers having rich possibilities for
"coming to know"— about the world— through the writer, the
text, and themselves. As individual, complex people
jconstituting meanings, -readers are seen in the
contextualist framework as importantly choosing
(self-consciously or not) to "know," which means,
variously, to grasp, to agree, or to believe.
Additionally, as a reflexive framework, contextualism
jenfranchises knowledge about that knowledge.
186
The practices of anonymous assessment do imply that
writers reveal themselves to readers (for how else could
differences be perceived?). But the practices also imply
that "successful" texts refer to a fixed, external reality.
What this means is that the practices suggest that texts do
not refer to how or how well a writer constitutes that
reality, this because the reader is allowed only to
recognize formal, classificatory features of texts, leaving
how and why the reader responds as he- does a complete
question mark. This assumes a general, rather than a
unique, quality of "coming to know" in such scenes and
acts. This impoverished sense of how knowing occurs and
what is known means that readers are forced to pretend that
they only "know" enough to narrowly respond with a
numerical score. This restricted sense of knowing replaces
the full, human range of response that they potentially
have. Anonymous assessment makes trivial and irrelevant
the nature of readers coming to know. Their response is
robbed of important evidence of the writer’s ethos because
the response is "decontextualized" from both the production
of the text and, as much as possible, even from their'own
reading of the text. The test methodology falls short of
its epistemological responsibilities in 1) its lack of
representativeness of a student, and 2) its inability to
187
reflect on its own activities. As to the . first, I have
already pointed out in Chapter Three that, of the ten
standard qualities of a good test,. "specificity,"
"difficulty," and "discrimination" are all problematic in
anonymous assessment. Those three qualities all require
knowing what educational content is being tested. This
cannot be established in essay tests of this kind, where
the past learning of the student is unknown. . The readers
are forced to assume they "know" important information
about the student they can't possibly know. While of
course it could be argued that they are equally in the dark
about all the writers, this enforced ignorance still means
that they are responding in a vague, general way to an
ethos that is individual and unique. Another sense of
"knowing" the readers are deprived of and must just try to
ignore regards the single, isolated text on which they must
base their judgment of the writer's writing ability. The
practices of assessment seem to imply that these texts
could be taken as intended for a "universal audience," and
as such could be normally and legitimately anonymous. But
ethos is texts meant for a universal audience is more
specific to the texts and their function, and not
generalized to a class of all possible texts written by the
writer, which is the (ungrounded) extrapolation that
188
anonymous assessment makes. The contextualist framework
argues that only by taking what the readers "know" the
student did, given his or her possibility of making other
choices, can comparisons be drawn against which a
definition of "ability" can be established. If the
test-makers recognized this competence-performance
distinction an explicit, if somewhat arbitrary, definition
of competence could be made as a standard for a; given
assessment. Finally, the denial of a reflexive dimension
in test methodology also reduces the readers' responses to
a text and a writer. ETS' Ina Mullis almost inadvertently
addresses this issue when she comments on the methdological.
difficulties in replicating holistic scores.for the same
essays in subsequent rating sessions.
The reasons why holistic scores are difficult to
replicate and to report in terms of specific
skills are found in the nature of the
methodology. It is true that broad criteria for
score points are often specified and that through
rating example papers, scorers can be taught to
apply criteria consistently. However, the
relationship between the scorers' internalized
criteria and the external or specified criteria
is never described" [empahsis added] (3).
189
4.4 CONCLUSION
Because the anonymous mass assessment of .student
writing ability is pervasive, costly, time-consuming, and
consequential, I thought it important to analyze those
practices. As rhetorical (discursive) acts, those
practices should be open to consistent and thorough
rhetorical explanation. It is my position that, on the
contrary, the rhetorical framework implied by the
assessment practices is full of theoretical gaps and even
inconsistent on its own terms. Since the object of
assessment is. to establish the writer's writing ability,
and since the only access we have to that writing ability
is the reader's judgment of it, the main issue in
assessment is how and how well the readers are able to
establish that ability. Since, further, the reader's
judgment of the student's writing ability is a construction
of ethos, and since all that can be known about that
writing ability is only available through people
interacting "personally", with, texts (ethos in a broad
sense), the key to clarifying what is vague and incomplete
in the theory and practice of writing assessment is ethos,
taken both narrowly and broadly. So a broader rhetorical
framework was needed to analyze ethos in this situation. I
developed a modern-classicist contextualist rhetorical
190
framework for this purpose in Chapter Two and inferred, in
Chapter Three, the contrasting rhetorical framework implied
by the practices of anonymous assessment. Chapter Four has
reinterpreted those practices from that contextualist point
of view, which has been apt, considering the power of the
modern-classicists in particular and contextualism in
general to describe the assessment scene.
Modern-classicist rhetoric is a descriptive, social, and
performative; ("dramatistic") rhetoric well suited to
analyzing the practical nature of such writing tasks, while
contextualism in general is well able to handle issues of
event perception and prose comprehension in a unique
ecological setting. As a contextualist framework, the
modern-classicists' / subscription to holism, relativism,,
interactionism, - dynamism, and reflexivism- are crucial
orienting perspectives. As rhetorics, their use of ethos,
trust, value, and choice as interpretive probes for
writers, texts, and readers is similarly appropriate and
penetrating. The modern-classicists; as we have seen,
argue for the reinstatement of ethos in a balanced
ethos-logos-pathos triad, against a too-logos-based
rhetoric. This is a revival of the discredited territory
of the "personal" in discourse transactions; The suasory
quality of discourse can be understood only by
191
acknowledging the rich resources of ethos. They argue for
the importance of trust in discourse as a necessary article
of faith in the discourse form and practically, as a useful
strategy in dealing with texts, against skeptical and
sophistic rhetorics. They argue for the inevitability of
value in language as a whole and in people and their
discourses in particular, against rhetorics that seek to
make language and texts value-less, neutral, or objective;
Finally, they argue for choice in discourse, claiming that
only by assuming human, ' active, committed choice
(self-conscious or not) are we acknowledging the genuinely
personal interactions that constitute discourse. They deny
views that make language or language-users mechanical or
determinate.
In contrast, the rhetorical framework implied by
anonymous assessment is .unable to describe ethos in
relation to writers, texts, and readers precisely and
consistently. Reinterpreting assessment practices from the
contextualist perspective reveals that first, anonymous
assessment can little account for how. writers' choices,
motivations, and intentions constitute the ethos they seek
to assess; second, that the practices of anonymous
assessment offer scant acknowledgment of the mediating role
of ethos in.texts; and third that anonymous assessment
192
tells us little about how readers' ethical processes work.
Anonymous assessment treats the holistic, complex event of
discourse as an atomistic and finally trivial, non-event.
Anonymous assessment makes the futile attempt to
"de-contextualize" a text. • More broadly, they try to
depersonalize the situation, and assume incorrectly they
have actually done so.
It is not only tempting but also necessary to try npw
to say how such assessments should be done differently,
given that I have developed a strikingly alternative way of
looking at what I feel really occurs in these aissessments.
But we must take care: I have already pointed out the
tendency of large assessing institutions to take aspects of
rhetorical theories and apply them piecemeal to existing
assessment practice. Unprincipled, isolated "improvements"
to test methodology only skew the results in another,
equally inadequate, direction. Given the "institutional"
status of these assessments, two kinds of change must both
occur for there to be genuine improvements in. the
assessment of student writing ability. First, there must
be a deeply-realized change in attitude. That is, the
contextualist framework operates within a thoroughly new
paradigm for thinking about .the event of discourse that
contrasts sharply with the "current-traditional" paradigm
193
still controlling the assessments. But second, the
assessors will probably; only be able to accept a
contextualist assessment of writing insofar as they come to
consider it as methodologically formalized as their current
"science" of testing. I believe that writing assessment
will change: 1) to the extent that the attitudes required
by a contextualist view are accepted by the assessors, and
2) to the extent that such attitudes come to have the kind
of empirical grounding that the assessors take their
current "science" to have. I predict that both changes
will occur, in the following ways. First, we must-
acknowledge that contextualism and the contextualist
implications for texts of the modern-classicists represent
an emerging paradigm in discourse studies. We need not be
surprised that an essentially conservative institution like
institutional testing only reflects the existing paradigm,
even though theoretical advances have deeply criticized it
and moved beyond it. There is not yet a fully emergent
contextualist assessment of student writing in terms
understandable to the assessors. Composition research as a
whole is far from having an explicit paradigm. So what we
must do is not be dogmatic, neither rejecting the explicit
but inadequate "mechanistic" paradigm currently informing
assessment nor the emerging but still controversial
194
contextualist of the modern-classicists. Rather, we must
keep an open mind for the eclecticism necessary in the
early stages of a paradigm. At,the very least, we must
face the challenges put forth by a new, competing
framework. We must at least ask how Burke, Perelman, and
the others have become indisputably major influences in the
rhetorical thinking of this century and yet how innocent of
the implications of their views those involved with
assessment have remained.
Surely the contextualist assessment of student writing
will appeal to the assessing institutions as various
strands of research formalize their methods and; findings.
There is early work in diverse disciplines that shows such
promise. For example, some of Clifford Geertz' work takes
an "ecological" and "anthropological" approach suggestive
of a contextualist perspective. Cognitive psychologists
Robert R. Hoffman and James M. Need have specifically
pursued contextualism in cognitive research. Psychologists
Michael Cole and Sylvia Scribner have presented an
ethnographic psychology of cognition congruent with
contextualist requirements. Sociolinguists M. A. K.
Sialliday, in Language as Social Semiotic, and Dell Hymes,
in Pirections in Sociolinquisties: The Ethnography of
Communication, focus on the situational context of language
195
use in order to analyze the ways context affects our
representation and understanding of the world. Discourse
theorisit James L. Kinneavy, in his "Translating Theory into
Practice in Teaching Composition," speaks for the
ethnomethodologists' assumption that "the methodology to be
used in describing and analyzing a culture is the
methodology which that group itself uses in its internal
interactions to accomplish its own goals"(79).
Ethnographers Kenneth J. Kantor, Dan R. Kirby, and Judith
P. Goetz, have made, according to Louise Rosenblatt, a
lively presentation of the appropriateness of ethnographic
methods for problems of language and literature. Drawing
nearer to educational assessment, Egon G. Guba, in Toward a
Methdoloqy of Naturalistic Inquiry in Educational
Evaluation, and Patrician Carini, in Observations and
Description; An Alternative Methodology for the
Investigation of Human Phenomena, emphasize an ecological
approach in education studies. In assessment per se,
Robert Calfee has stressed ecological wholeness in oral
language assessment. In written language and literacy,
Janet Emig's The Composing Process of Twelfth Graders
exemplifies a case study approach congruent with
contextualism, while her "Inquiry Paradigms and Writing"
establishes appropriate terminology for contrasting
196
contextualist with other approaches to writing research and
instruction. Compositionist Louise Wetherbee Phelps, ■ in
"Composition in a New Key," writes about the "new
ethnOmethodology" in writing research that unites
laboratory methods with ethnographic methods, of
participant-oriented research. : Finally, in writing
assessment, H. C. Clarke, in "The Language-as-Fixed-Effect
Fallacy," has critiqued the limits of non-contextualist
writing assessment.
As the strands of research coalesce into empirical
grounding for the contextualist assessment of writing, we
may see modifications in assessment procedures. For
example, it is likely that multiple texts will be gathered
to broaden their representativeness of the student's
writing ability. Several texts might be taken ort a single
occasion,* or texts might be taken on several occasions. A
choice of topics or completely free choice of topics might
be introduced as an innovation to enhance the writer's
investment in the text-event. Time limits will probably be
increased or eliminated in order to reduce the interference
that time constraints introduce. More of the student's
virtual and actual texts will probably be evaluated: notes,
drafts, and texts that consist of.writer and reader reports
on the writing and reading of the texts. Short of
197
thoroughly overhauling assessment procedures, the readers
might at least be given a contextualist explanation by. the
rater-trainers of what they're doing. That is, the readers
could be told that the results obtained are directly
related to the methods used. Perhaps anonymous assessment,
even while ignoring individual uniqueness, can still
legitimately perform a rough sorting function among large
groups or writing programs. Readers should be aware that
their response is not about the individuals it purports to
be about, but might be able to establish rough distinctions
in groups. Perhaps some "consciousness-raising" in the
rater-training process would at least, clarify the nature of
this unusual kind of "reading." It might help the
assessors to look to _actual, "unscientific" classroom
practice. The same teachers who are hired for those
marathon rating sessions have been acting all along in
their classrooms in a more thoroughly contextualist way
than they are allowed to do when they work for ETS and
NAEP. Actual classroom settings, because they inevitably
contain more knowledge of reader for writer (and vice
versa) are an ongoing "solution" to the "problem" of
anonymous assessment. With greater feedback flowing in
both directions than is allowed in anonymous assessment,
teachers and their student writers already approach ethical
198
concerns in a way more congruent with a contextualist view
than anonymous assessment does. So what should be done
until contextualist assessment presents the kind of
empirical grounding acceptable to large-scale institutional
practices begins with a change in point of view: we must
acknowledge the richer possibilities of writers, texts, and
readers--their complexities, and the necessity of a
multi-perspectival approach in assessing ethos. While
immeasurably more complex and as yet yielding few of the
neat certitudes we saw in'anonymous assessment, the thrust
of this new thinking is that we must try to ground our
understanding of communication in a more thorough view of
the unique conditions of a communicative act.
This is perhaps the final "charge" of the
contextualist framework— that it challenges us to take a
discourse as the total recreation of the writer's total
experience in creating the text. That totality includes
awareness of our own creative act of that
experience— self-reflection on the act. The argument here
has all along been that this is not an ideal removed from
actual occurrence. Rather, it is what we know to be
occurring, consciously or not, when we process discourse.
Our challenge in understanding writing and reading acts is
to bring our theories of that processing into congruence
199
with our practices of that processing, a goal both elusive
and critical.
200
Works Cited
Anastasi, Anne, ed. Testing Problems in Perspective,
Washington, D. C.: American Council of Education,
1966. Originally published by ETS.
Bailey, Richard W., Richard T. Brengle, and Edward L.
Smith, Jr. "Measuring Student Writing Ability."
Reinventing the Rhetorical Tradition. Ed. Aviva
Freedman and Ian Pringle. Conway, AKs L and S Books,
1980. 138-44.
Bamberg, Betty. "Multiple Choice and Holistic Scores: What
Are They Measuring?" College Composition and
Communication 33 (1982): 404-06.
3erlin, James A., and Robert P. Inkster.
"Current-Traditional Rhetoric: Paradigm and Practice."
Freshman English News 8,3 (1980): 1+.
Bernstein, Richard J. Beyond Objectivism and Relativism:
Science, Hermeneutics, and Praxis. Philadelphia: U of
Pennsy1vania P, 1983.
3erthoff, Ann E. Forming, Thinking, Writing: The Composing
Imagination. Rochelle Park, NJ: Hayden, 1978.
Bishop, Arthur, ed. Focus 5^: The Concern for Writing.
Princeton: ETS, 1978.
Booth, Wayne. Modern Dogma and the Rhetoric of Assent.
Chicago; U of Chicago P, 1974.
Britton, James, T. Burgess, N. Martin, et al. The
Development of Writing Abilities (11-18). London;
Macmillan Education, 1975.
Brown, Rexford. "National Assessments of Writing Ability.”
Writing: The Nature, Development, and Teaching of
Writtten Communication. Volume 2 of wFi'ting: Process,
Development, and Communication. Ed. Carl H.
Frederiksen and Joseph P. Dominic. Hillsdale, NJ:
Erlbaum, 1981. 31-38.
Burke, Kenneth. Language as Symbolic Action. Berkeley: U
of California P, 1968.
Burke, Kenneth. The Philosophy of Literary Form.
Berkeley: U of California P, 1973.
201
Burke, Kenneth. A Rhetoric of Motives. New York:
Braziller, 1955.
Calfee, Robert. "Oral Assessment Through Formal
Discussion." Topics in Language Disorders 2 (1982):
Carini, Patricia. Observations and Description; An
Alternative Methodology for the Investigation of Human
Phenomena. North Dakota Study Group on Evaluation
Monographs. Grand Forks, ND: U of North Dakota P,
1975.
Clark, H. C. "The Language-as-Fixed-Effect Fallacy; A
Critique of Language Statistics in Psychological!
Research." Journal of Verbal Learning and Verbal!
Behavior 12 (April, 1973); 335-359.
Cole, Michael. "An Ethnographic Psychology of Cognition."
Thinking. Ed. P. N. Johnson-Laird and P. C. Wason.
Cambridge; Cambridge UP, 1977. 468-82.
Conlan, Gertrude. How the Essay in the College Board
English Composition Test is Scored; An Introduction to!
the Reading for Readers. N.p.; ETS, 1978.
Cooper, Charles, and Lee Odell, eds. Evaluating Writing.
Urbana, IL; NCTE, 1977.
Cooper, Charles, ed. The Nature and Measurement of
Competency in English. Urbana, IL; NCTE, 1981.
Crowhurst, Marion. "Syntactic Complexity and Teachers'
Quality Ratings of Narratives and Arguments."
Research in the Teaching of English 14 (1980); 223-31.
Descartes, Rene. The Discourse on the Method. Paris;
Vrin, 1946.
Diederich, Paul B., John W. French, and Sydell T. Carlton.
Factors in the Judgments of Writing Ability. Research
Bulletin 61-15. Princeton; ETS, 1961’ .
Diederich, Paul B. Measuring Growth in English. Urbana,
IL; NCTE, 1974.
Downie, N. M. Fundamentals of Measurement. 2nd ed. New
York; Oxford UP, 1967.
202
Emig, Janet. The Composing Process of Twelfth Graders.
Urbana, IL: NCTE, 1971.
Emig, Janet. "Inquiry Paradigms and Writing." College
Composition and Communication 33 (1982): 64-75.
Emig, Janet. "The Tacit Tradition: The Inevitability of A
Multi-Disciplinary Approach to Writing Research."
Reinventing the Rhetorical Tradition. Ed. Aviva
Freedman and Ian Pringle. Conway, AK: L and S Books,
1980. 9-17.
Flower, Linda, and John Hayes. A Process Model of
Composition. Document Design Project Technical Report
No. 1. Pittsburgh: Carnegie-Mellon U, 1979.
Freshman Writing Program. Guide to the Freshman Writing
Program. Unpublished document.
Freshman Writing Program. Summer Session I_ Freshman
Writing Program Final Examination. Unpublished
document.
Geertz, Clifford. "From the Native's Point of View: On the
Nature of Anthropological Understanding."
Interpretive Social Science: A Reader. Ed. Paul
Rabinow and William M. Sullivan. Berkeley: U of
California P, 1979. 225-241.
Geertz, Clifford. The Interpretation of Cultures. New
York: Basic Books, 1973.
iGodshalk, Fred I. The Measurement of Writing Ability. New
York: College Entrance Examination Board, 1966.
Graves, Donald H. "An Examination of the Writing Processes
of Seven Year Old Children." Research in the Teaching
of English 9 (1975): 227-41.
Grice, H. P. "Logic and Conversation." Syntax and
Semantics 3 (Speech Acts). Ed. P. Cole and J. Morgan.
New York: Academic, 1975. 41-58.
Grobe, Gary. "Syntactic Maturity, Mechanics, and
Vocabulary as Predictors of Quality Ratings."
Research in the Teaching of English 15 (1981): 75-85.
3uba, Egon G. Toward a Methdology of Naturalistic Inquiry
in Educational Evaluation. UCLA Graduate School of
203
Education Monograph Seris, No. 8. Los Angeles: U of
California, 1978.
Hairston, Maxine. "The Winds of Change: Thomas Kuhn and
the Revolution in the Teaching of Writing." College
Composition and Communication 33 (1982): 76-88.
Halliday, M. A. K. Language as Social Semiotic.
Baltimore: U Park P, 1978.
Hirsch, E. D.,-Jr., and David P. Harrington. "Measuring
the Communicative Effectiveness of Prose." Writing:
The Nature, Development, and Teaching of Written
Communication. Volume 2 of Writing: Process,
Development, and Communication. Ed. Carl H.
Frederiksen and Joseph P. Dominic. Hillsdale, NJ:
Erlbaum, 1981. 189-207.
Hoffman, Robert R., and James M. Need. "General
Contextualism, Ecological Sciences and Cognitive
Research." The Journal of Mind and Behavior 4 (1983):
507-59.
Hymes, Dell. "Models of the Interaction of Language and
Social Life." Directions in Sociolinguistics: The
Ethnography of Communication. Ed. John J. Gumperz and
Dell Hymes. New York: Holt, 1972. 59-65.
Jakobson, Roman. "Concluding Statement: Linguistics and
Poetics." Style in Language. Ed. Thomas A. Sebeok.
Cambridge, MA: MIT P, 1960. 350-78.
Kantor, Kenneth J., Dan R. Kirby, and Judith P. Goetz.
"Research in Context." Research in the Teaching of
English 15 (1981): 293-309.
Kinneavy, James L. A Theory of Discourse. Englewood
Cliffs, NJ: Prentice-Hall, 1971.
Kinneavy, James L. "Translating Theory into Practice in
Teaching Composition: A Historical View and A
Contemporary View." Essays on Classical Rhetoric and
Modern Discourse. Ed. Robert J. Connors, Lisa S. Ede,
and Andrea A. Lunsford. Carbondale, IL: Southern
Illinois UP, 1984. 69-81.
Kirrie, Marjorie. The English Composition Test with Essay.
Princeton: ETS, 1979.
204
Knoblauch, C. H., and Lil Brannon. Rhetorical Traditions
and the Teaching of Writing. Upper Montclair, NJ:
Boynton, 1984.
Labov, William. Sociolinguistic Patterns. Philadelphia: U
of Pennsylvania P, 1972.
Lindemann, Erika. A Rhetoric for Writing Teachers. New
York: Oxford UP, 1982.
Lloyd-Jones, Richard. "Primary Trait Scoring." Evaluating
Writing. Ed. Charles Cooper and Lee Odell. Urbana,
IL: NCTE, 1977. 33-66.
Lunsford, Andrea A., and Lisa S. Ede. "Classical Rhetoric,
Modern Rhetoric, and Contemporary Discourse Studies."
Written Communication 1 (1984): 78-100.
Mellon, John C. National Assessment and the Teaching of
English. Urbana, IL: NCTE, 1974.
Moffett, .James. Teaching the Universe of Discourse.
Boston: Houghton Mifflin, 1968.
Morgan, Jerry L., and Manfred B. Sellner. "Discourse and
Linguistic Theory." Theoretical Issues in Reading
Comprehension. Ed. Rand J. Spiro, Bertram C. Bruce,
and William F. Brewer. Hillsdale, NJ: Erlbaum, 1981.
Mullis, Ina V. S. Using the Primary Trait System for
Evaluating Writing. Denver: NAEP, 1980.
"NAEP Grant Awarded to ETS." NAEP Newsletter 16 (1983):
1+.
National Assessment of Educational Progress. Procedural
Handbook: 1978-1979 Writing Assessment. Denver:
Education Commission of the States, 1981.
National Assessment of Educational Progress. Writing
Achievement, 1969-1979: Results from the Third
National Writing Assessment. Volume I— 17-Year-Olds.
Denver: NAEP, 1980.
Nielsen, Lorraine, and Gene L. Pich^f. "The Influence of
Headed Nominal Complexity and Lexical Complexity on
Teachers' Evaluation of Writing." Research in the
Teaching of English 15 (1981): 65-73.
205
Odell, Lee. "Defining and Assessing Competence in
Writing." The Nature and Measurement of Competency in'
English. Ed. Charles Cooper. Urbana, IL: NCTE, 1981.
Odell, Lee, and Charles Cooper. "Procedures for Evaluating
Writing: Assumptions and Needed Research." College1
English 42 (1980): 35-43.
Payne, David. A. The Assessment of Learning: Cognitive and
Affective. Lexington, MA: Heath, 1974.
Pepper, Stephen C. World Hypotheses: A Study in Evidence.
Berkeley: U of California P, 1942.
Perelman, Chaim, and L. Olbrechts-Tyteca. The New
Rhetoric: A Treatise on Argumentation. Trans. John
Wilkinson and Purcell Weaver. Notre Dame, IN: U of
Notre Dame P, 1969.
Phelps, Louise Wetherbee. "Composition in a New Key: A
Conceptual Framework for Teaching." ms. U of Southern
California. 137-139.
Phelps, Louise Wetherbee. "The Dance of Discourse: A
Dynamic, Relativistic View of Structure." PRE/TEXT 3
(1982): 51-83.
Plato. Gorgias. HammOndsworth, England: Penguin, 1960.
Plato. "Phaedrus." The Works of Plato. Ed. Irwin Edman.
New York: Modern Library, 1956. 263-329.
Polanyi, Michael. Personal Knowledge: Towards a
Post-Critical Philosophy. New York: Harper, 1964.
Purves, Alan C. "In Search of an Internationally-Valid
Scheme for Scoring Compositions." College Composition
and Communication 35 (1984): 426-38.
Rabinow, Paul, and William M. Sullivan. "The Interpretive
Turn: Emergence of an Approach." Interpretive Social
Science: A Reader. Ed. Paul Rabinow and William M.
Sullivan. Berkeley: U of California P, 1979. 1-21.
Richards, I. A. The Philosophy of Rhetoric. New York:
Oxford UP, 1936.
Rohman, D. Gordon, and Albert O. Wlecke. Pre-Writing: The
Construction and Application of Models for Concept
206
Formation in Writing. East Lansing, MI: Michigan
State Ur 1964.
Rosenblatt, Louise M. "Viewpoints: Transaction versus
Interaction— A Terminological Rescue Operation."
Research in the Teaching of English 19 (1985): 96-107.
Scribner, Sylvia. "Modes of Thinking and Ways of Speaking:
Culture and Logic Reconsidered." Thinking. Ed. P. N.
Johnson-Laird and P. C. Wason. Cambridge: Cambridge
UP, 1977. 483-500.
Searle, John R. Speech Acts. London: Cambridge UP, 1969.
Spiro, Rand J., Bertram C. Bruce, and William F. Brewer,
eds. Theoretical Issues in Reading Comprehension.
Hillsdale, NJ: Erlbaum, 1981.
Stallard, Charles. "An Analysis of the Writing Behavior of
Good Writers." Research in the Teaching of English 8
(1974): 206-218.
Stevenson, Charles L. Ethics and Language. New Haven:
Yale UP, 1960.
Tyler, Ralph W., and Richard M. Wolf, eds. Crucial Issues
in Testing. Berkeley: McCutchan, 1974.
Weaver, Richard. "Language is Sermonic." The Rhetoric of
Western Thought. 2nd ed. Ed. James L. Golden,
Goodwin F. Berquist, and William E. Coleman. Dubuque,
IA: Kendall/Hunt, 1978. 202-211.
Wheelwright, Philip. A Critical Introduction to Ethics.
3rd ed. New York: Odyssey, 1959.
White, Edward M., and Leon L. Thomas. "Racial Minorities
and Writing Skills Assessment in the California State
Universities and Colleges." College English 43
(1981): 276-83.
Winterowd, W. Ross. "Dear Peter Elbow." PRE/TEXT 4
(1983): 96-97.
Winterowd, W. Ross. "The Rhetoric of Beneficence,
Authority, Ethical Commitment, and the Negative."
Philosophy and Rhetoric 9.2 (Spring, 1976): 65-83.
207
Winterowd, W. Ross. "The Rhetorical Transaction of
Reading." Colleqe Composition and Communication 27
(1976): 185-91.
Witte, Stephen P., and Lester Faigley. "Coherence,
Cohesion, and Writing Quality." Colleqe Composition
and Communication 32 (1981): 189-204.
Young, Richard. "Arts, Crafts, Gifts and Knacks: Some
Disharmonies in the New Rhetoric." Reinventing the
Rhetorical Tradition. Ed. Aviva Freedman and Ian
Pringle. Conway, AK: L and S Books, 1980. 53-60.
Young, Richard E. "Paradigms and Problems: Needed Research
in Rhetorical Invention." Research on Composing:
Points of Departure. Ed. Charles R. Cooper and Lee
Odell. Urbana, IL: NCTE, 1978. 29-47.
208

Asset Metadata

Core Title 00001.tif

Tag OAI-PMH Harvest

Permanent Link (DOI) https://doi.org/10.25549/usctheses-oUC11257830

Unique identifier UC11257830

Legacy Identifier DP23101

Linked assets

University of Southern California Dissertations and Theses