University of Southern California Dissertations and Theses

Correlation of factors related to writing behaviors and student -developed rubrics on writing performance and pedagogy in ninth-grade students

(USC Thesis Other)

Correlation of factors related to writing behaviors and student -developed rubrics on writing performance and pedagogy in ninth-grade students

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content CORRELATION OF FACTORS RELATED TO
WRITING BEHAVIORS AND
STUDENT-DEVELOPED RUBRICS
ON WRITING PERFORMANCE AND PEDAGOGY
IN NINTH GRADE STUDENTS
by
Kathy Ann Green
A Dissertation Presented to the
FACULTY OF THE ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
in Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF EDUCATION
May 2001
Copyright 2001 Kathy Ann Green
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
UM I Number: 3027723
Copyright 2001 by
Green, Kathy Ann
All rights reserved.
___ ®
UMI
UMI Microform 3027723
Copyright 2001 by Bell & Howell Information and Learning Company.
All rights reserved. This microform edition is protected against
unauthorized copying under Title 17, United States Code.
Bell & Howell Information and Learning Company
300 North Zeeb Road
P.O. Box 1346
Ann Arbor, Ml 48106-1346
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
UNIVERSITY OF SOUTHERN CALIFORNIA
School of Education
Los Angeles, California 90089-0031
This dissertation, written by
Kathy Ann Green
under the direction o f h JULDissertation Committee, and
approved by all members o f the Committee, has been
presented to and accepted by the Faculty of the School
o f Education in partialfulfillment o f the requirementsfor
the degree o f
D o c t o r o f E d u c a t i o n
Date
Dean
Dissertation Committee
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
ii
Table of Contents
List of Tables ....... vii
Abstract ............................................................................................................. ix
Chapter
1. Introduction................................................... 1
Background of the Problem........................................................... 1
Statement of the Problem.............................................................. 3
Importance of the Study............................................................................. 4
Purpose of the Study...................................................................... 4
Significance of the Study................................................................ 4
Research Questions..................................................................................... 5
Overview of the Methodology.................................................................... 6
Assumptions.................................................................... 11
Limitations/Delimitations.............................................................12
Summary........................................................................................................13
2. Review of Literature ........ 14
Introduction ............................................... 14
Performance Assessment.............................................................. 14
Linkage of Performance Assessment to School Restructuring ....15
Historical Precedents of Performance Assessment.........................21
Use of Performance Assessment to Guide Curricular and
Pedagogical Decisions.......................................................... 27
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Chapter
Areas of Conflict Between Advocates and Critics of Performance
Assessment............................. 29
Relationship of Performance Assessment to Graduation
Requirements, Instructional Time, Policy Decisions, and
Accountability ......................................................................... 38
Written Composition ............................................................. 49
Assessment of Written Composition ............ 50
Use of Rubrics as a Teaching Device.................... 54
Conclusion.........................................................................................55
3. Methods and Procedures............................................................................ 57
Introduction.................................................................................... 57
Instrumentation................................................................................ 57
Development of the Student Writing Survey.....................57
Administration of the Student Writing Survey ....... 58
Scoring of the Student Writing Survey..............................58
Classroom Observation.............................................. 58
Interviews............................................ 61
Data Collection................................................................ 62
Description of the Sample ........................................ 62
Method ......................................................................... 64
Limitations and Delimitations...................................................... 65
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Chapter
Data Analysis.............................................................................. 66
Clinical Use of the Results for Improvement of Writing
Instruction............................................................ 68
4. Findings and Discussion............................................................... 69
Introduction ....................................................................... 69
The Student Writing Survey........................................................... 69
Description of the Scales.................................................................72
Scale 1: Value Placed on W riting..................................... 72
Scale 2: Attitudes Toward W riting.................................... 72
Scale 3: Use of Planning Strategies................................ 75
Scale 4: Use of Revising and Editing Strategies...............75
Scale 5: Teachers’ Comments on Completed Papers 78
Scale 6: Teacher Feedback on Writing ........................ 78
Scale 7: Use of a Rubric......................................................78
Findings......................................................................................... 82
Findings—Scale 1: Value Placed on Writing ..............82
Findings—Scale 2: Attitudes Toward W riting................. 83
Findings—Scale 3: Use of Planning Strategies................ 89
Findings—-Scale 4: Use Of Revising and Editing
Strategies..................................................................90
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Chapter
Findings—Scale 5: Teachers’ Comments on Completed
Papers............................................ 98
Findings—Scale 6: Teacher Feedback on W riting...........99
Findings—Scale 7: Use of a Rubric in W riting.............. 108
Summary ......... I l l
5. Summary.................... 113
The Problem...................... 113
Methodology............................................................ 114
Selected Findings........................................................................................114
Conclusions..................................................................................................115
Recommendations.......................................................................................116
Expansion of Study to Other Grade Levels............................. 116
Demographic Factors..................................................................... 116
Longitudinal Effects.................................................................... 117
Correlation to Other Measures............................ 117
Effect of Teacher Training.......................................................... 118
Types of Writing ...... 118
Final Thoughts............................................................................................ 118
Works Cited ......... 120
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Appendices ........................... 131
A. Student Writing Survey..................................... .......................... 132
B. Six-Point Rubric: Writing Performance Levels for
Comparison Contrast Essay............................. 137
C. Student Writing Samples from High, Medium, and Low Score
Levels Based upon Student-Developed Rubric (Table 3.1).. 148
D. Initial Interview Questions for Teachers.................................... 158
E. Follow-up Interview Questions for Students ............................ 160
F. Follow-up Interview Questions for Teachers............................ 162
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
List o f Tables
Table
1.1 Inquiry Matrix ......... 10
2.1 National Testing Chronology............................................................................ 24
2.2 Total Estimated Costs of Kentucky’s Assessments System, 1995-96 .......... 37
2.3 National Assessment of Educational Progress 1998 Writing Performance ... 53
3.1 Sample Student-Developed Rubric.................................................................. 60
4.1 Empirically Derived Scales................................................................................. 71
4.2 Scale 1: Value Placed on W riting........................................................................73
4.3 Scale 2: Attitudes Toward W riting..................................................................... 74
4.4 Scale 3: Use of Planning Strategies................................................................. 76
4.5 Scale 4: Use of Revising and Editing Strategies............................................. 77
4.6 Scale 5: Teachers’ Comments on Completed Papers........................................ 79
4.7 Scale 6: Teacher Feedback on W riting............................................................... 80
4.8 Scale 7: Use of a Rubric in W riting.................................................................... 81
4.9 Findings - Scale 1: Value Placed on W riting..................................................... 84
4.10 Findings - Scale 2: Attitudes Toward W riting................................................. 85
4.11 Relationship of Value Placed on Writing and Attitudes Toward Writing
to Writing Performance ................................................................... 88
4.12 Findings - Scale 3: Use of Planning Strategies............................................. 91
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table
4.13 Findings - Scale 4: Use of Revising and Editing Strategies ........... 92
4.14 Relationship of Use of Writing Process Strategies (Scales 3 and 4) to
Writing Performance................................................................................ 95
4.15 Relationship of Ethnicity to Writing Performance...................................... 97
4.16 Findings - Scale 5: Teachers’ Comments on Completed Papers ......... 101
4.17 Findings - Scale 6: Teacher Feedback on W riting......................... 102
4.18 Relationship of Use of Conventions to Writing Performance.......................106
4.19 Relationship of Use of Conventions, Writing Performance, and SAT 9
Scores........................................................................................................107
4.20 Findings - Scale 7: Use of a Rubric in W riting......................................... 110
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Abstract
Cognizant of the need for students to improve their ability to write, this study’s
purpose was to identify and describe factors which influence student writing; investigate
how involvement of students in developing rubrics of accepted written work affect their
writing performance; and examine the pedagogical effects of rubric-based writing
instruction on students and teachers. Relationships between and among these factors to
writing performance and writing instruction were analyzed.
Background measures identified on the National Assessment for Educational
Progress (NAEP) tests of writing achievement, along with this study’s emphasis on
student-developed rubrics, were used to define variables relevant to student writing.
Demographic factors, including ethnicity and language proficiency, also were considered.
Selected high school teachers and their ninth grade classes located in one school district
on the Central Coast of California participated in this study.
Factors related to student writing were identified and developed into empirically
derived scales entitled: Value Placed on Writing, Attitudes Toward Writing, Use of
Planning Strategies, Use of Revising and Editing Strategies, Teachers’ Comments on
Completed Papers, Teacher Feedback on Writing, and Use of a Rubric in Writing.
Data gathered through administration of a Student Writing Survey, interviews,
examination of student writing samples, review of teacher lesson plans, longitudinal
district assessment results, longitudinal nationally norm-referenced test scores, student
demographic data, and classroom observations formed the basis for examination.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
X
Profiles of each school were described in terms High, Medium, and Low response
to survey items and test results. Analyses were conducted between and among scales and
test data.
The findings of this study and the development of the Student Writing Survey
establish a methodology and instrumentation which may be used by districts, schools, and
teachers to identify factors that contribute to changes in the techniques students use to
evaluate written work and the pedagogy teachers use to instruct writing.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
1
Chapter 1
INTRODUCTION
Background of the Problem
One of the major responsibilities of public education in the United States has been
and continues to be the teaching of the “3 R’s: Reading, writing, and ‘rithmetic.” Of
these, writing has occupied a central place in American curricula for well over one
hundred years. As the National Council of Teachers of English Commission on
Composition stated, “Writing is a powerful instrument of thought. In the act of
composing, writers learn about themselves and their world and communicate their
insights to others. Writing confers the power to grow personally and to effect change in
the world” (Commission on Composition, National Council of Teachers of English,
1979). The importance of writing as a “powerful instrument” for students to express their
thoughts is a critical attribute of a well educated person. However, great concern and
consternation regarding the levels of achievement of students in written composition have
arisen over the past twenty five years.
According to the Office of Educational Research and Improvement (as cited in
Alkin, M. C., ed, 1992), “students continue to perform at ‘minimum level,’ and the
proportion of competent writers among advantaged and disadvantaged students alike
remains low” (p.57). This finding has been substantiated by the National Assessment of
Educational Progress (NAEP) which showed significant declines in the quality of writing
for thirteen- and seventeen-year olds between 1969 and the 1980’s (Brown, R., 1981;
Perl, S., 1979). Throughout the 1980’s and early 1990’s, writing performance remained
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
2
relatively stable at fourth and eleventh grades, with a slight upturn at eighth grade
(Mullis, I. et al., 1994).
This finding continued with the 1996 NAEP writing assessment. “The
percentages of students attaining each of the performance levels on the writing scale in
1996 were not significantly different from those in 1984...However, the percentages of
students in grade 8 who attained at least Levels 200 and 250 in 1996 were lower than the
percentages in 1984...Almost all eleventh graders reached at least Level 150 and 200,
and the vast majority reached at least Level 250, in both 1984 and 1996. However, there
was a decrease between the two assessment years in the percentages of students at grade
11 who demonstrated performance at or above Levels 250 and 300” (National Center for
Education Statistics, 1998.) Unfortunately, the relative stability indicates a large number
of students operating at unacceptably low levels of literacy.
Accompanying this decline in writing achievement are public and political
demands for high standards for all students, accountability, and a world-class education.
These demands are evidenced in program such as: Goals 2000, Improving America’s
Schools Act (IASA), recent changes in Title I requirements, comparative international
data, social promotion legislation, high school exit examinations, and numerous high-
stakes tests, school accreditation, and state-imposed rewards and sanctions. Many of
these initiatives share a common theme: The quality of a program shall be judged by
student achievement.
As measures of student achievement stretch “beyond the bubble” of standardized
testing to more authentic, performance-based assessments, writing samples take center
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
3
stage as one of the premier types of student work which can be used to evaluate
individual student achievement, as well as school, district, state, and national objectives.
Several high stakes writing assessments are currently used in the United States, including
the Educational Testing Service’s Advanced Placement (AP) tests, College Board’s
Scholastic Assessment Tests (SAT), the National Assessment of Educational Progress
(NAEP) program, and various state writing assessments, such as California’s Golden
State Examinations in English-language arts and history/social science. The one element
shared by all of the above-named assessments is their use of holistic scoring protocols
based on a rubric (criteria) to determine the level of writing performance.
Statement of the Problem
“Assessment has moved to the forefront in the search for solutions to our
country’s educational problems. Faced with the jumbled demands for new teaching
strategies, new instructional objectives, and new performance standards, progressive
states, school districts, and individual schools are experimenting with new forms of
assessment that can do more than merely measure learning.. .Parents and policymakers
want relevant assessments that reflect the capabilities our children will need to become
successful adults in a complex society. Educators want assessments that model high
quality instruction and promote student learning” (Educational Testing Service, 1995).
No entity in the testing arena (from individual teachers to educational associations
to commercial test publishers) argues with the need to include performance assessments
in a comprehensive, well-balanced assessment program. A cornerstone of the educational
reform movement is performance assessment with the ostensible raison d ’ etre to provide
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
4
better methods of evaluating student learning. If quality performance assessments are to
have a positive effect on teaching and learning, then we must thoroughly and critically
examine the implementation of these assessments in the classroom.
Given unsatisfactory levels of student achievement in writing, educators must
seek more effective instructional methods. Rubrics currently used in local, state, and
national assessments clearly establish the standards and criteria against which student
writing is evaluated. Since rubrics capture the essence of quality writing, focusing
instruction on those criteria should improve writing skills. Even more effective, however,
may be to teach students to internalize those criteria through developing their own rubrics
and applying those criteria to written work.
IMPORTANCE OF THE STUDY
Purpose of the Study
This study shall identify and analyze factors which relevant to writing, with an
emphasis on describing the relationship between using student-developed rubrics and
writing performance and pedagogy. By targeting instruction on rubric criteria, writing
achievement may be improved. This improvement may be further increased by having
the student as an active participant in identifying the criteria for quality written work and
by applying those criteria to improve his writing.
Significance of the Study
Writing is taught in every school to every student across the United States. The
ability of students, and by extension, citizens and employees, to communicate effectively
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
5
in written form is a national imperative. It takes on even more importance as demands of
the Information Age are factored in.
Writing continues to be an essential part of English-language arts instruction and
applied across the curriculum. Writing instruction is one area in which classroom
practice reflects research as evidenced by the widespread implementation of “process
writing” or “writing as a process” (Applebee, A. N., 1986). Absent, however, is a
generally accepted notion of what constitutes effective assessment of student writing
(Huot, B„ 1990; Witte, S. P., Trachael, M„ & Walters, K., 1986).
Previous experience in the evaluation of educational programs underscores the
need to attend to specifics of implementation at the local level. The “Black Box”
approach, where a program was introduced and then evaluated in terms of student
outcome measures with little or no attention to what went on in-between, proved to be
sterile in the early days of mandated evaluations of federal educational programs in the
1960’s. The actual implementation of a program in the classroom is crucial in
determining its impact. Affecting real change in classroom teaching and assessment
practices requires time, the learning of new skills, the development of new
understandings, and a change in beliefs. Therefore, this study will focus on the
classroom, and specifically, the intervention of student-developed rubrics as an effective
instructional strategy to promote improvement of writing.
Research Questions
1. What factors influence student writing, i.e., value placed on writing, attitudes toward
writing, writing process strategies, teacher comments and feedback, use of a rubric?
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
6
2. In what ways do involvement of students in developing rubrics of accepted written
work affect their writing performance?
• In what ways did students change techniques for evaluating their written work?
3. What pedagogical effects do rubric-based writing instruction have on students and
teachers?
• What strategies are teachers using to focus students on critiquing their own
written work?
• What have teachers learned as a result of using student-developed rubrics as an
instructional strategy?
Overview of the Methodology
Selected high school ninth grade English classes, 141 students, participated in this
study. Students attended classes in the same district, but at two different high school
campuses on the Central Coast of California. This was a descriptive study; no control
group will be designated.
Two teachers were involved in the study. One teacher had over fourteen years of
teaching experience and was a fellow of the Central Coast Writing Project; the second
teacher had two years of experience and had not participated in a Writing Project. Each
teacher was trained in holistic scoring protocol, and participated in holistically scoring
student work using district writing rubrics. Teachers received a set of materials,
including descriptions of grade-appropriate writing types, a rubric for each writing type,
and resource books containing effective instructional strategies for teaching writing.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
7
These materials, as well as teacher-developed materials, were used with classes involved
in the study.
Each participating teacher was visited at three times. The purposes of these visits
was to observe classroom instruction, to analyze lessons, to administer the Student
Writing Survey, and to conduct interviews of teachers and students. Specific data
gathering techniques included:
1. An initial interview of teachers (Appendix D) to identify classroom
instructional practices used to prepare students for classroom writing
assignments and the district writing assessment;
2. A writing survey was administered to students to elicit previous experience in
classroom writing assignments and district writing assessments, and effects of
rubric-based writing instruction on writing performance; and
3. Structured follow-up interviews with students (Appendix E) and teachers
(Appendix F) to provide context for responses to interviews and survey and to
give feedback on experimental sessions.
Teachers provided extensive instruction to students on using rubrics to evaluate
writing and using student-developed rubrics to evaluate and to improve writing. This
instruction included the following activities:
• Sharing and evaluating (as a group) samples of literature to illustrate strong
and weak models of writing in specific writing types;
• Evaluating the work o f students (anonymously) and discussing how to
improve that work;
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
8
• Analyzing rubric-related skills; and
• Applying rubrics to students’ own and others’ writing.
Baseline information on student writing achievement was ascertained using scores
from students’ middle school and high school performance on district writing
assessments (dating from September 1997 - May 1999), in-class writing samples
beginning September 1999, and the Stanford Achievement Tests (SAT 9) for 1997-98
and 1998-99.
Student writing was evaluated based on classroom assignments and district
writing assessments. Classroom assignments were graded by the teacher; district writing
assessments were scored by district teachers calibrated for inter-rater reliability. District
assessments were read using a double blind scoring protocol to control for possible rater
bias.
Prior to the beginning of the study and at the end of the study, students wrote
essays under the following conditions: Students had three 45-minute periods to write
essays on assigned topics. Students were provided time to prewrite, create a rough draft,
read over the rough draft, revise, and edit. Students were allowed to use dictionaries and
thesauri during testing, but could not confer with the teacher or with each other. The
pretest and district assessment prompts were the same for each classroom. Classroom
assignments targeted the same writing genre, but prompts differed.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
9
An Inquiry Matrix (Table 1.1) served as the structural foundation for the study.
The Matrix provided a visual overview showing the relationships among the questions
considered essential to the research, the data sets necessary to answer the inquiry
questions, and the types of information upon which the findings of the study were based.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced w ith permission o f th e copyright owner. Further reproduction prohibited without permission.
10
Table 1.1 Inquiry Matrix
Inquiry Questions D ata Needs M eans of Gathering Data
1. What factors influence writing behaviors and
performance?
Student responses to survey; Writing samples;
Demographic profile; Assessment data
Student survey and interviews; Student
writing samples; District writing
assessment scores
2. In what ways do involvement of students in
developing rubrics of quality written work
affect their writing performance?
• In what ways have students changed their
techniques for evaluating their written
work?
a. Students’ past writing performance Scores on previous district writing
assessments; Grades in previous
English classes; Student survey and
interviews; Interviews with past
English teachers; Student writing
samples prior to this study
Students’ past experiences with using rubrics Student survey and interviews;
Interviews with past English teachers;
Student writing samples prior to this
study
Students’ current writing experiences Student survey and interviews;
Interviews with current English
teachers; Current student writing
samples; Scores on current district
writing assessments
3. What pedagogical effects do rubric-based
writing instruction have on students and
teachers?
Previous instructional strategies used to teach
writing
Interviews with teachers; Previous
lesson plans; Teaching materials
Current instructional strategies used to teach
writing
Current lesson plans; Classroom
observation; Interviews with teachers;
Student survey and interviews;
Samples o f student writing and teacher
feedback
11
Assumptions
For the past five years, the district involved in this study administered direct
writing assessments which were scored by teachers using genre-specific rubrics. During
the course of this experience, it was been noted that those doing the scoring seemed to
learn a lot about writing. In addition to learning about how students write, teachers
became much clearer on the criteria which form the target for quality writing (i.e., the
rubrics). Through the use of anchor papers and calibrating for inter-rater reliability,
teachers quickly recognized levels of writing performance and identified specific areas
for improvement. As a result, teachers reported that they were better able to teach writing
because they understood what they needed to teach.
The first assumption of this study was that if teachers have learned so much about
writing through using rubrics to analyze student work, then those same benefits would
accrue to students. In addition, writing may be improved if students developed their own
rubrics based on district criteria, and then applied those rubrics to their own and others’
written work.
A second assumption maintained that student writing would improve to the extent
to which instruction was focused on the criteria established in rubrics. If those criteria
were directly taught to students and then combined with specific and immediate feedback
and self-evaluation on their work, writing would improve. Therefore an impact on
student achievement may be made by targeting instruction, aligning assessment with
classroom instruction, and using student self-assessment.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
12
Limitations/Delimitations
Limitations
1. Students involved in this study were those enrolled in the target teachers’
English classes. A true random sample was not ensured.
2. This study was conducted in the researcher’s own district. There was a
possibility that personal biases and perceptions may have influenced the
selection of teacher participants, data analysis, or classroom observations.
3. Findings of the study may not be generalizable to other high school students
in other demographic settings and for students significantly below average on
norm-referenced measures, such as the Stanford Achievement Test.
Delimitations
1. This study was delimited to the high school in a unified school district on the
Central Coast of California. Teachers instructed students on development
and application of rubrics to written work.
Classroom assignments and district assessments were used to determine
writing improvement. Teachers involved in the study graded classroom
writing assignments. District scorers evaluated the district assessment using a
double blind, holistic scoring technique.
2. The study was further delimited to two classrooms at the ninth grade taught by
teachers at two different high schools.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
13
Summary
Cognizant of the need for students to improve their ability to write, this study
identified and described factors which influence student writing, specifically the effects
of student-developed rubrics on students’ ability to evaluate and improve their own
writing and the work of others. This study added to the literature in the areas of
performance assessment and written composition. More importantly, this study
addressed the under-examined area of writing at the high school level, as well as
increased the virtually non-existent body of research which examines the relationship
between rubrics and improvements in student writing.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
14
Chapter 2
REVIEW OF LITERATURE
Introduction
Assessment, particularly performance assessment such as writing samples, are
central to the current focus on educational reform. Several bodies of literature relevant to
this study were reviewed:
• the linkage of performance assessment to school restructuring;
• historical precedents of performance assessment;
• use of performance assessment to inform curricular and pedagogical
decisions;
• areas of conflict between advocates and critics of performance assessment;
• how performance assessment affects graduation standards, instructional time,
policy decisions, and accountability;
• written composition and assessment of written composition; and
• the use of rubrics as a teaching device to improve student writing.
Performance Assessment
The rationale for moving from traditional, standardized testing formats to
performance-based assessments comes from legislators, policy makers, researchers, and
practitioners (Mitchell, 1992; Newman, 1991; Resnick and Resnick, 1989; Shepard,
1989). Common arguments against standardized testing include:
• Traditional assessment measures no longer reflect desired skills and outcomes
(Resnick and Resnick, 1989; Wiggins, 1989a).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
15
• Traditional measures limit rather than promote equal opportunities to learn.
Nationally, students from lower socioeconomic groups perform less well on
standardized achievement tests than do their more affluent peers (Garcia &
Pearson, 1991; Hiebert, 1991; Mitchell, 1992; Resnick & Resnick, 1989).
• Large-scale accountability measures are high-stakes for teachers, resulting in
negative unintended effects on classroom practice (Shepard, 1990). Incentives
and disincentives which accrue to such high-stakes testing influence teachers
to teach to the standardized test, thus narrowing content and instruction to
meet the expectations of the test (Darling-Hammond & Wise, 1985; Dorr-
Bremme & Herman, 1986).
If traditional assessment tools no longer measure valued outcomes, tend to
negatively affect instruction, and discriminate against large segments of the population,
why not design measures which will shape instruction in desirable ways? (Wiggins,
1989b, 1992). Resnick and Resnick advocate the same principle, “...build assessments
toward which you want educators to teach.” (p. 66). “Assessments should be designed so
that when teachers do the natural thing—that is, prepare their students to perform well—
they exercise the kinds of abilities and develop the skills and knowledge that are the real
goals of educational reform” (Resnick & Resnick, 1989).
Linkage of Performance Assessment to School Restructuring
Schlechty (1990) asserts that “restructuring means altering systems of rules, roles,
and relationships so that schools can serve existing purposes more effectively or serve
new purposes altogether.” The linkage of performance assessment and school
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
16
restructuring resides at the intersection of two questions: “What should students know
and be able to do?” and “What evidence shall we accept of student achievement?”
The current wave of educational reform may be traced to the publication of a
seminal report to the Secretary of Education entitled A Nation At Risk in 1983. This
document drastically changed the rhetoric of education reform by linking the financial
security and economic competitiveness of the nation and its education system. The
chilling opening words declared war on the status quo: “The educational foundations of
our society are presently being eroded by a rising tide of mediocrity that threatens our
very future as a nation and a people. ... We have, in effect been committing an act of
unthinkable, unilateral educational disarmament” (National Commission on Excellence in
Education, 1983). This clarion call was followed in September 1989 by an education
summit. Shepard (1993) explains that at this summit, President Bush and the nation’s
governors agreed upon six broad goals for education to be reached by the year 2000.
Reform initiatives which followed reflect important common features. First, they
presuppose that most policy-making should be made at the state and local levels (Keeves,
J. P., 1994). This may be attributable to the historical and legal precedents of local
control and the silence of the United States Constitution on the subject of education.
Despite the legally questionable role of the federal government in education, there have
been attempts by the national government to influence the reform movement through
linking federal funds, such as Goals 2000 and Improving America’s Schools Act (e.g.,
Title I), to the implementation of certain measures of accountability and high standards.
There has also been some interest in the concept of a national test, but with states taking
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
17
up the mantle of improving education in earnest through setting rigorous standards and
testing programs, most of the conversation around the creation of a national test has
subsided somewhat. Today every state in the United States either has state-level
standards or, as in the case of Iowa, has delegated that responsibility to local districts.
Therefore, policies with regard to standards and the assessments which measure student
achievement of those standards, are clearly within the domain of the state and local levels
of governance.
The second common feature of reform initiatives is the notion that education
quality can be promoted by the setting of stringent standards for educational achievement.
The standards movement, as an outgrowth of the restructuring effort, has exploded in the
past decade. Standards represent an attempt to clearly define what students should know
and be able to do.
As a result of the President Bush’s education summit, two groups were
established to implement the new education goals: The National Education Goals Panel
(NEGP) and the National Council on Education Standards and Testing (NCEST). These
two groups were charged with addressing questions such as: What is the subject matter to
be addressed? What types of assessments should be used? What standards of
performance should be set? (Marzano, R. J. and Kendall, J. S., 1996). These questions
energized an enormous amount of activity on the part of national subject-matter
organizations to establish standards relevant to their content areas. The National Council
of Teachers of Mathematics (NCTM) had actually preempted the public mandate for
standards with its publication of Curriculum and Evaluation Standards for School
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
18
Mathematics in 1989. The success of this work prompted the National Academy of
Sciences to urge Lamar Alexander to underwrite national standards-setting efforts in
other areas. According to Diane Ravitch, then Assistant Secretary of Education,
Alexander “bankrolled the projects out of his office’s discretionary budget (in
Diegmueller, K., 1995). The flurry of activity in the standards-setting field for the past
twenty five years has touched every state and literally every content area. Assessment is
inextricably linked to the clear definition of standards - what students should know and
be able to do. Standards which are targeted to complex thinking skills, such as problem
solving, or processes, such as writing, form the foundation for the use of performance
assessment in the reform movement.
The third common element of reform maintains that such standards are to be
closely monitored through mediums such as student examinations and strict adherence to
teacher certification requirements (Keeves, J. P. 1994). This point represents the nexus of
the restructuring movement and performance assessment.
One of the quintessential changes from a traditional education to a restructured
learning environment is the shift from the teacher-centered classroom, i.e., focus on what
the teacher is doing or teaching, to a student-centered learning environment, i.e., focus on
what the student is doing or learning. To accomplish this, education reformers see the
teacher’s role as changing from disseminator of information to facilitator of learning.
Teachers move from their role as “sage on the stage” to “guide on the side.” This change
in roles has been accompanied by a shift in emphasis from evaluating teaching behaviors
to evaluating student learning.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
19
Evaluating student achievement through performance assessments is not a new
strategy. Good teachers have always judged and monitored their students’ progress
through written assignments, experiments, observation, oral examination, and research
projects. “What is new in the current reform effort is the systematic shift toward
schoolwide performance assessments away from multiple-choice tests for measuring
instruction and accountability” (Khattri, N., M. B. Kane, & A. L. Reeve, 1995).
The use of performance assessments is often considered to be a reflection of
“good instruction,” and as such, congruent with the restructuring movement. However,
several concerns must be addressed before this relationship is accepted as axiomatic:
• Is there a trade-off between good instruction and measurability and equity?
Grant Wiggins believes that the trade-off between good instruction and
measurability is “a significant issue at the state level, but it’s a nonissue at
the local level.” As a teacher “you can factor out these things with your
professional judgment and assessments over time. Teachers can have
more leeway for complex assessments in their classrooms, where they do
not need the high level of precision necessary to make high-stakes tests
accurate and fair” (Association for Supervision and Curriculum
Development, 1995).
• How do restructuring goals such as student choice, interdisciplinary
instruction, and cooperative learning, synchronize with performance
assessments?
Student Choice'. According to Jay McTighe of the Maryland Assessment
Consortium, “Student choice has lots of benefits, but you want to make
sure that opportunities for choice don’t get in the way of what you’re
trying to assess” (Association for Supervision and Curriculum
Development, 1995). Joan Herman, Associate Director of the Center for
Research on Evaluation, Standards, and Student Testing, agrees with
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
20
McTighe, “From a measurement perspective, giving students choices is a
terrible dilemma” (Association for Supervision and Curriculum
Development, 1995). Some options or topics may yield easier projects
than others; allowing students to choose subjects, resources, or methods
which have instructional benefits, but complicate assessment.
Interdisciplinary Tasks: Interdisciplinary assessment is most effective when a
teacher is familiar with students’ progress in several areas. For example, a
writing assessment on a history subject is hard to evaluate unless the
teacher can distinguish the level of performance in writing versus the
students’ ability to communicate content knowledge in history.
Cooperative Learning: Herman acknowledges that “Any kind of group
activity confounds the measurement of individual ability” (Association for
Supervision and Curriculum Development, 1995). Even though individual
accountability may be included as a component in a cooperative project,
the performance of other students, and the students’ ability to work
together as a team, may affect the individual’s performance.
Performance assessment supports the goals of restructuring in a number of ways:
1. Performance assessments are often developed, administered, and scored on the
local or state level, thereby affording a higher degree of alignment among
curriculum, instruction, and assessment.
2. Performance assessments are able to measure the kinds of knowledge, skills,
and habits of mind found in many content area standards in a more authentic
manner than multiple-choice tests. Performance assessment allows students to
“show what they know” in ways that are compatible to their learning styles
and interests. In addition, performance assessment more closely mirrors what
we are discovering through cognitive psychology regarding how humans
learn. To that end, performance assessment reflects principles of
constructivism, rather than behaviorism.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
21
3. Performance assessment provides more robust and three-dimensional methods
by which to measure accountability than the dipstick provided by an on-
demand, single administration norm-referenced test.
4. The data derived from performance assessment often provides evidence of the
student’s thinking and understanding. These data may be used to diagnose
and prescribe instruction for individual students in a more effective manner.
Historical Precedents of Performance Assessment
The use of performance assessment is not new. As one reviews the history of
performance assessment, the saying, “Everything old is new again.” springs to mind. As
Stiggins (1994) points out, performance tasks have been used for decades, at least, and
even centuries if one defines them in a broad sense. After all, the Socratic Seminar which
is popular in classrooms today, stems from the kind of provocative questioning and
thoughtful oral debate engaged in by Socrates and Plato in Ancient Greece. There is
evidence that performance tasks were used in Ancient China to assess the qualifications
of candidates for government positions.
More recently, in formulating an assessment mechanism to be used in his
Coalition of Essential Schools, Ted Sizer (1984, 1992) reached back to the eighteenth
century to uncover a performance assessment used in the early American academies and
the common schools of that era.
The exhibition, as practiced then, was an occasion of public inspection when some
substantial portion of a school’s constituency might show up to hear students
recite, declaim, or otherwise perform. The constituency might thereby satisfy
itself that the year’s public funds or tuitions had been well spent and that some
cohort of young scholars was now ready to move on or out (McDonald, J. P.,
1991).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
22
Siegel (1986) explains that performance tasks were developed within domains
involving technical “jobs” such as carpentry, welding, brick laying, lathe-turning, and
electronic assembly. To demonstrate competence in these areas, students performed the
job-related skills. They used the lathe, laid bricks, welded, and so on.
In 1969, the federal government began implementing a cross-sectional study of
educational achievement called the National Assessment of Educational Progress
(NAEP). The purpose of this assessment was to study the knowledge, skills, and
performance of schoolchildren in the United States. NAEP assessed 9-, 13-, and 17-year
olds in the subjects of science, writing, and citizenship in its 1969 test administration.
The writing assessment was a major benchmark in the use of performance assessment on
a national level. The test is a cross-sectional study of samples of students in which scores
for individual students do not exist. The data are intended to provide information on the
system of education, not on individual students.
Two schools of thought exist as to the reliability of the data obtained by NAEP.
First, because it is not a mandated test, it does not carry the weight of a “high stakes
assessment.” As such, teachers tend not to teach to the test and the results may be
considered more accurate as a dipstick measurement of student performance without the
presence of external pressures to perform well or to suffer the attendant consequences of
poor performance. Interestingly, the exact same argument is presented to cast doubts on
the reliability of the NAEP as a national index of student achievement. Will a student
perform his best if there is no individual accountability - no incentives and no
consequences? Will students be motivated to do their best if “it doesn’t count?”
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
23
The history of testing in the United States demonstrates an interesting, spiraling
pattern of assessment from local, individual classroom performance assessment to large-
scale, standardized testing, to large-scale performance assessment, to a blending of large-
scale performance and standardized testing (Table 2.1). “The past 150 years have seen
notable developments in educational assessment that will continue to influence the field
in the next century: Emergence of external accountability; advances in the technology
and science of assessment; advent of large-scale testing in schools; impact of large-scale
assessment on instruction and the call for reform” (Asp, E., 2000).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 2.1 National Testing Chronology
24
1845
Horace Mann and Samuel Gridley Howe asked the Boston School Committee to
administer a written examination to the city’s students in place of oral exams.
These written tests were intended to be standardized measures of performance for
the purposes of objectively classifying students and providing a means to
externally monitor the effectiveness of the state school system.
1880-1920
Standardized tests in various subjects were introduced in United States schools.
These tests were designed to justify school funding to taxpayers who wanted
objective evidence of students achievement.
1900
College Entrance Examination Board (CEEB) was established. CEEB
administered essay exams to students entering northeastern schools with the
purpose of driving boarding school curriculum, not selecting applicants.
1915
First multiple choice format test, the Kansas Silent Reading Test, was developed
by Frederick Kelly, and later modified by Arthur Otis, a psychologist at Stanford
University. Otis is credited as the father of multiple-choice testing, which
provided the format for most large-scale tests.
1914-18
First mass administration of objective mental tests, Army Alpha, was given to
more than 1.7 million soldiers to measure their intelligence. The test was used to
sort soldiers and assign them to various positions, including the selection of
officer candidates from the pool of Army recruits. These tests led to widespread
use of intelligence testing in schools.
1923 -26
The Scholastic Aptitude Test (SAT) was developed. The test was renamed the
Scholastic Assessment Test in 1994. Early tests and current tests include
performance assessments such as constructed response and writing sample.
1929
The University of Iowa created the first statewide student tests, the Iowa Test of
Basic Skills and the Iowa Test of Educational Development. These tests were
offered to schools on a voluntary basis.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
1933
Reynold Johnson parlayed a childhood prank, scratching pencil marks on the
outside of spark plugs to keep the engine from turning over, into a machine that
could electrically sense whether pencil marks on a scoring sheet were in the right
places.
1941
Scoring of College Board’s SAT was based on performance of a norm group of
test-takers.
1947
Educational Testing Service (ETS) was established as a nonprofit corporation
devoted to measurement and research.
1950’s
The electronic scoring machine was developed by E. F. Lindquist.
1963-64
The First International Mathematics Study tested students ages 13 and 17/18.
1965
Elementary and Secondary Education Act of 1965 was passed. This act required
school districts receiving Title I funds to use objective measures to identify
children eligible to receive services and to evaluate program goals. This
requirement accelerated large-scale testing nationally.
1969
National Assessment of Educational Progress (NAEP) was created to evaluate and
report on student performance in the United States at the state and national levels.
NAEP, a criterion-referenced test which includes performance assessments,
pioneered several new measurement techniques such as matrix sampling and
open-ended items with multiple choice questions.
1977
Average SAT scores declined by a total of 81 points, shocking the public into
rethinking the competence of the United States education system.
1983
A Nation at Risk, a report commissioned by the Department of Education to
examine the quality of education, was published warning of the “rising tide” of
mediocrity in education. It called for standardized achievement tests to be
administered at major transition points from one level of schooling to another.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
26
1989
The National Governors Association and then-President Bush convened the first
educational summit in Charlottesville, Virginia.
1990
President bush announced the National Education Goals for the year 2000; then he
and Congress established the National Education Goals Panel.
1991
Secretary of Education Lamar Alexander asked Congress to establish the National
Council on Education Standards and Testing which was charged to reach a
bipartisan consensus on national standards and testing.
1995
The Third International Mathematics and Science Study was given to 4th and 8th
grade students.
46 states have assessment programs; 41 report using multiple-choice items as part
of their state assessments.
1998
President Clinton issued his “Call for Voluntary National Testing,” rigorous
national tests of each fourth-grade student’s reading skills and eighth-grade
student’s mathematical skills.
(Association for Supervision and Curriculum Development, 1997;
Taylor, K. and Walton, S., 1998; Asp, E., 2000)
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
27
Use of Performance Assessment to Guide Curricular and Pedagogical Decisions.
Although the use of performance assessments is not new in education, their
effects have only recently been studied in depth. There is a growing body of research that
the use of performance assessment alters what happens in the classroom (Borko, H.,
Flory, M., & Cumbo, K, 1993; Falk, B. & Darling-Hammond, L., 1993; Kentucky
Institute for Education Research, 1995; Smith, M. L. et al., 1994).
Research is beginning to show that performance assessments can be an effective
instructional tool, but only if teachers receive sufficient training and support...
Performance assessments inspire teachers to integrate instructional practices so
that they become more ‘seamless,’ use a wide array of instructional strategies, ask
students to describe their thinking as they worked on their assessments, and ask
students to evaluate their own thinking and their own skill” (Khattri, N, M. B.
Kane, & A. L. Reeve, 1995).
Evidence is beginning to accrue that performance assessments provide the means
for improving teaching and learning (Borko et al.1993, Falk and Darling-Hammond
1993, Gearhart et al. 1993, Kentucky Institute for Educational Research 1995, Koretz et
al. 1993, and Smith et al. 1994). For example, research indicates that teachers in
Vermont and Kentucky are asking their students to write more and to do more work
together in groups.
In a review of research, Newmann found that performance tasks generate more
engagement from students from all types of backgrounds and engender in students a
deeper understanding of the content being studied (Newmann, F. M., Secado, W. G., and
Wehlage, G. G., 1995). In studies conducted at the Mid-Continent Regional Educational
Laboratory, it was found that students’ abilities to do performance tasks can be
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
28
dramatically increased if teachers systematically use these tasks in the classroom
(Marzano, R. J. and Kendall, J. S., 1996).
In a three-year study conducted for the Office of Educational Research and
Improvement, researchers presented the following findings:
• Curriculum: When teachers use performance assessments, the content and
sequencing of the subject matter remain largely unchanged. This is because
existing state and district frameworks dominate the curriculum choices
teachers make.
• Instructional Strategies: In general, performance assessments have had a
significant impact on instruction. Teachers are using a wide array of
instructional strategies modeled on performance assessments that their states,
district, and schools have developed. The success of their efforts, however,
depends upon the opportunities they have to experiment with the assessments.
• Learning: The use of performance assessments in conjunction with more
challenging content have been found to make students more motivated to learn
and more engaged with project-based tasks than with other types of
assignments. Teachers who use writing portfolios and literacy-based
assessments say that students are developing better writing skills and habits.
• Barriers: The effects of performance assessments on teaching and learning
were not uniform in all schools studied. Factors which contributed to less
effectiveness or success included: lack of time for planning, developing tasks,
and scoring; poorly defined content and performance standards; inadequate
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
29
professional development for teachers; and assessments which were marginal,
rather than integral to the instructional program, i.e., one-shot accountability tests
vs. ongoing instructional practice (Khattri, N., A. L. Reeve, M. B. Kane, and R.
Adamson, 1996).
Areas of Conflict Between Advocates and Critics of Performance Assessment
Educational assessments are instruments which, among other things, provide
information on student achievement. Data derived from these tests may be used by
students, teachers, parents, schools, district, communities, and nations to guide many
kinds of decisions from individual to international; from placement to policy. In today’s
global marketplace, assessments of student achievement take on an even more important
role: Judging our ability as a country to compete successfully with other industrialized
nations. Results from international assessments are being used as a call to arms to
improve the American educational system.
The key characteristic of performance assessments as compared to standardized
testing is that they require students to construct their responses, as opposed to selecting
from a list of options. Constructed-response items, such as those tasks found in
performance assessments, require students to do much more than do selected-response
items. This trait has resulted in an enormous amount of support for the use of
performance assessments as supplements to standardized tests or as alternatives to
standardized tests (Marzano, R. J., 1996).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
30
The areas of conflict between advocates and critics of performance assessment fall
broadly into the following categories:
1. Content
Narrowing of Content
Critics of traditional multiple choice tests charge that conventional tests measure
only a small portion of what students know, i.e., bits and pieces of information. These
types of tests are severely limited in the types of knowledge they can assess.
Test content is.. .constrained by the emphasis on basic skills, limiting the ‘height’
as well as the depth and breadth of permissible content. Even advocates of high-
stakes testing acknowledge that the tests do not cover the full range of important
instructional objectives...The extent of this narrowing — that is, whether 10
percent or 50 percent of desirable content is sacrificed — will depend on the
particular test and the accountability of purpose or market that shaped it. For
example, minimum competency or promotional gates tests are even narrower than
standardized tests of basic skills” (Shepard, L. A., 1989).
Although tests have been developed that assess higher level thinking and
application of knowledge (Arter, J. A. and Salmon, J. R, 1987), generally off-the-shelf
tests assess only surface understanding and lower level skills (Frederiksen, J. R. and
Collins, A., 1989; Marzano, R. J., 1990; Marzano, R. J. and Costa, A. L., 1988; Mitchell,
R. and Neill, M., 1992; Resnick, L. B., 1987b; Shepard, L., A., 1989; Stiggins, R. J.,
1994; Wiggins, G. 1993; Williams, P., Phillips, G. W., and Yen, W. M., 1991). Lauren
Resnick and Daniel Resnick (1992) conclude that standardized tests are designed to
assess low-level algorithmic skills and isolated pieces of factual information.
Negotiating of Content
The contents of such tests must be negotiated. Most state-developed tests go
through a consensus-building process: teams of curriculum experts and teachers
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
31
agree to the content of the test. Publishers of standardized tests...do content
analyses to ensure that test objectives are well matched to widely used textbooks.
These procedures are sensible, but they have a homogenizing effect, limiting both
the breadth and depth of content coverage. The most imaginative and challenging
problem-solving tasks offered uniquely by a single district are likely to be
negotiated out of the test content domain. Ironically, the textbooks that
commercial test framers look to for verification of coverage are governed by the
same sore of consensus model, to ensure market appeal, and are in turn matched
to the outlines of standardized tests (Tyson-Bemstein, H., 1988).
Traditional tests encourage teachers to teach to the test, that is, teachers select
instructional activities which will prepare students to pass mandated tests. Certainly this
practice is understandable since teachers want their students to perform well. However,
preparing students for multiple choice tests may result in emphasizing acquisition of
discreet knowledge over higher order thinking skills. Some tests have been criticized for
including items that are biased against certain kinds of students, e.g., ethnic minorities,
limited English proficient, rural, inner-city. The basis for this criticism is that the items
on the test reflect the language, culture, and/or learning style of the middle class majority
(Neill, D. M. andN. J. Medina, 1989).
In place of, or in addition to traditional tests, reformers advocate the use of
performance assessments which require students to construct answers, to create products
such as portfolios or computer databases, or carry out activities relevant to the discipline,
such as interviewing an elected official, conducting an experiment, producing a
newspaper, writing a letter of complaint to a company, etc. (Center for Civic Education,
1995).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
32
2. Format
Perhaps the most obvious difference between standardized and performance based
assessments lies in their format. Standardized tests are almost exclusively
multiple choice, while performance assessments take many forms and tend to
reflect authentic, real world behaviors. The multiple choice, also called “forced
choice” format is largely a practical, cost-effective consideration, however, the
multiple choice format, together with other narrowing elements noted above,
further constrains content.
Although multiple choice questions can elicit important conceptual
distinctions from students, tests composed entirely of such items do not
measure a respondent’s ability to organize relevant information and
present a coherent argument. Time limits and the need to survey as much
content as possible also preclude problem sets aimed at the interrelation of
facets of the same topic (Shepard, L. A., 1989).
In addition, reading passages are often significantly shorter and less complex than
the texts students use for daily instruction. “Moreover, to avoid confusing student by
frequent changes in format, each test uses a small number of item types - despite evidence
that students may appear to know a concept or skill when it is measured in one format,
but not know it if measured another way (Shepard, L. A., 1988). If the same test or a
similar test is given each year, the effect of these format limitations for the content
validity of the test will increase over time.
3. Purpose/Use
A brief review of the history of standardized tests show that they were initially
designed for the purpose of identifying which students would most likely succeed
in school and which students would most likely fail. This purpose was the direct
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
33
consequence of the assumption that intelligence, or aptitude is a fixed,
unidimensional characteristic (Marzano, R. J. and Kendall, J. S., 1996).
Howard Gardner (1992) traces the widespread use of standardized tests in the
United States to the work of psychologist Alfred Binet in the early twentieth century.
[Biner] hit upon the inspired idea of administering a large set of items to young
school children and identifying which of the items proved more discriminating in
light of his particular goal. The work carried out by the Binet team ultimately led
to the first intelligence tests, and the construct of intelligence quotient or IQ.
So great was the appeal of the Binet method that it soon became a dominant
feature of the American educational and assessment landscape. ...In the United
States, especially with its focus on quantitative markers and its cult of educational
efficiency, there has been a virtual mania for producing tests for every possible
social purpose (pg. 79).
Proponents of performance assessments argue that assessment and instruction
must form a seamless web that promotes teacher/student collaboration, active learning,
critical thinking skills, and multidisciplinary understanding. Performance assessments,
they reason, have a positive influence in the classroom. Performance assessments
provide pedagogical templates that help teachers to develop effective instructional
techniques; and provide comprehensive information about student progress, including
students’ strengths and weaknesses.
4. Technical Considerations
Performance assessments have been shown to have at least one major flaw. They
are not as generalizable as originally thought. In a number of studies, Richard Shavelson
and his colleagues, have shown that a student’s performance on a single performance task
is not necessarily a very good indicator of the student’s knowledge and skill within the
subject area that the task is designed to represent. For example, when the same science
knowledge was tested in a hands-on format, a computer simulation, and a written
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
34
description derived from a hands-on experience, it was found that the students might
perform well on one format, but not perform well on the other two formats. These
findings have led Shavelson and his co-researchers to conclude that a single performance
assessment is not a good general indicator of how well students can perform within a
content area (Shavelson, R. J. and Baxter, G. P., 1992; Shavelson, R. J., Gao, X., and
Baxter, G. R.,1993; Shavelson, R. J. and Webb, N. M., 1991; Shavelson, R. J., Webb, N.
M., and Rowley, G., 1989).
In an extension of Shavelson’s research, several measurement experts believe that
between 10 and 36 performance tasks are necessary to assess accurately students’
competence within a single subject area (Lane, S., Liu, M., Ankenmann, R. D., and
Stone, C. A., 1996; Linn, R., 1994; Shavelson, R. J., Gao, X., and Baxter, G. R.,1993).
5. Cost
Simplistically, the cost of assessments is often thought of as the price tag,
however, to be truly accurate and thorough in the analysis of cost, it is critical that we
consider the economic costs of assessment, including time and opportunities foregone.
These costs are significant and may account for the majority of costs of a testing system
(Haney, W. H., Madaus, G. F., and Lyons, R., 1993).
The assessment of true economic costs includes estimates of the benefits foregone
from the next best alternative and compares the benefits received with the costs
incurred. However, many of the benefits of educational assessment are difficult to
identify, hard to measure, and not easily valued in monetary terms. In addition,
knowledge of costs is hampered by the multiplicity of possible benefits and the
numerous, and sometimes contradictory, goals of assessment systems (Picus, L.
O., 1997).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
35
It has been shown that a considerable amount of time is devoted to teaching to test
objectives, test-taking skills, and even specific test items (Haney, W. H., Madaus, G. F.,
and Lyons, R., 1993). It is even more important that tests are “worth taking” (Wiggins,
G., 1989).
Critics of performance assessment cite the cost of development and scoring as
prohibitive. In 1992, it was reported that assessment experts feared that performance
assessments would be two to three times more expensive that their machine-scorable,
multiple-choice-format counterpart (O’Neil, J., 1992). In a study conducted four years
later, it was found that performance assessments could cost as much as 20 times more
than the machine-scorable tests (RAND Corporation, 1996).
One example of performance tests given on a national scope may be found in the
Advanced Placement examinations, particularly writing samples, art portfolios, and
foreign language tests. In 1992, it was estimated that the Advanced Placement tests cost
$65 per subject test, compared to $2 to $5 per subject test for a commercial standardized
test (Koretz, D. M., Madaus, G. F., Haertel, E., and Beaton, A. E., 1992). Stecher (1995)
reported that the cost of the complete California Test of Basic Skills (CTBS) battery of
multiple choice tests ran $2.80 per student, with significant increases in cost with the
addition of open-ended written responses. The cost of the CTB writing test was $4.80 per
student for a single prompt. The General Accounting Office estimated that the total
operational cost of a national performance-based assessment system would total $33 per
student (U.S. General Accounting Office, 1993).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
36
In a study of the costs of alternative assessment programs in Kentucky and
Vermont, Picus (1997) distinguishes between the simple “price tag” of tests and the more
difficult and complex task of determining opportunity or economic costs associated with
the assessment programs. Picus (1994) suggests that simple analysis of state
expenditures for assessment program may mask the true cost of theses programs — both
the newer forms of assessment and the more traditional multiple-choice methods.
Using Monk’s (1995) definitions: costs as “measures of what must be foregone to
realize some consequence;” expenditures as “measures of resource flows regardless of
their consequence,” Picus (1997) provides a comprehensive list of the elements needed
to estimate the expenditures and a list of ingredients that need to be considered in the
identification of opportunity costs at the district and school level. Components included
in the first dimension of cost-determination of an assessment program include such things
as development, production, administration, and scoring of the test instruments. The
second dimension costs relate to the level at which the expenditures are incurred, i.e.,
state, district, school, classroom, or private test market. The third dimension addresses
the kinds of expenditures, such as personnel, test materials, computer resources, or travel
and food for training sessions” (Picus, L. O., 1997).
The results of the Picus (1997) study (Table 2.2) show the significance of the cost
factor of performance assessments.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 2.2
37
Total Estimated Costs of Kentucky’s Assessment System, 1995-96
Estimated Costs (in dollars)
Category Low High
Direct Expenditures
State expenditures 7,401,000 7,401,000
District expenditures 2,978,250 3,001,250
Sub-total direct expenditures 10,379,250 10,402,250
Opportunity Costs
School level
Accountability teachers 49,083,000 161,210,000
Non-accountability teachers 12,532,500 12,532,500
Principals 24,488,583 38,365,447
Assessment Coordinators 15,794,342 22,112,078
District level
Superintendents 1,710,628 1,710,628
Assessment coordinators 2,877,087 3,452,505
Other central office staff 2,397,583 3,226,931
Clerical staff 1,200,600 1,400,700
Sub-total opportunity costs 11,084,323 244,010,789
Total Costs
Kentucky Assessment Program 120,463,573 254,413.039
Total number of students 658,896 658,896
Total students tested 141,975 141,975
Total tests administered 851,850 851,850
Total cost per student enrolled 182.83 386.12
Total cost per student tested 848.59 1,791.96
Total cost of test administered 141.41 298.66
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
38
“In comparison with more traditional multiple-choice examinations, performance
assessment and portfolio requirements are time-intensive since fewer tasks can be
administered in a single time period and they take longer to score and evaluate” (Picus, L.
0., 1997). The result of the cost of time that it takes for performance assessments is that
fewer assessments can be administered, with the net effect of fewer samples upon which
to judge reliability and validity (Burger, S. E., 1994; Mehrens, W. A., 1992, Viadero, D.,
1994).
The power of performance-based assessments is their capability to focus on
higher order thinking skills and complex tasks, however, researchers caution that the cost,
practicality, comparability, generalizability, objectivity, and administrative convenience
should be considered (Koretz, D. M., Madaus, G. F., Haertel, E., and Beaton, A. E.,
1992). “What is surprising is, given the tremendous emphasis placed on assessment
systems to measure school accountability, the relatively minuscule portion of educational
expenditures devoted to this important, and highly visible, component of the educational
system” (Picus, L. O., 1997).
Relationship of Performance Assessment to Graduation Requirements.
Instructional Time. Policy Decisions, and Accountability
1. Graduation Requirements
“The New York Regents Examinations, a set of subject-specific exit examinations
administered to college-bound students, were begun in the 1860’s” (Resnick, 1982). One
hundred years later, beginning in the mid-1970’s, states significantly expanded their
testing programs to focus on minimum competency tests. The majority of those
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
39
implemented by the early 1980’s were used as exit exams to set minimum standards for
graduation from high school (Jaeger, R. M.,1982).
California legislators have approved the use of an exit examination for high
school students beginning with the graduating class of 2004. At this point in time, it is
not known what will be on the test, nor what format(s) will be used to assess graduating
students.
Other states may provide some indication of the effect of a high-stakes
performance-based graduation test. In the spring of 1996, Michigan began using the
Michigan High School Proficiency Test. This examination is a nine- to twelve-hour
assessment in social sciences, natural sciences, mathematics, and communications.
Students are asked to provide a series of written responses in all content areas tested. For
example, the language arts writing section requires students to produce two essays, one in
twenty minutes and one in forty minutes. Almost all agree that the test is a rigorous
instrument of evaluation.
The use of assessments as a gatekeeper for high school graduation is
administered, analyzed, researched, and discussed on a large scale, the catalyst being
accountability or some other policy issue. The danger in discussing large-scale
assessments in a detached, dispassionate manner, is that the human face and voice of
those impacted by these policy decisions is often unseen and unheard. It is interesting
and enlightening to see this type of test through the eyes of a teacher responsible for
preparing students to be successful on such a high-stakes test. Following is one teacher’s
report on his experience with the high school proficiency test in Michigan.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
40
The following anecdotal evidence provides a glimpse into some of the effects of
performance assessments on graduation requirements and one teacher’s attitude toward
this initiative.
It is amazing how completely this single test has gripped the state, transformed
the dialogue, and redefined the concept of success. .. .Needless to say, our
department has already rewritten our school calendar to allow for test preparation
and motivational sessions. Even more alarming is how the test has thwarted the
student-centered, process-oriented character of our classes. Where before one was
encouraged to engage students in a plethora of whole language experiences - ones
that appeal to the interests and motivations of the student - we now are designing
our assignments to parallel the narrow composition requirements on the test.
...With an entire community poised and ready to see if certain districts are going
to raise exam scores, the onus is clearly on teachers to equate success with this
new standardized monster of educational reform (Shafer, G., 1997).
2. Instructional Time
One case study from Vermont may be generalized to some degree in order to
illustrate the effect of performance assessment on instructional time. For the purpose of
the following analysis, instructional time will be considered as a commodity which can
be measured, and to which opportunity costs can be applied. If instructional time is being
devoted to performance assessments, then other choices, such as teaching other
curriculum objectives, are foregone.
In 1988, Vermont began to develop a voluntary portfolio assessment system and
was the first state to use portfolios as a major component of its statewide assessment
system (Koretz, D., Stecher, B., and Deibert, E., 1992). The purposes of the new
Vermont Portfolio Program (VPP) were to provide rich data on student performance, to
encourage better teaching, to adopt higher standards, to coexist with the state’s tradition
of local control, and to encourage greater equity of educational opportunity (Koretz, D.,
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
41
McCaffrey,D., Klein, S., Bell, R. and Stecher, B., 1993). The portfolio program was
intended to be a compromise among the many goals of the system (Koretz, D., Stecher,
B., and Deibert, E., 1992). In a cost analysis of the Vermont Portfolio Project (VPP)
during the 1995-96 school year, Picus (1997) found that “the estimated annual time the
teachers spent on various components of the VPP was 214.11 hours and the median,
154.0 hours. Assuming an average work year of 10 months, these estimates represent
between 15.4 and 21.4 hours per month, and are lower than the 30 hours a month
estimated by Koretz, Stecher, Klein, & McCaffrey (1994) for both math and writing
during the second and third year of the programs implementation. Of this total time,
teachers reported that they spent, on average, 84.0 hours of classroom instructional time a
year working with students and administering the VPP. This time may represent an
opportunity cost in that other curriculum may not be covered during the time the teachers
are working with students on their portfolios (Picus, L. O., 1977).
The 1991-92 survey conducted by Koretz, Stecher, and Diebert (1992) reported on
student time commitments. Students spent about three and one-half hours per week on
portfolios: approximately two hours doing portfolio tasks, one hour in revision, and one-
half hour organizing their portfolios.
The cost of instructional time devoted to performance assessment is relative to
benefits gained. Did the benefits of instructional time spent by teachers and students in
completing portfolios outweigh the opportunities foregone? In a 1991-92 survey, Koretz,
Stecher, Klein, and MccCaffrey (1994) found that teachers and principals found the VPP
as a worthwhile burden. The study went on to report that many educators found the
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
42
program to be a powerful and positive influence on instruction. Four years later, a survey
of superintendents and teachers found strong support for the VPP (ACSU, 1995). Eighty
two percent of superintendents supported the state’s development of a performance
assessment program. Seventy three percent felt that the VPP “provided teachers with
valuable information to help them improve their instructional practices” (p. 1). Sixty nine
percent of the teachers expressed support for the state assessment program and seventy
percent found the VPP helpful in improving instructional practices. Almost seventy
percent of the teachers found portfolios useful in communicating with students about
their performance.
Koretz, Stecher, Klein, McCaffrey, and Deibert (1993) found positive changes in
attitudes regarding mathematics and learning. Teachers reported that both they and their
students were more enthusiastic about mathematics as a result of the portfolios. When
principals were asked to list the benefits of the VPP, three-fourths stated that portfolios
had a tremendous impact on instruction, and were a positive teaching tool (Koretz, D.,
Stecher, B., Klein, S. & McCaffrey, D., and Diebert, E., 1993).
3. Policy Decisions
In a country founded on principles of states’ rights and local control with regard to
education, it took a rising voice of concern for the academic performance of students and
fears for the country’s economic competitiveness to make the idea of national education
standards feasible.
In 1989, at the Charlottesville Education Summit, President Bush and the nation’s
governors agreed upon six national education goals, known as Goals 2000. These goals
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
43
stress quality education from early childhood through lifelong learning. Subsequently,
Congress enacted the Educate America Act which was signed into law on March 31,
1994. This law codified the original six National Education Goals and added two goal
related to teacher training and increasing parental involvement. The law provided first-
year funds totaling $105 million for states to submit applications describing the process
by which their state would develop a school improvement plan aligned to Goals 2000.
The law also established oversight panels, including: The National Education Standards
and Improvement Council, the National Education Goals Panel, and the National Skill
Standards Board (Donovan, F. and Sneider, C., 1994). These events are important
because they changed and broadened the role of the federal government in education from
funding categorical programs for special needs to providing incentives in the area of
general education.
There quickly followed a flurry of activity by professional organizations to
produce content standards similar to the work of the highly successful National Council
of Teachers of Mathematics (NCTM). According to NCTM, by 1994, forty one states
had adopted the standards for mathematics education, a mere five years after their
publication (Harrington-Lueker, D., 1994). Many standards contained in these
documents, such as speaking and writing, require the use of some kind of performance
assessment to evaluate student knowledge and skills in a valid manner.
One example of the link between policy and testing was the Clinton
Administration’s proposal for voluntary national tests (VNT). The proposal triggered
three research studies undertaken by the National Research Council (NRC).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
44
The NRC group evaluating the design of the voluntary national tests concluded
that the proposed VNT assessment development process was technically sound
but needed better up-front decision making on issues likely to affect validity, such
as how to report the results. The second NRC study found that there was little
feasibility in linking existing state and/or commercial tests to either the National
Assessment of Educational Progress or the proposed VNT. Finally, the third
NRC study - on the appropriate use of testing - concluded that tests can help
promote student learning and accountability, but when used inappropriately, can
undermine those same goals. A voluntary national test, the study emphasized,
should not be used to make high-stakes decisions about individual students, such
as retention (Lewis, A., 1998).
Changes in the federal government’s approach to categorical funding provides
another example of the linkage between performance assessment and policy. Title I
requirements are pushing states to adopt assessments for the early grades. This effort has
resulted in a number of new performance-based assessments, such as the Literacy
Development Checklist for kindergarten teachers. A pilot study on the Checklist is being
conducted through a partnership among the University Elementary School at the
University of California at Los Angeles and two local districts in the Los Angeles area.
The research is already making obvious the “we are learning as much about the
professional development needs of teachers as we are about children” (Imbens-Bailey, A.,
1998).
The nexus of assessment, policy, and accountability can be seen clearly in the new
accountability system for California Schools. The guiding principles of this system are:
A state system built on local standards; assessing achievement to standards; a
comprehensive accountability system; rewards and recognition; support and sanctions.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
45
To effect such a system, California lawmakers enacted AB 265, Statues of 1995:
It is the intent of the Legislature.. .to provide a system of individual assessment of
pupils that has as its primary purpose, assisting pupils, their parents, and teachers
to identify individual academic strengths and weaknesses, in order to improve
teaching and learning. It is further the intent of the Legislature to determine the
effectiveness of school district and schools, as measured by the extent to which
pupils demonstrate knowledge of the fundamental academic skills, as well as the
ability to apply to skills” (California Department of Education, 1997).
AB 265 was replaced by SB 376. The purpose of this legislation was to establish
a new statewide testing program entitled Standardized Testing and Reporting (STAR).
Ultimately, the program will consist of a number of multiple measures including a norm-
referenced test, Stanford Achievement Test (SAT 9) in English and SABE/2 in Spanish; a
standards-based augmentation to the norm-referenced test; Physical Fitness Testing;
Golden State Examinations, exit tests for selected secondary courses which may result in
a Golden State Seal Merit Diploma; Assessments in Career Education; and Assessment of
Applied Academic Skills.
Several assessments within California’s STAR program require evidence of
student proficiency through performance assessments:
• Written essays and constructed response items on the Golden State
Examinations in English composition, History, and Mathematics
• Strength, endurance, and cardiovascular exercises on the Physical Fitness Test
• Written responses and problem solving on Assessments in Career Education
• Performance assessments on Assessment of Applied Academic Skills
• Writing samples on the standards-based test and High School Exit Exam
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
46
California’ STAR program will require multiple measures to evaluate student
achievement. This strategy of multiple assessments is in concert with the Guidance on
Standards, Assessments, and Accountability for Title I from the U. S. Department of
Education. Title I requires that assessments must:
• Measure the proficiency of students in academic subjects in which the State
has adopted challenging content and student performance standards
• Involve multiple approaches within an assessment system with up-to-date
measures of student performance, including measures that assess complex
thinking skills and understanding of challenging content
• Include multiple approaches to assess student achievement in at least
reading/language arts and mathematics. A measurement approach is a
procedure for measuring student achievement. Measurement approaches
encompass a wide variety of strategies, including traditional tests that use
items and tasks such as multiple-choice items, short-answer forms, and
extended-response tasks. Other approaches emphasize student performance
and include the use of portfolios and projects designed to be assessment tools,
observational checklists, interviews, running records, and many other
techniques (California Department of Education, 1997).
4. Accountability
“What you test is what you get.” The use of performance assessments in high-
stakes tests aimed at providing accountability has very powerful and observable effects.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
47
It increases the incentives to focus instruction on skills and processes such as thinking,
problem solving, communicating, and creating. It moves performance assessments into
daily classroom instruction.
Robert Linn (1999) suggests that history has shown that testing is a popular
instrument of accountability and reform for a number of key reasons:
• Tests are relatively inexpensive.
Compared to changes involving increased instructional time, reduced class
size, training and attracting new teachers, assessment is very low-cost.
• Testing changes can be implemented relatively quickly.
Other school reforms may take years to implement, and it may take even
longer to know if they have improved schooling.
• Test results are visible and draw media attention.
Poor results in the first year of a new testing program are usually followed
by increasing scores in subsequent years, giving the appearance that
schools are improving.
• Testing can create other changes that would be difficult to legislate.
Research has shown that state- or district-level testing and assessment
requirements motivate changes in curriculum and teaching at the school
and classroom levels. It is much more difficult to directly legislate
changes in the classroom.
A desire for accountability provided much of the impetus for the recent growth of
state testing programs and federal compliance reporting. A significant increase in testing
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
48
at the state level began in the early 1970’s. States wished to impose accountability
through mandating minimum competency testing. These tests threatened serious
consequences for failure; they were intended not only to measure performance, but also to
improve it. The reach of accountability stretched from individual students, to teachers,
schools, and districts (Jaeger, 1982).
The convergence of a number of forces, all decrying the deplorable state of
student achievement, resulted in an even greater reliance on assessment as an
accountability tool. The National Assessment of Educational Progress, which used a
combination of multiple choice and performance assessment types, revealed that many
students were failing to master even rudimentary skills. A Nation at Risk riveted the
attention of policy makers in the United States. President Bush and the nation’s
governors set forth Goals 2000 demanding high standards for all students, accountability,
and a world-class education. Goals 2000 provided financial incentives to states to get on
the bandwagon. Legislators passed the Improving America’s Schools Act with much
stronger focus on student achievement and the use of multiple measures, including
performance assessments, to evaluate that achievement.
In the late 1980’s, a major milestone in the use of performance assessments for
accountability made its debut on the national scene. In response to the need for data to
compare and judge the performance of states’ educational systems, the Alexander-James
report (Alexander, L. and James, H. T., 1987) called for the expansion of the National
Assessment for Educational Progress (NAEP).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
49
Most assessments used for accountability purposes are externally mandated and
rely on objective, paper-and-pencil tests, which are usually machine scorable.
“The primary application of performance measurement in externally mandated testing has
been for the assessment of writing skills. Students write brief essays on specified topics
(prompts), and these are then rated” (Alkin, M. C., 1992). However, of late, increasing
attention has been paid to the use of performance measures other than writing samples in
externally mandated assessments. For example, open-ended problems calling for
mathematical thinking were used in the California Learning Assessment System
mathematics tests; science tasks were used at the state level in Connecticut and New
York; portfolio assessment was pioneered in Vermont; and a variety of performance
assessments continue to be an integral part of the Maryland Assessment Program.
Lauren Resnick, Director of the New Standards Project, was asked if performance
assessments should be used for accountability purposes and teachers or schools judged by
how well students do on those assessments - her response: “Yes, for a simple reason.
You shouldn’t hold people accountable for something they can’t directly work on, as has
frequently been the case in our education system. The accountability system has to use
those assessments that you’re willing to have kids keep practicing (O’Neil, J., 1993).
Written Composition
Three decades of research on writing and writing instruction have witnessed “a
steady evolution in the beliefs of writing researchers and practitioners about what writing
is and how it can best be taught” (see Durst, 1990; Dyson & Freedman, 1991; Freedman,
Dyson, Flower, & Chafe, 1987; Hairston, 1982; Nystrand, Greene, & Wiemelt, 1993).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
50
These shifts in viewing writing: from text to cognitive process to sociocultural
construction, parallel other educational trends (Sperling, 1996).
Students’ ability to communicate effectively in school settings is often equated
with academic success. Research findings show that students who do not clearly
understand the contextual demands of a task are unable to produce adequate written
composition which meets the requirements of the assignment (Shaughnessy, M., 1977;
Bartholomae, 1980; Daiute, 1981; Hull, 1987). Students who do not have a “contextual
rubric” for academic discourse are at risk. The research is clear that all students, but
particularly non-White, non-middle-class students, must to be taught those styles and
conventions of written English valued by and reflected in academic settings, and to
accomplish that, schools must explicitly teach those characteristics (Bartholomae, 1985;
Bizzell, 1982; Delpit, 1986; Harris, 1989; Rose, 1989; Walters, 1984).
If students clearly understand the criteria for proficient written performance;
indeed, if the student has internalized the demands of a variety of writing genres through
the conscious and explicit instruction of those rubrics, student writing might improve.
The implications of this research bear directly on the use of rubrics to orient students to
the demands of various tasks and to arm students with a framework of standards upon
which to self-assess written work.
Assessment of Written Composition
Assessing literacy is no easy matter (Stedman & Kaestle, 1991). Unlike other
comparable nations, the United States has no national curriculum or assessment.
Evidence comparing student performance on a national level is, therefore, scant.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
51
In the absence of a true “national test,” The National Assessment of Educational
Progress (NAEP) represents one excellent source of information on writing achievement
at selected grade levels. NAEP has been assessing student writing since 1969. Despite
declines on certain tasks in the 1970’s, overall writing performance has remained roughly
stable at all three age levels through 1988 (Applebee, Langer, Mullis, & Jenkins, 1990;
National Assessment of Educational Progress, 1982; National Center for Education
Statistics, 1995).
A closer analysis of the NAEP data conducted by Applebee (1994) reveals that a
variety of writing tasks were given to students at grades 4, 8, and 11, including a mix of
grade-appropriate informative, persuasive, and narrative writing. At each grade
approximately 1,500 students responded to the prompts. “Results indicate that given time
and familiarity with the topic, the best students can write relatively effective informative
and narrative pieces (Applebee, 1994). However, “even the best students have difficulty
in writing tasks that require them to muster arguments and evidence in persuasive writing
(Applebee, 1994). In addition, “the performance of the best students remained far ahead
of the performance of most of their classmates. Whatever successes schools may claim in
writing instruction, many students at each grade level continue to have serious difficulty
in producing effective information, persuasive, or narrative writing” (Applebee, 1994).
The most recent administration of the NAEP writing assessment occurred in 1998.
Data from that year (Table 2.3) reveal that over half of students at grades 4, 8, and 12
performed within the Basic level of writing performance (National Center for Education
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
52
Statistics, 2000). These statistics represent a continuation of trends for the past twenty
five years with a majority of students continuing to perform at minimal levels.
Some correlation seems to exist between the emphasis given different writing
domains and student achievement on those domains. For example, teachers and students
report that persuasive writing, advancing evidence and arguments to influence readers to
change their thinking, received less emphasis in their classes than did informative or
narrative writing (Applebee, 1994). The Office of Educational Research and
Improvement (1993) came to similar conclusions in stating that “students do not write
much, and they do not have an accurate idea of what constituted ‘good’ writing. ..Schools
need to: provide students with frequent writing assignments; give students a wide variety
of assignments; provide students with examples of good writing and teach students how
‘good’ writers approach writing.” It follows that if classroom emphasis is devoted to
effective writing in a variety of genres, and if students understand the expectations, i.e.,
criteria for excellence — rubrics, demanded by those writing tasks, then writing
achievement may be influenced positively.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
53
Table 2.3 National Assessment of Educational Progress 1998 Writing Performance
Percentage of students within each achievement level range
Grade Below Basic Basic Proficient Advanced
4 16 61 22 1
8 16 58 25 1
12 22 57 21 1
NOTE: Percentages may not add to 100, or to the exact percentages at or above
achievement levels, due to rounding.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
54
Use of Rubrics as a Teaching Device
The literature on student use of rubrics to improve writing performance is scant in
general and almost non-existent at the high school level. In one informal study, White
(1985) used holistic scoring in a college classroom, asking that his classes develop the
rubric. “Classes do a good job in defining what is important in a paper.” He concluded
that holistic scoring had a “...positive value on student writing” particularly in the areas
of “...fluency, organization, development, and details.”
A more thorough study of training students to be self-assessors of writing is found
in work being conducted by the Northwest Regional Education Laboratory (NREL). The
major difference between NREL’s and White’s study is that NREL’s rubric was given to
students to apply, while White’s students actually created the rubric.
Since 1984, NREL has trained teachers to assess student writing using six-trait
analytic scoring guides focusing on: ideas, organization, voice, word choice, sentence
fluency, and conventions. After six years of training teachers, NREL realized that great
benefits were accruing to teachers who systematically analyzed and rated student work.
NREL also “...noticed that those doing the scoring [teachers] seem to learn a lot about
writing. In addition, to learning about how students write, they become better writers
themselves. They also report that they are better able to teach writing, because now they
understand what they need to teach” (Arter, J. A. et al, 1994).
Given this rationale, NREL began to train teachers to teach students to assess their
own and others’ writing. To determine if this approach improved student writing, a study
of 132 fifth grade students randomly divided into Integration and Control groups was
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
55
conducted. The conclusions drawn by NREL are similar to those made by Wiggins,
Resnick and Resnick, White, and the Office of Educational Research and Improvement:
“...student writing improves to the extent that we address instruction to the features of
writing we deem to be the most important, and to the extent that we directly teach
students what good and poor writing looks like in each of these dimensions.. .Therefore
we can impact student achievement by improving classroom assessment techniques,
teacher skills in using them, and student self-assessment (Arter, J. A., 1994).
Recent studies have demonstrated the relationship between rubrics and self-
assessment on learning and metacognition. In this study, forty 7th graders were assigned a
classification task. Half of the class was given a rubric and periodically asked to assess
their work. The other half of the class was asked to assess their work without being given
the rubric. On a traditional quiz for basic content knowledge, students who had used the
rubric learned more (Goodrich, H., 1996).
A second study showed the effects of rubrics on 8th graders’ writing skills with
one group receiving the rubric before they began writing, and the other group not
receiving the rubric. “The first group tended to receive better scores on two of the three
essays; for one essay, the differences were statistically significant. Simply handing our
and explaining a rubric seemed to help students write better...It appeared that more
intensive work with the rubric might be helpful (Andrade, H., 1999)..
Conclusion
The extant literature in the areas of written composition and performance
assessment provides a solid foundation for this study. The literature on written
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
56
composition is clear: direct, explicit, contextualized classroom instruction on the
elements of quality writing impact positively on student writing. Research in the area of
performance assessment demonstrates the need to devise assessments worth assessing - to
teach what is valued. Writing samples based on relevant, authentic topics provide such
assessment tools. Lastly, the use of rubrics as an effective instructional device is
sustained by the literature, but further research in this area needs to be conducted,
particularly at the high school level.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
57
Chapter 3
METHODS AND PROCEDURES
Introduction
Student achievement in writing is an important indicator of whether young adults
in the twenty first century will have the writing and thinking skills necessary to express
themselves clearly as individuals, workers, and citizens. This chapter describes the
procedures followed in identifying factors which influence writing, developing
instruments, gathering information, and analyzing data relevant to the factors and the use
of student-developed rubrics on writing performance and instructional practice.
Instrumentation
Development of the Student Writing Survey
The Student Writing Survey (Appendix A) was based on questions used by the
National Assessment of Educational Progress from 1984 to the present. Factors
considered relevant to this study include:
• Value placed on writing: Questions 1-6
• Attitudes toward writing: Questions 7-11
• Use of planning strategies: Questions 12-16
• Use of revising and editing strategies: Questions 17-24
• Teachers’ comments on completed papers: Questions 25-31
• Teacher feedback on writing: Questions 32-36
• Use of a rubric in writing: Questions 37-40
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
58
Administration of the Student Writing Survey
The Student Writing Survey was a self-report questionnaire. The directions
printed on the first page of the questionnaire emphasized that student responses were to
be honest and thoughtful; responses were to be used for research purposes; and responses
were to be kept confidential. The questionnaire was distributed by the students’ English
teacher during the course of a regular class period. The survey took between 15-20
minutes to complete. Scantron forms were used to capture student responses, thus
facilitating item analysis.
Scoring of the Student Writing Survey
The scoring procedure for the Student Writing Survey was based on a Lickert
value (A-E) for each item response. Each value was assigned a descriptor: A = Always,
B = Mostly, C = Sometimes, D = Rarely, E = Never. Items were analyzed in two ways:
each item was reported as a single variable; questions were grouped and reported by
factor, e.g., value placed on writing, attitudes toward writing, etc.
Classroom Observation
Each class involved in the study was observed by this researcher during the
intervention period. The focus of the observations was the instructional strategies
employed by teachers to teach writing skills and evaluation of writing, including the use
of student-developed rubrics. A clinical supervision model was used by the researcher:
(1) Pre-observation conference with the teacher to determine the objective of the lesson;
lesson design; instructional sequence, i.e., what came before this lesson and what will
come after; strategies to assess student learning, student work products; (2) Observation
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
59
of the lesson using various data capture techniques, including scripting, student-teacher
and student-student interaction charts; movement diagrams, and time on task record; (3)
Post-observation conference with the teacher to review the lesson and achievement of
expected outcomes; review student work created during the lesson; analyze data;
determine next steps based on analysis of student work and data.
Student-developed rubrics and samples of writing were collected during the
course of this study. Table 3.1 illustrates one example of a rubric created by a team of
students at School 2. Examples of work in each score level based on the rubric presented
in Table 3.1 are included in Appendix C.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced w ith permission o f th e copyright owner. Further reproduction prohibited without permission.
60
Table 3.1 Sample Student-Developed Rubric
Domain High Miedium Low
Content Exceptional insight Attempted insight Missing
Organization Skillful
Consistent
Organized Inconsistent
Support Vivid details
Concrete details
Details Few details
Style Strong voice Whisper
Minimal
Mute
Silent
Conventions Minimal errors
Unnoticeable
Flows easily
Some/Few errors
Occasional
Careless
OOPS!
Detracts from content
61
Interviews
An initial interview with both teachers involved in the study was conducted by
this researcher in order to identify typical classroom instructional practices used to
prepare students for classroom writing assignments and for the district writing
assessments. Interviews of former English teachers of students involved in this study
were also conducted. Former teachers were asked to describe instructional practices they
used to prepare students for classroom writing assignments and for the district writing
assessments. They were specifically asked if they had ever used student-developed
rubrics in teaching writing or in evaluating student written work.
At the conclusion of the intervention, small groups of students involved in this
study were asked to participate in a follow-up interview. Student interviews focused on
providing a context for responses given on the Student Writing Survey, and on
determining if and how the use of a student-developed rubric changed their techniques for
evaluating written work.
A follow-up interview with the teachers also was conducted. The focus of this
interview was to identify if and how their instructional strategies had changed to
accommodate the use of student-developed rubrics. The teachers also were asked if
participation in this study had increased their interest in pursuing additional professional
development in this area.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
62
Data Collection
Description of the Sample
Selected high school English classes, 120 students of mixed ability levels,
participated in this study. Students attended classes in the same K-12 unified school
district, but at two different high school campuses on the central coast of California. The
two schools serve grades 9 through 12 and had a student enrollment of 960 and 1,560.
The socioeconomic status of the communities, and thus of the high school population, is
above the state’s average. Indicators of student achievement, such as the Scholastic
Assessment Tests, Advanced Placement Examinations, and norm-referenced Stanford
Achievement Test 9 and Iowa Test of Basic Skills, all indicate average and above average
student performance.
School 1 served a predominantly small town/rural population from two
communities. Approximately 75% of the students commuted 10 to 20 minutes to school.
The school was located on a 43-acre campus on the California coast with unrestricted
beach access.
School 2 was located in a medium-sized city which served as the county seat and
home to a state university. The school served three communities as well as outlying
farming and residential areas.
The ethnic distribution of student reflected the make-up of the communities in
which the high schools reside. With minor variations, the ethnic composition of the
student body was similar at both schools. The ethnicity of students at School 1 had been
relatively constant over the past five years with approximately 83% Euro-American, 10%
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
63
Hispanic, 3% Filipino, 1.5% African American, and the remaining 2.5% split among
Asian, Pacific Islander, and American Indian. Like School 1, the ethnic distribution of
students at School 2 remained stable over time with approximately 83% Euro-American,
10% Hispanic, 5% Asian, 2% African American, and .05% American Indian.
Primary language constituted the second demographic factor addressed by this
study. The number of students reporting a primary language other than English
increased slightly at School 1 and declined by half at School 2 over four years. The
predominant primary language identified was Spanish. School 1 reported approximately
5% of students with primary languages other than English. School 2 reported
approximately 3% of students with primary languages other than English. Many of the
students reporting a primary language other than English spoken in the home were
themselves Fluent English Proficient. Therefore, English Language Learners constituted
only 1.3% of the total student population at School 1, and 1.6% of the total student
population at School 2.
Two teachers participated in the study. Each teacher has been trained in holistic
and analytic scoring protocols. Teachers participated in district-provided professional
development inservices on effective instructional strategies and the writing process.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
64
Method
This research is designed as a qualitative, descriptive study. No control group
was designated. An Inquiry Matrix (Table 1.1) was developed to serve as a guide for
determining critical questions, data needs, and data sources.
As part of the implementation of the district writing assessment, teachers received
an extensive set of materials, including descriptions of grade-appropriate writing types, a
rubric for each writing type, and resource books containing effective instructional
strategies for teaching writing. These materials, as well as teacher-developed materials,
were used with classes involved in the study.
Each participating teacher was observed three times during March and April 2000.
The purposes of these observations were to analyze lessons and to interview teachers and
students. A survey of participating students was completed during March 2000.
Teachers provided instruction to students on using rubrics to evaluate writing.
This instruction included the following activities:
1. Sharing and evaluating (as a group) pieces of literature to illustrate strong and
weak examples of writing in specific writing types.
2. Evaluating the work of anonymous students and discussing how to improve
that work.
3. Instructing lessons in which teachers and students analyze rubric-related skills.
4. Developing student rubrics to use in revising and evaluating writing.
5. Applying student-developed rubrics to students’ own and others’ writing
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
65
6. Teacher scoring, peer scoring, and student self scoring of writing samples
based on rubric criteria
Baseline information on student writing achievement was gathered from various
sources, including in-class writing assignments and scores from students’ middle school
and high school on district writing assessments. District writing assessments are
administered once a school year. The assessment is to be completed in one sitting and is
scored by district teachers and trained substitute teachers who are calibrated for interrater-
reliability. Student papers are scored using a blind, double-read protocol.
Demographic factors, i.e., ethnicity and primary language, were used to better
understand the profile of learners involved in this study. These data were obtained by
querying the district student administration system for the fields considered relevant to
this study. As explained in Chapter 4, due to the small sample size contained in this
study relevant to selected demographic factors, a description of the relationship among
ethnicity, language proficiency, and writing performance was not possible.
Limitations and Delimitations
Limitations
1. Students involved in this study were enrolled in the target teachers’ English
classes. A true random sample was not ensured.
2. This study was conducted in the researcher’s own district. There was a
possibility that personal biases and perceptions may have influenced the
selection of teacher participants, data analysis, and classroom observations.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
66
3. Findings of the study may not be generalizable to other high school students
in other demographic settings and for students significantly below average on
norm-referenced measures, such as the Stanford Achievement Test.
Delimitations
1. This study was delimited to the high school in a unified school district on the
central coast of California. Teachers instructed students on development and
application of rubrics to written work.
Classroom assignments and district assessments were used to determine
writing improvement. Teachers involved in the study graded classroom
writing assignments. District scorers evaluated the district assessment using a
double blind, holistic scoring technique.
2. The study was further delimited to two classrooms at the ninth grade taught by
teachers at two different high schools.
Data Analysis
Information gained from initial teacher interviews and review of previous lesson
plans was categorized to determine the instructional strategies used to prepare students
for classroom writing assignments and for the district writing assessments. Interviews
with students’ former English teachers were used to provide a more complete longitudinal
picture of the range of instructional strategies used to teach writing in the district.
Students involved in the study were identified and demographic data were used to
identify factors which may affect writing performance, i.e., ethnicity and primary
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
67
language. Writing performances on previous and current assessments were triangulated
with the demographic data to reveal patterns relevant to this study.
Samples of student writing prior to, during, and after the intervention were
analyzed to determine changes in techniques students used for evaluating written work.
Student scores on previous district writing assessments were compared to scores on
current district writing assessments and in-class writing assignments to determine
changes in quality of written work.
Stanford Achievement Test (SAT 9) results, specifically the subtests of reading,
language, and spelling were reviewed to determine relationships between student
performance on district writing assessments and performance on a norm-referenced test.
Student writing performance on district writing assessments were correlated to
their responses on the Student Writing Survey. Survey questions were grouped by the
following scales to establish baseline information on student self-reported perceptions of
writing:
• Value placed on writing
• Attitudes toward writing
• Use of planning strategies
• Use of revising and editing strategies
• Teachers’ comments on completed papers
• Teacher feedback on writing
• Use of a rubric in writing
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
68
Clinical Use of the Results for Improvement of Writing Instruction
The clinical use of results for improving writing instruction focused on a
combination of factors obtained from questionnaires, interviews, classroom observations,
student writing samples, demographic data, and assessments. Using profiles of data, the
district, site, and individual teachers will be able to begin a discussion of how the use of
student-developed rubrics might affect instructional strategies and student techniques for
evaluating written work.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
69
C hapter 4
FINDINGS AND DISCUSSION
Introduction
This chapter describes the findings to the following research questions:
1. What factors influence student writing, i.e., value placed on writing, attitudes
toward writing, writing process strategies, teacher comments and feedback, use of
a rubric?
2. In what ways do involvement of students in developing rubrics of accepted
written work affect their writing performance?
• In what ways did students change techniques for evaluating their written
work?
3. What pedagogical effects do rubric-based writing instruction have on students and
teachers?
• What strategies are teachers using to focus students on critiquing their own
written work?
• What have teachers learned as a result of using student-developed rubrics as an
instructional strategy?
The Student Writing Survey
As described in Chapter 3, a forty item survey (Appendix A) based upon factors
related to writing behaviors and instruction was administered to 141 ninth grade students
in two high schools in the same unified school district on the Central Coast of California.
Students participating in this study were asked to complete a series of questions relevant
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
70
to the value placed on writing, attitudes toward writing, use of the writing process, and
instructional experiences.
Data sets from the forty item survey were categorized into seven empirically
derived scales. Each scale was then aligned to the Research Question(s) which it
addressed. Table 4.1 shows the empirically derived scales.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 4.1 Empirically Derived Scales
71
Scale Scale Title Research Question
1 Value placed on writing 1
2 Attitudes toward writing 1
3 Use of planning strategies 1 and 2
4 Use of revising and editing strategies 1 and 2
5 Teachers’ comments on completed
papers
1 and 3
6 Teacher feedback on writing 1 and 3
7 Use of a rubric in writing 1 ,2, and 3
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
72
Description of the Scales
Scale 1: Value Placed on Writing
This set of questions asked students to select the extent to which they valued
writing (Table 4.2). High scores on this scale, i.e., selecting “Always” and “Mostly” on
the majority of questions, indicate a perception that writing is important and serves as a
mechanism to facilitate studying, thinking, and sharing ideas. A high scoring student also
perceives that writing is a valued skill in getting a good job.
Scale 2: Attitudes Toward Writing
Another series of items asked students to respond to questions about their
attitudes toward writing (Table 4.3). A student scoring high, i.e., a majority of responses
in the “Always” and “Mostly” ranges, on this scale holds a positive attitude toward
writing. A high score on Items 7, 8, and 9 reflects a student who likes to write and has a
good self image as a writer. A high score here also suggests that the student’s writing is
appreciated by others.
A student with a high score on Item 10 does not like to have his writing graded,
which may suggest that the student does not receive good grades on his written work, or
simply that the student enjoys writing without an evaluation being imposed upon it. A
student with a high score on Item 11 suggests the feeling that writing is a required
activity to be done in the school setting.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 4.2 Scale 1: Value Placed on Writing
Item# How often is each of the following sentences true for you?
1 Writing is important.
2 Writing helps me learn about myself.
3 Writing helps me study.
4 Writing helps me think more clearly
5 Writing can help me get a good job.
6 Writing helps me share my ideas.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
74
Table 4.3 Scale 2: Attitudes Toward Writing
Item # How often is each of the following sentences true for you?
7 I like to write.
8 I am a good writer.
9 People like what I write.
10 I don’t like to write things that will be graded
11 If I didn’t have to write for school, I wouldn’t write anything.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
75
Scale 3: Use of Planning Strategies
This scale includes five items that center on the use of writing process strategies,
specifically, planning/prewriting (Table 4.4). High scores on this scale reflect application
of skills such as jotting down ideas, outlining major points, and considering audience and
purpose. A student who scores high suggests a writer who understands that prewriting is
an important step in the writing process which can help to yield more effective
organization, coherence of ideas, and proper tone and style based on purpose and
audience.
Scale 4: Use of Revising and Editing Strategies
Scale 4, like Scale 3, is composed of questions relevant to the use of writing
process strategies. The focus of Scale 4 items is on editing, and revising skills (Table
4.5). A student scoring high on this scale employs various techniques to improve the
rhetorical effectiveness and conventions of written work. Revising and editing methods
include moving sentences or paragraphs to different parts of the paper, correcting
mistakes in spelling, grammar, and punctuation, and using a thesaurus.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 4.4 Scale 3: Use of Planning Strategies
Item # When you write essays in class, how often do you:
12 Think about what you want to say before you start writing?
13 Jot down ideas for writing?
14 Outline major points before writing?
15 Write in different ways for different audiences?
16 Write in different ways for different purposes?
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
77
Table 4.5 Scale 4: Use of Revising and Editing Strategies
Item # How often do you use each of the following to make your papers better?
17 Move some sentences or paragraphs to different parts of the paper
18 Add new ideas or information
19 Take out parts of the paper that you don’t like
20 Change some words for other words
21 Correct mistakes in spelling, grammar, and punctuation
22 Make changes as you write
23 Read what you have written aloud to yourself
24 Use a thesaurus
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
78
Scale 5: Teachers’ Comments on Completed Papers
Scale 5 items inquired as to the frequency and kinds of comments students receive
from their teachers (Table 4.6). A student with a high score on Scale 5 indicates that
teachers often provide oral and written comments on written work. The focus of these
comments is on such areas as following directions, the quality of the ideas contained in
the paper, the organization of the paper, word choice, and language conventions.
Scale 6: Teacher Feedback on Writing
Scales 5 and 6 may be considered to be companion scales in that they both focus
on teacher behavior in evaluating written work. Scale 5 reflects the kind of things upon
which teachers choose to comment, while Scale 6 identifies various methods by which
teachers provide feedback on student writing.
High scores on Scale 6 suggest that teachers use a variety of methods of providing
information to students (Table 4.7). A student with a high score in this area has had
teachers who mark mistakes, write notes on the paper, point out what was well done and
what was poorly done, and make suggestions for improvement the next time.
Scale 7: Use of a Rubric
Scale 7 consists of items relevant to students’ experience in using rubrics or
scoring guides (Table 4.8). High scores on this scale indicate extensive experience using
rubrics. A student with a high score uses rubrics as part of the revision process and as
criteria to evaluate his own and others’ writing. In addition, a student with a high score
on this scale has or has had teachers who use/used rubrics to grade his own or others’
writing.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 4.6 Scale 5: Teachers’ Comments on Completed Papers
79
Item # After you have written papers, how often does the teacher talk or write to
you about each of the following things?
25 How you followed directions
26 Whether you wrote enough in the paper
27 The ideas in your paper
28 The way you explained your ideas
29 The way you organized your paper
30 The words that you used
31 Your spelling, punctuation, and grammar
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
80
Table 4.7 Scale 6: Teacher Feedback on Writing
Item # How often does the teacher do each of the following things with ]
writing?
32 Mark mistakes in your paper
33 Write notes on your paper
34 Point out what you did well
35 Point out what you did not do well
36 Make suggestions for the next time you write
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
81
Table 4.8 Scale 7: Use of a Rubric in Writing
Item #
37 How often has a teacher used a rubric or scoring guide to grade your
writing?
38 How often have you used a rubric or scoring guide to revise your writing?
39 How often have you used a rubric or scoring guide to evaluate your own
writing?
40 How often have you used a rubric or scoring guide to evaluate the writing
of others?
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
82
Findings
Each scale represented on the Student Writing Survey was analyzed by school
involved in the study (Tables 4.9, 4.10, 4.12, 4.13, 4.16, 4.17, 4.20). Additionally,
analyses were conducted to describe relationships between or among a scales or scales
and writing performances (Tables 4.11, 4.14, 4.15, 4.18, 4.19). Indicators of writing
performance included scores on district assessments and subtests of the Stanford
Achievement Test form T (SAT 9). Other data sources, such as interviews, classroom
observations, and student writing samples, served to confirm information gleaned from
the Survey, as well as to provide context and clarification for Survey responses.
Findings— Scale 1: Value Placed on Writing
The Value Placed on Writing (Scale 1) was one of the factors considered in
answering the research question, “What factors influence student writing?” On Scale 1,
students at both schools placed a high value on writing as evidenced by responses in the
“Always” and “Mostly” categories (Table 4.9). Students consider writing important and
perceive that writing skills assist in studying, thinking more clearly, and sharing ideas.
The items valued most highly by two thirds of the students were that writing helped in
getting a good job and that writing was important. At both schools, 68.3% of the students
perceived that writing is a skill to getting a good job always or most of the time.; 65.0%
of students felt that writing was important always or most of the time.
Interviews with students and teachers confirmed the high value placed on writing
at both schools. This value is visible throughout the culture of the district in Board-
approved Student Outcomes and each high school’s accreditation report which targets the
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
83
“student as an effective communicator” as a major goal which is the responsibility of all
teachers in the instructional program.
Findings— Scale 2: Attitudes Toward Writing
The Attitudes Toward Writing (Scale 2) was a second factor examined in
answering the research question, “What factors influence student writing?” Responses
show that approximately one third of students at both schools have positive attitudes
toward writing as demonstrated by responses in the “Always” and “Mostly” categories
(Table 4.10). Students most strongly agreed with the statement, “I am a good writer.”
Even though students perceive themselves to be good writers, they do not write outside of
school. Of the 123 respondents to the Survey, 44.7% reported that if they didn’t have to
write for school, they wouldn’t write anything. Interestingly, in follow-up interviews
students spoke of differences they perceived in writing within the school setting and the
use of email for personal communication. In fact, most students did not consider e-mail
as a form of writing, and therefore did not factor that type of communication into their
survey response. This is an area requiring further study.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced w ith permission o f th e copyright owner. Further reproduction prohibited without permission.
84
Table 4.9 Findings - Scale 1: Value Placed on Writing
Percentage of Students Responding in Each Level.
School 1: N=49 School 2: N=74
Statement Always Mostly Sometimes Rarely Never
School 1 School 2 School 1 School 2 School 1 School 2 School 1 School 2 School 1 School 2
Writing is important. 24.5% 24.3% 28.6% 48.6% 22.4% 25.7% 14.3% 1.4% 0.0% 0.0%
Writing helps me
learn about myself.
4.1 13.5 14.3 25.7 44.9 41.9 28.6 14.9 8.2 4.1
Writing helps me
study.
12.2 12.2 16.3 37.8 30.6 33.8 30.6 10.8 10.2 4.1
Writing helps me
think more clearly.
8.2 20.3 22.4 32.4 32.7 25.7 26.5 17.6 10.2 2.7
Writing can help me
get a good job.
22.4 33.8 40.8 37.8 20.3 20.3 2.0 2.7 4.1 2.7
Writing helps me
share my ideas. 20.4 23.0 22.4 40.5 23.0 27.0 8.2 6.8 2.7
Percents may not total 100 percents due to rounding.
Reproduced w ith permission o f th e copyright owner. Further reproduction prohibited without permission.
85
Table 4.10 Findings - Scale 2: Attitudes Toward Writing
Percentage of Students Responding in Each Level.
School 1: N= 49 School 2: N=74
Statement Always Mostly Sometimes Rarely Never
School 1 School 2 School 1 School 2 School 1 School 2 School 1 School 2 School 1 School 2
I like to write. 8.2% 18.9% 22.4% 29.7% 40.8% 33.8% 16.3% 10.8% 14.3% 6.8%
I am a good writer. 14.3 14.9 26.5 48.6 24.5 29.7 30.6 4.1 10.2 2.7
People like what I
write.
10.2 9.5 18.4 39.2 42.9 41.0 8.2 6.8 18.4 2.7
I don’t like to write
things that will be
graded.
24.5 12.2 24.5 20.3 20.4 45.0 10.2 19.0 20.4 4.1
If I don’t have to
write for school, I
wouldn’t write
anything.
18.4 12.2 24.5 14.9 20.4 25.7 24.5 20.3 16.3 27.0
Percents may not total 100 percents due to rounding.
86
To explore the relationship between the value students place on writing and their
attitudes toward writing to their writing proficiency, a comparison was made of students
reporting a majority of responses as low value placed on writing and attitudes toward
writing (“Rarely” and “Never”), medium value placed on writing and attitudes toward
writing (“Sometimes”), and high value placed on writing and attitudes toward writing
(“Mostly” and “Always”) with rhetorical effectiveness scores on district writing
assessments from 1997 to 1999.
The district writing assessment is an on-demand test which asks students to
respond to a common grade-level prompt. The test is administered annually to first
through twelfth grade students in the district. Essays are scored on two components: One
score is given for rhetorical effectiveness, i.e., organization, word choice, sentence
structure, purpose, audience, and style. A separate score is given for conventions, i.e.,
spelling, grammar, capitalization, and punctuation. District writing assessment scores are
reported based on a 6-point rubric (Appendix B). Student performances were categorized
as weak (1 and 2 on the 6-point rubric), medium (3 and 4 on the 6-point rubric), and
strong (5 and 6 on the 6-point rubric).
As shown in Table 4.4, there does not appear to be a correlation between the value
students place on writing and their attitudes toward writing to their writing proficiency.
The discrepancy is particularly apparent at School 2 where a majority of students, 51.8%,
responded in the high range on value and attitudes scales, but only 23.2% demonstrated
strong performance on two or more district writing assessments. Less than one third of
the students placed a medium value on writing and attitudes toward writing, but 70%
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
87
demonstrated a medium performance on two or more district writing assessments. Less
than a quarter of the students, 22.7%, placed a low value on writing and attitudes toward
writing, while their writing performance showed only 11.3% with weak performance on
two or more district writing assessments.
Scales 1 and 2 might be considered affective domains. Values and attitudes are
often part of the “hidden curriculum,” as opposed to specifically selected cognitive
objectives upon which lessons are developed and teacher feedback is given. As shown in
Table 4.4, data suggest that student perceptions with regard to the value they place on
writing and the positive attitudes they hold about writing do not, in fact, strongly
correlate with their writing performances.
In interviews with teachers, these data were shared. Noting that affective
measures are considered important, teachers postulated that students’ perceptions about
the value and attitudes they hold about writing can and do influence their motivation to
improve their writing, and to do their best on writing assignments. As a result of seeing
the data, teachers began to overtly emphasize the value of writing and positive attitudes
toward writing in lesson plans and in the feedback given to students on written
assignments.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced w ith permission o f th e copyright owner. Further reproduction prohibited without permission.
88
Table 4.11 Relationship of Value Placed on Writing and Attitudes Toward Writing to Writing Performance
Percentage of Students in Each Level.
School 1: N= 49 School 2: N=74
Value Placed on Writing and Attitudes Toward Writing Writing Performance
High Medium Low Strong Medium Weak
School
1
School
2
School
1
School
2
School
1
School
2
School
1
School
2
School
1
School
2
School
1
School
2
39.0% 51.8% 29.7% 31.7% 29.5% 15.8% 13.8% 23.2% 72.4% 68.1% 13.8% 8.7%
Percents may not total 100 percents due to rounding.
89
Findings— Scale 3: Use of Planning Strategies
The Use of Planning Strategies (Scale 3) was the third factor considered in
answering the research question, “What factors influence student writing?” In addition,
the Use of Planning Strategies was found to be one way in which students changed
techniques for evaluating their written work (Research Question 2).
Students at both schools report high use of planning/prewriting strategies (Table
4.12). They think about what they want to say before starting to write, they jot down
ideas, and outline major points. They consider the audience and purpose for their writing.
The strategy used by 71.5% of students was thinking before writing. More than half of
the students, 56.1%, reported writing differently based on purpose. Almost half of the
students surveyed, 49.6%, reported that the strategy used least was outlining major
points before writing.
In follow-up interviews, teachers report that the strategies they use to teach
prewriting focus on the use of devices as clustering, webbing, quick writes, Venn
diagrams, and other graphical methods for capturing ideas and relationships of ideas.
Outlining is not used for on-demand assessments or classroom writing assignments.
However, outlining is a requirement for longer written work, such as research papers. As
a side note, some software programs, e.g. Inspiration, use clustering techniques to unlock
ideas and then transform those clusters into an outline form. This type of technology
application may have an effect on students’ ability to plan writing and their organization
of ideas into final written form. This is an area for future study.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
90
Findings— Scale 4: Use of Revising and Editing Strategies
The Use of Revising and Editing Strategies (Scale 4) constitutes the fourth factor
examined for its influence on student writing. This Scale responds to the question, “In
what ways did students change techniques for evaluating their written work?”
Scale 4 items inquired into student use of revising and editing strategies (Table
4.13). A large number of students, 86.2%, reported that they always or mostly corrected
mistakes in spelling, grammar, and punctuation to make their papers better. Other
strategies often used by students included making changes as they wrote (76.4%),
changing some words for other words (68.9%), adding new ideas or information (64.2%),
and taking out parts of the paper (63.4%). The revising strategy used least by students,
57.7%, was using a thesaurus.
Observations indicated that reference materials were not readily accessible to
students in classrooms. In some classrooms, thesauri were not available at all. In other
classrooms, limited numbers of dictionaries and thesauri were kept on bookshelves along
the walls of the room. Students engaged in writing were required to get up from their
seats and walk to the back or side of the classroom to get the materials needed. In follow-
up interviews, students reported that it often “wasn’t worth it” due to the attention drawn
by getting up, disturbance caused to other students, and the time it took to use the
reference materials. Interviews with teachers confirmed that the use of thesauri is not
specifically encouraged, although it was the teachers’ perception that students knew
where the reference materials were stored and students would use them as needed.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced w ith permission o f th e copyright owner. Further reproduction prohibited without permission.
91
Table 4.12 Findings - Scale 3: Use of Planning Strategies
Percentage of Students Responding in Each Level.
School 1: N= 49 School 2: N=74
Statement
When you write
essays in class, how
often do you:
Always Mostly Sometimes Rarely Never
School 1 School 2 School 1 School 2 School 1 School 2 School 1 School 2 School 1 School 2
Think about what you
want to say before
you start writing?
42.9% 443.2% 30.6% 27.0% 10.2% 16.2% 8.2% 9.5% 8.2% 2.7%
Jot down ideas for
writing?
10.2 10.8 28.6 17.6 28.6 28.4 22.4 31.1 10.2 12.2
Outline major points
before writing? 0.0 5.4 20.4 17.6 26.5 29.7 36.7 27.0 16.3 20.3
Write different ways
for different
audiences?
6.1 9.5 12.2 31.1 46.9 36.5 20.4 16.2 14.3 8.1
Write different ways
for different
purposes?
20.4 20.3 24.5 43.2 40.8 24.3 10.2 8.1 6.1 8.1
Percents may not total 00 percents due to rouncling.
Reproduced w ith permission o f th e copyright owner. Further reproduction prohibited without permission.
92
Table 4.13 Findings - Scale 4: Use of Revising and Editing Strategies
Percentage of Students Responding in Each Level.
School 1: N= 49 School 2: N=74 Percents may not total 100 percents due to rounding.
Statement
How often do you use
each of the following
to make your papers
better?
Always Mostly Sometimes Rarely Never
School 1 School 2 School 1 School 2 School 1 School 2 School 1 School 2 School 1 School 2
Move some sentences
or paragraphs to
different parts of the
paper
6.1% 8.1% 26.5% 39.2% 38.8% 35.1% 16.3% 12.2% 14.3% 5.4%
Add new ideas or
information 16.3 29.7 28.6 47.3 51.0 18.9 6.1 2.7 0.0 1.4
Take out parts of the
paper that you don’t
like
24.5 32.4 32.7 35.1 28.6 23.0 8.2 5.4 6.1 4.1
Change some words
for other words 22.4 33.8 40.8 37.8 22.4 18.9 6.1 6.8 8.2 0.0
Correct mistakes in
spelling, grammar,
and punctuation
51.0
66.2 32.7 21.6 8.2 8.1 6.1 2.7 2.0 1.4
Make changes as you
write 30.6 41.9 38.8 39.2 24.5 23.0 4.1 4.1 2.0 0.0
Read what you have
written aloud yourself 14.3 27.0 24.5 24.3 26.5 18.9 20.4 18.9 16.3 9.5
Use a thesaurus
6.1 27.0 8.2 10.8 22.4 25.7 30.6 36.5 32.7 17.6
93
To explore the relationship between use of writing process strategies to writing
proficiency, a comparison was made of students’ responses to items on Scales 3 and 4 to
their performances on district writing assessments from 1997 to 1999.
Survey items relevant to planning/prewriting strategies and revising strategies
were compared to rhetorical effectiveness scores on district writing assessments. A
comparison was made of students reporting a majority of responses as low use of
planning/prewriting and revising and editing strategies (“Rarely” and “Never”), medium
use of planning/prewriting and revising and editing strategies (“Sometimes”), and high
use of planning/prewriting and revising and editing strategies (“Mostly” and “Always”)
with rhetorical effectiveness scores on district writing assessments from 1997 to 1999.
District writing assessment scores are reported based on a 6-point rubric (Appendix B).
Student performances were categorized as weak (1 and 2 on the 6-point rubric), medium
(3 and 4 on the 6-point rubric), and strong (5 and 6 on the 6-point rubric).
As shown in Table 4.14, a majority of students, 51.1%, reported a high use of
planning/ prewriting and re vising/editing strategies, with 37% demonstrating strong
performance on two or more district writing assessments. Of those students who reported
a medium use of writing strategies, 26.3%, a large number of those students, 70.3%,
demonstrated a medium performance on two or more district writing assessments. Of the
17.4% of students who reported a low use of writing strategies, only 11.3%, demonstrated
weak performance on two or more district writing assessments. There appears to be a
closer relationship of Scales 3 and 4 to writing performance than shown in Scales 1 and 2
to writing performance.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
94
Through interviews and observations, several areas of interest were noted. First, a
significant amount of instructional time is spent on teaching skills of prewriting/planning,
editing, and revising. Second, these skills form the basis for peer revision and teacher
evaluation of written work. Third, the application of these skills is honed through
continual practice and feedback. Fourth, prewriting and editing/revising skills are taught
similarly from teacher to teacher, i.e., writing process strategies form a standard
throughout the district. Fifth, the skills of prewriting and editing/revising that are used in
classroom assignments closely mirror the context of the district writing assessment.
These factors combined provide one explanation of the correlation of student responses to
items on Scales 3 and 4 with their actual writing performance.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced w ith permission o f th e copyright owner. Further reproduction prohibited without permission.
95
Table 4.14 Relationship of Use of Writing Process Strategies (Scales 3 and 4) to Writing Performance
Percentage of Students in Each Level.
School 1: N= 49 School 2: N=74
Use of Writing Process Strategies Writing Performance
High Medium Low Strong Medium Weak
School
1
School
2
School
1
School
2
School
1
School
2
School
1
School
2
School
1
School
2
School
1
School
2
46.2% 56.1% 28.9% 23.6% 13.8% 20.9% 13.8% 23.2% 72.4% 68.1% 13.8% 8.7%
Percents may not total 100 percents due to rounding.
96
Writing strategies such as prewriting, editing, and revising, may be associated with ethnic
styles of writing and proficiency in using the English language. As shown in Table 4.15,
an examination of the correlation of ethnicity to writing performance data was conducted
to determine if a relationship could be described.
The sample population used in this study did not reflect the ethnic make-up of the
schools as a whole. For example, in each school, approximately 10% of the total student
body was Hispanic. However, in this study’s sample population, no generalizable
findings could be made due to the small number of Hispanic, Filipino, and Asian
students. Likewise, only one English Language Learner was present in the sample
population at School 1, and none at School 2, thus prohibiting any conclusive analysis.
The relationship of ethnicity and language proficiency to writing performance is an area
for future study.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced w ith permission o f th e copyright owner. Further reproduction prohibited without permission.
97
Table 4.15 Relationship of Ethnicity to Writing Performance
Number and Percentage of Students
Ethnicity Writing Performance
Strong Medium Weak
School 1 School 2 School 1 School 2 School 1 School 2 School 1 School 2
Euro-American N=45
91.8%
N=69
93.2% 14% 23% 72% 70% 14% 9%
Hispanic N=3
6.1%
N=4
5.4% 0% 50% 66.6% 50% 33.3% 0%
Other:
Filipino N=1
2%
—
0% 100%
Asian ““ N=1
1.4%
—
100%
—
0% 0%
Percents may not total 100 percents due to rounding.
98
Findings— Scale 5: Teachers’ Comments on Completed Papers
Scale 5 focuses on instructional strategies. Teachers’ Comments on Completed
Papers provide information on the fifth factor analyzed for its influence on student
writing, Research Question 1, as well as examining the pedagogical effects of using
student-developed rubrics (Research Question 3).
Unlike Scales 1 through 4 which focus on students, Scale 5 focuses on teacher
behavior, specifically items ask students to report their experiences relevant to teachers’
comments on completed papers. Students scoring high on this scale always or mostly
receive teacher comments on their written work. Comments made by teachers inform
students on such areas as following directions, the ideas is their papers, organization of
their papers, the words that they use, and spelling, punctuation, and grammar.
Students reported that most of their teachers’ comments were in the areas of
spelling, punctuation, and grammar (50.4%), the ideas contained in the paper (48.8%),
and the organization of the paper (43.1%) as shown in Table 4.16. The item which
students rated lowest on this scale was whether the student had written enough.
Interviews and analysis of teaching materials and classroom assignments confirm these
findings. In addition, it was found that one area of teacher emphasis in providing
comments on student writing was the format of the paper, e.g., margins, type font, etc. It
should be noted that the Survey did not query students on this topic.
The type and extent of comments used by teachers communicates to the student
what elements of writing or presentation are valued. In comparing survey data with
teacher assignments and evaluation forms, the researcher noted an alignment of areas
with permission of the copyright owner. Further reproduction prohibited without permission.
99
typically commented on by teachers. It was also noted that teachers did not weigh one
area as more valuable than another area. In interviews, students reported that they viewed
all evaluation criteria as equal, even though the amount of instructional time devoted to
formatting criteria was minor compared to the amount of instructional time devoted to
organization and ideas contained in written work. As a result, students perceived that
margins and formatting requirements were as important as ideas and conventions in a
written assignment.
Findings— Scale 6: Teacher Feedback on Writing
Teacher Feedback on Writing (Scale 6) constitutes the sixth factor examined to
answer the question, “What factors influence student writing?” Like Scale 5, this scale
also informs the question of what pedagogical effects rubric-based writing instruction had
on students and teachers (Research Question 3).
Items on Scale 6 focus on the types of feedback teachers provide to students.
Methods of teacher feedback include marking mistakes on papers, writing notes on
papers, pointing out what was done well and what was not well done, and making
suggestions for improvement the next time.
As shown in Table 4.17, three fourths of students, 75.6%, reported that the type of
feedback they received most from teachers was marking mistakes. This was closely
followed by 73.2% of students who responded that teachers wrote notes on their papers.
Additionally, over two thirds of students had teachers who pointed out what was not done
well (67.5%) and teachers who pointed out what was done well (66.7%). The method
used least by teachers to provide feedback on written work was making suggestions for
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
1 0 0
the next time. Over one fifth of students, 21.2%, reported that they rarely or never had
teachers make suggestions for the next time they wrote.
These findings were confirmed in interviews with teachers and students and
through analysis of papers which had been graded by teachers. Teachers reported that
they spend most of their time in writing notes and correcting mistakes on papers, and
very little time in asking probing questions or making suggestions to improve student
writing.
The researcher noted one major difference between the types of feedback
provided by district elementary and high school teachers. A standard practice with upper
elementary grade teachers is the use of Writer’s Workshop in which each student’s work
is analyzed in a one-to-one setting with the teacher. The focus of the Workshop is to
provide suggestions for improving the student’s writing. Elementary teachers report that
this technique has been extremely effective. High school teachers in the district, in
general, do not use this technique, or if it is used, peer editors act as the evaluator in
making suggestions for improvement. The teachers involved in this study did not use the
Writer’s Workshop method. In interviews with the teachers, it was reported that peer
editing, other than on language conventions, was not an effective means of providing
advice to improve writing. An analysis of student samples at various phases of the
writing process confirm this statement. An area for future study might be to examine the
use of Writer’s Workshop techniques at the high school level to improve writing.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced w ith permission o f th e copyright owner. Further reproduction prohibited without permission.
101
Table 4.16 Findings - Scale 5: Teachers’ Comments on Completed Papers
Percentage of Students Responding in Each Level
School 1: N= 49 School 2: N=74 Percents may not total 100 percents due to rounding.
Statement
After you have
written
papers, how often
does the teacher talk
or write to you
about each of the
following things?
Always Mostly Sometimes Rarely Never
School 1 School 2 School 1 School 2 School 1 School 2 School 1 School 2 School 1 School 2
How you followed
directions
18.4% 8.1% 30.6% 21.6% 20.4% 31.1% 20.4% 18.9% 8.2% 16.2%
Whether you wrote
enough in the paper 16.3 6.8 32.7 12.2 26.5 20.3 18.4 36.5 12.2 24.3
The ideas in your
paper 22.4 16.2 30.6 44.9 32.7 44.9 8.2 22.4 6.1 9.5
The way you
explained your ideas 22.4 10.8 20.4 40.8 34.7 28.4 16.3 21.6 6.1 9.5
The way you
organized your paper 24.5 13.5 26.5 24.3 30.6 31.1 14.3 16.2 4.1 9.5
The words that you
used
18.4
5.4 22.4 16.2 24.5 40.5 24.5 21.6 10.2 22.4
Your spelling,
punctuation, and
grammar
49.0 12.2 20.4 25.7 20.4 28.4 8.2 18.9 2.0 13.5
Reproduced w ith permission o f th e copyright owner. Further reproduction prohibited without permission.
102
Table 4.17 Findings - Scale 6: Teacher Feedback on Writing
Percentage of Students Responding in Each Level.
School 1: N= 49 School 2: N=74 Percents may not total 100 percents due to rounding.
Statement
How often does the
teacher do each of
the following things
with your writing?
Always Mostly Sometimes Rarely Never
School 1 School 2 School 1 School 2 School 1 School 2 School 1 School 2 School 1 School 2
Mark mistakes in
you paper
65.3% 37.8% 24.5% 28.4% 10.2% 20.3% 0.0% 12.2% 0.0% 0.0%
Write notes on your
paper 49.0 41.9 20.4 3.8 30.6 24.5 6.1 5.4 0.0 1.4
Point out what you
did well 22.4 36.5 38.8 33.8 20.4 13.5 10.2 8.1 8.2 4.1
Point out what you
did not do well 32.7 36.5 34.7 31.1 28.6 24.3 2.0 1.4 0.0 4.1
Make suggestions
for the next time
you write
34.7 24.3 18.4 28.4 22.4 25.7 16.3 12.2 8.2 6.8
103
Within Scales 4 and 5 was a specific area of interest to the researcher, conventions
of writing, i.e., spelling, punctuation, grammar, and capitalization. One of the arguments
made by critics of Whole Language Movement was that writing conventions were not as
valued, and as a result, students’ spelling, grammar, and punctuation suffered. For
example, students were encouraged to use “inventive spelling” because communication
was more highly valued than correct spelling. Critics of Whole Language contended that
teachers no longer provided formal, systematic instruction in conventions, and that, in
general, student work was not corrected as thoroughly for conventions in the Whole
Language Approach as it had been in traditional programs.
Although Whole Language supporters never advocated abandoning the teaching
of spelling and conventions, as evidenced in the California English-Language Arts
Framework (English-Language Arts Curriculum Framework and Criteria Committee,
1987) which called for skills to be taught in context, the criticism remained. The
ramifications of the backlash against Whole Language are evident in current California
Education Code Section 9543 which stipulates that instructional materials adopted by the
state of California shall be based upon fundamental skills of English-language arts
including systematic, explicit phonics, and spelling (West’s California Codes, 1999) and
in the construction of the Academic Performance Index (API) in California which assigns
a weight of 15% for spelling at grades 2 to 8, and an additional 15% for language skills,
which is largely composed of conventions-related questions. At grades 9-12, the spelling
subtest is not given, but “language” is accorded a 20% weight, the equivalent value as
assigned to mathematics, history-social science or science.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
104
Since current ninth grade students were in elementary school during the Whole
Language Movement, it was enlightening to examine their editing skills relevant to
writing conventions. Question 2 Ion the Student Writing Survey asked students to rate
themselves on how often they corrected errors in conventions to make their papers better.
Question 31 on the Survey asked students how often the teacher provided feedback on
conventions. Responses to these two questions were compared to students’ conventions
scores on the district writing assessments.
A comparison was made of students reporting a low frequency on Questions 21
and 31 (“Rarely” and “Never”), medium frequency (“Sometimes”), and high frequency
(“Mostly” and “Always”) with conventions scores on district writing assessments from
1997 to the present. District writing assessment scores were categorized as weak (1 and 2
on the 6-point rubric), medium (3 and 4 on the 6-point rubric), and strong (5 and 6 on the
6-point rubric). Table 4.18 summarizes the findings of this subsection of Scales 4 and 5.
It appears that definitive conclusions may not be drawn by comparing survey data
with writing performance. Although a high percentage of students (70.1%) report that
they frequently correct mistakes in spelling, grammar, and punctuation and their teachers
frequently comment on conventions, only 18.2% of those students demonstrate strong
control of conventions on district writing assessments. Of students who report sometimes
correcting convention errors and sometimes having teachers comment on conventions,
59.3% score in the middle range on writing assessments. Of the students who report that
they rarely or never correct errors in conventions and their teachers rarely or never
comment on conventions on their papers (27.4%), only 10.8% demonstrate weak
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
105
performance on writing assessments. Interviews and analysis of classroom writing
assignments shed no light on these discrepancies.
A second comparison was made to determine if there was any relationship among
student responses to Survey items 21 and 31, writing performance on district writing
assessments, and scores on the Stanford Achievement Test form T (SAT 9) (Table 4.19).
Test scores from spring 1998 and spring 1999. The subtests of language and spelling
from the SAT 9 were used as the basis for this measure.
The data suggest that there may be a relationship between Survey findings and
SAT 9 scores. Students scored above the mean on all subtests, except spelling at School
1 in 1999. It appears that students are able to select correct responses on a multiple
choice test, far better than they are able to produce writing on a performance-based test.
One might conclude that student knowledge of language conventions, including spelling,
has not been adversely impacted by the Whole Language approach. It is interesting to
note that less than one fifth of students, 18.2%, were able to achieve a strong performance
on writing assessments, but these same students score near or above the national mean in
their knowledge of conventions.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced w ith permission o f th e copyright owner. Further reproduction prohibited without permission.
106
Table 4.18 Relationship of Use of Conventions to Writing Performance
Percentage of Students in Each Level.
School 1: N= 49 School 2: N=74
Use of Conventions Writing Performance
High Medium Low Strong Medium Weak
School
1
School
2
School
1
School
2
School
1
School
2
School
1
School
2
School
1
School
2
School
1
School
2
76.5% 63.8% 26.5% 18.2% 9.2% 18.2% 16.4% 19.9% 69.8% 49.6% 13.8% 7.7%
Percents may not total 100 percents due to rounding.
Reproduced w ith permission o f th e copyright owner. Further reproduction prohibited without permission.
107
Table 4.19 Relationship of Use of Conventions, Writing Performance, and SAT 9 Scores
Percentage of Students in Each Level.
School 1: N= 49 School 2: N=74
Use of Conventions Writing Performance
High Medium Low Strong Medium Weak
School 1 School 2 School 1 School 2 School 1 School 2 School 1 School 2 School 1 School 2 School 1 School 2
76.5% 63.8% 26.5% 18.2% 9.2% 18.2% 16.4% 19.9% 69.8% 49.6% 13.8% 7.7%
Percents may not total 100 percents due to rounding.
National Percentile Ranks (NPR)
SAT 9
Spring 1998 (7th grade) Spring 1999 (8th grade)
Language Spelling Language Spelling
School 1 School 2 School 1 School 2 School 1 School 2 School 1 School 2
59 75 51 62 57 75 43 55
108
Findings— Scale 7: Use of a Rubric in Writing
The Use of a Rubric in Writing (Scale 7) responds to the three Research Questions
presented in this study. The use of a rubric was examined as the seventh factor
influencing student writing. This Scale also provided information on the ways in which
student-developed rubrics affected writing performance and pedagogy.
One of the most surprising findings from the survey data was the majority of
students, 51.0%, reported rarely or never using rubrics (Table 4.20). This result was
particularly surprising in view of the fact that 81%of the students in the study had
attended schools in the district since the fourth grade. Since 1995, when the students
involved in this study were in the 4th grade, rubrics have been used to score student
writing on district assessments. Teachers were calibrated to apply the rubric across the
district at grade-level as part of the district professional development program. Essays
were scored and returned to teachers for use in diagnosing student needs. Scores were
reported to principals and district administrators and used as a measure of program
evaluation. Teachers were encouraged to share the district rubric, or a student version of
the district rubric, with students and to use the rubric to guide writing instruction.
Interviews with students and teachers, classroom observations, and analysis of
student work revealed that district rubrics were not used to establish criteria to evaluate
writing. Instead, teachers involved in the study use Evaluation Guides. These Guides
provide a type of checklist for the student. Categories created by the teachers include
such areas as format, organization, mechanics, writing style and content. A second type
of Evaluation Guide used by teachers is divided into four categories: Beginning, middle,
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
109
conclusion, and mechanics/manuscript form. These evaluation guides are used in peer
editing, self evaluation, and teacher evaluation. Scoring on the Evaluation Guides differ
from points assigned to a category, points allowed for a complete assignment, yes/no
responses, holistic score responses, number of errors, and continua of effectiveness.
Often a number of methods of grading are used for one assignment.
In interviews, teachers reported that they perceived the Evaluation Guides to be
rubrics, and that the Guides were used as rubrics to instruct writing. Teachers also stated
that they did not use the term “rubric” with students. This statement was confirmed in
interviews with students. Students reported that they did not recognize the terminology,
“rubric or scoring guide” on the Survey, even though teachers did explain the term before
students completed the Survey. Due to the misunderstanding in the terms used, students
responded that they rarely or never used rubrics. When the teachers’ Evaluation Guides
were shown to the students, they acknowledged that they had used the Guides in
classroom writing assignments.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced w ith permission o f th e copyright owner. Further reproduction prohibited without permission.
110
Table 4.20 Findings - Scale 7: Use of a Rubric in Writing
Percentage of Students Responding in Each Level.
School 1: N= 49 School 2: N=74 Percents may not total 100 percents due to rounding.
Statement Always Mostly Sometimes Rarely Never
School 1 School 2 School 1 School 2 School 1 School 2 School 1 School 2 School 1 School 2
How often has a
teacher used a rubric
or scoring guide to
grade your writing?
12.2% 6.8% 24.5% 31.1% 24.5% 32.4% 28.6% 23.0% 10.2% 5.4%
How often have you
used a rubric or
scoring guide to
revise your writing?
8.2 2.7 16.3 4.1 24.5 29.7 16.3 20.3 34.7 39.2
How often have you
used a rubric or
scoring guide to
evaluate your own
writing?
12.2 2.7 6.1 8.1 22.4 27.0 22.4 23.0 32.7 37.8
How often have you
used a rubric or
scoring guide to
evaluate the writing
of others?
2.0 0.0 18.4 16.2 22.4 25.7 32.7 29.7 24.5 27.0
I l l
Summary
The use of scales derived from the Student Writing Survey provided a framework
upon which various factors relevant to writing instruction could be analyzed. Responses
from the Survey, along with classroom observations, interviews with students and
teachers, assessment data, and analysis of lesson plans and writing samples, supplied
multiple measures for describing the ways students were involved in developing rubrics
and how those rubrics affected their writing performance. Secondly, these data sets
facilitated the description of the pedagogical effects rubric-based writing instruction had
on students and teachers.
Students became active participants in constructing the meaning of the criteria
upon which their work was being evaluated. Students created rubrics based upon
metaphorical comparisons to score levels and descriptors to label each score level. These
student-developed rubrics served as touchstones to evaluating their own and others’
written work.
The language used by students and teachers to critique written work was based
upon the rubrics students had developed. Students reported changes in the techniques
they applied in their writing, particularly with regard to revising and editing. For
example, student understanding that the content of the writing was paramount to
formatting requirements, shifted the balance of time they spent on creating ideas and
organizing those ideas to better meet the demands of purpose and audience. Students also
felt more empowered by being able to design rubrics. Students reported that deeper
understanding of the criteria they gained by developing rubrics allowed them to take
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
112
more chances with their writing, especially with regard to style features. They reported
making more changes in word choice and in moving sentences around in their paper to
provide for better organization and transitions.
Putting data derived from this study in the hands of teachers also proved to be a
powerful, objective method for analyzing and modifying pedagogy. Teachers knew that
using their rubrics as evaluation guides had only effected 50% of their students in
revising written work. With the use of student-developed rubrics, teachers noticed that
there was a greater degree of understanding of the criteria and more willingness to use the
criteria to improve writing. Teachers also reported that the student-developed rubrics
provided a common language and set of images upon which discussions centered. On the
whole, however, teachers noted that, in general, the quality of student writing had not
improved significantly, but the motivation to complete revisions based on teacher
feedback had improved somewhat. Teachers felt that student-developed rubrics was a
promising strategy to use to instruct writing, and they reported that they would be
interested in using the technique again.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
113
Chapter 5
SUMMARY
The Problem
Writing continues to be a fundamental skill which schools are expected to teach
and students are expected to master. A great deal of instructional time and professional
preparation is devoted to the teaching of writing throughout the United States. Despite
these efforts, in general, students continue to perform at minimal levels on national
assessments of writing at the elementary, middle, and high school levels.
The accountability movement and its accompanying reliance on standardized test
measures often leave writing samples as the only form of performance-based assessment
in a “bubble-in” battery of tests. Therefore, writing continues to provide a robust
measure of a student’s ability to craft an organized, thoughtful, sustained piece of
communication. Writing as assessment provides evidence of thinking and writing skills
in which rhetorical effectiveness and use of language conventions may be viewed in
context. Therefore, it is critical that students not only understand the criteria of quality
writing, but are able to transfer that understanding into application. The student as
evaluator of one’s own writing and the writing of others depends on the internalization of
these criteria. Rubrics and the systematic instruction, understanding, and application of
writing rubrics provide one strategy toward achieving that goal. Furthermore, it is clear
from a review of the literature that scant research has been conducted in the area of using
student-developed rubrics to improve student writing and pedagogy.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
114
Methodology
This study described the use of rubrics to instruct writing and the effect of
student-developed rubrics on writing performance and pedagogy. A Student Writing
Survey was developed based on scales created by the National Assessment of Educational
Progress. Using information gleaned from the survey, classroom observation, writing
samples, assessment data, and interviews, the researcher investigated the ways in which
student-developed rubrics affected student writing performance and pedagogy.
SELECTED FINDINGS
The following findings were selected to summarize the most important data of the
study and to answer the research questions posed by this study.
1. By using rubrics, and particularly by involving students in developing rubrics,
students and teachers created a mutually agreed-upon set of criteria upon
which to base evaluative decisions.
2. The empirically-derived scales found in the Student Writing Survey provided
a set of factors which were correlated to student performance on classroom
writing assignments, writing assessments, and standardized tests.
3. Teachers involved in the study had used rubrics to evaluate student writing
and to have students evaluate their own and their peer’s writing, but had never
used student-developed rubrics as an instructional tool to improve writing.
After including student-developed rubrics as an instructional strategy, teachers
reported that they found it effective in improving student writing and they
would use the strategy again.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
115
4. Survey data revealed the following:
• Students in general hold place high value on writing and hold positive
attitudes toward writing.
• Students in general also report high frequency use of writing process
skills, such as planning/prewriting, revising, and editing strategies
5. Survey and interview data revealed the following:
• Students and teachers report a high frequency of teacher comments on
completed papers and feedback on writing.
• Students and teachers report high frequency use of teacher-developed
rubrics in writing instruction.
6. Differences in student responses as related to each scale of the survey did not
vary significantly between schools involved in the study.
CONCLUSIONS
The following conclusions have been drawn based upon the findings and
discussion presented in Chapter 4.
1. Use of the scales found in the Student Writing Survey provide a vehicle to
assess student perception on a variety of factors relevant to writing.
Furthermore, these scales could be correlated to writing achievement on
performance-based assessments and to subtests of norm-referenced measures.
2. As a result of the use of student-developed rubrics in writing, students
reported a deeper understanding of the criteria upon which their work was
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
116
evaluated. Students were able to apply the rubric to revise their own work and
to critique the work of others.
3. As a result of the use of student-developed rubrics as an instructional tool,
teachers modified their instruction by devoting class time for students to
create rubrics and by using those rubrics to discuss student work.
RECOMMENDATIONS
Findings from this study are encouraging. The use of student-developed rubrics
did affect writing performance in that students changed the techniques they used to
evaluate their own written work. Teachers learned that students demonstrated deeper
understanding of the criteria of quality writing when they were engaged in developing
their own rubrics.
As a result of this study, there are a number of areas which would benefit from
further investigation and research:
Expansion of Study to Other Grade Levels
• Replication of this study at elementary and middle school grade levels to
determine if similar findings are observed.
• Comparison of the sophistication and complexity of student-developed rubrics
at selected grade levels.
Demographic Factors
• Adaptation of the methodology of this study to reflect research on ethnic
styles of writing.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
117
• Replication of this study with a more ethnically diverse student population
and greater numbers of English Language Learners.
• Analysis of additional demographic factors, such as gender, mobility, and
parent education level.
Longitudinal Effects
• Use of the methodology of this study over time to determine longitudinal
patterns and trends on various scales found in the survey.
• Follow-up studies to determine the long term effects of using student-
developed rubrics on writing performance and pedagogy.
• Examination of the consistency of instructional practices in teaching writing
over course of a student’s educational experience.
• Examination of the use of instructional practices at various phases of a
teacher’s career.
Correlation to Other Measures
• Use of assessment data, e.g., California high school exit exam, Scholastic
Assessment Test, Golden State Examinations, to determine if there is a
correlation of student-developed rubrics to writing performance on high stakes
tests.
• Comparison of locally gathered data on the scales found in the survey with
nationally collected data, such as the National Assessment of Educational
Progress writing assessments.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
118
Effect of Teacher Training
• Examination of the effect of professional development on instructional
strategies used to teach and to evaluate writing.
• Differences in teacher preparation, e.g., emergency credentialed teachers,
beginning teachers, teachers with advanced degrees in English, teachers with
more than five years of teaching experience, etc.
Types of Writing
• Separation of types of writing, e.g. written assignments in school, personal
notes/letters to friends, and email, to ascertain if students actually did write
outside of school, but did not define communication such as email as
“writing” on Question 11 of the Student Writing Survey.
• Tracing the long-term effects of the use of email on students’ writing
performance, specifically the use of conventions and the ability to organize a
sustained, lengthy piece of written discourse.
FINAL THOUGHTS
Writing will continue to be a fundamental skill taught in schools throughout the
United States. As a nation, we cannot tolerate the fact that almost three fourths of our
children are writing below grade level (National Center for Education Statistics, 2000). If
we are to improve student writing, we must open the “black box” of the classroom to
closely examine instructional practice and to apply pedagogical approaches which have
been found to be effective. Student-developed rubrics provides one such approach which
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
119
supports other areas of research, such as writing process strategies and constructivist
learning theory.
As this study has shown, the use of student-developed rubrics has evidenced
changes in the techniques students used to evaluate written work and the pedagogy
teachers used to instruct writing. It is clear that more research is needed in this area. This
study provides only the beginning.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
120
W orks Cited
Ackerman, J. (1990). Translating content into action. In L. Flower, V. Stein, J.
Ackerman, M. J. Kantz, K. McCormick, & W. C. Peek (Eds.). Reading-to-write:
Exploring a cognitive and social process (pp. 173-193). Oxford. England: Oxford
University Press.
ACSU. (1995). Memorandum from James D. Lombardo for the VS A Assessment
Committee to Vermont Superintendents. Dated May 8, 1995. Middlebury, VT:
Addison Central Supervisory Union, Office of the Superintendent.
Alexander, L. and James, H. T. (1987). The nation’s report card: Improving the
assessment of student achievement (Report of the Study Group). Washington D.
C.: National Academy of Education.
Alkin, M. C. (Ed.). (1992). Encyclopedia of educational research (6th ed., vol.2). New
York: Macmillan.
Andrade, H. (1999). The effects of instructional rubrics on student writing. Educational
Leadership. February 2000, 13-18.
Applebee, A. N. (1991). Informal reasoning and writing instruction. In J. Voss, D.
Perkins, & J. Segal (Eds.). Informal reasoning and education (pp. 401-414).
Hillsdale, NJ: Lawrence Erlbaum.
Applebee, A. N. (1994). NAEP 1992 writing report card. . Princeton, NJ: National
Assessment of Educational Progress.
Applebee, A. N. (1986). Problems in process approaches: Toward a reconceptualization
of process instruction. In Petrovksy, A. & Bartholamae, D. (Eds.). The teaching
of writing: Eighty fifth yearbook of the National Society for the Study of
Education. Chicago: University of Chicago Press, 95-113.
Applebee, A., Langer, J., Mullis, I., & Jenkins, L. (1990). The writing report card. 1984-
1988. Princeton, NJ: National Assessment of Educational Progress.
Arter, J. A. et al. (1994). The impact of training students to be self-assessors of writing.
Paper presented at the Annual Meeting of the American Educational Research
Association, New Orleans. (ERIC Document Reproduction Service No. ED
370975).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
121
Arter, J. A. and Salmon, J. R. (1987). Assessing higher order thinking skills: A
consumer’s guide. Portland, OR: Northwest Regional Educational Laboratory.
Association for Supervision and Curriculum Development. (December 1997). Voluntary
National Testing. AS CD Infobrief. Issue II. 4-5.
Association for Supervision and Curriculum Development. (August 1995). Education
Update. 37. no. 6, 1,4-5,8.
Asp, E. (2000). Assessment in education: Where have we been? Where are we headed?
In Brandt, R. S. (Ed.), Education in a New Era. 123-157. Alexandria, VA:
Association for Supervision and Curriculum Development.
Bartholomae, D. (1985). Inventing the university. In M. Rose (Ed.), When a writer
can’t write: Studies in writer’s bloc and other composing problems (pp. 134-165).
New York: Guilford.
Batholomae, D. (1980). The study of error. College Composition and Communication.
31 (3), 253-269.
Bereiter, C. & Scardamalia, M. (1987). The psychology of written comprehension.
Hillsdale, NJ: Lawrence Erlbaum Associates.
Bizell, P. (1982). College composition: Initiation into the academic discourse
community. Curriculum Inquiry. 12 (2). 191-207.
Borko, H., Flory, M., and Cumbo, K. (October 1993). Teachers’ ideas and practices
about assessment and instruction: A case study of the effects of alternative
assessment in instruction, student learning, and accountability practice. CSE
Technical Report, no. 366. Los Angeles: National Center for Research On
Evaluation, Standards, and Student Testing (CRESST), University of California at
Los Angeles.
Brown, R. (1981). National assessments of writing ability. In E. D. Gagne, A. W.
Yerkovich, & F. R. Yerkovich, The cognitive psychology of school learning (2n d
ed.). New York: Harper Collins College Publishers.
Burger, S. E. (1994). Determining the validity of performance-based assessment.
Educational Measurement: Issues and Practice. 13. no,l, 9-15.
California Department of Education. (August 1997). A new accountability system for
California schools. Paper presented at School’s IN!: The State Superintendent’s
symposium on critical issues for California Schools, Sacramento, CA.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
122
Center for Civic Education. (1995). Issues concerning a national assessment of civics:
Executive summary. New York: Center for Civic Education.
Commission on Composition. Teaching Composition: A position statement. Urbana, IE:
National Council of Teachers of English, 1979.
Daiute, C. (1981). Psycholinguistic foundations of the writing process. Research in the
Teaching of English. 15 (1), 5-22.
Darling-Hammond, L. & Wise, A. (1985). Beyond standardization: State standards and
school improvement. The Elementary School Journal. 85. 315-336.
Delpit, L. (1986). Skills and other dilemmas of a progressive black educator. Harvard
Educational Review. 56 (4), 379-385.
Diegmueller, K. (April 12, 1995). Running out of steam. Struggling for standards: an
Education Week special report. Washington, D. C.: Education Week.
Dietel, R. J., J. L. Herman, and R. A. Knuth. (1991). What does research sav about
assessment? Oakbrook, IL: NCREL.
Donovan, F. and Sneideer, C. (September 1994). Setting and meeting the national
standards. Technology and Learning. 40-48.
Dorr-Bremme, D. W. & Herman, J. L. (1986). Assessing student achievement: A profile
of classroom practices. Center for the Study of Evaluation, University of
California, Los Angeles.
Durst, R. K. (1990). The mongoose and the rat in composition research: Insights from
the RTE annotated bibliography. College Composition and Communication. 4
(4), 393-408.
Dyson, A. H. & Freedman, S. W. (1990). On teaching writing: A review of the literature
(Occasional Paper No. 20). Berkeley, CA: University of California, National
Center or the Study of Writing and Literacy.
Dyson, A. H. & Freedman, S. W. (1991). Writing. In J. Flood, J. Jensen, D. Lapp, & J.
Squire (Eds,), Handbook of research on teaching the English language arts
(pp.787-802). New York: Macmillan.
Educational Testing Service. (1995). Performance assessment: Different needs, difficult
answers. Princeton, NJ: Educational Testing Service.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
123
English-Language Arts Curriculum Framework and Criteria Committee. (1987).
English-language arts framework for California public schools kindergarten
through grade twelve. Sacramento, CA: California State Department of
Education.
Falk, B. and Darling-Hammond, L. (March 1993). The primary language record at P. S.
261: How assessment transforms teaching and learning. New York: National
Center for Restructuring Education, Schools, and Teaching.
Frederiksen, J. R. and Collins, A. (1989). A systems approach to educational testing.
Educational Researcher. 19. no. 9.
Freedman, S. W., Dyson, A. H., Flower, L. & Chafe, W. (1987). Research on writing:
Past, present, and future (Tech. Rep. No. 1). Berkeley, CA: University of
California, Center of the Study of Writing.
Gagne, E. D., Yekovich, A. W., &Yekovich, F. R. (1993). The cognitive psychology of
school learning (2n d ed.). New York: HarperCollins College Publishers.
Garcia, G. E. & Pearson, P. D. (1991). The role of assessment in a diverse society. In
Hiebert, E. (Ed.). (1991). Literacy for a diverse society. New York: Teachers
College Press, 253-276.
Gardner, H. (1992). Assessment in context: The alternative to standardized testing. In
Gifford, B. R. & O’Connor, M. C. (Eds.), Changing assessments: Alternative
views of aptitude, achievement and instruction. 77-120. Boston: Kluwer
Academic Press.
Goodrich, H. (1996). Student self-assessment: At the intersection of metacognition and
authentic assessment. Educational Leadership. February 2000,13-18.
Hairston, M. (1982). The winds of change: Tomas Kuhn and the revolution in the
teaching of writing. College Composition and Communication. 33. (1), 76-88.
Haney, W. H., Madaus, G. F., and Lyons, R. (1993). The fractured marketplace for
standardized testing. Boston, MA: Kluwer Academic Publishers.
Harrington-Lueker, D. (February 1994). Toward a new national yardstick. The
American School Board Journal. The American School Board, 41-43.
Harris, J. (1989). The idea of community in the study of writing. College Composition
and Communication. 40 (TL 11-22.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
124
Hembree, R. (1987). Effects of noncontent variables on mathematics test performance.
Journal for Research in Mathematics Education. 18. 197-214.
Hiebert, E. (Ed.). (1991). Literacy for a diverse society. New York: Teachers College
Press.
Hull, G. (1987). The editing process in writing: A performance study of more skilled
and less skilled college writers. Research in the Teaching of English. 21 (1), 8-29.
Huot, B. (1990). The literature of direct writing assessment: Major concerns and
prevailing trends. Review of Educational Research. 60 (2 ), 237-263.
Imbens-Bailey, A. (Fall 1998). 1998 CRESST conference focused on urgent issues. The
CRESST Line. 4.
Jaeger, R. M. (1982). The final hurdle: Minimum competency achievement testing. In
Austin, G. R. and Garber, H. (Eds.). The rise and fall of national test scores
(pp.223-246). New York: Academic Press.
Keeves, J. P. (1994). United States: System of education. In International Encyclopedia
of Education (Vol. 11, 6515-6523). New York: Pergamon.
Kentucky Institute for Education Research, The. (January 1995). An independent
evaluation of the Kentucky Instructional Results Information System (KIRIS):
Executive summary. Frankfort, KY: The Kentucky Institute for Education
Research.
Khattri, N., A. L. Reeve, M. B. Kane, and R. Adamson. (1996). Studies of education
reform: Assessment of student performance. Washington, D. C.: U. S.
Department of Education, Office of Educational Research and Improvement.
Khattri, N., M. B. Kane, and A. L. Reeve. (November 1995). Research report: How
performance assessments affect teaching and learning. Educational Leadership.
53,80-83.
Koretz, D., Stecher, B., Klein, S. & McCaffrey, D. (1994). The evolution of a portfolio
program: The impact and quality of the Vermont program in is second year. Los
Angeles, CA: RAND Institute on Education and Training, Center for Research on
Evaluation, Standards and Student Testing (CRESST).
Koretz, D., Stecher, B., Klein, S. & McCaffrey, D., and Diebert, E. (1993). Can
portfolios assess student performance and influence instruction? The 1991-92
Vermont experience. Los Angeles, CA: RAND Institute on Education and
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
125
Training, Center for Research on Evaluation, Standards and Student Testing
(CRESST).
Koretz, D. M., Madaus, G. F., Haertel, E., and Beaton, A. E. (1992). National education
standards and testing: A response to the recommendations of the National Council
on Education Standards and Testing. Santa Monica, CA: RAND.
Koretz, D., McCaffrey,D., Klein, S., Bell, R. and Stecher, B. (1993). The reliability of
scores from the 1992 Vermont Portfolio Assessment Program. Los Angeles, CA:
RAND Institute on Education and Training, Center for Research on Evaluation,
Standards and Student Testing (CRESST).
Koretz, D., Stecher, B., and Deibert, E. (1992). The Vermont portfolio assessment
program: Interim report on implementation and impact. 1991-92 school year. Los
Angeles, CA: RAND Institute on Education and Training Center for Research on
Evaluation, Standards and Student Testing (CRESST).
Kubistant, T. (1981). Test performance: The neglected skill. Education. 102. 53-55.
Lane, S., Liu, M., Ankenmann, R. D., and Stone, C. A. (1996). Generalizability and
validity of a mathematics performance assessment. Journal of Educational
Measurement. 33. no. 1, 71-92.
Lewis, A. (Fall 1998). 1998 CRESST conference focused on Urgent Issues. The
CRESST Line. 4.
Linn, R. (1994). Performance assessment: Policy promises and technical measurement
standards. Educational Researcher. 23. no. 9,4-14.
Linn, R. (1999). Standards-based accountability: Ten suggestions. CRESST Technical
Report 490. Assessment and Accountability. 1998. . Los Angeles: National
Center for Research On Evaluation, Standards, and Student Testing (CRESST),
University of California at Los Angeles.
Marzano, R. J. (1990). Standardized tests: Do they measure general cognitive abilities?
NASSP Bulletin. 74. no. 526., 93-101.
Marzano, R. J. and Costa, A. L. (May 1988). Question: Do standardized tests measure
cognitive skills? Answer: No. Educational Leadership. 45. 66-73.
Marzano, R. J. and Kendall, J. S. (1996). A comprehensive guide to designing
standards-based districts, schools, and classrooms. Alexandria, VA: Association
for Supervision and Curriculum Development.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
126
Mathematical Science Education Board. (1993). Measuring what counts. New York:
National Research Council.
McDonald, J. P. (1991). Exhibitions of mastery: Some preliminary considerations.
Research paper prepared for the Coalition of Essential Schools. Provident, RI:
Brown University.
Mehrens, W. A. (1992). Using performance assessment for accountability purposes.
Educational Measurement: Issues and Practice. 9. no. 11, 3-9, 20.
Mitchell, R. (1992). Testing for learning. New York: The Free Press.
Mitchell, R. and Neill, M. (1992). Criteria for evaluation of student assessment.
Washington, D. C., National Forum On Assessment.
Monk, D. H. (1995). The costs of pupil performance assessment: A summary report.
Journal of Education Finance. 20. no. 3, 363-371.
Mullis, I. et al. (1994). NAEP trends in academic progress: Achievement of U. S.
students in science 1969 to 1992 - mathematics. 1973 to 1992 - reading. 1971 to
1992 - writing. 1984 to 1992. Princeton, NJ: National Assessment of Educational
Progress.
National Assessment for Educational Progress. (1982). Writing achievement. 1969-1979
(Report No. 10-W-35). Denver, CO: Education Commission of the States.
National Center for Education Statistics. (1995). The condition of education 1995.
Washington, D. C.: Education Commission of the States.
National Center for Education Statistics. (2000). NAEP 1998 Writing Report Card
Highlights. Washington, D. C.: National Center for Educational Statistics.
National Center for Education Statistics. The Nation’s Report Card: Report in Brief:
NAEP 1996 Trends in Academic Progress. National Center for Educational
Statistics: Washington, D. C., 1998.
National Commission on Excellence in Education. (1983). A nation at risk: The
imperative for educational reform. Washington, D. C.: Government Printing
Office.
Neill, D. M. and Medina, N. J. (February 1989). Standardized testing: harmful to
educational health. Phi Delta Kanpan. 70. 688-697.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
127
Newman, F. (1991). Linking restructuring to authentic achievement. Phi Delta Kappan.
72(6), 458-463.
Newmann, F. M., Secado, W. G., and Wehlage, G. G. (1995). A guide to authentic
instruction and assessment: Vision, standards, and scoring. Madison, WI:
Wisconsin Center for Educational Research, University of Wisconsin.
Nystrand, M, Greene, S., & Wiemelt, J. (1993). Where did composition studies come
from? An intellectual history. Written Composition. 10 (3), 267-333.
O’Neill, J. (February 1993). On the New Standards Project: A conversation with Lauren
Resnick and Warren Simmons. Educational Leadership. 50. 17-21.
O’Neil, J. (May 1992). Putting performance assessments to the test. Educational
Leadership. 49. 14-19.
Pea, R. D. & Kurland, D. M. (1989). Cognitive technologies for writing. Review of
Research in Education. 14, 277-326.
Perl, S. (1971). The composing processes of unskilled college writers. In E. D. Gagne,
A. W. Yerkovich, & F. R. Yerkovich, The cognitive psychology of school
learning (2n d ed.). New York: Harper Collins College Publishers.
Picus, L. O. (March 1997). Estimating the costs of alternative assessment programs:
Case studies in Kentucky and Vermont. Paper presented at the annual meeting of
the American Education Research Association, Chicago.
Picus, L. O. (1994). A conceptual framework for analyzing the costs of alternative
assessment. Los Angeles: National Center for Research On Evaluation, Standards,
and Student Testing (CRESST), University of California at Los Angeles.
RAND Corporation. (January 4, 1996). Educators weight high price of performance
assessments. Education Daily. 29. 1-3.
Resnick, D. (1982). History of educational testing. In Wigdor, A. K. and Gamer, W. R.
(Eds.), Ability testing: Uses, consequences, and controversies. Part II (pp.173-
194). Washington, D. C.: National Academy Press.
Resnick, L. B. (1987a). Education and learning to think. Washington, DC: National
Academy Press.
Resnick, L. B. (1987b). Learning in school and out. Educational Researcher. 16. no. 9.
13-20.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
128
Resnick, L. B. and Resnick, D. P. (1992). Assessing the thinking curriculum: New tools
for educational reform. In Gifford, B. R. and O’Connor, M. C. (Eds.), Changing
assessments: alternative views of aptitude, achievement and instruction. 37-76.
Boston, Kluwer Academic Press.
Resnick, L. and Resnick, D. (1989). Tests as standards of achievement in schools. Paper
presented at the Invitational Conference of the Educational Testing Service, New
York. (ERIC Document Reproduction Service No. ED 335421).
Rose, M. (1989). Lives on the boundary: The struggles and achievements of America’s
underprepared. New York: The Free Press.
Schlechty, P. C. (1990). Schools for the 21s t century: Leadership imperatives for
educational reform. San Francisco: Jossey Bass.
Shafer, G. (October 1997). School reform and the high school proficiency test. English
Journal. 39-41.
Shaughnessy, M. (1977), Errors and expectations: A guide for the teacher of basic
writing. New York: Oxford University Press.
Shepard, L. (1990). Inflated test score gains: Is the problem old norms or teaching to the
test? Educational Measurement Issues and Practice. 9 (3), 15-22.
Shepard, L. (1989). Why we need better assessments. Educational Leadership. 46 (7),
4-9.
Shavelson, R. J., and Baxter, G. P. (May 1992). What we’ve learned about assessing
hands-on science. Educational Leadership. 49. 21-25.
Shavelson, R. J., Gao, X., and Baxter, G. R. (1993). Sampling variability of
performance assessments. CSE Technical Report, no 361. Santa Barbara, CA:
UCLA, National Center for Research in Evaluation, Standards, and Student
Testing.
Shavelson, R. J. and Webb, N. M. (1991). Generalizabilitv theory: A primer. Newbury
Park, CA: Sage Publishing.
Shavelson, R. J., Webb, N. M., and Rowley, G. (1989). Generalizability theory.
American Psychologist. 44. 922-932.
Shepard, L. A. (April 1989). Why we need better assessments. Educational Leadership.
46* 4-9.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
129
Siegel, A. I. (1986). Performance tests. In Berk, R. A.(Ed.). Performance assessment:
Methods and applications (pp. 121-142). Baltimore, MD: The Johns Hopkins
University Press.
Sizer, T. (1984). Horace’s compromise: The dilemma of the American high school.
Boston: Houghton Miflin.
Sizer, T. (1992). Horace’s school: Redesigning the American high school. Boston:
Houghton Mifflin.
Smith, M. L., Noble, A. J., Cabay, M., Heinecke, W., Junker, M. S., and Saffron, Y.
(July 1994). What happens when the test mandates changes? Results of a
multiple case study. CSE Technical Report, no. 380. Los Angeles: National
Center for Research On Evaluation, Standards, and Student Testing (CRESST),
University of California at Los Angeles.
Sperling, M. (1996, Spring). Revisiting the writing-speaking connection: Challenges for
research on writing and writing instruction. Review of Educational Research. 66.
(1), 53-86.
Stecher, B. (1995). The cost of performance assessment in science: The RAND
perspective. Paper presented at the National Council on Measurement in
Education, San Francisco, CA.
Stedman, L. C. & Kaestle, C. F. (1991). Literacy and reading performance in the United
States from 1880 to the present. In C. F. Kaestle, H. Damon-Moore, L. C.
Stedman, K. Tinsley, & W. V. Trollinger (Eds.), Literacy in the United States:
Readers and reading since 1880 (pp. 75-128). New Haven, CT: Yale University
Press.
Stiggins, R. J. (1994). Student-centered classroom assessment. New York: Merrill.
Taylor, K. and Walton, S. (1998). Children at the center: A workshop approach to
standardized test preparation. K-8. Portsmouth, NH: Heinemann.
Tyson-Bemstein, H. (1988). America’s textbook fiasco: A conspiracy of good
intentions. American Educator. 12, 20-27, 39.
U.S. General Accounting Office. (1993). Student testing: Current extent and
expenditures, with cost estimates for a national examination. Report
GAO/PEMD-93-8. Washington, D. C.: U.S. General Accounting Office.
Viadero, D. (July 13,1994). Teaching to the test. Education Week. 21-25.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
130
Walters, K. (1984). It’s like playing password, right?: Socratic questioning and
questioning at school. Texas Linguistics Forum. 24. 157-188.
White, J. O. (1985, November). Students learn by doing holistic scoring. English
Journal. 71. 50-57.
Wiggins, G. (November 1993). Assessment, authenticity, context and validity. Phi
Delta Kannan. 200-214.
Wiggins, G. (1992). Creating tests worth taking. Educational Leadership. 49. (8), 26-
34.
Wiggins, G. (April 1989). Teaching to the (authentic) task. Educational Leadership. 46.
(7), 41-47.
Wiggins, G. (1989a). A true test: Toward more authentic and equitable assessment. Phi
Delta Kanpan. 70 (9), 703-713.
Williams, P., Phillips, G. W., and Yen, W. M. (1991). Some measurement issues in
performance assessment. Paper presented at the annual meeting of the American
Educational Research Association, Chicago.
Witte, WS. P., Trachael, M., & Walters, K. (1986). Literacy and the direct assessment of
writing: A diachronic perspective. In K. L Greenberg, A. D. Wiener, & R. A.
Donovan (Eds.), Writing assessment: Issues and strategies (pp. 13-34). New
York: Longman.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Appendices
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Appendix A
Student Writing Survey
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
use
SCHOOL OF
EDUCATION
Student Writing Survey
Welcome 1
Thanks for being part o f our stu d y .
Instructions
Please read each question carefully. Be honest and thoughtful in selecting your responses. There are
no right or wrong answers. Your answers will be used for research purposes.
Mark the answer that best describes you. Remember, your answers will be kept confidential.
Use the following scale to bubble in the letter that most accurately describes your answer for each
question.
a = Always b = Mostly c = Sometimes d = Rarely e = Never
Ready? Let’s Go!
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
134
a = Always b = Mostly c = Sometimes d = Rarely e = Never
How often is each of the following sentences true for you?
1. Writing is important.
2. Writing helps me learn about myself.
3. Writing helps me study.
4. Writing helps me think more clearly.
5. Writing can help me get a good job.
6. Writing helps me share my ideas.
How often is each of the following sentences true for you?
7. I like to write.
8. I am a good writer.
9. People like what I write.
10. I don’t like to write things that will be graded.
11. If I didn’t have to write for school, I wouldn’t write anything.
When you write essays in class, how often do you:
12. Think about what you want to say before you start writing?
13. Jot down ideas for writing?
14. Outline major points before writing?
15. Write in different ways for different audiences?
16. Write in different ways for different purposes?
You ’ re doing great! Keep going,..
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
135
a = Always b = Mostly c = Sometimes d = Rarely e = Never
How often do you use each of the following to make your papers better?
17. Move some sentences or paragraphs to different parts of the paper
18. Add new ideas or information
19. Take out parts of the paper that you don’t like
20. Change some words for other words
21. Correct mistakes in spelling, grammar, and punctuation
22. Make changes as you write
23. Read what you have written aloud to yourself
24. Use a thesaurus
After you have written papers, how often does the teacher talk or write to you about each of the
following things?
25. How you followed directions
26. Whether you wrote enough in the paper
27. The ideas in your paper
28. The way you explained your ideas
29. The way you organized your paper
30. The words that you used
31. Your spelling, punctuation, and grammar
The end is in sight! Just a little more.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
136
a = Always b = Mostly c = Sometimes d = Rarely e = Never
How often does the teacher do each of the following things with your writing?
32. Mark mistakes in your paper
33. Write notes on your paper
34. Point out what you did well
35. Point out what you did not do well
36. Make suggestions for the next time you write
Use of a rubric
37. How often has a teacher used a rubric or scoring guide to grade your writing?
38. How often have you used a rubric or scoring guide to revise your writing?
39. How often have you used a rubric or scoring guide to evaluate your own
writing?
40. How often have you used a rubric or scoring guide to evaluate the writing of
others?
You9 re Finished!
Great Job !
Thank you very much for completing this survey.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Appendix B
Six-Point Rubric
Writing Performance Levels for Comparison Contrast Essay
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
138
Six-Point Rubric
Writing Performance Levels for Comparison Contrast Essay
LEVEL 6: Exceptional Achievement
Student Work:
• consistently pursues a strong central purpose across a complex range of ideas,
skillfully engages the reader, and shows exceptional insight into the subject
• Introduces or orients reader too subject and controlling idea (or narrative) in
an engaging and creative manner.
• Controlling idea is either stated or implied, but in either case, is clear to the
reader.
• Controlling idea and central purpose are implicitly or explicitly evident
throughout the essay, providing unity and focus.
• Essay flows well for reader and does not leave the reader adrift. Reader is
captivated, intrigued, and propelled forward throughout the essay. Reader is
eager to read further.
• Writer demonstrates and communicates a clear and insightful understanding of
the matters being compared and/or contrasted, and strongly supports any
judgments or conclusions being made on the basis of this understanding.
• The comparison and/or contrast engenders or develops an insightful or
unusually perceptive realization, and interesting “so what” or other
implication.
• includes main ideas that are developed comprehensively and supported with a
variety of logical reasons and detailed examples
• Quality and/or quantity of examples clearly reveal the distinctions and
similarities being developed.
• Examples are concrete, specific, and clearly suited to the point being made.
• Observation, narrative, or personal experience examples use vivid, concrete
details, sensory language, and appropriate narrative techniques.
• Each point of comparison and/or contrast is convincingly developed with
detailed, specific examples.
• Very specific, descriptive, ad relevant Information that strongly supports
major claims.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
139
• is skillfully organized
• Ideas flow naturally. Essay is coherent and unified.
• Organization enables writer to powerfully clarify similarities and/or
differences.
• Organization helps reader stay engaged and prevents reader from getting
bogged down or confused.
• Organization is appropriate to the complexity or depth of ideas being
developed and may include such techniques as parallel sequenced paragraphs.
• Transitions are clear and engaging. Writer demonstrates competency in
transitional expressions.
• Effectively develops a strong opening claim or focus and builds to key
insights, emphases, claims, or realizations.
• Concludes in a satisfying way. Conclusion flows naturally from and is well-
grounded in the previous text. Offers a “so what” or other type of implication
or even epiphany.
• shows distinctive style through skillful and expressive use of vocabulary,
phrasing, and sentence structure
• Style and tone appropriate to purpose and audience.
• Confident, authoritative, lively, interesting, compelling, convincing,
thoughtful, and distinctive.
• Word choice, phrasing, and sentence structure appropriate to purpose,
audience, style, and tone.
• Coherence is maintained through skillful control of sentence structure and
word choice.
• demonstrates essentially error-free control of grammar, punctuation,
capitalization, and spelling
• Virtual mastery of the conventions of usage, mechanics, and spelling.
• Minor errors do not divert the reader’s attention or cause4 confusion about
meaning.
• Occasional lapse in conventions due to complexities in ideas or style.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
140
Six-Point Rubric
Writing Performance Levels for Comparison Contrast Essay
LEVEL 5: Commendable Achievement
Student Work:
• consistently pursues a central purpose, holds the interest of the reader, and
shows insight into the subject
• Introduces or orients reader to subject and controlling idea in an engaging and
creative manner.
• Controlling idea is either stated or implied, but in either case, is clear to the
reader.
• Comparisons and/or contrasts are generally engaging and compelling.
• Essay is focused and consistent, but may briefly lack the same coherence,
unity, or focus of a 6 essay.
• Delineation of similarities and/or differences is thorough, descriptive, and
convincing.
• May lack insight of a 6 paper, but claims are perceptive and convincingly
supported.
• includes main ideas that are developed and supported by a variety of reasons
and examples
• Details and evidence are relevant and convincing, but not quite as thorough,
specific, or revealing as in a 6 essay.
• A specific point of comparison or contrast may not be as well-developed
supported or elaborated.
• is effectively organized
• Organization may not be as tailored to the complexity and depth of the ideas
being developed, but keeps reader focused on the controlling idea and the
examples being offered.
• Organization does not lead reader astray.
• Paper is cohesive and well-connected with clear direction and purpose, but
lacks the unfaltering smoothness of a 6 essay,
• May not show the same coherence, unity, or control of a 6 essay.
• Organization may not propel reader to key insights or claims as powerfully as
a 6 essay.
• Conclusion still flows naturally from previous text, but may not be as
satisfying as in a 6 paper. It is likely to be perceptive, but lack the “aha” or
punch of a 6.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
141
• shows emerging style through effective use of vocabulary, phrasing, and sentence
structure
• Confident, authoritative, thoughtful, convincing, and interesting, but is less
compelling, lively, and distinctive than a 6 essay.
• Shows writer’s ability to use words, phrasing, and sentence structure
appropriately.
• Word choice shows less versatility than a 6, however, diction is strong and
appropriate.
• Exhibits sustained control of a variety of sentence structures.
• consistently demonstrated the accurate use of grammar, punctuation,
capitalization, and spelling.
• Minimal range of errors with some errors being repeated.
• Shows strong and clear control of conventions of usage, mechanics, and
spelling.
• Errors do not divert the reader’s attention or cause confusion about meaning.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
142
Six-Point Rubric
Writing Performance Levels for Comparison Contrast Essay
LEVEL 4: Adequate Achievement
Student Work:
• shows a consistent purpose, communicates to the reader, and connects the
writer’s knowledge to the subject
• Adequately orients reader to the specific subject matter and controlling idea.
• Introductory presentation of controlling idea definitely moves beyond simple
restatement or rephrasing of the subject.
• Essay makes clear the parameters along which the comparison and contrast
will be conducted.
• Controlling idea provides clear and consistent focus and flow to essay.
• At a minimum, presentation is competent, informative, and complete, though
it may be pedestrian, mundane, and uninspired.
• Writer clearly delineates similarities and/or differences.
• Though less thorough than a 5 paper, the essay does not lapse into
superficialities.
• Claims are well-supported, but the essay fails to demonstrate significant
insight or particularly perceptive realizations.
• includes main ideas that are organized, developed, and supported by reasons
and examples
• Clearly organized though perhaps not as effectively as possible.
• Writer has obviously given thought to order of presentation.
• Organizational scheme and presentation of evidence do not lead reader astray.
• Presentation flows due to implicit or explicit logic; reader does not have to
labor to discern this logic.
• No lapses in general unity, coherence, or control. If a lapse occurs, it is brief
and inconsequential, otherwise the paper is no longer operating at level 4.
• Most specific points of comparison or contrast will be well-developed,
supported, and/or elaborated.
• Details and evidence are relevant and specific, but may be less thorough or
convincing than a 5 essay.
• Ending achieves closure. Conclusion is clearly based on previous text, but
moves little beyond a summary.
• Informative
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
143
• shows suitable vocabulary, phrasing, and sentence structure.
• Word choice and sentence structure are appropriate to points being made.
• Adequate variation in sentence structure.
• generally demonstrates the accurate use of grammar, punctuation,
capitalization, and spelling
• Writing shows adequate control of the conventions of usage, mechanics, and
spelling.
• Several kinds of errors may be repeated throughout the writing.
• Errors may be distracting, but will not cause significant confusion about
meaning.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
144
Six-Point Rubric
Writing Performance Levels for Comparison Contrast Essay
LEVEL 3: Some Evidence of Achievement
Student Work:
• shows a purpose, but may be inconsistent in communicating to the reader or
connecting the w riter’s knowledge to the subject
• Controlling idea may only be a simple restatement of the subject, or r the
writer may establish a thesis, but the thesis falls outside or otherwise neglects
the primary scope of the prompt.
• Simplistic introduction may not engage reader and does not adequately orient
reader to a controlling idea or narrative.
• The parameters of comparison may not be clear.
• Stays on subject or related topic, though focus may be lacking.
• Controlling idea may fail to provide clear and consistent focus and flow to
essay.
• Presentation seems incomplete or not very informative.
• Presentation may lack clarity.
• Writer attempts to delineate similarities and/or differences, but does not
consistently support claims in a more than superficial fashion.
• Less informative than a 4 essay.
• includes main ideas that may be organized or partially developed and supported
by some reasons and examples
• Organizational scheme is lacking, unclear, or simplistic.
• Not obvious that reader has given adequate attention to the structure and order
of presentation.
• May lead reader astray. May wander, meander, or otherwise feel bumpy or
erratic.
• Lapses in coherence, unity, or control. Reader may have to do the work to fill
in or provide order or flow to the essay.
• Writer attempts to develop and support ideas, but elaboration is neither
thorough nor convincing.
• Support tends to lack specificity or relevance.
• Reader may not be clear on why the writer has included particular examples or
details.
• Moves beyond being a simple list, but examples are typically general or brief.
• Not well substantiated
• Closure is absent or superficial. Essay may feel truncated. Conclusion often
fails to move much beyond a summary.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
145
• may show imprecise use of vocabulary, phrasing, and sentence structure
• Basic control of simple sentences with little or no variety.
• Word choice is usually appropriate to content, however, language may be
more general than specific.
•
• includes some errors in the use of grammar, punctuation, capitalization, and
spelling
• Shows some control in the conventions of usage, mechanics, and spelling.
• Exhibits frequent errors which are distracting to the reader and may cause
some confusion about meaning.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
146
Six-Point Rubric
Writing Performance Levels for Comparison Contrast Essay
LEVEL 2: Limited Evidence o f Achievement
Student Work:
• attempts to connect writer’s knowledge to the subject; may not show a purpose
• Controlling idea may be only a simple statement of subject.
• May not stay on topic.
• Writer notes similarities and/or differences, but provides little or no support.
• Generally uninformative.
® is typically brief, unorganized, and underdeveloped
• Brief, sketchy.
• Thin development, if any at all.
• Little, if any, specific details.
• Organization is simplistic or missing.
• Underdeveloped, not specific, or unclear.
• includes consistent and serious errors in the use of grammar, capitalization,
punctuation, and spelling
• Writing shows little control of the conventions of usage, mechanics, and
spelling.
• Writing demonstrates serious and numerous problems which distract the
reader and lead to confusion.
• Writing may be incoherent in places, but is otherwise understandable.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
147
Six-Point Rubric
Writing Performance Levels for Comparison Contrast Essay
LEVEL 1; Minimal Evidence of Achievement
Student Work:
• is too brief or disorganized to communicate to the reader
• Responds to prompt, though may not stay on topic.
• Badly organized, or too little to organize.
• may not connect the writer’s knowledge to the subject
• Few or no supporting details or relevant examples.
• typically includes many errors in the use of grammar, punctuation,
capitalization, and spelling
• Writing shows minimal control of the conventions of usage, mechanics, and
spelling.
• Writing may be incoherent.
• Will have serious errors in almost every sentence.
No Score (NS1
Student Work:
• is illegible or unintelligible
• uses inappropriate language for school setting
• is written in a language in which scorer is not fluent
• is blank or nearly so
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
148
Appendix C
Student Writing Samples from High, Medium, and Low Score Levels
Based upon Student-Developed Rubric (Table 3.1)
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
149
Student Samples from High, Medium, and Low Score Levels
Based upon Student-Developed Rubric (Table 3.1)
Writing Assignment: Romeo and Juliet
Among the characters you have read about in Romeo and Juliet, you should have a good
understanding of the characters of Benvolio, Tybalt, Romeo, and Friar Lawrence. These
characters act as foils to one another within this play. Combine your knowledge of the
play with your experience writing comparison/contrast paragraphs to respond to the
directions below.
For each question:
• Write a two-paragraph response that compares/contrasts the two characters.
You may use either of the two formats:
A. Paragraph #1 = About character #1
Paragraph #2 = About character #2
B. Paragraph #1 — Similarities between the 2 characters
Paragraph #2 = Differences between the 2 characters
• Use appropriate transitions between the two paragraphs.
• Support your descriptions of the characters with examples and quotes from the
play.
1. Compare/contrast the characters of Benvolio and Tybalt.
2. Compare/contrast yourself with either Romeo or Juliet. (Use your own personal
examples to compare/contrast with Romeo or Juliet’s actions and lines in the play.)
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
150
NOTE: This writing sample has been transcribed exactly as written by the student.
Sample Student Writing: High Score Level
Benvolio is the peacekeeper of the play. Lord Montague trusts Benvolio to tell
him what he does not know about his son Romeo. Benvolio does not like Tybalt at all.
Benvolio does not get into any fights in the play, he just tries to stop them from
happening. Benvolio is Romeo’s best friend. He is trusted by Romeo not to tell any of
his secrets to anyone.
Tybalt, on the other hand, is the trouble maker in the play. Tybalt joined the fight
in the beginning of the play when it was just getting started. Also, he started the fight in
which he and Mercutio lost their lives. He likes starting fights with members of the
Montague family, especially Romeo and Benvolio. Tybalt is Juliet’s cousin. He has a
short temper compared to the other characters in the play.
Romeo is one of the two main characters in the play. He is the son of Lord
Montague, cousin of Benvolio and friend of Mercutio. He falls in love with Juliet
Capulet, daughter of the family enemy. He makes some very life threatening decisions
throughout the course of the play. One such choice is when he killed Tybalt. He should
have not done that even though Tybalt had slain Mercutio. Another bad choice is when
Romeo bought the poison from the apothecary. The worst mistake Romeo made was to
drink the poison. If he had not drunk the poison he would be alive and happy.
Friar Lawrence is wise and old compared to Romeo. He knows what he is doing
and thinks about what might occur before it happens. Friar goes very slowly and finished
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
151
all the work that needs to be done. He has been around for a lot longer than Romeo, so he
has accumulated more knowledge and has more experience when it comes to decision
making. Friar Lawrence believes that if you do something too quickly, then you will
have problems. However, if you go slow, you will eventually finish, have done a good
job, and have had very few problems. Also, Friar is one of the few characters, besides
Romeo and Juliet, who knows everything that is going on.
Romeo is like me in many ways. When he gets excited, he tends to go too fast
and get carried away with his actions. This is like me because I sometimes act like this.
Oftentimes, I try to find the easiest and quickest way to solve a problem before I have
time to figure out all the solutions. This usually leads to trouble for me that I can not
avoid. These are some of the ways Romeo and I are the same.
Most of the time, I do not need help from older, wiser people to solve my
problems. Romeo needed help from Mercutio, Friar, Benvolio, and Juliet to help him
solve his problems. Most of the time I tell my parents what is going on in my life.
Romeo seems to never tell anyone what is going on in his life except for the people he
trusts most. Our families are very different from each other. My family is pretty well off
and we have an adequate amount of money. Also, we do not have a sworn family enemy.
The Montague family is very wealth and in the higher class of society and they do have a
family enemy. These are the things that make Romeo and me different from each other.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
152
NOTE: This writing sample has been transcribed exactly as written by the student.
Sample Student Writing: Medium Score Level
Within the story Romeo and Juliet, there are many characters that act as foils to
one another. A dramatic foil is a character who brings out the characteristics of another
character in a play. Two such foils are Benvolio vs. Tybalt, and the Friar vs. Romeo.
Without one of the characters, the other character’s role wouldn’t have such a significant
meaning and wouldn’t be seen as powerful. The Friar brought out the youth and
inexperience of Romeo, Romeo brought the elderliness and wisdom in the Friar.
Whereas Benvolio brought out the fire and hostility of Tybalt. Tybalt brought out the
calmness and the leadership capabilities in Benvolio.
Benvolio vs. Tybalt
Benvolio is a very close friend of Romeo’s as well as being Lord Montague’s
nephew. He is a good hearted man. His name comes from the Latin word “Ben” which
means good. He is very truthful and is trusted in the city of Verona. Whenever there is
an incident that the Prince arrived on, he would always ask Benvolio for the truth about
what had happened. Within the family feud between the Monague’s and the Capulet’s,
he attempted to play the peacemaker, but the rage and carnage that had built up over the
years of battle between the families was far too much for Benvolio to contain. In one of
the fights, Benvolio said, “Part; fools! Put up your swords. Your know not what you
do.” Although he was unsuccessful in trying to break up the fight, he was persistent in
trying to bring peace to the two families.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
153
On the other hand, Tybalt is a fiery, youthful man. (I don’t know if it has
anything to do with his name but it is similar to the word tyrant which is bad person.)
Tybalt is the nephew of Lord Capulet. He is always looking for a Montague to fight.
When Benvolio attempted to break up the fight (above), Tybalt said, “What drawn, and
talk of peace? I hate the word as I hate hell, all Montagues , and thee. Have thee
coward.” Tybalt has no tolerance for anyone on the opposite side of the battle field.
Either you are behind him or he’ll fight you until you or he dies. Although he is not a
very benevolent person, he is very tenacious and will not give up.
Romeo vs. Friar
Romeo is a young man who is the son of Lord and lady Capulet. He obviously is
a true believer I love at first sight because he had never met Rosaline and he fell in love
with her, and he had never met Juliet when he had fallen in love again. He is respected
by the entire town of Verona. Even Lord Capulet called him, “...a virtuous and well-
governed youth”, about Romeo. Romeo is an extremely brave man and proved it when
he fell in love and married a Capulet. Not only did he fall in love with one but he would
visit the house fully aware that if a guard saw him, he would be killed. Romeo is also
very impatient. When he and Juliet decided to get married he wanted to get the marriage
said and done with as quickly as possibly. He said, “O, let us hence! I stand on sudden
haste.”
Friar, however is just the opposite. He is an elderly man and is full of knowledge.
He does not believe in love at first sight. He said “Young men’s love then lies not truly
in their hearts, but in their eyes.” This means that he feels that young men only love what
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
154
they see and don’t love the person. He believes in taking things slow and letting things
come to you. When Romeo was so rushed to get married Friar said “Wisely and slow.
They stumble that run fast.” Friar has a way of always looking at the potential positive
outcomes in any situation. He agreed to wed Romeo and Juliet becuaeS “...For this
alliance may so happily prove to turn your households’ rancor to pure love.”
Myself vs. Romeo
I would compare my self to Romeo more than Juliet. Romeo seems very cautious
that something could happen at anytime. When he went to the Capulet party, he tells
Mercutio and Benvolio that he had a dream that someone would die tonight as a result of
them going to the Capulet party. I am the same way. I always think of the things that
could happen in the future as a result of my actions regardless of the situation. I thought
of him to be more of a social person. He seemed to be more likely and awake when he
was around his friends. Laughing more and telling jokes. But when he was around
Juliet, he was very deep, thoughtful and also seemed cocky and very full of himself. I am
the same way around groups of people. I am more alive when I am around groups of
people, laughing and talking more than I usually go.
However I don’t change the way that I am with my peers, boy or girl. They are all
young and love to laugh and have a good time. I don’t necessarily think of myself as
being in the “limelight” more as a side act that people laugh at. Romeo’s beliefs are so
strong that he is willing to die for them. He loved Juliet so much that when he saw her
“dead”, he committed suicide so that their souls would be together in heaven. I would
never put something that I believe in before my life. I believe that there is no belief,
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
155
feeling or moral standard that is worth dying for in my eyes. I guess that I have never had
the feeling that I would do anything in defense because I felt so strongly about my
feelings and emotions.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
156
NOTE: This writing sample has been transcribed exactly as written by the student.
Sample Student Writing: Low Score Level
1) The character Benvolio was Romeo’s friend, and a devoted friend to the Montegues.
Tybalt was loyal to Juliet and the Capulet family, he was Juliet’s cousin. Benveolio
and Tybalt had common inter4est as they were both loyal to the members of the
familys. The Montegues and Capulets were enemys. Romeo kills Tybalt in self
defense. These two characters were much alike as they just wanted to protect and
watch after the people in there familys that they were loyal too and die for them if
need be and Tybalt did die trying to protect Juliet from Romeo.
2) Romeo belived in love at first sight and that is how he fell in love with Juliet, by just
seeing her once and in away Friar Lawrence must of believe the same because he
married Romeo and Juliet knowing that their families hated each other and would
never approve of there relationship. When Juliets Family decided that she should be
marry to Paris, Friar gave Juliet the potion and did die. Friar was there but arrived to
late to tell Romeo the truth about Juliet. Friar had warned Romeo about the violience
that could come of his relationship with Juliet but really had no idea how it would be.
Both characters were strong and belived in what they believe in with out any doubts.
3) I guess because I am a guy I should compare my sle3f with Romeo, but this is hard
because I have never been in love with a girl or would I know how this feels, to be in
love.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
157
But maybe we are a little alike because if someone hurt my friend I would fight for
him too, If I did fall in love with someone who was a enemy to my family I do not
think I would betray my family and married them. I think maybe there is such a thing
as forbidden love and in this case if Romeno and Juliet had forgotten about each other
all these people would not of died.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Appendix D
Initial Interview Questions for Teachers
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
159
Initial Interview Questions for Teachers
1. What strategies do you use to prepare students for classroom writing assignments?
2. What strategies do you use to prepare students for the District Writing Assessment?
3. How long have you taught?
4. How long have you taught in this district?
5. Have you attended a Writing Project?
5a. Have you instructed for the Writing Project?
5b. How many times have you participated in the Writing Project?
6. What types of feedback do you give students on their writing, e.g., revision ideas,
errors in spelling/grammar/punctuation, editing changes, etc.?
7. When evaluating student writing, do you use a rubric?
8. Have your students ever participated in developing a rubric to evaluate writing?
9. Is there anything else you would like to share about your teaching of writing?
10. Would you share some lesson plans, teaching materials, and rubrics you use to teach
writing?
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Appendix E
Follow-up Interview Questions for Students
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
161
Follow-up Interview Questions for Students
1. In this assignment, you developed your own rubric. Had you ever heard the word,
“rubric” before? Do you see any relationship to the rubric you developed and the
“Evaluation Guide” your teacher uses? If so, what are the differences?
2. Do you consider all criteria on a rubric or evaluation guide of equal weight? For
example, does “format” have the same value as organization or spelling?
3. What criteria on a rubric do you consider most important?
4. How did you use the student-developed rubric to evaluate your own work?
5. Each year you take district writing assessments. Have the rubrics used to score your
essays on those assessments been used in your class as criteria to evaluate your
written work?
6. On the Student Writing Survey, a number of you stated that you did not use a
thesaurus in writing or revising your written work. Why?
7. Do you consider email a form of writing? How is email different from writing
assignments you do at school?
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Appendix F
Follow-up Interview Questions for Teachers
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
163
Follow-up Interview Questions for Teachers
1. How has the use of student-developed rubrics changed the way you get students to
think how they evaluate written work?
2. In what ways has the use of student-developed rubrics changed the instructional
strategies you use to evaluate student essays?
3. What effect(s) did asking students to develop rubrics have on your lesson
planning/scheduling? Do you feel the time allocated to having students develop their
own rubrics resulted in better writing or a better understanding quality writing?
4. When using their own rubrics, what strategies did students use to revise written work?
5. What areas in students’ written work improved as a result of the use of student-
developed rubrics?
6. Were there any unintended negative consequences of using student developed rubrics
in your classroom?
7. In what ways has your participation in this study increased your interest in pursuing
additional professional development opportunities in the area of evaluating writing or
the use of rubrics?
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

How has the requirement to implement content standards affected the instructional program in schools and classrooms?

PDF

Factors influencing teachers' and administrators' identification of diverse students for gifted programs in Title I schools

PDF

An analysis of Long Beach Unified School District's EXCEL model of providing gifted programs for urban students and their effect on student achievement

PDF

Implementation of literature circles in a rural high school English class: One teacher's journey of changing student attitudes toward reading

PDF

A history of the development of the California Science Content Standards: 1990--2005

PDF

Comparing social studies planning processes of experienced teachers and student teachers

PDF

A case study of a successful urban school: Climate, culture and leadership factors that impact student achievement

PDF

Differential characteristics of beginning teachers

PDF

An analysis of teachers' perceptions of the efficacy of an instructional method for English language development using music and singing paired with movement

PDF

An analysis of the use of data to increase student achievement in public schools

PDF

A grounded model of writing: A qualitative investigation into complex systems of creativity and cognitive constructs associated with the writing of poetry

PDF

Higher education's responses to economic development: Vietnam

PDF

Correlates of job satisfaction among California school principals

PDF

Correlates of job satisfaction among high school principals

PDF

Examining the factors that predict the academic success of minority students in the remedial mathematics pipeline in an urban community college

PDF

Districtwide instructional improvement: A case study of an elementary school in the Beach Promenade Unified School District

PDF

Connecting districts and schools to improve teaching and learning: A case study of district efforts in the Los Coyotes High School District

PDF

Effects of interactive multimedia for the prevention of obesity on self-efficacy, beliefs about physical activity, and social influence

PDF

Cognitive coaching training for master teachers and its effects on student teachers' ability to reflect on practice

PDF

A study of the relationship between student achievement and mathematics program congruence in select secondary schools of the Archdiocese of Los Angeles

Asset Metadata

Creator Green, Kathy Ann (author)

Core Title Correlation of factors related to writing behaviors and student -developed rubrics on writing performance and pedagogy in ninth-grade students

School Rossier School of Education

Degree Doctor of Education

Degree Program Education

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag education, curriculum and instruction,language, rhetoric and composition,OAI-PMH Harvest

Language English

Contributor Digitized by ProQuest (provenance)

Advisor Lemlech, Johanna (committee chair), Baker, Robert (committee member), Kaplan, Sandra (committee member)

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c16-98422

Unique identifier UC11338223

Identifier 3027723.pdf (filename),usctheses-c16-98422 (legacy record id)

Legacy Identifier 3027723.pdf

Dmrecord 98422

Document Type Dissertation

Rights Green, Kathy Ann

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the au...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus, Los Angeles, California 90089, USA