University of Southern California Dissertations and Theses

A Comparison Of Selected Writing Criteria Used To Evaluate Nonnative Speakers Of English At A California State University

(USC Thesis Other)

A Comparison Of Selected Writing Criteria Used To Evaluate Nonnative Speakers Of English At A California State University

PDF

Download

Open document

Flip pages

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content INFORMATION TO USERS This manuscripthas been reproduced from the microfilm master. UMI films the text directly from the original or copy submitted. Thus, some thesis and dissertation copies are in typewriter face, while others may be from any type of computer printer. The quality of this reproduction Is dependent upon the quality of the copy submitted. Broken or indistinct print, colored or poor quality illustrations and photographs, print bleed through, substandard margin*, and improper alignment can adversely affect reproduction. In the unlikely, event that the author did not send UMI a complete manuscript and there are missing pages, these will be noted. Also, if unauthorized copyright material had to be removed, a note win indicate the deletion. Oversize materials (e.g^ maps, drawings, charts) are reproduced by sectioning the original, beginning at the upper left-hand comer and continuing from left to right in equal sections with small overlaps. Each original is also photographed in one exposure and is included in reduced form at the bade of the book. Photographs included in the original manuscript have been reproduced xerographically In this copy. Higher quality 6" x 9” black and white photographic prints are available for any photographs or illustrations appearing in this copy for an additional charge. Contact UMI directly to order. A Bell & Howell Information Company 300 North Zeeb Road. Ann Arbor. M l 43106*1346 USA 313.'761*4700 800.'521-0600 A COMPARISON OF SELECTED WRITING CRITERIA USED TO EVALUATE NON-NATIVE SPEAKERS OF ENGLISH AT A CALIFORNIA STATE UNIVERSITY by Karen Ann Russikoff A Dissertation Presented to the FACULTY OF THE GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (Education) December 1994 0HI Humber: 9601051 Copyright 1996 by Russlkoff, Karen Ann All rights reserved. OHI Microform 9601051 Copyright 1995* by UMI Company. All rights reserved. This microform edition is protected against unauthorized copying under Title 17t United States Code. UMI 300 North Xeeb Road Ann Arbor, MI 48103 UNIVERSITY OF SOUTHERN CALIFORNIA THE GRADUATE SCHOOL UNIVERSITY PARK LOS ANGELES, CALIFORNIA 9000? This dissertation, written by Karen A. R u ssik o ff under the direction of h.$£...... Dissertation Committee, and approved by all its members, has been presented to and accepted by The Graduate School, in partial fulfillment of re quirements for the degree of DOCTOR OF PHILOSOPHY C . . / K __ — — ' / Dean of Graduate Studies Date ..... DISSERTATION COMMITTEE ( . \ I w U — ' 1 c . Chairperson Dedication This research is dedicated to my husband, Lanny, who h a s offered unconditional love and support throughout all my endeavors, including this research ; to my children, Jen n ifer and M atthew , w ho have provided so m uch joy to my life; and to my m other, Sally Colpitts, who h a s serv ed a s a m odel of support an d strength/ A cknowledgm ents T his re s e a rc h w a s d e p e n d e n t upon th e skills, know ledge a n d generosity of m any people, to whom I am grateful: to David E. Eskey, Ph.D., for constant support, feedback and encouragem ent; to Robert B. Kaplan, Ph.D ., for his considerable guidance, g en ero u s resp o n se, an d ability to keep m e focused; to Bert Rivas, Ph.D. and Sidney Ribeau, Ph.D. for cam p u s support; to the readers, panelists, and pilot respondents: Rita H am ada-K ahn, M.A. D iane Luehrs, M.A. Jo s e p h Farrell, Ph.D. S u san R eed-Jones, Ph.D. Larry K. Robinson, M.A. Linda Olson, M.A. Phyllis W healon, Ph.D. D uane Sharp, Ph.D. S haron Hilles, Ph.D. Judith R ose, M.A. Donald K raem er, Ph.D. Frank T orres, Ph.D. Ja n ice Pilgreen, Ph.D. Frank Ja n g er, M.E., P.E. Donald A m broson, Ph.D. Sheila McCoy, Ph.D. iv Table o fC o n ten ts __________________________________________ Eage I. Preliminary Pages Dedication ii Acknowledgments iii List of Tables and Figures vi List of Appendices vii Abstract viii II. Chapter I. The Problem 1 Introduction 1 Background 1 The GWAR on One Cam pus 2 Statem ent of the Problem 12 Significance of the Study 15 Purpose of the Study 15 Research Questions 16 Delimitations of the Study 16 Limitations of the Study 17 Definitions of the Terms 18 Organization of the Remainder of the Study 19 III. Chapter It. A Review of the Literature 21 Organization of the Chapter 21 The California State University Writing A ssessm ent Mandate 21 Writing A ssessm ent Scoring 28 Summary 63 V ___________________________________________Eage IV. Chapter III. Methodology 65 Organization of the Chapter 65 P hase I: The Questionnaire 65 P hase II: The Analytic Rubric 76 P hase III: Re-Scoring of the GWT E ssays 79 V. Chapter IV. Results 89 Organization of the Chapter 89 P hase I: The Questionnaire 90 P hase It: The Analytic Rubric 121 P hase III: Re-Scoring of the GWT E ssays 137 VI. Chapter V. Summary, Conclusions, Implications, and Recommendations 145 Organization of the Chapter 145 Summary 145 Conclusions 154 Implications for Teaching And Testing 162 Suggestions for Change 163 Recom mendations for Future R esearch 165 VII. References VIII. Appendices 166 180 vi List of Tables and Figures ___________________________________________Eaae Table 1. Sam ple Distribution by College & School 92 Table 2. Faculty Respondent Ranking 93 Table 3. Length of Teaching Experience 94 Table 4. GWT Scoring Participation 95 Table 5. Faculty-Perceived C au ses of ESL Problems 100 Table 6. Faculty-Perceived Language Problems . 101 Table 7. Percentage of Faculty-Assigned Writing 105 Table 8. Course Writing M odes 109 Table 9. Standards versus Grading 111 Table 10. Ten Most-Used Course Writing Criteria 115 Table 11. Attributed Importance to Writing Criteria 117 Table 12. Survey Criteria for Analytic Factors 123 Table 13. Correlation Matrix for Scoring 129 Table 14. Analytic Criteria Stepwise Regression Results 133 Table 15. Descriptive Statistics of Analytic Scoring 134 Table 16. GWT Holistic/Analytic Stepwise Regressions Results 136 Table 17. Original GWT Scored Essay Distribution by Batch 137 Table 18. Analytic Scoring Pass/Fail (P/F) Ratios 139 Figure 4.1 ESL Analytic Scores R egressed on Original GWT Holistic Scores 130 Figure 4.2 GWT Analytic Scores R egressed on Original GWT Holistic Scores 131 List of Appendices ________________________________________________Eage A. Faculty Questionnaire 181 B. Faculty Questionnaire R esponses 186 C < Faculty Questionnaire Open-Ended R esponses 193 D. Pilot Questionnaire and R esponses 206 E. Graduation Writing Test Rubric 211 F. Cover Letter for Faculty Questionnaire 213 G. Voice Mail Follow-Up Text 215 H. Questionnaire Distribution to Departments, Colleges, 217 School I. Department Return 219 J. Faculty Questionnaire Native Language R esponses 222 K. Faculty Questionnaire Contingency Tables 224 L. Writing Criteria Factor Totals 240 M. GWT Essay Request 242 N. ESL Panel Essay Selection 244 O. Detailed Analytic Rubric 250 P. Brief Analytic Rubric 252 Q. Cover Letter/Directions for Holistic Scoring 254 R. Cover Letters/Directions for Analytic Scoring 256 S. Reader Scoring Configurations and Reliabilities 259 T. Batch Scores 262 U. Table: Faculty Tenure to Assigned Writing 267 V. California State University GWAR Options by Cam pus 269 viii Abstract The study investigated the efficacy of the Graduation Writing Test (GWT), the instrum ent used to implement the California S tate University Graduation Writing A ssessm ent Requirement (CSU-GWAR) at California State Polytechnic University, Pom ona (Cal Poly Pomona), a s a method of assessin g the writing com petency of non-native speakers of English (NNS) enrolled in the University's undergraduate and graduate programs. The major focus w as the investigation of the relationship between two sets of criteria used for evaluation of NNS student writing at the University: 1) criteria used for regular academ ic coursework, gathered through a faculty questionnaire (N»392) (i.e., analytic criteria) and 2) criteria used for the GWT (i.e., holistic criteria). Five factors, com posed of faculty responses, comprised the analytic rubric: Content, weighted at 25%, Organization, Language Use, and Mechanics, each at 20%, and Vocabulary at 15%. One hundred previously-scored GWT essay s were re-scored using the analytic rubric by two sets of raters, ESL (English-as-second-language) specialists and GWT regular raters. Qualitative and quantitative analyses confirmed similarities and differences in the criterion application: Although holistic and analytic scoring criteria were similar in that they appeared to a s se ss the sam e underlying writing abilities, analytic criteria resulted in significantly increased numbers of passing scores for NNS student essays by both rater sets. The single criterion responsible for predicting holistic scores w as Language Use, confirming a difference between the academic course criteria which indicated Content a s most important. These results ix reveal a lack of content and educational validity for the GWT and suggest that holistic assessm ent which em phasizes Language Use (i.e., grammar control) may bias the scoring against NNS students. A 1 CHAPTER I THE PROBLEM Introduction C hapter I explains the u se of holistic a sse ssm e n t of student e ssa y s written by non-native sp e ak e rs of English (NNS), a s required for m eeting th e California S ta te University (CSU) G raduation Writing A ssessm en t R equirem ent (GWAR). A g en eral background of the sta te -m a n d ate d testing program and a detailed exam ple of com pliance at a single cam pus, California S ta te Polytechnic University, P om ona, a re p re se n te d . In addition, the specific outline of the research project is included in th e form of the problem statem en t, with additional su b sectio n s delineating the significance and p u rp o se of the study, an d th e re se a rc h q u estio n s g en erated by the literature and the problem , to which this study provides answ ers. A brief glossary is included to clarify term s used in this study. Background In May 1976, th e California S ta te University Board of T ru stees responded in the form of a system -w ide m andate to dem an d s originating In th e state legislature for improved writing by stu d en ts educated in state* funded universities. T he result w as th e W riting Skills Im provem ent Program , which included two types of writing com petency exam inations: (1) th e E nglish P la ce m e n t T e st (E PT ), req u ired of all fresh m an un d erg rad u ate stu d e n ts upon adm ission in order to determ ine English proficiency levels and su b seq u en t coursew ork; and (2) the G raduation Writing A ssessm ent Requirem ent (GWAR), required of all upper-division and graduate students to confirm English writing com petence a s an exit requirem ent for all university d eg rees ( Executive Order 514,1987). At the time, all nineteen cam p u ses of the California S tate University system agreed to u se the sam e EPT exam ination, with the stipulation that cut sc o res would b e se t by the individual cam puses; however, no single test form at for the GWAR w as a ccep tab le to all. C onsequently, th e T rustees allowed each cam pus to devise its own version of a com petency exam ination to confirm college-level academ ic writing proficiency (Moye, 1983), implying th e need for each cam p u s to reflect its curriculum in the GWAR testing. Currently, eighteen of the now-twenty cam p u se s require an e ssa y exam ination which is hollstically scored, while six c am p u se s additionally include a m ach in e-sco red objective te st; ele v en allow fulfillment of the requirem ent by a course option or the choice of a course or e ssa y exam ination (Borowiec, 1988). (S e e Appendix V for cam p u s options.) T he GWAR on O ne C am pus O n one of the California S tate University cam p u se s, at California S tate Polytechnic University, P om ona (Cal Poly P om ona), th e GWAR m a n d a te is know n a s th e G rad u atio n W riting T est (GW T). All undergraduates, following completion of 135 units, and graduate students, following completion of 8 units, m ust "dem onstrate com petency in writing skills a s a requirem ent for g rad u atio n ...assessed by m ea n s of a written test "(Cat Poly Pomona Catalog, 1994; p. 99). In 1980 at Cat Poty Pom ona, the initial administration of the GWT included both a faculty-scored e ssa y and a m achine-scored gram m ar com ponent. However, after several years, the Testing Office found no positive correlation betw een th e proficiencies d em o n strated on the objective gram m ar test and those on the essay exam . Although students who w ere non-native sp eak ers of English (NNS) w ere, in fact, generally able to p a ss the gram m ar test, they w ere not able to p a ss the essay, while English native-speaking (NS) students w ere m ost often able to p a ss the essay exam but not the gram m ar portion (G aschler, 1992). Recognizing this fact, test evaluators eliminated the gram m ar com ponent, leaving only one 75-minute essay exam ination, which is holisticaliy scored by cam pus faculty from diverse disciplines. All student essa y s, w hether written by native or non-native sp eak ers of English, are scored at the sam e time by the sam e group of readers, using the sam e scoring guidelines. At present, the NNS student failure rate for this test is approximately 80% on the first attempt ( CSU Survey, 1982; 1990). O nce each quarter the test is adm inistered, and scoring follows two w eeks afterward over one w eekend. Generally requiring eight hours on Saturday and betw een six and eight hours on Sunday, each scoring session involves approximately two to three thousand pap ers which are read and scored holisticaliy. This form of assessm en t purports to evaluate student com position by viewing th e writing a s a w hole in stead of exam ining its se p a ra te elem ents (White, 1985), and it requires that readers assign scores based on a guide, commonly referred to a s a rubric. A num ber of calibrating e ssa y s are used to assist read ers in recognizing the range of applications for descriptors in the rubric and to confirm a distribution of scores. Using a version of the scoring guide created for the California S ta te University English Equivalency Exam ination (a placement/challenge test), two independent faculty raters read each essay and assign a score from a 1-6 scale, with 6 representing a superior paper. (S ee Appendix E for copy.) Any discrepancy of m ore than one point results in a third reader adjudicating the scoring. A cumulative minimum score of 7 (3-4 combination from two readers) is required for a student to p a ss the test, while a score lower than 7 requires the student to re-take the examination. B ecause the Chancellor's Executive Order 514 m andates that the test be "an all-cam pus responsibility" (1987; p.3), faculty who score the Graduation Writing T est include a group of veteran readers, a rotating core of twenty to thirty faculty m em bers from a variety of disciplines, including history, social sciences, business, com puter sciences, environm ental design, languages, arts, liberal studies, hotel and restaurant m anagem ent, agriculture and engineering, along with occasional others who are invited to augm ent the group. Six to eight of the readers who have scored the examination since its inception on this cam pus function a s table leaders who give guidance to new readers, and based on the test’s origins which were dependent upon English Department faculty, an English Department professor serves a s the room leader. Due to the considerable num ber of veteran readers, scoring of Cat Poly Pom ona's test h as always resulted in great inter-reader reliability, with the Testing Office consistently announcing interrater correlations on the order of .97 or .98. An analysis of the num ber of holisticaliy scored tests in California (e.g., EEE, EPT, CBEST, PPST, ECT, NTE, AP, MCAT, and WPE) concluded that the California State University system may have m ore experienced holistic read ers than can be found anyw here else, lending credibility to Cal Poly Pom ona's small discrepancy rate (Borowiec, 1988). But the reliability betw een test administrations offers no such clear confirm ation a s the Testing Office records only g en eral pass/fail percentages, which support only the norm-referencing aspect of the test and not adherence to criterion-referencing elem ents (Klammer, 1983). Graduation Writing Test.Criterla While all student pap ers (i.e., native and non-native) are scored using the sam e scoring guide and by th e sam e group of readers, it rem ains unclear how readers use the general criteria of the Graduation Writing Test (GWT) scoring rubric, which in Cal Poly Pom ona’s case w as devised from a rubric used to a s s e s s a challenge examination for native sp eak ers of English. In the course of creating a scoring guide for the GWT, the test originators reduced the original rubric for the English Equivalency Exam ination from a six-point to a five-point sc ale by combining sco res 4 and 5 with a hyphen and a single description. At present, even though the GWT is described a s having scores that range along a six-point scale, in fact, the sco res range only along a five-point scale with high and low deg rees of the single 4-5 level. The result of this scaling is that scores tend to aggregate in the mid-range level, which is the 3 score in a five-point scale (Oppenheim , 1966; White. 1985; Allaei & Connor, 1991). This concentration is problematic because if two readers assign scores of 3, the result is a total score of 6, a failing GWT score, in addition, the descriptors for level-3 are considerably longer, m ore detailed, and more encom passing (e.g., "for papers m arred by m ore than a few minor grammatical errors"); a s a result, the 3 score may draw increased aw areness and attention from the readers (Quellmalz, 1980). With the 3 score attached to a disproportionate num ber of essays, the d ata suggest that a disproportionate number of NNS essay s may also be included. The Test Office staff d o e s not track results in this m anner, but anecdotal evidence provided by NNS students seem s to confirm the magnetic lure of this mid-range failing score. To ensure the appropriate application of the broad criteria, two p ro cesses involve the careful selection of read ers and diligent holistic training. GWT scoring participation is by invitation only, at the discretion of the Testing Officer who h as no special expertise in testing NNS writing. Few GWT readers, in fact, have such professional training; only one or two of the twenty to thirty re a d e rs have either linguistic or TESL backgrounds, and they are not alw ays included in any single grading session. Therefore, the training that precedes the scoring of actual "live" papers is critical. T hree to four hours with practice p a p ers introduces read ers to holistic scoring and is devised to sensitize readers. However, research indicates that training m ay not influence nor dissuade som e raters from using individual standards and expectations when scoring mixed batches of e ssa y s written by English native-speakers (NS) and non-native speakers (NNS), resulting in an em phasis on surface errors of essays written by non-native speakers of English (Janopoulos, 1993). A related and additional problem, is that, although seem ingly counterintuitive, training may, in fact, cem ent expectation and attract additional attention to surface errors b ecau se the rubric and guidance provided readers may actually encourage such scoring (e.g., Freedm an, 1981). Ju st a s the psychological te a se of "Don't think of a white bear" cau se s a listener to immediately imagine the bear, the warning issued to GWT readers of "Don't penalize for typical ESL errors, such a s missing or m isused articles, prepositions or idioms" may cause readers to concentrate on precisely these errors. A number of attitudes supportive of this attention, yet counter to the philosophy of holistic assessm ent, are confirmed by holistic readers' comments, represented in statem ents such as: •This handwriting is so hard to read. •Spelling is really my pet peeve. •It's only a page and a half-too short. •Punctuation is merely a demonstration of academic discipline. •If it can't be sent out to a client, it cannot pass with my scoring. Nowhere on the designated holistic rubric are any such reductionist concerns noted. However, research has docum ented high positive correlations with such surface characteristics and holistic scores (e.g., Harris. 1977; Fein, 1980; Freedman & Calfee, 1984; McGirt, 1984; Johns. 1991b). Indeed, individual reader strateg ies that focus on th e superficial characteristics of the texts, although distinct from the scoring guidelines, m ay directly result in lower sco res for NNS student essay s. It is not clear w hat other explicit or implicit criteria (e.g., rea d er background) affect faculty-scoring of the GWT specifically. R e ad e rs in holistic scoring se ssio n s m ay simply b e contending with easily identifiable NNS su rfa ce erro rs in stead of reading for m eaning from a holistic perspective (cf. Ham p-Lyons, 1991c; Bochner, Albertini, Sam ar, & Metz, 1992). NNS Student W riters In order to ap p reciate the sc o p e of this concern, this re se a rc h attem pted to discover p recise num bers of NNS stu d e n ts within th e California S tate University (CSU) system but found that statistics of NNS stu d en ts are not m aintained by th e CSU nor by individual c am p u se s. However, three other m ea su re s w ere available to g au g e th e increasing NNS student population at the University. In 1992, the CSU C hancellor's Office reported a 50% in cre ase in enrollm ent in E ng lish -as-seco n d - lan g u ag e (ESL) program s since 1985 (K erschner). W hile significant growth is evidenced by this fact, students enrolled in CSU ESL (usually lower-division) co u rses a re only a portion of the NNS students affected by c o n ce rn s with th e G raduation W riting A sse ssm e n t R equirem ent. To a u g m e n t this nu m b er, th e d e m o g ra p h ic s of th e c a m p u s u n d e r consideration in this study w ere found to b e m oderately helpful; at Cal Poly Pom ona, during the Fall Q uarter of 1993, over 10,000 (63%) of the 17,000 stu d en ts enrolled w ere from ethnic minority gro u p s (C a/ Poly Pomona Strategic Plan, 1993). Again, however, b ecau se no records ot language background a re m aintained, an exact num ber of non-native sp eak ers of English within th ese groups is unknown. The third and m ost precise approxim ation of NNS student num bers w as provided by the Testing Office at Cal Poly Pom ona, which d o e s require student self-report of language background and estim ates th e NNS student population of test-takers at approximately 50% of each GWT administration (i.e., from five to fifteen hundred NNS students, depending upon the quarter). The problem s inherent in su ch testing, therefore, involve and co ncern a significant num ber of students at Cal Poly Pom ona. GWT a s Barrier O ne tacit assum ption of the GWAR testing and a general prevailing attitude of the GWT readers is that, if NNS students expect a degree from a university in the United S tates of America, then they m ust display native* like proficiency in English and participation in the basic culture com m on to a high proportion of residents in North America (ESL W orkgroup, 1988; S antos, 1988; Jo h n s, 1991a; Vann, Lorenz, & M eyers 1991). Yet the reality recognized by professionals in the fields of linguistics and the teaching of English a s a second language (TESL) is that com petency in formal academ ic English com position is not a sim ple m atter for NNS students: A variety of factors influence the sp eed and d eg ree of second- language acquisition and written-discourse proficiency (Schum ann, 1978; K rashen, 1981; Klammer 1983; Ughtbown, 1985; R oss, Burne, Callen, Eskey & McKay, 1984; Hamp-Lyons, 1991a; Kerschner, 1992; Leki, 1992), 10 T h ese factors are com plicated further by the effects of test anxiety c au sed by tim e constraints, artificial or unfam iliar topics and m odes, lack of reference opportunities (i.e., dictionaries and com puter-assisted spelling, gram m ar or style checks a re not perm itted), and heightened aw aren ess of the outcom e a s a potential barrier to university graduation (Brown, 1986; Jo hns, 1991a). At present, the majority of non-native sp e a k e rs of English at Cal Poly Pom ona derive from Asian language backgrounds, and their writing p re se n ts unique co n cern s for GWT rea d ers and th e w ays in which the re a d e rs sco re th e e ssa y s. Few of th e re a d e rs are in any w ay formally trained to handle NNS writing and, while sensitizing re a d e rs to NNS writing problem s h a s often b e en d iscu ssed during th e scoring se ssio n s, re se a rc h indicates that th e potential for sensitization is limited (e.g., Klamm er, 1983; Stansfield & R oss, 1988; Kaplan, 1990; H am p-Lyons, 1991a; B ochner, Albertini, S am ar, & Metz, 1992). Thus, holistic rea d ers often ap p ear to focus only on superficial a sp e c ts of NNS writing. The lack of understanding or familiarity with this type of writing m ay contribute to faculty readers/raters* e x p re ssio n s of an n o y an ce with NNS stu d e n ts1 inability to attain native-like writing proficiency, causing th e re a d e rs to draw upon their own intuitive criteria and em phasize problem s with surface featu res, su ch a s handwriting, text length, punctuation, idiomaticity and gram m atical control. Since nearly 80% of the NNS students who take the test at Cal Poly Pom ona do not p a s s it on the first attem pt (CSU Survey, 1982; G aschler, 1990; T orres, 1994) and a re required to k eep taking it until they p a s s 11 . (som e taking it a s many a s seventeen tim es-and requiring additional years of university enrollment), it h as becom e a gate-keeping apparatus for a distinct portion of the student population (ESL Workgroup, 1988; Kerschner, 1992). A single cam pus location, the Learning R esource Center, offers designated assistance for students who fail the GWT. However, this assistance consists of only one full-time tutor and one part- time tutor who attempt to respond each quarter to hundreds of students, both NS and NNS (with an increasing dem and by NNS students) (Torres, 1994). For students not passing the test after multiple attem pts, a waiver system is in place which requires a student take the test three tim es, receiving scores of 6 on at least two of the attem pts, and which requires documented attendance at additional tutoring sessions, such as the limited opportunities offered by the Learning Resource Center. However, with the completion of these requirements, the student's transcript is stam ped with the words: "GWT Waiver Granted--Did Not P a ss the Graduation Writing Test." For som e students, the waiver is sufficient; that is, they obtain their degrees. But a significant num ber of students are not willing to settle for the future consequences of this practice because graduate schools will not accept them, especially any in the CSU system since graduate students must also take and p a ss the GWAR tests; most engineering firms which hire from the eight engineering majors on this polytechnic cam pus refuse students with the stam p since it signals a distinct impediment to their professional futures; and foreign students who have received financial support from governm ents or major corporations in their hom elands to 12 study in the United S ta te s cannot return to their countries with this perm anent negative mark because English-language ability is an expected outcom e of the financial support they have received (Borowiec, 1988). With the increasing num ber of NNS students at the University, the high failure rate for NNS student e ssa y s on the Graduation Writing Test, and the multiple difficulties associated with obtaining a GWT waiver, valid holistic scoring of NNS student essa y s in the high-stakes state-m andated exit a s s e s s m e n t is a se rio u s m atter for stu d e n ts, faculty, and adm inistrators at Cal Poly Pom ona. Having potential implications for all holistically-assessed testing which includes NNS students, the scoring of Cal Poly Pom ona's version of the GWAR, the Graduation Writing Test, especially the criteria used for evaluation and its application by faculty readers, must be validated. Statem ent of the Problem Since the 1970s, holistic a sse ss m e n t h a s becom e a widely accepted method of evaluating student writing at the university level for a variety of purposes, including adm ission, placem ent, proficiency and certification of com petency for graduation. At present, it is the m ethod used by eighteen of the twenty cam puses of the California State University system for evaluating student e ssa y tests used to satisfy the m andated Graduation Writing A ssessm ent Requirem ent (GWT) (Borowiec, 1988; CSU Survey, 1990). The original decision by the Board of T rustees to allow each cam pus to devise its own format in the context of the individual cam pus curriculum confirms the aw aren ess of the educational validity 13 n ecessary for such testing (i.e., the results of testing reflected in effects "on the curriculum, m aterials design, teacher developm ent, and student study patterns" [Horowitz, 1991; p.82]). The underlying implicit assum ption is that, reciprocally, the curriculum will prepare students for the Graduation Writing Test, and the backw ash effect of the writing test will Improve the overall quality of student writing in academ ic coursework. By contextualizing such testing within its educational setting, it is possible to a s s e s s the validity of the holistic assessm en t m ethod used to evaluate student writing. Referring to the holistic assessm ent form ats used by the cam puses comprising the California State University system , White (1985) urged, "Tests them selves should be tested"(p. 75). However, the validity of such scoring, specifically the interpretation and application of scoring criteria, h a s rem ained unconfirmed for NNS students in general (Henning, 1991). and for NNS students at Cal Poly Pom ona. The examination of the holistic assessm en t used for the Graduation Writing Test at Cal Poly Pom ona is necessitated by the growing population of California S tate University students for whom English is not a native language and who face discrimination for two reaso n s related directly to . their language backgrounds when they sit for the Graduation Writing Test: 1.)Criteria and Application: C onsiderable variability m ay exist in how holistic asse ssm e n t criteria are applied by faculty raters scoring th e Graduation Writing Test (GWT). T hese criteria may be overly broad and, a s such, d ep en d en t upon a num ber of inconsistent an d intangible variables, including rater background, variable holistic training, and individual interpretation of the rubric. In spite of claims m ade for this form 14 of evaluation-that it a s s e s s e s the whole and not the p a rts-th e u se of holistic criteria m ay em phasize one criterion over another, thereby unduly penalizing NNS student e ssa y s w hen they are included in a mixed scoring with NS student writing, a s they are in GWT testing at Cat Poly Pom ona; and 2.) Iim e _ a n d Support; A se rio u s d isa d v an tag e in tim e and opportunity m ay exist for NNS stu d en t w riters. Prior to university adm ission, native sp e a k e rs of English generally have twelve or m ore y e a rs to learn and practice English academ ic writing, followed by validation in university coursew ork; conversely, NNS stu d e n ts have considerably le ss time with limited direct attention to their academ ic literacy n eed s (Cumming, 1990). R esearch indicates that seven to nine y ears is the required length of time for NNS students to attain cognitive acad em ic lan g u ag e proficiency, th e level required for university coursew ork (Cummins, 1979; Collier, 1994) and that, for NNS students, th e academ ic d e m a n d s of coursew ork a re th e m ea n s of developing English writing skills (Bereiter, 1980; M endelsohn & Cumming, 1987; Hamp-Lyons, 1991c; Leki, 1992). Consequently, the relationship betw een University curriculum and GWT testing m ay b e all the m ore profound for NNS students who, by virtue of their limited time at the University, may be disproportionately dependent on their regular coursew ork for academ ic discourse learning opportunities a s a m ean s of GWT preparation. T hus, to confirm th e valid u se of holistic a sse ss m e n t of NNS stu d en ts by the G raduation Writing T est, a clear understanding of the similarities and differences represented by coursew ork and by the writing 15 test is necessary. The constructs for com parison are operationalized for the curriculum, by the criteria which faculty u se for assessing NNS writing in academ ic coursew ork, and for the GWT, by the holistic criteria which raters u se for the high-stakes evaluation that perm its, delays or denies university graduation. Significance of the Study Currently, the G raduation Writing A ssessm ent Requirem ent is the state m andate to certify English writing com petence for all students at all California S tate University cam p u ses. The increasing num ber of NNS students for whom the essay format for m eeting this requirem ent presents a serious barrier requires that, when holistic scoring of student e ssa y s is the m ean s of evaluation, a s it Is at Cal Poly Pom ona, the validity of the assessm en t scoring must be confirmed. Should it be found that this test form is biased against NNS student writers, another form of evaluation m ust be devised to provide appropriate a s s e s s m e n t for NNS stu d e n ts to obtain th e G raduation Writing A ssessm ent Requirement certification. Purpose of the Study The main focus of the study w as to investigate the relationship betw een two se ts of criteria used to a s s e s s NNS student writing: (1) the criteria u sed by faculty to a s s e s s NNS student writing in academ ic coursew ork acro ss the curriculum, and (2) the criteria used to a s s e s s NNS student writing on the university-wide G raduation Writing T est 16 (GWT), a form of th e California S ta te U niversity G raduation Writing A ssessm ent Requirem ent (CSU-GWAR). The investigation w as conducted in o rd er to validate th e form of holistic a sse ss m e n t u se d for certifying English writing co m p eten ce of NNS stu d e n ts at o n e California S ta te University cam pus, California S tate Polytechnic University, Pom ona. R esearch Q uestions A review of the literature concerning NNS com position and holistic a sse ssm e n t suggested th e following research questions: 1. W hat criteria do faculty at the University u se to m easu re NNS student writing in their academ ic courses? 2. Are the course criteria different in kind or d e g re e from the criteria used to a s s e s s th e G raduation Writing T est, o n e version of the California S tate U niversity-m andated G raduation Writing A ssessm ent R equirem ent? If so, in what way(s) are th ese two s e ts of criteria different? 3. How would NNS student scores, especially the pass/fail ratio, b e altered if the sam e criteria u sed by faculty for evaluation of coursew ork w ere used to a s s e s s student perform ance oi. the G raduation Writing T est? Delimitations of the Study T he following boundaries w ere th e delimitations of this research: 1. Only faculty on o n e CSU c am p u s, C alifornia S ta te Polytechnic University, Pom ona, w ere surveyed. 2. T he study w as limited by those who responded to the questionnaire. 17 3. The Testing Office staff provided a limited number of previously-scored essays from one GWT administration. 4. The Testing Office staff had no m eans of determining which essay s were written by NNS students. 5. Due to the creation of a new analytical rubric, no calibrating essay s were available for use a s training papers during the analytical scoring. 6. The corpus of 100 GWT essay s included essays written by native (11) and by non-native (89) speakers of English. 7. The survey m ethods w ere designed to gather data concerning the writing and assessm ent of only the target population of NNS students. Limitations of the Study The following limitations may have affected the extent to which the results of this research may be generalized: 1. The sample for the questionnaire response w as limited to faculty at one California State University campus. 2. The sam ple size for the questionnaire response w as further limited by those who chose not to respond to the questionnaire. 3. Validity of the survey responses w as limited by the individual faculty m em ber's aw areness of his/her own practices and by the veracity of his/her responses. 4. Both sets of readers involved in the re-scoring of GWT essay s (ESL Specialists and GWT Raters) had considerable prior holistic scoring experience. 16 Definitions of the Term s For the purpose of this study, the following term s are used with these meanings: Analytic Assessm ent: A method of direct evaluation which considers elem ents or factors within student writing in order to generate separate subscores which are later totaled for a cumulative score. EPT (English Placement Examination): The title of the test given to all admitted freshmen at the California State University in order to determine entry-level English skills and to assign required college English coursework. At present, Educational Testing Services, a private corporation, scores and reports the combined results of an objective machine-scored gram m ar test and a holistically-scored essay to individual cam puses. ESL Specialists: Faculty m em bers who teach in the English-as-a-Second- Language Program at Cal Poly Pom ona and who participated in the re-scoring of essay s for this research. GWAR (Graduation Writing A ssessm ent Requirement): The state- m andated exit writing requirement for conferring undergraduate and graduate degrees and certificates from any of the twenty cam puses in the California State University system. GWT (Graduation Writing Test): The nam e given to Cal Poly Pom ona's form of the state-m andated graduation writing requirement, which is a single 75-minute essay examination, scored holistically by faculty m em bers on a combined .12-point scale, requiring a 7 a s a passing score. 19 GWT Raters: Faculty members who regularly score the Graduation Writing Test at Cal Poly Pomona (i.e., more than 6 times) and participated In the re-scoring of essays for this research. Holistic Assessm ent: A popular method of direct evaluation of student writing sam ples, commonly using sam ple essays to assist faculty readers in obtaining an overall impression of the writing which corresponds to a single number on a scoring guide. NS (Native Speaking) Students: Students for whom English is a native language. NNS (Non-native Speaking) Students: Students for whom English is not a native language. Rubric: A synonym for the scoring guide used in writing evaluation, a term commonly used by Educational Testing Services. TESL (Teaching of Engtish a s a Second Language): The term commonly used to describe the teaching of English to speakers of other languages. Organization of the Remainder of the Study The remainder of the study is organized in the following manner: Chapter II reviews the literature on the California State University system GWAR mandate, the issues for NNS students, the practices of holistic and analytic assessm ent, specifically criteria interpretation and application through faculty rating, with an em phasis on issues affecting validity and NNS student writing, Chapter II) explains the three stag es of the process used to obtain the data analyzed in this study, including data collection of faculty course criteria by m eans of a faculty questionnaire, the creation of the analytic rubric used to operationalize the faculty course criteria, and the re*scoring of GWT e s s a y s by two m ethods for p u rp o se s of com parison. C hapter IV p resen ts the significant findings obtained as a result of the three p h ases of this research, and the subsequent analyses. C hapter V contains a sum m ary of the study, conclusions b ased on the findings, implications for teaching and testing, suggestions for change, and recom m endations for further research. Completing the study are the cited references and appendices. 21 CHAPTER II REVIEW OF THE LITERATURE Organization of the Chapter Chapter II reviews the literature in three general are as in order to present the relevant issu es regarding the scoring of student writing assessm ent, especially for university-level non-native-English-speaking (NNS) students. The first part presents the background of the California State University writing assessm en t requirem ents and the present-day concerns which the assessm ent causes for university NNS student writers, exemplified by the problem s at one S tate University cam pus. The subsequent section discusses the research on direct scoring, first, by the current practice of holistic a ssessm en t, its popularity and problem s, including concerns for NNS writers, and second, by contrasts to an alternative m ethod of assessm en t, analytic scoring. The third section considers the psychometric properties of reliability and validity associated with direct scoring of student writing. The California State University Writing A ssessm ent Mandate The Graduation Writing A ssessm ent Requirement In 1975, by legislative dem and, a task force w as established to study the cau ses and solutions for declining writing abilities of students educated in the California S tate University system (Obrecht, 1992). Responding to its findings, the,Board of T rustees m andated the system - wide Writing Skills Program , which includes two types of writing com petency examinations: (1) the English Placem ent Test (EPT), which is adm inistered to all freshm an undergraduate students upon adm ission in order to diagnose English proficiency levels and determ ine subsequent coursework; and (2) the G raduation Writing A ssessm ent Requirem ent (GWAR), which is required of all upper-dlvision and graduate students to confirm English writing com petence a s an exit requirem ent for all university degrees (Moye, 1983). The nineteen State University cam puses at that time agreed to use the sam e English Placem ent Test, but no single test format for the GWAR w as acceptable to all; consequently, the T rustees allowed each cam pus to devise, within the context of its own curriculum, its own version of a certification m easure to confirm college-level academ ic writing proficiency (Moye, 1983). Currently, while six cam puses include a m achine-scored objective test and eleven allow fulfillment of the requirem ent by writing included a s part of a course option or the choice of a course or essay examination, eighteen of the present twenty cam puses require an essay exam ination which is holistically scored (S ee Appendix V for cam pus options.) (Borowiec, 1988). NNS Students at the California State University W hen the GWAR w as implem ented in 1976, the population of students in California differed from the current student population in a significant way. In 1970, 27% of the school-age population (ag es 5-17) consisted of language m inority.students, yet by 1990 the percentage of this student group had grown to 46%, and an increase to 52% is projected 23 by the year 2000 (Cort6s, 1992). This growth in California's school-age population of language minority students is reflected in the increasing num bers of students currently enrolled in the California State University (CSU) system who are non-native speakers of English (NNS). No records on general student language background are compiled by any CSU cam pus, but the overall in crease in NNS stu d en ts in California is evident on the cam pus examined in this research, California State Polytechnic University, Pom ona (Cal Poly Pomona). While ethnicity is not a reliable indicator of language background, C hapa (1990) h as dem onstrated that race and ethnicity of students in California are related to limited English language proficiency, especially for Hispanic and Aslan students. Therefore, the dem ographics of this University cam pus may provide applicable general information since in Fall Quarter, 1993, 63% (10,826) of the students enrolled at Cal Poly Pom ona were from ethnic minorities, including 18% Hispanic and 35% Asian. Additional data include the fact that over 20% of the students at Cal Poly Pom ona w ere born outside the United S tates ("Toward the 21st Century," 1994).Therefore, while specific figures for the non-native-English-speaking population enrolled in courses at Cal Poly Pom ona are not available, the inference is reasonable that a significant num ber of students are non-native speakers of English. English-as-a-second-language (ESL) course enrollment data are an accurate indicator of growth within the non-native-English-speaking student population, (although these d ata are limited to enrollment figures only). In the 1970's, only twelve CSU cam puses offered ESL courses with a total enrollment of 3,421. By 1991, all twenty cam puses offered courses to a total enrollment of 16,999, nearly five tim es that of the 1970s. (ESL W orkgroup, 1988). However, identifying only a segm ent of the NNS students, the California State University Chancellor’s Office describes students enrolled in English-as-a-second-language (ESL) courses at the University to include: 1) Visa students, who are admitted to the U.S. with a foreign student visa and pay nonresident fees; 2) R esidents aliens, who are considered perm anent residents, holding green cards; 3) Refugees, who are admitted to the U.S. a s residents for political reasons; and 4) U.S. citizens with limited English proficiency who com e from bilingual or non- English speaking hom es (Kerschner, 1992). Easily traced through records, the first three NNS groups have increased considerably since the late 1970s. CSU enrollment statistics for resident aliens, visa students, and refugees totaled 21,677 in 1979 but more than doubled to 49,896 by 1990 (Kerschner, 1992). The fourth and presumably the largest group, however, Is not tracked by devices such a s visas, TOEFL scores, and required ESL course enrollment, so specific data on this group are unavailable. A num ber of indicators confirm the growth of the NNS student population in the California State University system , including the student population at Cat Poly Pom ona, since the inception of the GWAR in the 1970s. NNS Student GWAR Contentions The growing population of California State University students for whom English is not a native language face special problem s a s they encounter the G raduation Writing A ssessm ent Requirem ent. T h ese 25 problem s, a s p re se n te d in th e limited literature, focus on th e writing abilities of this population, th e n eed for in creased writing opportunities, and th e co n cern s for valid criteria application and re a d e r sensitivity a s scoring is conducted in th e holistic a sse ssm e n t of th e GWAR, including th e form Known at Cal Poly P o m o n a a s th e G raduation W riting T est (GWT). As broad indicator of th e co n cern s for English language writing abilities, the NNS student population w as acknow ledged in the results of the English P lacem ent T est in 1989 which found that over 84% of the students who reported that English w as not their primary language did not receive sufficiently high sc o re s to b e placed into cotlege-level composition co u rse s (K erschner, 1992) and w ere th u s assig n ed to pre*college level coursework. C ollege coursew ork writing oppo rtu n ities for NNS university stu d e n ts are of im portance, a s they are to all university stu d e n ts. However, b e c a u se student writing skills develop a s a direct result of the work d o n e in school (Bereiter, 1980), that is, no o n e is a native w riter- everyone m ust b e taught (Leki, 1992)-th e re m ay a greater need to offer coursew ork opportunities for those w hose English language abilities are developing, and for whom coursew ork opportunities m ay b e their primary m e a n s of acquiring English acad em ic writing skills (M endelsohn & Cumming, 1987; Hamp*Lyons, 1991c). T he a w a re n e ss of this coursew ork dem and for NNS stu d en ts w as confirm ed in a report com m issioned by th e Cal Poly Pom ona cam p u s for evaluation of its form of the GWAR m andate, known a s th e G raduation 26 Writing T est (GWT), which found that students, faculty, and adm inistrators all voiced concern over NNS student writing abilities (G reene, 1985). The recom m endations in the report su g g e ste d in cre ased opportunities for GWT preparation for NNS stu d en ts in cam pus-supported w orkshops and increased a w are n ess of NNS student n e e d s in th e GWT a p p eals p ro cess offered by the University. In a m ore com prehensive m anner, NNS stu d en t contentions with the GWAR w ere p resen ted in a report authored by th e ESL W orkgroup, a com m ittee of E ngtlsh-as-a-S econd-L anguage (ESL) ed u cato rs from the California S ta te University sy stem . S eeking to discover w hether th e GWAR w a s a barrier for NNS student graduation, their work reported that nine of th e fo u rteen c a m p u s e s w hich re sp o n d e d to th is q u estio n considered the test to be a barrier ( ESL W orkgroup, 1988). The resp o n se to the com m ission from Cal Poly P om ona confirm ed that its version of the exam ination, the G raduation Writing T est (GWT), is "a definite barrier" but added that th e a p p eals p ro ce ss in place for NNS stu d en ts co m p en sated for this problem (p. 53). However, a s statistics provided by the California S tate University C hancellor's Office (Survey, 1982, 1990) indicate that, at Cal Poly Pom ona, 80% of the NNS stu d en ts fail the test, th e m agnitude of the problem m ay not b e fully a d d re ssed by the ap p eals process. A ddressing a m ajor concern for holistic scoring of NNS student e ssa y s, Klamm er (1983) advocated sensitive treatm ent of NNS student writers at th e CSU, "who, though they m ay never b e able to do a s well a s th e m ost com petent native speaking stu d e n ts, c an n ev erth eless often dem o n strate th e qualities of good writing that w e consider essential in 27 acceptable college-level essay s" (p.6). Concurring with th e n eed for sensitivity, the ESL W orkgroup contended that, to e n su re fair scoring, read ers "need to be sensitive to the writing of second language students and not penalize them for m istakes in idiom or other minor errors”(p.9). As a rater for CSU holistically-scored e ssa y tests, Klammer agreed, stating that the "intuitions that enable [the raters] to apply a scoring guide to e ssa y s written by native sp eak ers don't work a s well when [the raters] are confronted by the writing of non-native speakers" (p. 1). Re-exam ination of criterion application for NNS student writing is necessary and is illustrated by the u se of the rubric for the GWT at Cal Poly Pom ona. This 6-point scale is a slightly altered form of the scoring guide devised for the English Equivalency Examination, a test which when it originated in 1973 (Moye, 1983) had been norm ed for native speakers of English (Kerschner, 1992). And yet test evaluations usually specify that "tests developed for or norm ed on native sp eak ers of a language are not valid or reliable indicators of the language knowledge or skills of a person who is not a native speaker of that language" (R oss et al., 1984; p. 12). Disputing an additional practice of holistic asse ssm e n t, that is, its dependence on com bined criterion- and norm -referenced scoring for the Graduation Writing A ssessm ent Requirem ent, Klammer (1983) argued that ranking of student writing h a s no place in a test that se e k s to certify abilities, especlaliy b e ca u se the ranking penalizes NNS stu d en ts when they a re scored with native students, a s is the practice at Cal Poly Pom ona. 28 Of the many concerns associated with holistic assessm en t, perhaps the single g re a te st issu e confronting NNS stu d e n ts is the general underlying assum ption of the QWAR, which contends that "...because th ere is no distinction in th e value of d e g re e s stu d e n ts can e a r n - regardless of their native lan g u a g es-th e re should b e no distinction in expectations for stu d en ts on the GWAR" (ESL W orkgroup, 1988; p.9). This issue is a major concern b e c a u se it also underlies faculty readers' attitudes a s they holistically sc o re th e G raduation Writing T est. In concluding that the GWAR "is unfair in that ESL stu d e n ts m ay never achieve native fluency or sophistication of expression in English...and should not be expected to do so," the ESL W orkgroup suggested course options to fulfill the state requirement (p. 11). Writing A ssessm ent Scoring The California S tate University G raduation Writing A ssessm ent Requirem ent, the prescription for improved student writing in the 1970s, grew out of the sam e popular aw aren ess in that time period that caused educators and resea rch e rs to re*evaluate their instrum ents for student writing assessm ent. In rejecting indirect m easures, such a s multiple choice or fill-in-the*blank ty p es of te s ts , ev alu ato rs advocated th e u se of a sse ssm e n t instrum ents which would m ore closely m atch perform ance objectives of composition coursew ork (Quellmalz, 1980). The required use of holistically-scored student e ssa y s by eighteen of the twenty California S tate University cam p u ses to m eet the G raduation Writing A ssessm ent 29 R equirem ent is'th e current attem pt to m atch testing to the coursew ork objectives of the University. HoJisticAssessm ent The now-popular practice of direct evaluation of student writing by the u se of written sam ples h a s g en erated a num ber of innovations, the m ost com m on known a s holistic a ssessm en t. By interpreting the whole a s being g reater than the sum of Its parts, holistfcism (a term coined by White, 1984) purports to allow evaluation of the actual product of writing in the attem pt to "get u s closer to w hat is essen tial in...com m unication" (Cooper, 1977, p. 3). During th e early d ev elopm ent of direct a s s e s s m e n t, m any ap p ro ach es w ere considered holistic; that is, they relied upon rea d ers com ing to a c o n se n su s in criterion application and e s s a y ranking. Exam ples ranged from dichotom ous sc a le s which included a se ries of yes/no questions asking w hether a feature or requirem ent had b een included in the writing (Cooper, 1977) to primary trait scoring which asked raters to construct a scoring guide for analysis of relevant main features to a type of discourse (Lloyd-Jones, 1977). Today, however, the concept of holistic a sse ss m e n t is generally acknow ledged a s a com bination of criterion-based and norm -referenced scoring in which prototypical e ssa y s are identified and ranked. T he rem aining e ssa y s in th e batch a re then m atched to ch aracteristics noted in the perceived prototypes or to generally desirable characteristics confirm ed by specified criteria, and assigned a ranking score (Myers, 1980; White, 1985; Huot, 1990a). 30 The common and accepted m eans of ranking student essa y s by holistic a ssessm en t serv es a variety of purposes, such a s adm ission, placem ent, c o u rse exit, proficiency, research and com petency for graduation from many high schools, colleges and universities around the world (White, 1985; McNamara, 1990; Hamp*Lyons, 1991a; Williamson, 1993). The differing purposes dem and a variety of processing strategies and malleable expectations from the readers (White, 1986; Huot, 1990a; Sm ith, 1993), an d th e evaluation p u rp o se influences the scoring (Williamson, 1993). Placem ent, for exam ple, requires a reader to decide to which class, from a limited selection, a student should be assig n ed , b ased on determination of the student's specific skills. Ranking is norm ed within the testing sam ple, with the reader's understanding that all levels, including sufficient num bers at both top and bottom, must be selected to fill classes. R eaders are aw are of the importance of placem ent decisions since these decisions will require the student to take certain c la sses (am ounts and levels), and correct placem ent is assum ed in teacher, program , and departm ent planning (Pula & Huot, 1993). Placem ent decisions, however, may be countered by classroom teach ers' diagnosis a s a safeguard (White, 1986; Smith, 1993). On the other hand, the decisions for certifying competency, a s m andated by the California State University system in its G raduation Writing A ssessm ent Requirem ent, require that th e reader identify and judge a significantly more complex range of student writing abilities in order to arrive at a rank assignm ent. This type of assessm ent cau se the rater to respond in a different m anner than when making placem ent decisions since the result m ay determ ine w hether or not a student graduates from a school or university, a higfvstakes evaluation consideration (Klammer 1983; W hite, 1985; Huot, 1993; W illiamson, 1993). This variable of sorting holistic judgm ents for different purposes is only one of the limitations of holistic assessm ent. Limitations and Concerns Even with its popular and multiple applications, holistic assessm en t has distinct limitations, which som e researchers warn, have often been overlooked (e.g., White, 1985; Huot,1990a; Hamp-Lyons, 1991a). First, holistic a sse ssm e n t only ranks papers; it d o e s not diagnose writing abilities or problem s, nor d o e s it explain reaso n s for scores. Thus, it is limited a s an academ ic testing m ethod since it does not identify students for the purposes of helping them (Williamson, 1993); it offers no recourse for stu d e n ts who m ay n e ed a ssista n c e or clarification afterw ard (Quelimalz, 1980; Winters, 1980; Elbow, 1993). Second, a holistic score is dependent on the particular student population and e ssa y topic; it h as no absolute value and should not be used to "establish proficiency in a concrete sense" (Mullis, 1984; p.17). Higher sco res do not necessarily m ean the writing is good, only that it is better than other writing with which it w as judged (Mullis). Thus, a score of 7 does not m ean the a sse sse d student writer is a 7-level writer. This lack of absolute value also m e a n s th at th e p assin g g rad e for e a c h administration of a writing examination should be reviewed to determ ine 32 appropriate cut-off scores for each testing session, a reliability issue that is frequently ignored (While, 1985; Hamp-Lyons, 1991a). R elated to the second limitation, the third limitation is that the scoring evaluation is overly dependent on the reliability property of testing, and the reported reliabilities are often over-rated and difficult to trace. A num ber of statistical procedures (at least eight for interrater reliability alone) have been used to report testing reliability, but researchers have found it often reported with little clarity or understanding (Coffman, 1971; Cherry & Meyer, 1993). Even the method most commonly used to resolve scoring discrepancies, the third reader, has produced inflated reliabilities, "a kind of fraud," according to Cherry and Meyer (p. 122). T he m ajor limitation of holistic a sse ssm e n t, how ever, is its dependence on subjective judgm ents of student writing m ade by faculty readers. Oiler (1979) rationalized that subjectivity is "indispensable in decisions concerning whether the writer h as expressed well his intended m eaning a n d .Jn determining what that intended meaning is" (p. 392). But since this subjectivity may range from subtle aspects, such a s a reader's background and training, to m ore overt reactions, such a s b iases and inability to adhere to scoring guides, the scope of such judgm ents may be difficult to evaluate. For this reason, it is important to understand how raters process student writing and the scoring rubric a s the faculty raters score holistically. 33 Faculty. R eaders B ecause holistic asse ssm e n t is an interactive process, what the readers bring to the task and what the text offers is synergistic. R esearch h a s contended that "meaning exists in the reader as well a s in the text" (G ere, 1980; p. 58); that "reading...can never be sep arated from the purposes, prior knowledge, and feelings of the person engaged in the activity" (Smith, 1988; p. 179); and that "readers...have a s m uch to do with the 'su ccess' of a student essay as...the student (who] h a s m anaged to use the language...to sh ap e m eaning " (Stock & Robinson, 1987; p. 119). Even with this aw areness, very little is known about holistic readers d u e to certain faulty assum ptions. In general, selecting read ers from faculty who have a vested interest in the outcom e of the test (whether by accepting future students, acknow ledging levels and stan d ard s, or confirming com petency for their institution) has resulted in accepting readers about whom insufficient information exists (Coffman, 1971; Myers, 1980; V aughan, 1991). M oreover, the assum ption that te a c h e rs are autom atically appropriate judges constitutes an unresolved concern; interacting with students in ways that motivate, enlighten, and alter thought or behavior, a s teachers do daily, is not com parable to standing in distant, anonym ous judgment, as raters are required to do occasionally (Rodby, 1992). Further, readers' highly developed professional schem ata is not com parable to the very narrow view required by holistic raters (Horowitz, 1991). And a strange dichotomy, is created by holistic scoring when faculty teach using one system (e.g., process writing), then a sse ss students using a different system (e.g., product emphasis) (Coxwell, 1991). Ironically, the quality of a testing program that u ses holistic scoring is only a s strong as its readers (Jacobs, Zlngraf, Wormuth, Hartfiel, & Hughey, 1981), so precise information about these faculty m em bers is necessary in order to evaluate how they are assessing student writing. Reader_backoround. The fact that readers share common values and recognize common dem ands and possibilities constitutes the criterion that is critical to an appropriate group of readers for holistic assessm ent (Coffman, 1971; Myers, 1980; White, 1985). But faculty raters come from heterogeneous backgrounds, "producing] different attitudes and values which operate significantly in their evaluations of essays," and which result in the single greatest variable within this form of assessm en t (Follman & Anderson, 1967, p. 199). Janopoutos (1993) labels this combined effect of these attitudes and values as the readers1 expectations, and he contends that it is these which are responsible for the variability in readers1 application of scoring scales. Among the many differences accounting for th ese expectations, resea rch h a s show n that ag e, gen d er, language background, educational background and teaching experience influence holistic scoring of student writing. R esearch su ggests that with age com es tolerance for language control problem s (Santos, 1988; Vann et al., 1991). G ender may also m ake a difference since wom.en in hum anities, education, and social sciences appear more tolerant of errors than men in the sam e disciplines 3 5 (Vann at al., 1991). Additionally linked to gender is the ability to ag ree with other readers, according to McColly (1970), who found that college m ale faculty w ere m ore reliable in their scoring than college fem ale faculty, and that college m ale faculty w ere m ore reliable than all high school tea ch e rs regardless of gender. T h e lan g u ag e background of ra te rs also s e e m s to Influence scoring. Suspicion of this finding c a u se d Oiler (1979) to announce that only native or near-native sp e a k e rs of English should be included a s holistic rea d ers. For student writing a sse ssm e n t, a re a s of content and u sa g e seem m ost affected by th e raters' language backgrounds. S antos' (1988) study confirmed that language errors of native sp e ak e rs (NS) w ere penalized m ore severely by non-native-E nglish-speaking (NNS) raters, seem ing to indicate their bias that native stu d e n ts should have control over their native language. G reen and Hecht (1985) discovered that raters who w ere English native s p e a k e rs an d th o se w ho w ere non-native sp e a k e rs w ere unable to a g re e on w hat constituted an error in student writing, but they found th at native-speaking re a d e rs focused m ore on content. A nother factor of re a d e r background which a p p e a rs to affect scoring is the academ ic discipline In which a rater h a s b e e n educated. R aters from th e so-called "hard sciences" (physical, m athem atical and biological scien ces) have b e en show n to b e less tolerant of lan g u ag e errors than those in th e "soft disciplines" (hum anities, education and social sciences), a result which m ay be related to little recognition or concern 36 with co n cep ts of potential or growth in th e hard sc ien c es (S a n to s,1988; Vann, M eyer & Lorenz, 1991). An early study on holistic scoring (Deidrich, French, & Carlton, 1961) su g g e ste d a relationship to re sp o n se s b a se d on disciplines. This study u se d se v e n English te a c h e rs, a s well a s fifty-three journalists, editors, social and physical scientists, lawyers, and b u sin ess executives to s e e if people ex p o sed to v ast am ounts of writing would recognize and value proficiency in th e sa m e way. Lacking training and guidelines, the re a d e rs a g re ed very infrequently: "94% of the p a p e rs received either sev en , eight, or nine of the nine possible sc o re s and no p ap er received few er than five different grades" (p.1). But within their disciplines, they did find grounds for agreem ent: English te a c h e rs resp o n d ed m ore often to m ec h an ics an d structure, b u sin e ss ex ecu tiv es to organization, and journalists and editors to creative qualities. Teaching experience. In conjunction with acad em ic disciplines, length an d type of teaching ex p erien ce a p p e a rs to h av e an effect on re a d e rs for holistic scoring but, again, with great variability. For exam ple, M arshall (1967) and Harris (1977) found novice teach ers gave m ore weight to organization an d content while ex p erien ced te a c h e rs em p h asized m echanics and se n ten c e structure, although te a c h e rs w ho'assigned m ore writing in their c la s s e s ten d e d to b e le s s critical in their a sse ss m e n t of surface-level errors during holistic scoring (Vann et al.. 1991). 3 7 A good deal of research h a s b een devoted to o n e type of teaching experience in relation to holistic scoring, that is, com parisons of English departm ent faculty with other faculty a c ro ss th e disciplines. T he results generally confirm that, while English faculty a re m ore co n cern ed with language u sag e, th e rest of the faculty em phasize content (e.g., Halpern, Spreitzer & Givens, 1978; Conklin, 1982) an d logical organization (e.g., W eaver, 1982). In addition, research sh o w s that stu d e n ts who are not necessarily good w riters by English departm ent sta n d ard s but who can respond with factual information w ere rew arded m ore than the thoughtful "good" English writer who required "the p ro cess" to estab lish id e a s (Conklin, 1982). O ther research h a s confirm ed this finding of th e split in faculty expectations of student writing (e.g., Bridgem an & C arlson, 1983; Horowitz, 1986; Jo h n s, 1981, 1991a, 1991b; M endelsohn & Cum m ing, 1987; Ostler, 1980). Even within the general field of English teaching, differing view s are apparent for th o se who teach English a s a seco n d lan g u ag e (ESL) and th o se who te a c h com position for native s p e a k e rs . Brown (1991) investigated this a re a and, even though he found little difference in sco res for ESL an d English native student writers, h e discovered that the two g ro u p s of re a d e rs arrived at their s c o re s differently: English faculty focused m ore on m echanics; ESL faculty em phasized content. T hus, in this study, ESL faculty supported the attention to content dem onstrated in other research by faculty a cro ss the cam p u s (e.g., Bridgem an & Carlson, 1983) rather than the focus of English departm ent readers. T eaching experience may, in fact, be the key to "the real rubric" underlying holistic scoring (Pula & Huot, 1993; p. 252). R ead ers seem to read b e st w here they have th e m ost experience (Ham p-Lyons, 1991b). Sm ith's (1993) research found that for placem ent purposes, read ers with th e experience of having taught th e sam e English com position c o u rse s w ere found to b e particularly reliable scorers. In other research , English writing tea ch e rs have b e en described a s being continuously "in training" a s they read and respond to their students' written work. This accum ulated reading experience gives th e se read ers "an internal s e n s e of the features of quality writing" (Pula & Huot, 1993, p.248). A recent study exam ined the effect of teaching experience by studying reactions of experienced classroom teach ers (with teaching experience of six to over twenty years), who had different am ounts of holistic scoring experience. T he tea ch e rs w ere divided into novice holistic raters without rubrics an d experienced holistic raters with prescribed rubrics. The experienced raters with rubrics tended to ignore the student and focus solely on th e text; novice raters without rubrics kept returning to concern for the student's abilities or n eed s a s perceived by th e raters' experience a s te a c h e rs an d through th e students' writing. R egardless of the w hether they used a rubric or not, both s e ts of rea d ers arrived at nearly the sam e sco res, suggesting that within their sh ared discourse community, that is, English com position teaching, resided the underlying rubric for scoring (Huot, 1993). Experienced ESL te a c h e rs dem onstrated a sim ilar c o n se n su s in scoring. An early study of ESL te a c h e rs who w ere provided with no guidelines found a high d e g re e of agreem ent with th ree characteristics 39 acco u n tin g for their com m on ju d g m en ts: length of product a n d gram m atical an d spelling errors. (Arthur, 1979). In m ore recent stu d ies which include ESL te a c h e rs, how ever, su rface erro rs a re th e least reg ard ed characteristic of stu d en t writing. Faculty with th e g re a te st exposure to non-native sp e a k e rs of English had a tendency to b e m ore tolerant of surface errors (Vann et al., 1991), viewing the text a s a m eans of discovering an ESL student's particular learning needs; they focused on the am ount and developm ent of id eas a s the m ost important featu res of student writing (Cum m ing, 1990). A nother study found that experience w as the significant factor for ESL teach ers who overlooked typical errors in m id-range writing and boosted sc o re s to higher levels in contrast to the sc o re s of inexperienced tea ch e rs of second-language stu d en ts (Bochner et al.,1992). R esearch h a s found that this tolerance for surface-level errors a p p e a rs to com e through experience. Ham p-Lyons (1991a) defends this product of exp o su re to NNS writing a s an ad v an tag e w hereby ESL teach ers recognize that even when content is at a m astery level, second- language writers will inevitably have surface-level problem s, including w hat sh e d e scrib e s a s "fossilized error patterns" (p.13), or w hat other research h a s called "under-utilized" or inconsistently applied editing skills (R oss et al., 1984; p. 6). Rating experience. Experience in the classroom and exposure to student writing m ay have significant effects on faculty rea d ers' backgrounds which, in turn, m ay affect scoring. Participation in holistic scoring m ay further influence a re a d e r and th e re a d e r's judgm ent in su b se q u e n t holistic scoring. For exam ple, the am ount of participation a p p ea rs to c a u se different resp o n ses to scoring. D iscrepancy rate s in holistic a sse ssm e n t vary with novice and experienced read ers, with interrater discrepancies rates a s high a s 20% for novices an d a s low a s 1% for ex p erien ced re a d e rs (F reedm an & Robinson,1982). The research p resen ts mixed findings, however, a s to the em phasis which raters place on th e com ponents of writing a s determ ined by their holistic scoring experience. Breland and J o n e s (1984) found e x p erien ced ra te rs sc o red m ore harshly an d limited their focus to m echanics m ore than novice raters. And in Cum m ing's research (1990), novice raters tended to evaluate by using only a few skills from the listed criteria while experienced raters ap p ealed to a wider range of skills. It is unclear how th ese differences affect the overall scoring. E xperience gained from participation in other form s of holistic scoring a d d s to th e variability of rate rs and, thus, to the limitations of holistic a sse ssm e n t. R esearch indicates that veteran rea d ers m ay have difficulty converting to new scoring guides from o n e s they have previously used (Baker & Quellmalz, 1979; White & Polin, 1986). On the other hand, read ers who internalized a rubric from previous scoring and w ere asked to apply th e sa m e criteria again w ere m ore efficient raters, scoring m ore quickly and m ore reliably (Keech & McNelly, 1982; Huot,1993). However, the inclusion of new re a d e rs into a group of v eteran re a d e rs p re se n ts an o th er problem . Q uellm alz (1984) found th at w hen c o n se n su s over criteria w as reached by readers,in previous scoring sessions, new read ers could not u n d e rsta n d how th e criteria h ad b e e n d eterm in ed and 41 interpreted, and a s a result, they had difficulty applying th e criteria in the sa m e m anner a s th e experienced read ers. It is probably for this reaso n that general guidelines for holistic scoring suggest that no m ore than one* third of a reader group b e new raters (White, 1985). H plistic.Scp r in g . Process R ead er background a p p e a rs to influence scoring in a num ber of w ays, but so also do a se t of variables that are part of th e holistic scoring itself. T h ese include th e scoring guide (rubric), read er training, and the scoring m anner, which includes th e sp e e d of reading and the focus in reading. B ubr i s, At m ost holistic readings, a scoring guide (a rubric) is presented or created by th e read ers, which includes a specific list of characteristics designated a s th e appropriate criteria by which to a s s e s s student writing. According to research, th e rubric en co u rag es read ers to u se the full range of possible sc o re s (White, 1985), and it aids th e rea d ers in keeping track of th e w riter's points (C h ase, 1968). T he early study by Diedrich et al. (1961) is often cited a s proof of the n eed for a scoring guide, since in their study th e lack of sh a red guidelines resulted in widely d isp arate sco res. Ninety-four percent of th e e s s a y s in this study received five or m ore sc o re s out of a possible choice of nine, dem onstrating read ers' inability to ag ree without significant support. Confirm ation of re a d e rs' actu al u se of th e rubric, how ever, is available in few stu d ies. Sm ith (1993), o n e exam ple, found that, for placem ent pu rp o ses, w hen raters had a rubric they w ere able to assig n appropriate sc o re s w hether they w ere from the sa m e school or different institutions. But m ore com m only, th e literature includes evidence that re a d e rs a re d e p e n d e n t on their own intuitive or creative judgm ents, thereby rejecting explicit descriptions of e a c h sco re level (e.g., Barritt, Stock & Clark, 1986). In an early study, Scannell and M arshall (1966) instructed re a d e rs to ju dge solely on co n ten t, a n d yet p a p e rs with gram m atical errors w ere scored lowest, without errors highest, regardless of content. Since th e u se of sam ple p a p e rs is a com m on practice at holistic readings, it is unlikely that holistic rea d ers rely solely on th e guide. While rejection of th e guidelines m ay occur frequently, it rarely occurs overtly, but such w as th e c a s e of ETS rea d ers who insisted on using the provided sam ple p ap ers instead of th e scoring guide a s their criterion for ranking, which resulted in a m uch wider range of judgm ents (C arlson, Bridgman, Cam p & W aanders, 1985). Further evidence for resistan ce to reliance on only th e rubric m ay b e found in the frequent occurrence of rea d ers putting little plu ses or m inuses next to the sco res, indicating that student e s s a y s do not alw ays fall neatly into th e stated categories. T he cracks in betw een th e specified criteria are found to be important, since this is w here read ers are left to their own discretion and their own m ethods of judging (Vaughan, 1991; Smith, 1993). R e search .attrib u tes re a d e rs' varied an d individual re sp o n se s to the fact that no rubric can ev er "specify th e entire s e t of criteria used to judge an essay. There is always an V factor which stands for how a rater interprets the rubric"(Winters, 1980; p.18). The specification of the scoring guide may provide additional explanation for this variation. Length of the description for each category can affect the readers; If, for example, one score has multiple descriptors while another has only one or two, readers will be left with the impression that the more detailed score is actually more important; this imbalance will attract in creased attention during scoring and will result in a disproportionate num ber of e ssa y s at this level (Quellmalz, 1984). Considered a calibration problem, this imbalance is a serious testing concern which jeopardizes decisions regarding students (Davidson,1991). Beyond the explicit criteria, research su ggests that readers, in general, have a "distribution mindset," meaning that they do not expect an entire stack of papers to be all at top or all at bottom score levels; readers may instead have a subconscious plan that will "force the students into the expected distribution" even when the papers do not match the criteria. In addition, different readers may have varying widths plotted for particular scores along a mental scoring continuum; that is, for som e raters, a band of papers included in a score of 3 might be quite broad, drawing many more varieties of essay s than a score of 2, which could draw a more limited sam ple (Smith, 1993; p.154). Elbow (1993) recently acknowledged the dilemma presented in the scoring of mid*range papers caused by the fact that these papers often include qualities from both strong and weak descriptors on the rubric. Through his considerable experience of leading many holistic scoring sessions, he has found that "a reader's priorities or 44 mood or tem peram ent” will resolve the quandary of the mid-range papers, but not adherence to the scoring guide (p. 194). In the interpretation of scoring guides, a common practice, that of using generic scoring guidelines, h as drawn criticism. Failure to create new guides for new tests and new topics (Quellmalz, 1984; White, 1986), under the false assum ption "that the questions and students change so little from test to test that a new guide Is simply not needed" (White, 1986, p.71), may cau se additional problem s for scoring. Not only d o es this economy indicate an insensitivity to the writers and their writing (Stock & Robinson, 1987), but a s an even more serious issue, it leaves the readers to respond to "the more superficial aspects of writing at the expense of the more substantive ones" (White, 1986; p.71). This practice of using generic guidelines also overlaps with the concern of using rubrics for NNS writing that have been normed for native speaker writing (Klammer,1983). In fact, it is this combination of problems (resulting in v agueness of criterion application and usage) that cau se s ESL teach ers to feel uncomfortable with holistic scoring, according to Janopoulos (1989). Training. Because of the variability of reader background and the number of ways in which readers may apply the guidelines, training h as becom e a comm on part of the holistic assessm ent process. During th ese sessions, read ers are trained to use the explicit criteria for rating in order to overcom e their differing expectations (Gere, 1980; Charney, 1984). But 45 training presents a confounding variable in the assessm ent process, since research h as failed to dem onstrate that the effects of training are the sam e for all re a d e rs, that re a d e rs actually u s e all th e tech n iq u es and suggestions from the training, or that the effects of training have sufficient duration for the scoring sessio n . Indeed, holistic training m ay not be sufficient; instead, professional training may be required for scoring NNS student essay s. T he literature includes studies that are supportive, in various ways, of the influence of training. Freedm an (1981) found that a few words at the beginning of a scoring session by the trainer can set the tone for the entire reading; such a com m ent may c a u se higher sco res for e ssa y s b e ca u se the training lowers rater expectations. Connor and Carrell (1993) credited rigorous training and calibrations for keeping re a d e rs in their study focused on th e criteria, which resulted in high reliabilities (Connor & Carrell, 1993). And ETS d e p en d s on five hours of training with trial scoring of 50 practice e ssa y s to produce high reliabilities for its Test of Written English (TWE), a com ponent of the T est of English a s a Foreign L anguage (TOEFL) (Bridgeman & Carlson, 1983). Training is not alw ays considered a positive influence in scoring, however. Huot (1990b) argued that it "distorts the raters' ability to m ake sound choices concerning writing ability"(p. 202). R aim es (1990) also questioned the validity of the kind of writing that "readers are trained to reward" (p.439). Training h as additionally been shown to produce different expectations am ong rea d ers from th o se which the topic prom pt d o e s am ong writers (Ruth & Murphy, 1988). For exam ple, in a study in which 46 readers w ere told to "value fluency, vitality, and force," but writers had selected safe and clear routes in their writing, the writers w ere unduly penalized a s a direct result of the training (Hamp-Lyons,1991d; p. 88). R eaders who are involved with the scoring of NNS student papers need to be specifically trained, according to the research. But the training indicated m ust consist of m ore than a few hours of holistic scoring m ethods; rather, the training should be professional. Klammer (1983) described his own difficulty a s a reader and cautioned that ESL-trained re a d e rs should read ESL p ap ers. Ham p-Lyons (1991a) a g re ed by confirming that readers perform best at levels where they are most used to reading, and Stansfield and R oss (1988) concurred, suggesting that raters of such work should be experienced with foreign students. Also stressed in the literature Is the need for readers to recognize the differences in ESL writing, differences that "may persist even after good editing skills h a v e b e e n d eveloped, particularly in testing situations"(Ross et al., 1984; p. 17). The aw areness of the persistence of such differences resulted in the su g g estio n that only language com ponents which can be taught should b e considered in scoring; this suggestion, however, requires considerable knowledge of language learning by the reader a s prerequisite to scoring (Klammer, 1983; Ross et al., 1984). Holistic training ap p ears to hold extra punitive consequences for ESL student writing if L i and L2 student work Is inter-mixed for the sam e holistic scoring. R esearch points out that, in this situation, readers must be cautiously trained to recognize high and low level facets of ESL writing in 47 order not to b e caugh t up with only low-level problem s (H am p- Lyons, 1991 a). In addition, raters without training in second-language education often simply resist NNS student writing. Kaplan voiced concern over English native readers who refuse "to interact with a text a s the result of its 'foretgn-ness',* concluding that in such case s, the NNS student writer "is doom ed to failure from the start" (1990, p. 15). R eaders resisting foreign- sounding texts have also been found to assum e that, when writing does not sound quite right to their native-speaker ears, the text is wrong instead of different (Basham & Kwachka.1991; Land & Whitley, 1989). R esearch h a s contended with the potential for altering th ese positions through different d e g re es of training. During holistic training, discussion often turns to sensitization of readers to NNS student writing, but sensitizing is limited, according to the work of Bochner et al. (1992). They argued that increasing the readers' sensitivity to certain features in L2 writing is tem porary and superficial, while actual experience with L2 writers functions a s continuous training, allowing read ers to recognize characteristic writing behaviors. Therefore, while anyone might be trained to recognize the superficial aspects of L2 writing, not everyone will be able to recognize and respond to the d eep er dem an d s presented in NNS student writers' work. Hamp-Lyons (1991c) cited a similar concern for read ers who lacked background knowledge about L2 texts and could therefore not reasonably a s s e s s content. She pointed out that no amount of short-term training or changes in the criteria would com pensate for this lack. 48 Beadipa.speed, In addition to issues of sensitization for holistic readers, another issue that h as created controversy over its potential for effectiveness is the rapid reading of student essay s required for holistic assessm ent. Early research urged a speed of 400 words per minute so that readers would not be distracted by "tangential qualities" (McColly, 1970; p. 150). Sixteen years later, this concern w as echoed by researchers who found that slower reading resulted in the creation or inclusion of additional criteria (Barritt et al., 1986). In a study commissioned by ETS (which encourages rapid reading for their testing), researchers surprisingly disagreed with the requirement for speed, since discrepancies in their holistic processes were caused by the one reader who scored most quickly and, a s a result, hovered in the center of the range, a safe but inappropriate level. They raised the concern that sp eed seem ed detrimental to accuracy and that readers should be trained to be more reflective and to slow down (Bridgeman & Carlson,1983). In addition to the argum ent by ETS against rapid reading, other research confirms the problems of speed, especially regarding the scoring of NNS student writing. Results of studies show that additional time is important for accessing NNS students1 m eaning, especially "if that m eaning is obscured by the sort of errors that [ESL] writers typically commit" (Janopoulos.1993; p,306-7), or when native speakers a s readers initially perceive the information a s inconsistent or the plan a s difficult to follow. The results confirmed that, with am ple time (as a necessary 49 elem ent of scoring), native readers could attend to the meaning in NNS writing (Meyer, 1982). S peed w as se e n also a s an impediment to the read er's engagem ent with the text (G ere, 1980), and a s creating "an unnatural environment," allowing for only superficial features to catch the reader's eye (Charney,1984; p. 74). Reading focus. The sp eed of holistic reading may contribute to the most-widely criticized aspect of this evaluation tool, the focus on superficial features. D escribed a s a focus on the whole instead of th e parts, holistic assessm en t purports to disregard or diminish the effect that individual characteristics have on student writing, but the research in the literature questions this assertion. Carlson et al. (1985) reported on the suspicion that high inter reader reliabilities "do not indicate whether different readers are evaluating the sam e features" (p. 58), and a number of studies show that readers do not evaluate by the sam e criteria and that specific characteristics generate disproportionate attention. Charney (1984) warned that holistic ratings reflect only "superficial characteristics in the essays which are easy to pick out but which are irrelevant to true writing ability" (p. 84). Such suspicions over readers' attraction to surface features have frequently been noted. The attractiveness of a writing sam ple, for exam ple, se em s to have an immediate impact on the reader, suggesting that the score may be related to the quality of handwriting. A variety of studies h as docum ented the W ade, 1978; Hughes, Keeling & Tuck, 1983). R esearch has, in fact, found 50 that good quality handwriting is responded to with high scores and carries over a halo effect to other asp ects of the paper; conversely, poor script conveys an equal but dam aging effect (C hase,1968). Handwriting w as found to be the dominant factor in scoring in a persuasive study in which, from a sam ple of 500 papers, researchers randomly selected 45 essay s and rewrote them using the Palm er Handwriting method. All the rewritten papers were evaluated as significantly better than the others, regardless of content and control (Sloan & McGinnis, 1978). Focusing on another asp ect of general ap p earan ce, research indicates that holistic sco res correlate to length; that Is, longer papers received higher scores (Nold & Freedm an, 1977; Arthur, 1979; Stewart & Grobe, 1979; Brosell, 1986; Breland & Jones, 1984; Freedm an & Calfee, 1984; Carlson et al., 1985; Hillocks, 1986). Several explanations attempt to account for this phenom enon: Arthur (1979) hypothesized it w as not the total word count alone that attracted readers' attention and contributed to a higher score, but that longer papers w ere typically more articulate and fluent, length being a by-product of these dem onstrated qualities. Nold and Freedm an (1977) agreed, citing lack of development in shorter essa y s a s responsible for lower scores. And Freedm an and Calfee (1984), explained that all mechanical writing controls must be automatic for a writer to create a lengthy work. In sharp contrast to assessm ent which commonly focuses on error counts (i.e., indirect m easures), holistic evaluation is reported to s e e beyond (or through) discrete, errors; again, how ever, th e research indicates otherwise. N um erous studies of holistic scoring have reported 51 error categories noted by raters or highlighted by statistical analyses (e.g., Harris, 1977; Arthur, 1979; Purves, 1984; Vann et al„ 1984; Zamel, 1985; M endelsohn & Cumming, 1987; Santos, 1988; Cumming, 1990). Yet a s readers rate student writing, how they perceive a particular error is not always clear (Qreen & Hecht, 1985), with the result seem ing to indicate a great deal of variation from reader to reader (Cumming, 1990; Davidson, 1991; Elbow, 1993). In addition, a variety of other separate features have been noted to influence readers' judgment during scoring; for example, cohesion (Bamberg 1984), spelling errors (Stewart & Qrobe, 1979; Grobe, 1981), vocabulary (Mullen,1980; Grobe, 1981), subordination (Homborg, 1984) and m echanics (Diedrich et al., 1961; Freedm an, 1979). The d eg ree to which surface errors or sep arate characteristics contribute to holistic sco res a p p ea rs particularly pronounced in NNS student writing. Fein's study (1980) found that, while NNS students scored significantly lower in holistic scoring and had significantly m ore grammatical errors than native English speakers, there w as little overall difference in the quality of content and organization betw een NS and NNS writing. McGirt (1984) confirmed the fact that grammatical m istakes lower scores assigned to NNS student essays, explaining, "The non-native, but not the native, can significantly raise his/her holistic rating on a composition by the elimination of grammatical errors"(p.xi). Indeed, in his study, 40% of the previously failing NNS e ssa y s w ere p a sse d after controlling for grammatical errors. Holistic assessm ent has been advocated a s a m eans to a sse ss the whole instead of the p arts of stu d en t writing, but re se a rc h h as 52 dem onstrated that th e uncertain and varied influences of read er background, holistic training and holistic scoring p ro cesses (e.g., sp eed and rubric adherence) tend to create expectations which cause readers to em phasize specific com ponents of student writing, with negative results for NNS student essay s. The fact that research h a s been able to point to separate com ponents in writing h a s been a major factor in confirming the validity of alternative assessm ent-analytic scoring. Analytic A ssessm ent Early in the developm ent of direct writing assessm en t, analytic scoring w as considered a s one category of holistic evaluation (Cooper, 1977), but a s the term holistic assessment cam e to m ean evaluation based on an overall impression, supported by writing sam ples and rubrics, the difference betw een the two types of assessm ent becam e clear. Today analytic scoring has com e to m ean evaluation of student writing through separate scales (som etim es differentially weighted) for assessin g writing com ponents, such a s content, organization, vocabulary, gram m ar, and m echanics (Jacobs et al., 1981; Winters, 1982; Muot, 1990; Cohen, 1994). Analytic a sse ssm e n t is the m ost familiar m eans of dealing with student writing on an everyday basis since classroom teachers regularly comment on or correct student writing and assign letter grades by basing their judgm ents on different features of the writing (White, 1985). For formal a sse ssm e n t purposes, the categories of such featu res w ere originally devised a s a result of a factor analysis performed on responses to what independent judges considered good writing (Deiderich et al.. 1961), and the categories of ideas, organization, wording, m echanics and flavor, from this now-classic study, have rem ained som ew hat stable over time (Huot, 1990). Jo n e s replicated the study in 1978 and concluded that these categories were indeed an accurate description of faculty responses and their implicit criteria for judging student writing. Using international readers, Purves (1984) partially replicated the original study and agreed that three of the five (organization, flavor, ideas) and a fourth factor, rater's personal response, were the dom inant influences for scoring by world wide evaluators of English writing. The sam e original five factors w ere confirmed for reliability in a 1992 study by Blok and de Glopper which found that 79 other studies had indicated similar categories for writing evaluation, with slight variations in the labels (e.g., ideas for oonfenf; wording for vocabulary). Much of the research on analytic scoring h as been provided a s a contrast to holistic assessm ent. Studies have examined the advantages of holistic scoring extensively, and while it h as popularly been considered the m ost efficient tool for direct a sse ssm e n t in term s of cost, time and reliability, the literature h a s p resen ted mixed findings about th e se purported advantages. In considering expense, Veal and Hudson (1983) found a mid-range cost of 75 cents per paper for holistic scoring; this cost com pared with a cost of 53 cents for less expensive objective tests, 58 cents for primary trait, $1.06 for more expensive m echanics scoring, and $2.37 for analytic scoring. B auer (1981) also determ ined a m id-range training tim e of 52 m inutes for holistic evaluation with su b seq u en t individual essay s read at the rate of .96 minute each, while primary trait scoring required only 44 minutes for training and 1.08 minutes per essay, and analytic scoring required 114 minutes for training and 4.14 minutes per essay. However, a number of researchers have found analytic scoring to be more reliable (e.g., Bauer, 1981; M endelsohn & Cumming, 1987; Steele, 1979; W inters, 1980). Quellmalz (1984) questioned the time requirement a s her study found the time difference per paper betw een holistic and analytic scoring to be less than one minute apart. Spooner* Smith (1980) confirmed that training and reading using analytic scoring took slightly longer, but a number of researchers have agreed with her evaluation that the information yielded by analytic methods is much more useful to educators (Jacobs et al., 1980; Quellmalz, 1980; Cumming, 1990; Hamp-Lyons, 1991c). The use of holistic assessm ent as a norm-referencing m easure for high-stakes writing tests involving NNS students and NS students has also been questioned. The test results are not m eant to be a ranking of the best writers at a single testing session but rather the confirmation of a level of writing com petence expected of students prior to graduation from the University (Klammer, 1983). Distinct from holistic scoring, analytic evaluation provides an analysis of characteristics within the writing sam ple based on strong or weak points (Mullis, 1984). For example, if the content is strong and the punctuation weak, the analytic score will reflect this discrepancy, based on the weight attached to each of the characteristics (Jacobs, et al., 1981). The misuse of com m as or articles would not alone result in a failing score, a possibility with holistic scoring which has the 55 potential for collapsing its categories and offers no sum m ative rationale for the sc o re s (Cohen, 1994). W hile providing detailed information to stu d e n ts and raters, analytic scoring h a s not alw ays b e en advocated. W hite (1985) argued that experts h a v e found little support for analyzing writing a s a collection of various subskills to be exam ined independently an d totaled; in fact, th ere is som e controversy over which subskills should b e included. W riting is often perceived by com position ed u cato rs and resea rch e rs a s a com plex whole, g re a te r th a n th e sum of its p arts. O n e perceived draw back of analytic s c a le s w a s th at th e s e p a ra te se ctio n s m ight actually draw re a d e rs ' attention to errors, thereby em phasizing th e negative a s p e c ts a s th e p a p e r's overall im pact (Najimy, 1981). However, th e overarching fear of unreliable scoring, th e prim ary fo cu s in direct a s s e s s m e n t from its beginning, increased the negative attention on analytic scoring to contend with its m ultiple su b co m p o n en ts. T h e se sectio n s, it w as feared , would require th e re a d e rs to m ake m ore qualitative judgm ents than a single holistic sco re, and could, thereby, in cre ase th e potential for rater error (Cohen, 1994). At its b est, how ever, resea rch on analytic scoring h a s recognized an d tak e n into accoun t th e c o n ce rn s of th e critics, contending that th e elem en ts of a com position "should be evaluated only for their contribution to th e total com m unicative effect...a connected, co h eren t piece of writing that com m unicates effectively" (Ja c o b s et al., 1981; p.32) A num ber of a d v a n ta g e s for analytic a sse ss m e n t have b e e n docum ented. C ategories allow decisions about ratings to be m ad e in a relatively sim ple fashion, 56 and specific results a re isolated by criteria for later u se if n e ce ssa ry or desired (Najimy, 1981). Training of raters is also highly successful and sim ple once th e categories a re explicit (Jaco b s et al., 1981; C ohen, 1994). P e rh a p s th e stro n g est support, how ever, lies in the distinctions which analytic a sse ssm e n t m akes for difficult decisions; that is, while th e u se of different sc a le s (e.g., analytic, holistic, primary trait) a p p e a rs to a s s e s s high and low writers equally, analytic sc a le s ap p ear to be the b est m eans for discrim inating sc o re s in th e middle range (W inters, 1980). R esearch h a s offered an o th er ad v an tag e for m id-range NNS writing, finding that analytic scoring is a m ore appropriate and accurate m easure since holistic scoring conceals variation (M endelsohn & Cumming, 1987). An additional credit to analytic scoring h a s b e e n found in th e opportunities it c re a te s to overcom e rea d er expectation by providing resu lts regarding the writing sam ple an d not th e read er. F reed m an 's (1984) study d em onstrated that by mixing levels an d groups of writers, read ers' expectations w ere altered. In her study, during holistic scoring, professional w riters w ere penalized since their writing co u n tered th e expectations of the read ers, but in analytic scoring re a d e rs w ere able to a ssig n appropriately high sc o re s to the work of professional w riters. Another outcom e for th e u se of analytic sc a le s h a s b een docum ented a s controlling potential halo effects (a broad judgm ent that m ay b e either positive or negative and affect additional a s p e c ts of scoring, su c h a s handwriting quality). B ecause re a d e rs a re required to m ake independent ju d g m en ts of specific c a te g o rie s of criteria, th e carryover to o th er judgm ents is limited (Jacobs et al., 1981). 57 For non-native sp e ak e rs, an additional a d v an tag e of the u se of distinct categ o ries for judgm ents is that analytic scoring "can a s s e s s stu d en ts' language proficiency and writing expertise concurrently," while holistic scoring te n d s to em phasize one over the other or to collapse both for scoring so that only the w eaker aspect prevails (Cumming, 1990: p. 43; C ohen, 1994). T he u se of analytic or criterion-only sc a le s (without norming a s is th e com m on practice today in holistic scoring p ro ced u res) m ay b e n e ce ssa ry for th e evaluation of writing by NNS stu d e n ts b e c a u se "it is th e se stu d en ts who, though they m ay never b e able to do a s well a s the m ost co m p eten t native sp eak in g stu d e n ts, c an n e v e rth e le ss often d em o n strate th e qualities of good writing that w e consider essen tial in acceptable college-level writing" (Klammer, 1983; p.6). T he a d v a n ta g e s in th e afterm ath of testing m ay b e th e m ost persuasive for advocacy of analytic scoring. C arlson (1988) n o tes that if NNS students are not provided the m eans for recognizing which skills they are lacking (the condition com m on to holistic scoring), th en they face further penalities. Analytic criteria allow for explication and for work toward com petency. Psychom etric Properties Educational te s ts using analytic and holistic a sse ss m e n t do not exist in a vacuum . Therefore, th e context in which the G raduation Writing T est (Cal Poly P om ona’s form of com pliance with the GWAR m andate) is 58 p resen ted an d the p u rp o ses for which it w as devised contribute to its reliability and validity, properties which function a s the tests of the test. Reliability Since its beginning, direct a sse ssm e n t of writing h a s g en erated concern over the ability of read ers to a s s e s s reliably, that is, to ag ree with each other and with them selves at a future tim e (Diedrich, French & C arlton, 1961; Diedrich, 1974; W hite, 1985; H am p-Lyons, 1991a; Williamson, 1993). In fact, Diedrich (1974) w arned of "how commonly and seriously teach ers disagree in their judgm ents of writing ability" (p.5), and he designated the sole purpose for his book, Measuring Growth in English (1974), a s suggesting w ays for improving reliability. As a result of this w idespread concern, th o se seeking support for direct scoring have concentrated on creating a reliable m ethod of assessm en t. Subsequently, a num ber of studies established that reliability in holistic scoring is, indeed, possible (e.g., G odshalk, Swineford & Coffman, 1966; Coffman, 1971; Bridgeman & Carlson, 1983; Brown & Bailey, 1984). Although early u se of analytic scales initially involved only one rater (Najimy, 1981), later studies on analytic scoring support similar positive reliability results (S teele,1979; W inters, 1980; B auer, 1981; J a c o b s et al., 1981; Q uellm alz, 1984; M endelsohn & Cumming, 1987). The em phasis on reliability h a s created additional concerns. First, although reliability is necessary, it is not sufficient, and a s such is not a guarantee of validity; that is, confirming th e sam e incorrect results is again only the sam e incorrect results (H enerson, Morris & Fitz-Gibbon, 1978). S econd, even w hen direct scoring is found to b e reliable, resea rch h a s indicated there is no certainty that raters are scoring for the sa m e reaso n s (C arlson, B ridgem an, C am p & W aa n d ers, 1985). Holistic scoring, in particular, h a s provoked suspicion and skepticism (e.g., Barritt et al., 1986; Stock & Robinson, 1987) by its published high reliabilities w hen the sa m e re se a rc h e rs have b een unable to replicate th e se results with the sa m e raters and sam e pap ers (White, 1984). In addition, high reliabilities for holistic evaluation hav e b e e n a sso c ia te d with superficial writing characteristics a s the main focus for readers, since it is reasonably sim ple to gain c o n sen su s for th e se characteristics (Charney, 1984; IRA & NCTE, 1994). And a third concern is that th e writing c re a te d for holistic an d analytic a sse ss m e n t is not authentic, and consequently, the reliabilities c re ate d by this scoring reflect only "im posed ag reem en ts...in artificial settings" (Elbow, 1993). This finding is particularly acu te w hen only o n e instrum ent is used for testing since it is then difficult to generalize findings to authentic contexts and other read ers. A num ber of studies have argued against the reliability of read ers with only one writing sam ple from which to judge (e.g., Deidrich et al., 1961; G odshalk et al., 1966; J o n e s , 1978; Q uellm alz, 1984; B reland & J o n e s, 1984; P urves, 1984; Stansfield & R oss, 1988). Validity In direct writing a sse ssm e n t, student writing is evaluated through actual writing sam ples, and a s a c o n seq u e n ce , it is often perceived a s 60 having a high d egree of face validity (i.e., it ap p ears to test what it sa y s it tests). But b ecau se educators m ay find face validity a s sufficient, other types of validity are often still ignored (Hamp-Lyons, 1991a; McKendy, 1992; W illiamson, 1993). A general definition of validity is that an instrum ent actually te s ts w hat it purports to test, but b e c a u se this psychom etric property Is frequently slighted in s tu d ie s of direct assessm en t, concern for test bias may also be overlooked (Barritt et al., 1986; Huot, 1990b; McColly, 1970; Reynolds, 1982; Williamson, 1993). O ver twenty y e a rs ago, research w arned of the ex cessiv e focus on reliability and the neglect of validity (e.g., McColly, 1970). Recently, Huot confirmed that this neglect in its early sta g e s is "the major reason holistic scoring is now in a vulnerable position" (1990b, p. 202). The ability to m ake appropriate judgm ents b a sed on an ag reed - upon set of criteria rep resen ts a type of content and construct validity (Ham p-Lyons, 1991a), which often lack support since th ere is little co n sen su s am ong writing professionals a s to what writing actually is or w hat writing te s ts actually m ea su re (W hite, 1985; Elbow, 1993). A greem ent, at p resen t, co n sists of beliefs that writing is "complex, m ultifaceted, affected by cognitive and affective dem ands" a s well a s environmental conditions (Hamp-Lyons 1991a, p. 10). Yet even the choice of m e a su re s against which to weigh th e validity of holistically scored exam s (i.e., criterion validity) Is unclear (Godshalk et al., 1966; Hamp- Lyons, 1991a; Williamson, 1993). In researching a num ber of scoring system s, W iggins (1993) studied school writing a sse ss o rs and despite their claim s to the contrary found that the underlying criterion for their 61 assessm ent rating w as whether the paper w as interesting, yet ail denied that this descriptor should be part of a formal scoring guide. Thus, the judgm ents which faculty m ake about student writing a re laden with variables that are difficult to isolate and m easure, and thereby, difficult to validate. However, by contextualizing a writing assessm ent instrument within its educational setting and in consideration of its purpose, two types of validity m ay b e established, according to the research . With the institution's curriculum a s the context, educational validity and content validity m ay b e evaluated. R e se a rc h e rs have long arg u ed that educational assessm en t must to be tied to the curriculum (IRA & NCTE, 1994). Educational validity, therefore, extends beyond "numerical test results [to] the larger educational results, including the effect of a testing program on curriculum and m aterials design, teacher developm ent and student study patterns"(Horowitz, 1991; p. 82). Oiler (1979) stated, "The validity of te s ts in education m ust be referenced against the skill, perform ance, ability, or w hatever the educational program purports to instill" (p. 404). And C ronbach (1988) confirmed, "The worth of an instructional test lies in Its contribution to the learning of students working up to the test, or to next year's quality of instruction" (p. 5). Therefore, a related issue of content validity is determ ined by whether a test m easures the knowledge and skill that subjects might be expected to p o ssess and dem onstrate in that context (Allaei & Connor, 1991). T h ese sa m e requirem ents apply to tests, such a s the Graduation Writing Test at Cal Poly Pom ona, used to fulfill the S tate University m andate for student 62 writing com petence certification. The GWT offers the implicit assum ptions that the assessm ent reflects the curriculum and that the students are provided learning opportunities to p a ss the test, but the educational and content validity of the test in its context are unconfirmed. Re-examination of the GWT "Tests them selves should be te s te d ," White (1985; p. 75) urged in considering the num ber of versions of holistic assessm en t used by separate cam puses of the California State University system. Even though reliability issues (i.e., the degree of consistency within ratings and among raters [Hamp-Lyons,199la]) have been the major focus of holistic and analytic assessm ent procedures from their inception through the present, the educational and content validity of such scoring (i.e., the degree to which a test m easures what it purports to m easure), specifically in the interpretation and application of scoring criteria by faculty raters, has remained largely unconfirmed (Henning, 1991). In addition, the growing number of university student writers who are non-native sp eak ers of English has cau sed recent research to question the valid use of holistic scoring for this student population, (e.g., Huot, 1990a,b; Carlson, 1991; Hamp-Lyons, 1991a; Jo h n s, 1991a; Janopoulos, 1993; Williamson, 1993), such as, the process currently used to a sse ss native and NNS student writing on the Graduation Writing Test. O ne aspect addressed in validation exam inations of tests is a concern for test bias, defined a s "...a constant or systematic error...usually due to group membership...which occurs in the estimation of a score on 63 a[n]...educational test or perform ance criterion" (Reynolds, 1982; p. 199). In recognition of the potential for error, including bias, a num ber of researchers have concurred with W hite's advice in advocating frequent and on-going evaluations of large-scale testing program s to prevent "slippage" that may occur from the standards set in early stag es (Carlson, 1991; p. 303), to confirm sensitivity to changing dem ographics of student populations, which may require revision of original standards and criteria, and to prevent test bias resulting from insensitivity to changing student populations (White, 1985; Carlson, 1991; Hamp-Lyons, 1991c; Henning, 1991; Vaughan, 1991). To becom e vigilant surveyors of test validity and reliability, California S tate University cam pus faculty and adm inistration were ordered to maintain on-going evaluation of the first of the Writing Skills Tests, the English Placem ent Test (EFT); however, no such requirement or process w as part of the Graduation Writing A ssessm ent Requirement (GWAR) (Executive Order 514 ,1987). A 1988 review of the GWAR found that, of the nineteen cam puses screened, six, including Cat Poly Pomona, had no evaluation m ethods in place for their graduation testing instrument (Borowiec). "All writing a sse ss m e n t instrum ents d ecay over tim e-an d m aintenance is essential" (Hamp-Lyons. 1991c; p. 328). Summary Compliance with the Graduation Writing A ssessm ent Requirement is typically met, on eighteen of the twenty California State University cam puses, with student essay s which are holistically scored. While a considerable body of research has exam ined the holistic applications of criteria by various types of raters, the studies have confirmed that holistic scoring has a major flaw caused by its inconsistency across raters a s they apply the scoring criteria. Much of the research indicating variability in rater response includes reader-background studies which sug g est that differing asp ects of background constitute a variable which may not be overcom e by holistic training, and which may negatively affect the increasing num ber of NNS students. Analytic scoring, on the other hand, offers explicit criteria and may allow investigation of holistic raters' criterion application by comparison. Although most of the research on holistic assessm en t has focused on the writing of native speakers of English, the special concerns of NNS student writers, especially in contexts such as those represented by the California S tate University, and for purposes such a s the Graduation Writing A ssessm ent Requirement, need to be addressed. This research sought to confirm that due to rater variability, holistic scoring is not whole, and as a major consequence of raters' em phasis on specific co m p o n en ts of writing, it p en alizes NNS stu d e n ts in a disproportionate an d , th erefore, b iased m an n er. F urther, sin ce researchers warn of validity concerns in te sts which are disjointed from teaching, this research attem pted to dem onstrate the impact which lack of content and educational validity caused by this disjunction may hold for NNS students in holistic writing assessm ent at the University. CHAPTER III METHODS AND PROCEDURES 65 Organization of the Chapter This chapter details the three research p h a ses used to com pare two sets of criteria applied to the writing of non-native-Englistvspeaking (NNS) students at California State Polytechnic University. Pom ona. The first phase presents the development and distribution of a questionnaire instrument designed to collect criteria which faculty u se in academ ic coursew ork for NNS student writing evaluation. The second p h ase explains the creation of an analytical rubric used to operationalize the academ ic coursework criteria derived from the questionnaire responses for comparison with holistic criteria used on the Graduation Writing Test (GWT). And the third p h ase traces the acquisition and selection of previously scored GWT student essays, explains the selection of essay raters, and discusses the scoring procedures for applying two se ts of criteria to GWT essays. Also included are the quantitative and qualitative methods used to analyze the collected data. P hase I: The Questionnaire Development Due to the large num ber of faculty (N=796) to be surveyed at California State Polytechnic University, Pom ona (Cal Poly Pomona), a questionnaire instrum ent provided the m ost efficient m eans of data collection. The specific nature of the requested information on criteria used for NNS student writing assessm ent in regular academ ic coursework at 66 the University required the researcher to develop the instrument based on information provided in the literature and from discussions with composition, linguistics, and TESL professionals, a s well a s faculty who score the GWT. Twenty-eight questions sought information from the Cal Poly Pomona faculty in the following areas: faculty background; aw areness of NNS students in academ ic courses; the actual use of writing for coursework; and methodology and belief system s regarding NNS students and their writing in university courses. The questions, therefore, requested information that would reveal explicit and implicit criteria used by faculty to a sse ss NNS student writing in regular academ ic courses. (See Appendix A for copy of the questionnaire.) At Cal Poly Pomona, faculty most commonly refer to students for whom English is not a native language a s ESL or English as Second Language students, whether these students are actually enrolled in ESL courses or not. Therefore, the questionnaire w as entitled "ESL Writing Survey," and for purposes of discussion of the survey, the term ESL will be used to refer to students who are otherwise referred to in this research a s non-native speakers of English (NNS), The first section of the questionnaire asked for specific information on faculty background, requesting departm ent and rank a s optional. (Previous coding supplied departm ents and ranks.) Each respondent's professional information w as requested since an individual's background has been shown to affect grading of writing, (e.g., Vann et al., 1991). The departm ent information w as also important in order to confirm that all 67 surveys had been distributed, no group of surveys had been misplaced, and that a representative percentage of each departm ent w as included in the total response. Native language w as additionally requested since research show s faculty language background may correlate with expectation level and grading of non-native speakers of English (Santos, 1988). The subsequent three questions asked for the respondents' teaching background in number of years, levels of teaching, and number of years at Cal Poly Pomona. Again, research indicates that types and amount of experience may affect scoring (Vann et al., 1991). The sim ple form at in the beginning section p repared the respondents for the remainder of the survey instrument a s it w as possible to respond to the entire questionnaire simply by checking options in boxes. Although eight of the questions also included blank lines for additional explanation, it w as assum ed that few questions would generate open- ended responses. Questions 1 through 3 asked for faculty aw areness of ESL students in classes. T hese questions were designed to confirm the representative distribution of actual respondents with that of the university a s a whole. Therefore, faculty were asked if ESL students were in their classes, how frequently, and a s what percentage of the classes. Questions 4 through 6 dealt with trends that affect student writing. Question 4 sought to discover if on-going and recent events, such a s the cam pus-wide W riting-Across-the-Curriculum Program , other forms of faculty development with writing em phasis, or California State University budget cuts and/or increased class size, had impacted the amount of 68 writing regularly assigned in courses. If so, these conditions could affect writing opportunities for ESL students. Therefore, faculty were asked if they use writing more frequently, less frequently or to the sam e extent a s they did two to three years ago. Another important opportunity w as considered in the following question. Because the Graduation Writing Test does not allow computers or word processors (except under special circumstances, e.g., for certain disabled students), Q uestion 5 sought to discover the d eg ree of dependence on technology that regular coursework builds into student expectation and practice, which may act a s a limiting factor in the performance of ESL students on the GWT. In addition to technological trends affecting teaching, the current trend in composition has focused on the process of writing, that is the creation of multiple drafts, emphasizing stag es of writing instead of the final product, which is the sole concern of writing examinations such a s the GWT. Therefore, Question 6 asked if this trend also underlies work in other disciplines in order to se e if the 75-minute product required by the GWT deviates from general university coursework expectations. Questions 7 through 12 asked pairs of questions about general writing practice. Split into lower and upper-division contexts, the questions were purposefully worded in a similar m anner to simplify reading and response (Reid, 1990). Upon reaching their junior year (completion of 135 quarter units), students are expected to register for and take the GWT. Because lower-division courses precede this limit yet often do not require lengthy writing assignm ents, w hether or not work in th ese co u rses supports GWT preparation w as a concern; in addition, the amount and frequency of writing in both lower* and upper-division coursework were thought to provide NNS students opportunity for guided writing which could be reflected in GWT essay perform ance. Therefore, Questions 7 and 8 asked for the num ber of writing assignm ents of any sort for these levels. Q uestions 9 and 10 sought m ore detailed information on the number of assignm ents exceeding 250 words, the approximate length of a short essay . The word am ount stipulated in this pair of questions attempted to distinguish between assignm ents or examinations which use short answ ers (e.g., 25 words or fill-ln-the*blank responses) and more developed written work. Questions 11 and 12 asked for similar frequency data but em phasized writing under a time constraint, a serious concern for ESL students, and one which affects all students on the 75*minute GWT. O nce the individual faculty m em ber detailed his/her general standards and expectations for course writing assignm ents, the following questions, 13 and 14, requested information on faculty attitudes, specifically whether the respondents believed that non-native speakers of English (NNS) should be required to m eet the sam e academ ic standards for English usage a s native speakers of English (NS) and then whether faculty grade NNS students by the sam e standards a s they use for NS students. T hese questions attempted to uncover the general attitudes and grading policies used for ESL students across the university, since these students must com pete on the GWT m eeting the sam e criteria a s do native speakers of English, regardless of length of residence or any other variables which may affect written production in English. Previous 70 research (Bridgeman & Carlson, 1983) asked a similar question of faculty and found all academ ic departm ents, except one (Psychology), reported faculty m em bers who a sse sse d ESL students more leniently than native English speakers. This researcher believed that, should the sam e results be indicated by faculty on this cam pus, questions regarding faculty and student expectations and student opportunities for appropriate writing practice would need to be raised. Q uestions 15 and 16 w ere developed a s detailed lists of criteria used for assessm en t (compiled from the work of Deidrich et al., 1961; Jo n es, 1978; Quellmalz, Capell & Chou, 1980; W inters, 1980; Jaco b s et al., 1981; Purves, 1984; Brown & Bailey, 1984; W hite & Polin, 1986). Using a Likert-type scale, faculty w ere asked to check their frequency (always, often, sometimes, rarely, or never) of correcting or commenting on twenty specific a re a s of ESL student writing, su ch a s content, punctuation, or vocabulary usage. The results would reveal the criteria which faculty consider worth their time and effort for grading, that is, their actual practice. The early study by Diedrich et al. (1961) confirmed five factors (re-confirmed by Jo n es, 1978; Jacobs et al., 1981; Brown & Bailey, 1984; Purves, 1984; White & Polin, 1986; Blok & de Glopper, 1992) which created a basis for the selection of twenty criteria, each factor listing four criteria separately. The five factors for the newly developed rubric were pre-set a s C ontent, O rganization, Vocabulary, L anguage U se, and Mechanics (Jacobs et al., 1981). While Question 15 asked.for practice, Question 16 asked for similar criterion information but with an em phasis on beliefs; that is, whether 71 faculty valued certain criteria even if they did not mark them on student papers. This question included the sam e five factors with two general criteria listed for each, and the resp o n se option provided fell on the positive or negative sides of "Important" (very, somewhat, not very, not at all). B ased on prior research (Bridgeman & Carlson, 1983; Horowitz, 1991), Question 17 asked faculty which m odes of writing are used for coursework. The GWT most often u se s personal expository prose with elem ents of description, narration, and argum entation. Since research indicates that other disciplines rarely require the personal exposition mode and that this m ode m ay provide additional problem s for NNS students (Johns, 1991a), this study sought to discover the general expectation and practices across this cam pus in order to com pare or contrast with the GWT usage. Q uestions 18 through 21 w ere aim ed at uncovering faculty aw areness of ESL student English language problem s in class and for coursework. R espondents' attribution of the assu m ed c a u se s for any problem s w as thought potentially to provide information about faculty attitudes. And the last question on concerns, Question 21 asked how faculty en co u rag e or provide a ssista n c e with ESL student writing problems. The list of possible responses included information on student- a ssista n ce opportunities at Cal Poly Pom ona and w as additionally provided a s suggestions for faculty who m ay have been unaw are of available aid and tutoring on the.cam pus. T he closing inquiry, Q uestion 22, ask ed for participation in the scoring of the GWT. The responses to this question w ere thought to reveal possible additional information about the criteria in Q uestions 15 and 16 a s well a s overall attitudes about the need for writing. Since the GWT is a cam pus-wide test, the resp o n ses in num ber of faculty actually involved in its scoring could reflect the d e g re e to which writing is perceived a s n ecessary acro ss the disciplines in all academ ic courses, the d eg ree to which faculty sh are concern about certifying student writing com petence, and the potential effects of backw ash from the test scoring to academ ic coursew ork. Additionally, b e c a u se all faculty at this university m ay volunteer to score the GWT, all are potential readers. Since research h as indicated that with frequent scoring/grading of student writing, readers' e m p h ases shift from content or comm unication to surface-level controls (M arkham, 1976; Breland & Jo n es, 1984; Quelimalz, 1984), a concern in this research w as that those who often (up to four tim es a year) grade the GWT, the English com petency exit exam , m ay have different expectations and standards for non-native sp eak ers of English than those who do not regularly g rad e it. T his question on GWT scoring participation w as intentionally placed at the end of th e survey in order to avoid coloring resp o n ses to the other questions with a GWT focus instead of a focus on classroom practice and issues, b ased on the researcher's assum ption that the two m eans of addressing NNS writing are not the sam e. 73 Pilot As th e initial s ta g e of th e study, a pilot qu estio n n aire w as adm inistered at California S tate Polytechnic University, Pom ona, to seven faculty m em bers from three colleges, Agriculture, Arts, and Engineering, who rep resen ted six different d ep artm en ts1, five held doctorates, two m asters; four w ere full professors; three w ere lecturers. Two had never scored the GWT, two had participated betw een two and five tim es, and three had nearly alw ays scored the test. (No reader had scored at every reading, but a num ber had sco red m ost of the tim e.) Six had been teaching at Cal Poly for more than ten years; the seventh, for nine years. Individual discussions with the pilot respondents uncovered specific problem s with clarity and form at in the survey. T he am ount of tim e n ecessary for completion w as also discussed a s a contributing factor for potential resp o n se rate to the general survey distribution. The resp o n ses to m ost of the questions and their wording w ere generally favorable, but the instrum ent w as revised in three minor w ays to accom m odate pilot participants' concerns. First, Question 5 asking for com puter u sag e w as am ended to add "typewriter" a s two resp o n d en ts did not understand w hether com puter program m ing or the print technology for form atted presentation w as involved. In addition, the questions about the num ber of written assignm ents w ere also se e n a s difficult by those faculty m em bers who did not offer any writing in lower-divislon courses, thus the addition of "Not Applicable" to the choices in Q uestions 7 through 12. Last and p erh ap s m ost important, the unanim ous judgm ent of all respondents w as that the reversal of headings on the criteria questions 74 (15 and 16) w as confusing and unnecessary. Consequently, the headings (always, often, sometimes, rarely and never, very, somewhat, not very, not at all) w ere altered to b e presented in the sam e order for alt twenty criteria listed in Q uestion 15 and for all ten in Question 16. The potential concern that so m e faculty m em bers might simply check all of one colum n w as outw eighed by the fact that four of the seven faculty m em bers in the pilot group, who w ere ask ed to consider carefully all a sp e c ts of th e survey instrument, initially reversed the order of their resp o n ses b e ca u se of what they d e em ed a "tricky" part. Only individual d isc u ssio n s with the respondents allowed the research er to discover that the resp o n ses had b een reversed. Since this would have b een impractical with th e large sam ple to be surveyed, the format w as simplified. T he last asp ect considered w as the am ount of tim e n eed ed to com plete th e questionnaire. All faculty responded that it took betw een eight and ten m inutes, an acceptable length of time for busy faculty in fall quarter. (For Pilot Survey results, s e e Appendix D.) Analysis B e ca u se th e pilot questionnaire w as scrutinized for potential corrections and clarifications applicable to th e subsequent main survey, only frequencies and percentages of resp o n ses w ere totalled. Reliability of the resp o n ses to the general questionnaire distributed to the entire faculty w as confirmed by com parisons with pilot totals. PfetrM iofl After ch an g es b ased on the pilot resp o n ses w ere com pleted, a cover letter requesting faculty participation w as provided by the Vice President of Academ ic Affairs for the cam pus-w ide distribution of the questionnaire. (See Appendix P for copy.) As a sta te university of m oderate size, Cal Poly Pom ona is com posed of six colleges (Arts, Agriculture, B usiness, Engineering, Environmental Design, Science) and one school (Hotel and Restaurant M anagem ent). To gather current information on faculty assignm ents, rosters w ere collected early in the fall quarter from each of the colleges and school. Only faculty teaching academ ic c o u rses in that term were included in the survey; teaching assistan ts w ere not included since for many fall quarter is the first opportunity to teach, and it w as felt that they may not yet have the experience to respond to questions about their own classroom practices and beliefs; faculty on leave, in administrative-only positions, or on off-campus assignm ents were not included. The questionnaire w as devised a s a two-page, two-sided canary yellow form with the attached cover letter from the Vice President of A cadem ic Affairs. A total of 796 surveys w ere hand-delivered in individually addressed manila envelopes by the researcher to each of the forty-seven departm ent offices on cam p u s with oral req u e sts for distribution to individual faculty mail boxes. (S ee Appendix H for list of departm ents.) R esp o n se through cam pus mail w as encouraged by attached, self-addressed envelopes directed to the research er's on- 76 cam pus office. Two w eeks tater, a voice mail follow-up requested additional response. (See Appendix Q for text.) Analysis The resp o n se s to the questionnaire w ere analyzed through descriptive statistics, including frequencies and percentages of responses. In addition, cross tabulations were conducted on select questions in order to determ ine additional relationships betw een faculty responses. (See Appendix K for contingency tables.) P hase II: The Analytic Rubric The tabulated results of the returned questionnaires provided the necessary data to devise the assessm en t instrument for the subsequent phase of the research project, the analytic rubric. This scoring guide w as necessary for operationalizing the criteria which faculty use for regular academ ic coursework, in order to com pare them with the general holistic criteria used for the Graduation Writing Test. Five Factors Through analysis of resp o n ses to Question 15, a new scoring guide, an analytic rubric, w as created. The survey had asked faculty to respond on a Likert-type scale to their frequency of comm enting on or correcting ESL student writing using twenty criteria which previous research h as determ ined to belong to one of five factors: Content, Organization, Vocabulary, Language Use, or M echanics (Deidrich et al., 77 1961; Ja c o b s et al., 1981; Blok & de Qlopper, 1992). T he criteria for the factors w ere presented in an alternating pattern on the questionnaire to avoid grouping re sp o n se s that would not clearly discrim inate betw een choices and degree. For exam ple, criteria from the Content factor w ere listed a s every fourth or fifth choice: a) id eas and know ledge, e) original Ideas/factual support, i) detailed developm ent, and m) topic identification. The criteria from the other four factors w ere distributed in a similar pattern. Criteria for the O rganization criteria included: b) seq u en c in g , f) co h esio n , j) length/brevity, q) organization (incl. beginning, m iddle, ending); for Vocabulary: c) word deletion &/or unnecessary additions, g) register (level of voice, e.g., formal or informal tone), n) w riter's purpose/audience a w are n ess, r) appropriate word/idiom u sag e; for L anguage Use: d) preposition &/or article (a, an, the) u sag e, k) sen ten ce construction, o) gram m atical control (e.g., agreem ent of subject/verb/pronoun), s) verb u sag e (tense, modal, auxiliary, gerund, infinitive); and for M echanics: h) punctuation, I) spelling, p) capitalization, and t) form/format/handwriting. B ecause the purpose of the detailed inquiries for criterion u sag e w as to determ ine which criteria faculty actually u se to evaluate ESL student writing in coursework, only resp o n ses to Always and Often were included to com pute totals for the analytic rubric. The midpoint in the scale from Q uestion 15, Som etim es, w as considered neutral, with Rarely and Never a s negative; therefore, th ese three frequency levels w ere not used to com pile faculty criterion application. The factor w eights for the new analytic rubric w ere m athem atically derived from colum n sco re totals 78 weighted at 1.0 for Always and 0.5 for Often. Overall percentage of each factor w as rounded off to determ ine the resulting rubric. (S ee Appendix L for factor totals.) The new analytic rubric included the following five factors and percentages: Content 25% Organization 20% Vocabulary 15% Language Use 20% M echanics 20% Based on previous research (Jacobs et al., 1981), each factor w as designated levels to assist readers in scoring. High, middle and low levels w ere labeled, "Excellent to Very Good," "Good to Average," and "Fair to Poor," and the middle level within each factor w as designed to divide num erically at th e midpoint in its range. For exam ple, the factor of Organization had a range of 0 to 20 possible points so that the middle level of "Good to Average" included score possibilities of 7-13, covering the midpoint score of 10, with "Excellent to Very Good," including 14-20, and "Fair to Poor" including 0-6. (S e e Appendix O for detailed analytic rubric.) Cut-off Score Using the Graduation Writing Test cut-off score of 6 and below a s a failing score and 7 and above a s a passing score on a cumulative 12-point scale, the cut-off for the analytic scale w as se t at the sam e midpoint for pass/fail: The com bined 200-point scale w as set at failing with 50% (i.e., 79 100 points) and below and passing at over 50% (i.e., 100.5 and above). Thus, approximately the sam e proportions for pass/fail w ere maintained. P h ase lit: Re*scoring of the Graduation Writing T est The completion of the new analytic rubric b ased on faculty criteria for academ ic coursew ork led to the third p h ase of the research, which w as the re-scoring of previously-scored G raduation Writing T est essay s. The re-scoring w as conducted in o rd er to com pare faculty read ers' application of the two s e ts of criteria, th o se m ade explicit by faculty through their questionnaire re s p o n se s to criteria u se d to ev alu ate coursew ork writing of NNS students and compiled a s analytic criteria, and those used on the GWT which are broad holistic criteria. The com parisons consequently provided d a ta to calculate differences in pass/fail ratios for NNS student e ssa y s by the application of the analytic and the holistic assessm en t instruments. E ssav Acquisition In order to obtain th e n e ce ssa ry e ssa y s for rea ssessm en t, an adm inistrator from the C hancellor's Office assisted in gaining perm ission for anonym ous e ssa y s to be used for research purposes. B ecause no reco rd s a re m aintained a s to te st-ta k e rs' lan g u ag e background in connection with their essa y s, it w as not possible to identify which e ssa y s w ere written by non-native sp e a k e rs. T he T esting Office, how ever, estim ates that approximately 50% of the test-takers are NNS students at any given test administration; therefore, over twice th e num ber of e ssa y s needed w as requested in order to allow a panel of experts to judge which w ere authored by NNS stu d en ts and to obtain the eventual 100 NNS student essay s that would cover the entire range of scoring possibilities. The request for e ssa y s by sco res em phasized the m idrange a s it is considered the m ost problem atic score range for NNS student writers due to several reasons: 1.) E ssa y s which fall in the m idrange are often m ost difficult for read ers to a s s e s s since they usually contain characteristics of high and low levels of writing (Hamp-Lyons, 1991c; Elbow, 1993); 2.) The GWT rubric's broad descriptor sta te s that the m idscore (3) applies to "papers...m arred by m ore than a few minor gram m atical inconsistencies," thus, creating a band for scoring assignm ent that is wider than other bands, resulting in uneven calibration of the assessm en t instrument (Davidson,1991), thereby potentially drawing a disproportionate num ber of NNS essays; and 3.) Even though the 3 score ap p ears to be a bottom-half score, it is actually th e m idpoint d u e to th e h y p h e n ated 4 -5 single sco re; consequently, the assignm ent by two raters of the 3 score, even though a midpoint in the overall range, constitutes a failing score. T he Testing Office complied with the request and provided copies of e ssa y s with the following cumulative scores: 3-3 combination (Score 6) 120 essa y s 3-4 (Score 7) 40 essay s 3-2 (Score 5) 40 essay s Other sam ple essay s requested included the ends of the scoring range to complete the distribution: Scores totaling 2, or 3, or 4 20 essay s Scores totaling 8, or 9 ,1 0( 11,12 20 essay s Although essays from the extrem e ends of the scoring distribution, (i.e., 2, and 11 and 12) were requested, none w as included, and only one 3 paper w as offered. Therefore, the distribution of provided essay s ranged from 3 to 10. All test papers w ere from a single test administration and w ere written in response to the sam e topic prompt, one which would generate personal expository prose. (The specific topic is not provided in this research a s topic prom pts are held in confidence for possible u se on future tests.) Panel Selection of NNS E ssavs Even though the Testing Office requires language background statem ents by each student prior to taking the GWT for the Testing Office's follow-up statistical analyses, th ese self-report statem ents are immediately destroyed after the test is scored and recorded and w ere, therefore, not available for assista n ce in this research. In order to determ ine which essa y s were written by non-native speakers (NNS) and which were authored by native speakers (NS), three specialists in English a s a second language (not included in other ESL-specialist activity of re scoring essay s) w ere asked .to participate a s a panel for judging. Independently, they read the e ssa y s and then assigned papers to stacks 82 of either NNS or NS. Discrepancies were not adjudicated; any discrepancy resulted in the essay not being included in the scoring. (See Appendix N for panel results.) In addition, b ecau se few essay s written by NNS students in the upper range (i.e., scores of 8, 9 ,1 0 ) were m ade available by the Testing Office or were identified a s NNS-authored by the ESL panel, essay s judged to have been written by native speakers were used to complete this end of the distribution (i.e., eleven NS essays with scores of 8 ,9 and 10). The final corpus of 100 previously-scored GWT essays included 89 essays judged to have been written by non-native-speaking (NNS) student writers and 11 essay s judged to have been written by native-speaking (NS) student writers. Random Assignment to Batches within Distributions Selected NNS student essays and necessary additional NS student essays were coded and randomly assigned to one of four groups, W, X, V, and Z. The codes were such that the readers were unaware of the original score. Each batch included a range of essay s m eant to approximate a normal distribution with the greatest em phasis on the 6 score and the two adjacent scores which included the score of 3 (3+2=5; 3+4=7) a s part of their cumulative score. Each batch of 25 essay s included the following range of previously-scored essays: 3 essays with scores of 3 or 4 5 essays with the score of 5 10 essays with the score of 6 4 essays with the score of 7 3 essays with scores of 8 ,9 , or 10 83 H e a d e r s Reader. Selection Since research indicates that readers1 backgrounds, including professional and holistic scoring experience may affect readers1 use of criteria for scoring students essay s (e.g., Brown, 1991), two groups of readers from the University were invited to participate in re-scoring the previously-scored GWT essays. The first group w as purposely selected to represent readers who regularly score the GWT but have no special expertise with NNS student writing and the second group w as purposely selected to represent readers who regularly grade ESL writing, the two areas of concern to this research. The first group Included four faculty m em bers who have regularly scored the Graduation Writing Test (more than 6 times each) but have no special expertise in ESL composition. All four held doctorates (one in Spanish, one in Reading, and two in Composition/Rhetoric); two were full professors, one an associate professor, and one a lecturer. This group is labeled GWT Raters in this study. The second group of readers included four ESL specialists, that is University faculty who have regularly taught ESL composition courses; three held m aste rs degrees in Composition/Rhetoric and /or TESL and one had a doctorate in Linguistics; three were lecturers, one an associate professor. All were experienced holistic readers, but only one had scored the GWT. This group is labeled ESL Specialists in this study. 84 Directions to R ead ers Based on findings in the literature which confirm common faculty background a s sufficient for norming scoring and due to California State University faculty's heavy teaching schedules, no group reading w as attem pted. Instead, the researcher met individually with each reader and discussed the necessary step s of the process in order to assist and confirm independent scoring. As supplem entary rem inders, directions were detailed in a cover letter to each reader. (See Appendices O and R for copies of two versions.) In addition, scoring and recording of the scored data were simplified by the attachm ent to each essay of either a scoring sheet with 1 to 6 for holistic scoring or a brief analytic rubric for marking appropriate score ranges for each factor (See Appendix P for copy of brief analytic rubric.) GWT Rater Analytic Scoring The GWT Raters were faculty who regularly or frequently score the GWT. Consequently, they were not asked to re-score holistically since it was assum ed that their reading would only confirm prior scoring. With the newly devised analytic rubric, the GWT Raters were each asked to score two stacks of essays, (i.e., W, X, Y, and Z batches) with 25 essays per batch for a total of 50 essay s per reader. (See Appendix S for reader configuration.) No calibrating range-finder e ssa y s w ere available for this group and this stage of the study. Directions, both oral and written, preceded the independent scoring, and the suggested time limit for completing the task w as two w eeks. The two-week limit w as 85 * determined to allow readers sufficient time for the additional reading and scoring within their busy teaching schedules and to maintain the research project schedule. Two of these readers completed the scoring in two days; the other two readers required extensions to a total of four weeks from the time they received them. ESL Specialist Holistic Scoring As research indicates that ESL specialists may be more sensitive to ESL writing problems, a second group of readers w as included to se e if their reading would be comparable to the GWT Raters. ESL Specialist readers were asked to read and score duplicate essays with two different evaluation instruments, totaling 100 essays per reader for holistic and analytic readings. (See Appendix S for reader scoring configuration and inter-reader reliabilities.) The first reading involved holistic scoring of the previously-scored GWT essays by using the GWT holistic rubric. Range-finder essays were available to assist readers in norming their scoring to begin, just as is done during the GWT readings. Directions, both oral and written, confirmed the guidelines. Although readers were allowed two w eeks to complete the scoring, all readers related that, once they began, they completed the work of reading 50 essays each within a maximum of two days. ESLSpeciaiisLAnalytic Scoring One month after their holistic scoring, ESL Specialist readers were asked to read the sam e stacks of essays again, but this time with the 86 newly devised analytic rubric (50 essays per reader). Again, a s with the GWT Raters* analytic scoring, no calibrating range-finder essa y s were available for this stage. Directions, both oral and written, again preceded the independent scoring and a two-week time limit w as again imposed. Three readers returned the batches within one week; the other required more time, returning the batches three weeks after receiving them. Time Concern The three readers (one ESL Specialist and two GWT Raters) who required extra time explained that their heavy schedules (e.g., committee and administrative duties combined with teaching responsibilities) required the extension and not the actual essay scoring. In fact, the ESL Specialist who required the extension said that the analytic scoring was particularly rapid once she w as normed to the new rubric, and the analytic scoring took her no longer than holistic reading. Therefore, the suggested time limitation appeared to have had no effect on the raters. Analysis Quantitative analyses were used to com pare holistic scoring and analytic scoring for both GWT Raters and ESL Specialists. Correlations were used to com pare overall scoring with both groups of raters and to determine interrater reliability coefficients. Simple linear, multiple, and stepw ise regression analyses were used to com pare holistic and analytic criteria with both groups of raters and to determ ine which analytic criteria m ost influenced each type of scoring. In addition, paired M e sts analyzed th e statistical significance betw een groups and scoring m easures. Chi sq u a re w as u sed to a s s e s s th e difference in pass/fail ratios b etw een groups, and descriptive statistics w ere u se d to com pare th e applications of analytic scoring factors and differences in pass/fail ratios for each rater group. 88 Notes: 1. D epartm ents rep resen ted by respondents to the pilot questionnaire included Animal and Veterinary Science, T eacher Education, English (one from the Rhetoric Program and one from English-as-a-Second*Language Program ), M usic, Civil E ngineering, an d Electrical an d C om puter Engineering. 89 CHAPTER 4 RESULTS Organization of the Chapter In order to present com parisons of two types of assessm en t criteria used at California S tate Polytechnic University, P om ona (Cal Poly Pomona), Chapter 4 is divided into three sections representing each of the three ph ases of the research project. The first section provides data on the criteria faculty use to a s s e s s non-native*English-speaking (NNS) student writing in regular academ ic coursework by detailing the results of a faculty questionnaire, including frequencies, percentages and cro ss tabulation results. This part divides the resp o n se s into three categories: 1) The sam ple, including faculty background; 2) Faculty aw aren ess of NNS students; and 3) Faculty practice and beliefs regarding NNS student writing. The second section presents the ways in which regular academ ic course criteria are different from and simitar to criteria used to a s s e s s the Graduation Writing Test by explaining the results of Question 15 of the survey. The m eans by which th e se criteria w ere then converted to an analytic rubric for re*scoring GWT e s s a y s are also provided. The com parisons of criteria are presented through quantitative and qualitative analyses of the GWT holistic rubric and the analytic rubric. The third section reviews the changes in pass/fail ratios for NNS student e ssa y s from the original GWT holistic scoring to the results provided by using analytic scoring, b ased on a composite of criteria used to a s s e s s student writing in regular academ ic coursew ork. Both 90 quantitative and qualitative findings from holistic re*scoring and analytic scoring by both GWT Raters and ESL Specialists are presented. P h ase I: The Questionnaire The first research question for this study asked: W hat criteria do faculty at the University u se to m easure NNS student writing in their academ ic courses? Studies cited in the literature indicate that faculty u se both explicit and implicit criteria to a s s e s s student writing. Therefore, ail resp o n ses from the questionnaire, including information garnered on respondent background, attitudes, beliefs, and practices, yielded pertinent data for recognizing the criteria used by faculty to m easure NNS student writing in academ ic coursework. Findings are sum m arized in three categories. The first includes Questions 1A through 6A and 22 to present the respondent background. The second category introduces faculty aw areness of ESL students and related issues through resp o n ses to Q uestions 1 through 3, 18, 19, and 20. And the third section describes faculty practice and beliefs regarding ESL student writing in three clusters. Practices are detailed in the first two clusters: Q uestions 4 through 6 on m iscellaneous practices, and the second cluster, Questions 7 through 12,17, and 21 on class assignm ents; the third cluster, paired Q uestions 13 with 14 and 15 with 16 in order to com pare practice with beliefs. (All resp o n ses to the questionnaire listed a s frequencies and percentages ar.e included in Appendix B.) The questionnaire material refers to NNS students a s ESL students b ecau se faculty at Cal Poly Pom ona generally use the term English as second language or ESL students to refer to all students who are non native speakers of English. The Sample In Fall Q uarter 1993, a questionnaire w as distributed to the entire faculty at Cal Poly Pom ona (N=796), with the exceptions of faculty on leave, in administrative-only positions, on off-cam pus assignm ents, or teaching assistants. Within three w eeks, a total of 392 faculty m em bers responded to the questionnaire (49% of the entire faculty). All 47 academ ic d ep artm en ts in th e six colleges and o n e school w ere re p re se n te d . T he p e rc e n ta g e of th e sam ple from e ac h college approxim ates the percentage of each college within the University. The lowest num ber of respondents w as from the Theatre/D ance, Art, Music, and Social Work departm ents; the highest num ber of respondents w as from Agricultural E ngineering, Interdisciplinary G eneral Education, Agricultural B usiness M anagem ent, and Geology. Table 1 displays the breakdow n by colleges and school. (For detailed return rate by departm ent, se e Appendix I.) For exam ple, 27 faculty m em bers from the College of Agriculture returned surveys; they represented 7% of the entire sam ple of N=392. However, th ese 27 faculty respondents equal 60% of the College of Agriculture faculty, which h as a total n of 45 to whom questionnaires were sent. The faculty from this college com prise 6% of the entire University 92 faculty. Therefore, the 7% response rate closely approximates the 6% total faculty that the College of Agriculture has within the University. TABLE 1 ____________ SAMgLE_DISTBIBUTION_ B Y _ COLLEGE AND SCHOOL Colleoe Sample N Sam ple % College/Schl College/Schl % N University % Agriculture 27 7% 60% 45 6% Arts 111 28% 46% 243 31% Business 72 18% 52% 138 17% Engineering 66 17% 46% 145 18% Environ. Des. 27 7% 45% 60 7% Science School of 78 20% 52% 149 19% Hotel/Rest.. Mat ._11 3% 69% 16 2% TOTAL 392 100% 796 100% Requesting information on faculty ranks, the optional Question 2A found that senior faculty responded most often to the survey (40% of all full professors teaching at Cal Poly Pom ona during Fall Quarter 1993) and responded most often to this question, a s Table 2 indicates. Nearly half of all respondents for the sam ple were full professors (46%). 93 TABLE 2 PACULTY-BESeONPENT- RANKING Rank N %Total N Fall Qtr. 1993 University (NT Full Professors 179 46% 443 Associate Professors 40 10% 108 Assistant Professors 16 4% 60 Lecturers 50 13% (No data avail.) No ResDonse to Ques.2A 107 27% TOTAL .392 100% Further requested faculty background information w as faculty respondents' native languages, which research has shown to affect scoring and treatment of NNS student writing (e.g., Green & Hecht, 1985; Santos, 1988). These data were solicited in order to discover whether this factor influences the treatm ent of NNS student writing at Cal Poly Pomona. By far, the majority (73%) of respondents were native speakers of English (n=286), but 17% (n=65) listed languages other than English a s their native tongues. The single dominant language group consisted of Chinese (i.e., Mandarin, C antonese, Taiwanese, n=14). Farsi w as the second most often language listed (n=11). Forty-one (10%) did not respond. (See Appendix J for a complete list of all non-English language responses.) Since teaching experience may contribute to implicit criteria for writing assessm ent, two similar questions on faculty background asked for length of time teaching: The first (4A) asked for teaching in general, and the second (6A) inquired about teaching specifically at Cal Poly Pomona. As the faculty rankings suggest, over two-thirds of the responding faculty 94 had been teaching for m ore than a d ecad e and over half of the respondents had been teaching at Cat Poly Pom ona for that sam e amount of time (n^208). Table 3 presents a cross tabulation of the two questions which confirmed the decade-plus group of respondents. TABLE 3 LENQTH_OP_TEACHINQEXPERIENCE <1 vear Years of Teaching in General 1-5 vrs. 6-10 vrs. 10+vrs. Years at Cat Poly Pomona • < 1 Year 3 7 1 2 1-5 Years 1 27 24 36 6-10 Years 0 0 40 40 10+Years 0 0 0 208 TOTAL 4 34 65 286 No R eso o n se t3) Additional teaching experience was also queried by Question 5A, which asked for levels of prior teaching. Respondents indicated the entire range of offerings, since more than one level could be checked as appropriate. The greatest em phasis w as on College/University teaching (n-329, 84%). The mid-range included Community College level (29%), Professional Training (23%), Secondary level (20%), and Adult Education (17%). The least noted but also listed included Elementary, Vocational, and "Other" which included a range from Peace Corps to consulting. Due to the research goal, the single most important question in this study concerning faculty background w as faculty participation in scoring the GWT. The question w as purposely placed at the end of the survey in order to avoid confusion with inquiries on the subject of ESL writing in general. However, the large number of respondents who indicated that they had never scored the test (n*312) resulted in small num bers for the rem ainder of the breakdown. Only 19% of the responding faculty had scored the GWT. TABLE 4 GWT SCORING PARTICIPATION Freauencv N Percent Never 312 80% One time 24 6% 2*5 times 29 7% 6+times (but not almost always) 15 4% Almost always 8 2% No ResDonse 4 1% T O T A L . 392 100% To gather more details on faculty background, cross tabulations with a num ber of questions were conducted using the resp o n ses to Question 22 on faculty participation in the GWT scoring, and they are included a s the sections on ESL student issues and writing issues are presented. Pertinent to this a re a of faculty background is the cross tabulation that found 16 departm ents had no faculty who had scored the GWT (See Appendix K for detailed contingency table, Questions 1A and 22.) The small num ber who had scored the test were thinly spread out am ong the rem aining 31 departm ents; however, three departm ents appeared to have more involvement in the GWT scoring than did others. Forem ost w as the single English and Foreign L anguages Departm ent (n=13, 45% of the departm ent's respondents had scored; 38% multiple 96 tim es); the second m ost involved w as the Biology D epartm ent with 7 readers, 44% of its respondents; and third, Urban & Regional Planning with 4 readers, 57% of its respondents. B ecause one scoring session may have a limited effect on faculty pedagogy, this research looked at multiple scoring and faculty ranking through a seco n d cro ss tabulation. Given the large num ber of full professors responding to the survey and to the question on ranking, the research show ed, not surprisingly, that 36 of th ese full professors have scored the test multiple times, which is 47% of all faculty who have scored; additionally 5 junior tenure and tenure-eligible faculty and 2 lecturers have scored it repeatedly. Language background is another variable that h as b een shown to affect writing evaluation. Consequently, cross tabulations w ere run and they revealed that 23% (n=63) of the native sp eak ers of English who answ ered the question on native language had scored the GWT; 16% (n=10) of the NNS faculty had scored. However, native scorers dominated the scoring sessions since 83% of the responding faculty who had scored the test were native speakers of English. A further u se of cro ss tabulations revealed that length of time teaching at Cal Poly Pom ona a p p e a rs to allow for GWT scoring opportunities, since 25% of those who had been at Cal Poly Pom ona for over ten years had scored the test (n-42). Only 14% of those with six to ten years (n=11) had scored, while the group with one to five years included 15% (n-13) who had. scored. Even though an increased time teaching at Cal Poly Pom ona would allow for more opportunities to score 97 the test, an alternative explanation may be that the strongest em phasis for recruitment of readers may have been m ade in the test's early stag es (i.e., 1980, fourteen years ago) which would have included those faculty who have been at the University for m ore than ten years. (See Appendix K for contingency tables.) Faculty A w areness of ESL Students Following background inquiries, the beginning survey questions prepared faculty for issu es concerning ESL students by asking three questions on general aw areness, percentage, and frequency of ESL student presen ce in University courses. While the University does not m aintain records on language minority students, the questionnaire response data convey the impact of NNS on university courses. In Question 1, nearly all faculty (94%) confirmed that ESL students were present in their courses. T he few (n=17) who denied having this group of students were spread across 16 departm ents; only Electrical and Com puter Engineering had m ore than a single negative response. Question 2 responses indicated that 90% of the faculty had ESL students enrolled in their courses every quarter, and Question 3 answ ers showed that 87% of the faculty respondents w ere aw are that ESL students comprised from 5% to more than 50% of their classes. Interestingly, only 3% of the faculty responded that they never or only occasionally have ESL students. C ross tabulations revealed the further information that non-native* speaking (NNS) faculty respondents seem ed to be more aw are of ESL students, checking survey resp o n se s to only every quarter (89% of NNS faculty re sp o n se ) or occasionally (8% of NNS faculty resp o n se ). Additionally, recognizing a higher percentage of ESL stu d en ts in their c la s s e s (i.e., b etw een 26-50% ), 57% of NNS faculty w ere aw are com pared with 40% of native-English speaking faculty. This finding m ay be accounted for in several w ays. First, NNS faculty m ay b e m ore cognizant of ESL student p re se n c e b e c a u se of their own language backgrounds; and second, they m ay attract m ore ESL stu d en ts who believe they will be able to use their first language with the faculty m em ber or believe that a NNS faculty m em ber m ay not penalize ESL students for their English usage. Further information on faculty a w a re n e ss of ESL stu d en ts w as requested by Question 18 in which faculty w ere asked to respond to a broad inquiry, that is w hether ESL students encounter problem s in their courses. While the majority of resp o n ses w ere affirmative (n=231, 59%), the question did not ask for a com parison of native to NNS students and the results on their own a re not clear. C ross tabulations with other questions, however, provide som e interesting data. Of th o se who had graded the GWT, the majority (n=56, 74% of the total who had scored) said that ESL students do encounter problem s in their co u rses. But faculty who had never scored the GWT offered an entirely different view: 90 faculty respondents who had never scored the test said they w ere uncertain w hether students had problem s (87% of total unsure responses), and 42 faculty m em bers who had never scored the test said stu d en ts did not encounter problem s (88% of total negative 99 response). In addition, the total of th ese two groups, from the non-scoring segm ent of the sam ple, who replied with uncertain or negative answ ers represent one-third of all respondents. W hen native language background w as m easured against this general recognition of ESL students encountering problem s, faculty native sp eak ers of English responded affirmatively slightly more than did non native sp e a k e rs to the sta te m e n t that ESL stu d e n ts do en co u n ter problem s in their courses; that is 63% of th e native-English speaking faculty respondents said that ESL students encounter problem s, while 54% of the NNS faculty respondents said that the ESL stu d en ts have problems. Question 18, designed to obtain information on faculty perceptions of ESL problem s in coursework, served a s an introduction to Q uestions 19 and 20. In Question 19, faculty who had responded affirmatively to ESL students encountering course problem s were asked to detail the types of problem s that the students encountered. Even faculty who had failed to respond or had offered negative or unsure resp o n ses a s to w hether ESL stu d en ts had co u rse problem s responded to the following questions. R espondents w ere a sk ed to check a s m any a re a s a s they d eem ed appropriate; the totals for the question therefore exceed 100%, a s each subcategorical resp o n se w as m easured against the N=392. R esults are presen ted In Table 5. The em phasis, a s expected, w as on L anguage Difficulties (76%), and Inadequate Prior Academ ic Preparation drew over one-third (37%) of th e possible resp o n se. The category of Cultural Differences w as also an important factor for one-fourth of the faculty. 100 TABLE 5 FACULTY^ERCEIVEP CAUSES OF_ ESL PROBLEMS N 3DDX. % Language difficulties 299 76% Inadequate prior academ ic prep. 144 37% Cultural differences 99 25% Inattention 28 7% Other 27 7% Financial concerns 16 4% Emotional problem s 12 3% Basic Intelliaence. 10 . 2% B ased on the anticipated response to the prior question, Q uestion 20 asked faculty to narrow their focus to the types of language problem s that students encounter. R esults are presented in Table 6. By far, W eak Writing Skills w as the category m ost often checked (68%). Another skill, that of U nderstanding, offered three possibilities and drew resp o n se to each from more than one-third of the respondents: Understanding Written Q uestions 41%, Understanding Lectures 36%, and Understanding Spoken Q uestions 36%. Oral R esponse and Slow/Inefficient Reading w ere also perceived a s significant problem s by over one-third of the faculty. 101 TABLE 6 FACULTY_-PERCEIVEP LANGUAGE PROBLEMS N BDpX. % W eak writing skills 267 68% Understanding written questions 161 41% Responding orally to questions 144 37% Understanding lectures 141 36% Understanding spoken questions 140 36% Slow/inefficient reading 137 35% W eak communication with peers 82 21% Slow/inexact note-taking 62 16% O ther ___ 27...... 7% Faculty Practice and Beliefs Regarding ESL Writing As the primary focus of the questionnaire, faculty attitudes (i.e., implicit criteria) toward written work in the courses they teach, especially a s they assign, grade and consider ESL student writing, w ere queried by m ost of the questions. The following discussion is offered in two parts: the first includes questionnaire resp o n ses regarding faculty writing practice, in order to understand what faculty actually do a s they require writing in their co u rses; and th e second includes questionnaire re sp o n se s regarding faculty beliefs of ESL writing, in order to recognize conform ity or disjunction of classroom practice with faculty beliefs. 102 Practice W hat faculty actually do a s they assign, grade and consider ESL student writing is found in resp o n ses to the first cluster of questions (4, 5, and 6) which dealt with a variety of pedagogical considerations for writing assign m ents: current writing practice, technological expectation, and process-versus-product em phasis. Due to recent budget cuts at the University, the possibility that larger cla ss size might impact the am ount of assigned writing w as not borne out by th e re sp o n se s to Q uestion 4. Only 6% of th e faculty respondents said they u se writing less than they did 2 to 3 years ago. The negative effects of the budget cuts may have been offset by the impact of the Writing Across the Curriculum Program a s 30% of the faculty said they u se writing m ore. Five departm ents w ere found to have an increased num ber of faculty who assign m ore writing than others: Horticulture (100% of this d e p artm e n t's re sp o n d e n ts, n=6), A ccounting (68% of the departm ent, ns9), English & Foreign L anguages (31% of the departm ent. n=9), M anagem ent & Human R esources (39% of the departm ent, n=7), and M athem atics (32% of th e departm ent, n=6). (S ee Appendix K for c ro ss tabulation of departm ents and re sp o n se s to Q uestion 4.) Most faculty responded that they use writing to the sam e degree a s they did 2 to 3 years ago (63% of total survey response, n-247). But it is unclear if this response m eans they actually assign any writing in their co u rses since, if th ese faculty respondents had not required written work 2 to 3 years ago, the resp o n se to "Sam e" would, not b e a positive indicator of classroom writing practice. 103 In reconsidering faculty background with th e re sp o n se s to this question, this research found that 27 of those who had scored the GWT assign writing m ore often; 6 less; 43 the sam e. In addition, 34% of NNS faculty said they u se writing m ore while 28% of NS faculty said they u se it m ore than 2 to 3 y ears ago. And length of time at the University d o es not seem to affect willingness to change a s 60% of those who have been at Cal Poly-Pom ona for six to m ore than ten y ears responded that they assign writing more than they did 2 to 3 y ears ago. Moving from g e n e ra l writing a ssig n m e n t e x p e c ta tio n to technological expectation, Question 5 asked if faculty expect the u se of a com puter, word processor or typewriter for assignm ents that are to be turned in for a grade. In m arked contrast to the lack of opportunity for this technological support on the GWT, a total of 90% of the respondents expect this u sa g e with the following frequency: O ccasionally (22% ), Frequently (33%), or Always (35%). And attem pting to com pare content coursew ork tren d s with the current pedagogical em phasis in com position c o u rses on writing a s a process, Question 6 asked if faculty em phasize pro cess or product most in their co u rses. Since only product is the focus of th e GWT, it w as interesting to find the answ ers divided into three groups: P ro cess 33%, Product 38%, and Both or O ther (26%); no resp o n se (2%). T hus, the general expectation for coursew ork a p p ea rs to be b a sed on m ore than only a product. In order to consider writing practice in a m ore concrete se n se , the second cluster of questions, 7 through 12, presented paired inquiries 104 regarding am ount of writing in gen eral, length of writing assig n m en ts (exceeding 250 words), and opportunity for a specific condition for writing, (i.e., tim ed in-class writing). E ach pair w a s divided into lower and upper division coursew ork. Table 7 reports the p ercen tag es of response. Fifty-five percent of the faculty respondents said that during a ten- w eek quarter, the length of the University term , they assign betw een 1 and 6 assignm ents to lower division courses; an additional 13% of the faculty resp o n d en ts e x ce ed th is am ount, checking from 7 to m ore th an 10 assignm ents. For upper division courses, 67% of th e respondents assign 1 to 6 writing assignm ents; an additional 18% assign 7 or more. In considering length of writing, half of th e faculty resp o n d en ts (51%) require 1 to 6 assignm ents which exceed 250 words; an additional 6% of the respondents assign m ore than 7. For upper division, 72% of the faculty who resp o n d ed to th e survey expect 1 to 6 a ssig n m en ts that exceed 250 words; an additional 9% of the respondents require 7 or m ore. O pportunities for tim ed writing a re offered by 32% of the faculty resp o n d en ts who u se 1 to 6 tim ed assig n m en ts for their lower division co u rses, while an additional 5% of the resp o n d en ts u se 7 or m ore tim ed assignm ents per quarter. For upper division, 44% of the respondents give 1 to 6 timed assignm ents; an additional 4% of the respondents require 7 or more. In sh arp contrast to th e p e rc e n ta g e s of assig n m en ts given, of particular im portance is th e num ber and p ercen tag e of faculty who do not assign any writing, do not assign lengthy writing, and do not provide tim ed writing opportunities, especially for lower division co u rses. In fact, 118 (30% of total respondents) never assign writing for lower division coursew ork; 159 (40% of total respondents) never assign writing exceeding 250 words; and 239 (61% of total respondents) never assign writing under timed circum stances. For upper division, an increased num ber of opportunities for all types of writing appear to exist: only 51 faculty (13% of all respondents) do not assign writing, and 67 (17% of all respondents) do not assign writing exceeding 250 words. And last, 197 (50% of total respondents) do not expect timed writing. While this figure appears to indicate an improvement over lower division expectation of 61% of all faculty respondents who never assign timed writing, this response stitl indicates that at least half of the faculty respondents do not assign timed writing at any level. TABLE 7 PERCENTAGE OF FACULTY-ASSIGNED WRITING _None/NA Number of assignm ents 1-2 3*6 7*10 11+ NR General writina 7. Lower division 30% 29% , ,27% 9% 4% 2% (55%) (13%) 8. Upper Division 13% 22% 44% 15% 3% 2% (67%) (18%) Writina exc.250 wds 9. Lower division 40% 33% 18% 5% 1% 2% (51%) (6%) 10. Upper Division 17% 32% , , 35% 8% 1% 2% (72%) (9%) T im ecL W xitlng 11. Lower division 61% 20% 12% 3% 2% 2% (32%) (5%) 12. Upper Division 50% • 27% 17% 3% 1% 2% (44%) (4%) (NR/No Response) 106 C ross tabulations offered additional information on faculty writing practice in regard to th ese three pairs of questions. Native*English* speaking and non-native*English-speaking faculty assign writing to approximately the sam e degree and amount with the exception of timed writing; a higher percentage of NS faculty do not assign timed writing for lower division courses (62% of all NS faculty respondents) than NNS faculty (51% of all NNS faculty respondents). C ross tabulations also determ ined two im portant item s of information by comparing length of time teaching at Cal Poly Pom ona (Question 6A) with assigned writing (Q uestions 7 through 12). (S ee Appendix U for detailed table.) First, the small num ber of missing resp o n ses from th ese three pairs of questions in the middle of the questionnaire (n=3) suggests serious faculty engagem ent with the survey instrument and the subject of writing assignm ents in general. And second, a strong response was.evident for multiple writing assignm ents by faculty who have been teaching for over ten years. For example, 60% of the faculty who have been teaching for more than ten years assign between 1 and 6 writing assignm ents each quarter to lower division courses; 68% of the faculty teaching more than ten years assign 1 to 6 pieces of written work to upper division; an additional 10% of the ten-plus-year group assign 7 or more pieces of written work to lower division; 21% of this group assigns 7 or m ore to upper division. The category of writing assignm ents exceeding 250 words generated similar response rates by faculty teaching more than ten years a s responses to questions on amount of writing. But timed writing offered notably different returns; only 31% of 107 those teaching for more than ten years regularly assign 1 to 6 timed writing opportunities to lower division; 41% of this group assign 1 to 6 for upper division; 6% of this group give 7 or more timed writing assignm ents to lower division; 5% of this group require 7 or more timed assignm ents for upper division. Diminished opportunity for student writing is also seen in negative evidence of faculty involvement with writing produced by another set of cross tabulations which compared frequency of GWT scoring participation and amount and type of assigned writing. (The detailed cross tabulation may be found in Appendix K.) Of particular note is the finding that 109 faculty respondents (28%) who have never scored the test also never assign writing to lower division students; 143, (37%) who have never scored the test also never assign lengthy writing to lower division students; 203 (52%) of those who have never scored also never assign writing under timed conditions to lower division students, and 162 (42%) never assign timed writing to upper division students. Contingency tables also resolved a specific research concern within these paired responses. In looking at the num ber of respondents who assign general writing, cross tabulations revealed that of those who assign writing to lower division students, 56 (21%) do not assign lengthy writing (exceeding 250 words), and 127 (48%) do not assign timed writing. Of those who assign writing to upper division courses, 21 (6%) do not assign lengthy writing and 148 (45%) do not assign timed writing. (See Appendix K for comparisons of Questions 7/9,7/1 lan d 8/10 and 8/12.) Com pleting th e collection of additional inform ation on faculty practice, two questions, 17 and 21, w ere placed on the last p a g e of the survey. Due to th e limited offerings for writing m o d es p resen ted on the G raduation Writing Test, this research sought to discover and com pare the expectations and d e m a n d s for writing m o d es that faculty hav e for regular acad em ic coursew ork through re s p o n s e s to Q uestion 17. As resp o n d en ts w ere ask ed to check all th at applied, e ach subcategorical resp o n se offered a 100% possible return; the results a re listed in order of highest resp o n se frequency in Table 8. T he m ode m ost often u sed on the GWT, P e rso n a l E xposition, is n o ted by only 19% of th e faculty respondents a s a b asis for writing in their courses. On the other hand, the m ost frequently cited m ode, D ata Support an d Interpretation, (n«231, 59%) is not used on the G raduation Writing T est but is the m ode to which m ore stu d en ts a p p ea r to have exposure and opportunities for practice. The re sp o n se s to the "O ther” category included a variety of additional writing styles, such a s analysis, problem solving, and creativity. (S e e Appendix C for open-ended responses.) 109 TABLE 8 COURSE-WRITING MODES Mode ____ N _ aoox. % . . D ata support/interpretation 231 59% Description 210 54% C om parison/contrast 184 47% Sum m ary 165 42% P ro cess 149 38% C ause and effect 140 36% Argumentation 122 31% Definition 114 29% Personal exposition 74 19% Narration 66 17% Classification 52 13% Other 37 9% T he last question dealing specifically with faculty practice, Question 21, asked faculty how they assist ESL students with English writing skills. This question had a two-fold purpose: one, to gain information about faculty writing practice, and two, to offer su g g estio n s for future need. Specific locations and program s on cam pus w ere listed to a ssist faculty in recognizing th e num ber and variety of available opportunities. Most resp o n d en ts (n=211) su g g ested that stu d en ts obtain a ssista n ce from the Learning R esource C enter on cam pus; 201 su g g e ste d , in g eneral, that stu d en ts obtain help; and 178 personally conferred with students. O n the o ther hand, only 51 said that they do not e n co u rag e stu d e n ts to se e k a ssista n ce . Fifty-nine resp o n d en ts also added their own su g g estio n s to the "Other" category, which indicates that faculty are aw are of the need for m any w ays to a ssist ESL stu d e n ts with English writing skills. T h ese resp o n ses included a range of offerings such a s journal writing, accepting early drafts for feedback, encouraging ESL coursework, and encouraging students to speak with English-native-speaking students. (A com plete list of open-ended responses may be found in Appendix C.) Beliefs Beyond the am ounts and types of assigned writing, this research sought to uncover potential differences betw een faculty beliefs and practices, especially regarding criteria for evaluation of ESL student writing. T herefore, two pairs of questions dealt with th e se issu es. The questions and responses to the first pair, Q uestions 13 and 14, are: 13. In your opinion, should non-native speakers of English be required to m eet the sam e criteria for English writing skills a s native sp eak ers of English? (263) 67% a. Y es (67) 17 b. No (46) 12 c. Unsure (12) 3 d. Other (Please explain) (4) 1 No R esponse 14. How do you grade ESL student writing in com parison to native speakers of English? (115) 29% a. More leniently (0) 0 b. More severely (250) 64 c. Sam e (21) 5 d. Other (P lease explain) (6) 2 No R esponse At first glance, the frequencies and percentages appear to tell the entire story; that is, 67% of the respondents said that ESL students should 111 b e required to m eet the sam e criteria for writing skills a s native sp eak ers and nearly the sam e percentage (64%) of the respondents said they grade ESL student writing in the sam e m anner a s native sp eak ers' writing. The 17% for "No" an d 12% for "Other" total 29% , which is the sa m e p ercentage of resp o n se to grading "m ore leniently." However, a cro ss tabulation of th e se re sp o n se s found otherw ise u n se en yet im portant results (reported in Table 9) which are inconsistent with th e suggested response totals. Of the 263 who said that ESL students should be required to m eet the sam e criteria, 51 (nearly 20%) appear to counter this belief in practice a s they said they grade writing by th ese students m ore leniently. Conversely, 19 respondents, 28% of those who said ESL students should not be required to m eet the sa m e criteria, report grading ESL student writing in the sam e m anner a s they do that of native speakers. And of the 46 who were "Unsure," 16 grade more leniently (35%) and 23 (50%) grade with the sa m e d egree of severity. Nearly half a s m any (115, 46% ) grade more leniently than grade the sam e (250). TABLE 9 ____________________ STANDARDS VERSUS GRADING_______________ Question 13 (Same criteria) -------------------------- Yes____NaUnsure Other NoResp, Total Question 14 (Grading) More leniently 51 41 16 7 0 115 More severely 0 0 0 0 0 0 Sam e 204 19 23 3 1 250 Other 6 7 6 2 0 21 No resDonse 2 0 1 0 3 6 TOTAL ...... ........m , 67 46 12 . . 4 392 112 W hen individual university departm ents w ere considered, forty-one departm ents w ere divided in their resp o n ses a s to w hether ESL students should be required to m eet the sam e criteria; only six departm ents (with m ore than one respondent) unanimously agreed that ESL students should m eet the sam e criteria1. O ne departm ent-A rt-disagreed unanim ously. W hen it cam e to grading, though, m ost departm ents w ere divided a s to how the faculty grade ESL student writing. Only two (with m ore than one respondent) agreed unanimously that criteria and grading should be the sa m e for everyone: A erospace Engineering and Chem ical & M aterials Engineering; that is, they w ere the only two d epartm ents which w ere consistent with their beliefs and practices. Ten departm ents w ere divided nearly in half betw een grading more leniently or the sam e 2 . Beliefs also are affected implicitly by faculty language background. In comparing native and NNS faculty, it appears that native speakers more often grade leniently (34% of NS faculty) than non-native sp e ak e rs (18% of NNS faculty) even though they offered no explicit difference in opinions about criteria (both approxim ately 67%, Yes; both approxim ately 18%. No). Even with similar scoring experience, no co n sen su s on beliefs w as available. Of th e faculty who have scored the GWT, 64% who have scored the test (n=49) said that criteria should be th e sam e; 24% who have scored the test (n=18) said NNS should not be required to m eet the sa m e criteria; and 10% who have scored the test (n=7) replied that they w ere unsure. Yet w hen asked.how they grade, 30% who have scored 113 (n=23) said they grade more leniently, but twice a s many (61%) who have scored (n=46) said they grade the sam e, and 10% (n=7) w ere unsure. Interestingly, Q uestions 13 and 14 generated the greatest num ber of open-ended resp o n se s even w hen the resp o n se s could have been handled with only check boxes. (S e e Appendix C for com plete open- ended responses.) This response indicated strong em otions concerning the topic which faculty felt compelled to express. For exam ple, in response to Question 1 3 ,1 2 respondents checked "Other" and offered a wide range of com m ents, including "ESL 'accent* okay”, "I wish I knew...", and "I've alw ays had difficulty here...." In addition, 15 faculty respondents added com m ents beyond th e requisite check box to adam antly declare their opinions (e.g., "Absolutely"; "How could we expect less if they are to function in this society?"); 7 faculty resp o n d en ts w ere uncertain or sym pathetic (e.g., "With com passion"; "Not su re/u n ev en field of evaluation."); and 32 offered ex cu ses and/or exceptional conditions for their answ ers (e.g., "It Is probably unrealistic to expect ESL students to m eet sam e criteria but w e should strive for it"; "D epends-yes, on edited term papers; no, on in-class papers which are graded for content"). G enerating the sam e intensity, Q uestion 14 on grading practice garnered 21 checks to th e "Other" box with a range of com m ents that stretched from "Individual analysis" to "I grade for content only." Additional open-ended resp o n se s fell into th ree groupings: 8 faculty respondents said in som e fashion that they grade in a simitar m anner (e.g., "I grade the sam e for everyone. I have just relaxed the standards for gram m ar, syntax, etc. for everyone"); 27 faculty respondents offered exceptions (e.g., "All 114 exam s & reports a re tak e hom e; content g rad ed equally; gram m ar, punctuation, spelling, vocabulary forgiven for ESL's on th e se criteria; I em phasize content"); and 10 respondents explained the type of assistance they offer in consideration of grading (e.g., "But I help them edit"). The potential for conformity or disjunction betw een practices and beliefs w a s also th e focus of the co n trast b etw een Q uestion 15, em phasizing actual u sag e ("correct or com m ent on"), and Question 16, em phasizing value ("consider important"). B ecause Q uestion 15 is the b asis for the creation of th e analytic rubric used for re-scoring GWT essay s, a detailed discussion of resp o n ses to this question is presented in P h ase II of this chapter. But to sum m arize briefly, the ten most commonly corrected or com m ented-on criteria that faculty alw ays u se (from the twenty offered in Q uestion 15) are arranged by resp o n se frequency in T able 10. E m phasis on id eas, along with attention to content and originality or factual support, ap p ears to merit m ost of faculty respondents' tim e and effort. In im m ediate succession, how ever, w as a m echanical control, spelling, which nearly half of all respondents said they always correct. Pilot participants had not responded a s strongly to this criterion (14% always, 43% often', 43% sometimes) but in post-pilot discussions with faculty, the re se a rc h e r w as told that faculty who do not teach composition or gram m ar feel confident to mark only m echanics, such a s spelling, capitalization an d punctuation on stu d e n t writing. This explanation may help to account for the high response to the categories of spelling and capitalization; punctuation also received a significant response total (eleventh on the list, n-104, appx. 28%), 115 TABLE 10 JE N MOST-USED COURSE WRITINQCRITERIA________ N n > § X 1) Content 252 67% 2) Original ideas/factual support 186 51% 3) Spelling 165 44% 4) Organization 130 35% 5) Grammatical control 127 35% 6) Cohesion 124 34% 7) Verb usage 120 33% 8) Topic identification 120 33% 9) Capitalization 107 29% 10) Sentence construction 105 28% Within the ten m ost commonly used criteria, a total of six w ere attached to the two factors used in the analytic rubric (discussed in P hase II) of Content and Language Use, each with three. The three criteria cited for the factor of Content included: 1) content, 2) original ideas/factual support, and 8) topic identification. T he criteria cited for the factor of Language Use included: 5) Grammatical control, 7) Verb usage, and 10) Sentence construction. The factors of Organization and M echanics each garnered two of the criteria while the factor of Vocabulary did not attract sufficient attention to any of its criteria to be included in the ten m ost noted. Q uestion 16 asked faculty to attach d e g re e s of im portance in consideration, not necessarily in u sag e (as did Question 15), to each of ten general a re as for writing evaluation. (S ee Appendix A for frequency and percentage totals.) Two criteria from each of the original five factors in Question 15 were selected to elicit a general response and were evenly distributed in the question so that no two criteria from the sam e factor were placed next to one another. Instead of usage, faculty were asked to confirm, through a Likert-type scale, the degree of im portance they considered each criterion to bear on assessm ent of their students' writing. In order to analyze the responses, the checks for very and somewhat were both labeled positive; not very and not at all were labeled a s negative. The non-response total has also been included. The response percentages for each criterion are presented in Table 11. While Question 15 asked for criteria that faculty actually use to mark their students' papers, Question 16 asked which criteria faculty deem ed to be important. An overwhelming majority thought that most of the criteria listed were important, with a m ean of 152.35 for attributing positive d eg rees of importance while a m ean of 28.3 w as found for attributing negative degrees of importance. In contrast, the responses for Question 15 resulted in a m ean for positive use of criteria to be 111.2 and a negative m ean of 47.0 (including the neutral response of Sometimes! or 26.075 (without Som etim es!. This comparison indicates that while faculty may be inclined to consider m ost criteria im portant, they do not necessarily believe to the sam e degree that correcting or commenting on them is worth their time or effort. 117 TABLE 11 ATTRIBUTED IMPORTANCE TO_WRITINQ.CRITERIA Very CRITERIA aDDX.% n Somewhat % n N oL V .ery % n N o L .a t.A H _ _ % n CONTENT Originality 49% 180 41% 149 9% 32 1% 5 Specific dev. 54% 193 41% 146 4% 13 1% 2 ORGANIZATION Order 48% 174 47% 172 4% 14 1% 2 Logic 78% 288 20% 74 1% 2 1% 2 VOCABULARY Vocabulary range 17% 62 60% 219 21% 79 2% 6 Register/tone 10% 34 53% 181 28% 98 9% 31 LANGUAGE USE Grammar 55% 204 38% 138 6% 23 1% 3 Sentence variety 12% 42 51% 184 30% 109 7% 26 MECHANICS Mechanics/ 44% 159 43% 155 12% 45 1% 5 punctuation Lea._wrta/format 43% 154 38% 139 14% 50 5% 19 TOTAL 1490 1557 465 101 Summary of Questionnaire Findings In response to the first research question, "What criteria do faculty at the University use to m easure NNS student writing in their academ ic courses?" faculty offered a variety of expectations through beliefs and practices, which may account for implicit and explicit writing assessm ent criteria. To begin, the topic of writing by NNS students was perceived as a serious one, dem onstrated by the high return num ber from across the disciplines, the few omitted survey responses, a rapid return rate, and the involvement of senior faculty. 118 Nearly all faculty were aware of ESL students in their classes, and most felt that these students encountered problems. The main are as for concern were listed a s language difficulties, inadequate prior academ ic preparation, and cultural differences. A very high response also listed weak writing skills a s the most significant language problem, along with a number of others which dem onstrated faculty aw areness and concern. Many faculty respondents expected writing a s part of coursework but not necessarily the sam e type, amount, or m ode a s that which is expected on the Graduation Writing Test (GWT), and not under the sam e conditions. Interestingly, an increased em phasis on writing appeared in upper-division coursework. However, since NNS student writers have limited opportunities for developing academ ic discourse through guided practice with feedback, the general lack of writing w as significant, especially responses that indicated the lack of written work beyond short answ ers (i.e., exceeding 250 words) as well a s the lack of timed writing opportunities. Faculty expected ESL (i.e., non-native speaking) students to write as much or more than a few years ago; they also expected them to use technology, which includes spelling and grammar checkers available with computer usage; and they were divided over whether product or process received the greatest degree of em phasis in coursework. They offered specific types of writing assignm ents with m odes they were able to label, especially data support and interpretation, yet for coursework, they rarely expected the types of writing necessary for the GWT. And most faculty 119 generally en co u rag e d or offered ESL s tu d e n ts so m e a s s is ta n c e with writing problem s. Length of tim e teaching at Cal Poly P o m o n a had so m e additional effects on faculty perceptions toward writing. T hose with ten y e a rs or m ore w ere willing to ch an g e, saying they a ssig n e d m ore writing at th e tim e of th e questionnaire distribution th an a few y e a rs ag o an d th ey a ssig n e d m ore at all levels. Additionally, th o se with this length of teaching tim e w ere also m ore likely to sco re th e GWT. Faculty native v e rs u s n on-native s p e a k e rs of E nglish (NNS) occasionally p erceiv ed m atters differently. For ex am p le, faculty native sp e a k e rs (NS) m ore often believed that ESL stu d en ts encountered course problem s, faculty NS offered few er opportunities for tim ed writing, an d faculty NS said they grad ed ESL writing m ore leniently. T h o se w ho h ad sc o re d th e G raduation W riting T est sa w ESL student writing in a different light than did the v ast majority (80%) who had not sco red the test. T hose who had sco red th e test did, in fact, feel that ESL stu d en ts m ore often en co u n tered problem s in their c o u rse s than did non-scorers. Faculty who had sco red th e te st assig n ed writing m ore than they had a few y e a rs before, and they assig n ed m ore writing at all levels. H ow ever, GWT re a d e rs, along with th e majority of faculty re sp o n d e n ts who h ad never b e e n re a d e rs, w ere also split in deciding w h eth er ESL stu d e n ts should b e required to m ee t th e s a m e sta n d a rd s a s native stu d e n ts, an d GW T re a d e rs split In their re s p o n s e s to grading m ore leniently or with equal severity. , 120 Nearly all faculty w ere able to assig n u sa g e e m p h a se s to specific criteria, su ch a s content, organization, spelling, and gram m atical control, and did so to varying d eg rees. B ecau se the non-response rate to the very detailed criterion lists ranged from 17 to 50 on single item s, it ap p eared th a t m o st faculty re s p o n d e n ts b o th re c o g n iz e a n d apply th eir understanding of various evaluation criteria to their stu d en ts' writing for coursew ork. P e rh ap s th e m ost revealing findings draw n from th e questionnaire a re th o se which highlighted th e d isc re p a n c ie s b etw een beliefs and practice. T h ese are significant b e c a u se they m ay b e responsible for the rift in determ ining which criteria faculty actually u se to a s s e s s ESL student writing. Faculty attitudes tow ard sta n d ard s w ere contrasted with grading p ractices an d show ed that faculty hav e not resolved this concern for th em selv es; that is, they do not a g re e that all stu d e n ts (NS and NNS) should or should not be required to m eet the sa m e criteria for writing, and faculty do not a g re e that they should g rad e all student work in th e sam e m anner. Confirming this variability w ere com parisons betw een attributed u sa g e and im portance of selected criteria: While resp o n se s to Q uestion 16 rev ealed th at m ost faculty resp o n d en ts thought nearly all listed criteria w ere im portant, th e re sp o n se s to Q uestion 15 countered this belief by revealing th at faculty do not actually a s s e s s their own stu d en ts' writing using th e criteria to which they attribute im portance. 121 P h a se II: The Analytic Rubric T he seco n d research question for this study asked: "Are th e course criteria different in kind o r d e g re e from the criteria u se d to a s s e s s the G raduation Writing T est, o n e version of th e California S ta te University* m an d ated G raduation Writing A ssessm en t R equirem ent? If so , in w hat way(s) are th ese two s e ts of criteria different?" In order to answ er the seco n d research question, th e results from Q uestion 15 of th e survey w ere analyzed to devise a n analytic rubric, representing th e d e g re e of e m p h a sis which faculty attach to criteria for a s s e s s m e n t of NNS stu d e n t writing in acad em ic coursew ork. T hen qualitative an d quantitative co m p ariso n s and c o n tra sts of th e analytic rubric with the holistic rubric u sed for GWT evaluation confirmed the w ays in which the two s e ts of criteria w ere similar and different. Creation of the Analytic Rubric Factor h e ad in g s for e a c h s e t of criteria w ere pre-set, reflecting guidelines in th e re se a rc h by J a c o b s et al. (1981), and w ere applied to e ac h grouping. Criteria w ere an aly zed an d w eighted by 1.0 and 0.5 respectively for only positive re sp o n se s (Always an d Often) in order to determ ine overall u sag e. T he percentage of the whole w as m athem atically derived an d ro u n d ed off for factor grouping. B e c a u se th e negative resp o n se s of Rarely and Never and the neutral resp o n se of Som etim es did not offer additional positive information on faculty practice, they w ere not u se d . Reliability for th e d e g re e of u sa g e attributed to th e criterion , 122 factors w as achieved by com parisons to resp o n se s from th e pilot survey, which supported the totals for e ach factor. The result w as the five-factor listing presented in Table 13. Content w a s th e m ost im portant factor, with 25% e m p h a sis; O rganization, L anguage U se, an d M echanics w ere all indicated at about th e sa m e d e g re e, reflected on th e new rubric with 20% eac h . And according to faculty re s p o n se , th e le a st im portant facto r w a s th e c ateg o ry of Vocabulary, which resulted in 15% of the whole on the new scoring guide. (S ee Appendix L for Factor Totals.) 123 TABLE 12 SURVEY CRITERIA FOR ANALYTIC FACTORS FACTOR Always fn x1.01 Often fn xO.51 Content: 0-25 ots. Content (ideas, knowledge) 252 89 Original id ea s &/or factual 186 121 support Detailed developm ent 76 127 Topic identification (what topic requires) 120 121 Oraanization: 0-20 Dts. Sequencing 87 124 C ohesion (linking together) 124 146 Length/brevity 51 124 Organization (incl. beginning, middle, ending) 130 117 Vocabulary: 0-15_ots. W ord deletion &for 94 130 un necessary additions Register (level of voice, e.g. formal or informal tone) 35 52 Writer’s purpose/audience a w are n ess 91 112 Appropriate word/idiom u sa g e 88 121 L anauaae Use: 0-20 Dts. Preposition &/or 104 87 article (a,an,the) u sag e S entence construction 105 125 Gramm atical control (e.g., agreem ent:subj./verb/pronoun) 127 110 Verb u sag e (tense, modal, auxiliary, gerund, infinitive) 120 106 M echanics: 0-20 Dts. Punctuation 104 106 Spelling 165 105 Capitalization 107 87 Form/format/handwritina 82 90 Total: 0-100 ooints oossible 124 ' T h e A a lv tic Rubric and Criteria Only th e tw enty criteria from th e qu estio n n aire com prised th e analytic rubric; a s su ch , th ey w ere not m ean t to b e an ex h au stiv e exam ination of all a sp e c ts of writing. While research indicates that writing professionals have yet to reach c o n se n su s on w hat constitutes good writing (e.g., Q uellm alz, 1980; Ham p-Lyons, 1991; W illiamson, 1993), th ese twenty criteria w ere offered a s a g enerous sam pling for each of the writing factors which have b e en supported by research (e.g., Diedrich et al., 1966, J a c o b s , et al., 1981; Cum m ing, 1990). T h ese factors w ere additionally confirm ed in a study by Blok and deG lopper (1992) which found that in 79 a sse ss m e n t m odels, th e sa m e g en eral criteria w ere repeatedly recognized and used: content, organization, style (including vocabulary), and conventions (including gram m ar and m echanics). Loosely b a sed on the writing evaluation instrum ent developed by Ja co b s, et at. (1981), the rubric offered three levels for each factor ranging from excellent to very good, good to a v erag e , an d fair to poor. T he m idrange point for each factor w as se t with th e re se a rc h e r's a w are n ess that the cut score had b een established to duplicate the original midpoint of the GWT cut score. R ead ers w ere provided both a detailed description of th e criteria for e ach level a s a Detailed Analytic Rubric and a brief analytic scoring sh e e t w as coded and attached to each e ssa y to simplify scoring and the follow-up reporting. (S ee A ppendices O an d P for copies of D etailed and Brief Analytic Rubrics.) S c o res w ere determ ined by assigning points to e a c h se p a ra te factor and totaling th e factors for a cumulative score. The distinct advantage of the analytic rubric is its potential for explication. B ecause each category's subcom ponents are explicit, it is possible to explain to any student-writer how each rater scored separate sections and the step s which might be employed for the student's future study. For exam ple, if a student authored an essay with well-developed content, in an organized m anner, with effective vocabulary, but m ade gram m atical errors in agreem ent or verb ten se, while controlling mechanics, the rater would deduct points from the Language Use section. Even if the writer m ade num erous grammatical errors, only the possible points for that subsection, in this case 20, would be deducted from the total score. Using this rubric, if a second reader agreed that these were the only flaws in the test essay, each then assigning a score of 80 to the otherwise strong paper, the essay would p a ss with a cumulative total of 160, The use of the analytic scoring rubric and its criterion application offer three important advantages over holistic assessm ent: The student is provided an understanding of the scoring criteria for future need; the raters understand the specific guidelines and proportions for each criterion factor they are using for the assessm en t; and no single factor extends a disproportionate influence over passing or failing scores. The GWI Holistic R ubric and C riteria In contrast to the details provided in the analytic guide, the GWT holistic rubric offers little in the way of specific criteria: A good paper is one that is "generally well written," while a w eak p ap er "su g g ests 126 incompetence." (See Appendix E for copy.) Consequently, raters are left to m ake their own judgments by interpreting types and numbers of calibrating essay s presented during training and by applying their own criteria to sco res on the general rubric for assessm ent. D iscussions that ensue during scoring do not specify error counts or idea tallies. Therefore, the only criteria directly addressed in the holistic rubric are minimally specified in the following three general areas: 1. Organization (mentioned twice). 2. Content (detailed in both positive and negative ways; m ost in the 3 score descriptor: "detail," "specificity," "slight part of the question," "only descriptive or narrative," "overly cliched," "overly repetitious," "general or superficial"). 3. G ram m ar (noted via two general term s: "gram m atical inconsistencies" and "w eaknesses in structure, syntax, diction and/or development"). The w eakness mentioned in the last section could be extrapolated to include "w eakness in structure," applicable to organization; "weakness in... diction" could also include vocabulary range. No mention of m echanics (e.g., spelling, legibility, capitalization) is noted, but according to discussions during training, raters allow the catch* all "minor grammatical inconsistencies" in the 3*level score to subsum e these elem ents ifthe paper is flawed in these ways. One last point on the GWT holistic rubric needs to be reiterated: The top two descriptors (for scores of 6 and the 4*5 combination) offer general positive com m ents but leave the rater to determ ine what "well 127 written" m eans in 6 and what "competently written" m eans in 4-5. On the other hand, the bottom half descriptors are slightly m ore detailed yet also offer a wide gap to accom m odate interpretations of "marred by more than a few minor gram m atical inconsistencies." The rater is again left to determ ine his or h er own understanding of "a few," "minor," "gram m atical" an d "inconsistencies," a s well a s w hat "su g g e sts incompetence" m eans in the score of 1. Posing the greatest problem in the u se of the GWT holistic rubric is that it is hollow; without significant support by m eans of training sessions, calibrating e ssa y s, discussions with other raters, individual faculty aw areness of writing and writing terminology, a s well a s extensive rater practice, the holistic rubric is nearly m eaningless. Due to this num ber of variables, it is impossible to know how the general categories listed in the rubric are applied to selected student writing by the various GWT readers; therefore, consistency across raters' usage rem ains unclear. As an exam ple, consider again the sam e student-writer from the analytic rubric discussion, who h as written an essay with well-developed content, in an organized presentation, with effective vocabulary, but in which a num ber of grammatical errors in agreem ent or verb tense intrude, although m echanics are controlled. In this situation, the holistic rater could determ ine that th e re a re m ore th an "a few m inor gram m atical inconsistencies." The rater would then assign the 3 score to the essay. With the sam e determination by a second reader, assigning a cumulative total score of 6 from the possible 12, the student's essay would fail, based solely on one descriptor addressing grammatical accuracy. In contrast to analytic scoring in which the explanation of the scores is integral, holistic scoring offers no advantages. In attempting to remedy the results for a future testing situation, a student writer could only g uess at the types of m istakes that were considered important to the raters. Later, once removed from the training sessions and calibrating essays, the raters might have equal difficulty in attempting to explain their scoring since the rubric is far too general to offer reminders. And with the broad descriptors of the holistic scoring guide, a single negative focus could attract readers' attention to the detriment of the test-taker. R esearch has noted that for NNS student writers taking a test under timed and tense conditions, the minor grammatical errors could be sole reason for the failing results (Fein,1980; McGirt, 1984; R oss et al., 1984), thereby exerting a disproportionate influence over the holistic scoring which purports to rate the whole and not its parts. Quantitative Comparisons of A ssessm ent Applications While qualitative analyses of the two scoring instruments highlight their disparities and m ake visible the differences from a read er's perspective, further examination by quantitative analyses suggest som e overlap and distinctions not otherwise evident. The probability level for all statistical analyses w as set at p<.05. General Relationships of Scoring Applications The two types of assessm ent instruments for analytic and holistic scoring with both groups of raters (ESL Specialists and GWT Raters) were 129 examined as a matrix of correlations across original GWT holistic scores, ESL holistic scores, analytic GWT Raters* scores, and analytic ESL Specialists* scores. This matrix yielded data (presented in Table 13) which suggest that the scoring methods are related to a moderately high degree. TABLE 13 GWT Holistic ESL Holistic GWT Analytic ESL Analvtic * GWT Holistic 1 ESL Holistic .723 1 GWT Analytic .732 .593 1 ESL Analytic ,752 .733 .683 1 The two types of scoring procedures which provided the highest correlation were the original GWT holistic scoring and the analytic scoring by ESL Specialists (r^.752). The shared variance (r2=.523) indicates that more than half of the variance is accounted for by the relationship between the two variables; 47% is unaccounted for. Scoring which also indicated a substantial positive relationship included the intersect of correlations for ESL hotistic and ESL analytic (r=.733). Similar coefficent correlations were found for the relationship betw een GWT holistic and GWT analytic scoring (r=.732). This would indicate that the two scoring rubrics are assessing the sam e underlying academ ic discourse traits. The scoring m ethods which appeared to be least related were ESL holistic and GWT analytic scoring (by GWT Raters) (r=.593). 130 In order to exam ine the relationships betw een scoring m ethods further, simple linear regression analyses were performed to m easure how well the analytic variables predicted the holistic scores. The relationships indicated by the matrix d ata w ere supported by ESL analytic sco res regressed on original GWT holistic scores with the correlation coefficient of .752 and adjusted r2 of .561 (F-test*127.391, df 1/98, p<.001). (See Figure 4.1.) The d ata were also supported by GWT analytic scores regressed on original GWT holistic scores with a correlation coefficient of .732 and adjusted r2 =.531 (F*test=112.969, df 1/98, p<.001). (See Figure 4,2.) Figure 4.1 ESL Analytic Scores Regressed on Original GWT Holistic Scores y • 15.042x ♦ 20,654, r * .565 200 1 S O 160 f- 140 cc 120 -100 60 40 6 8 9 10 11 2 3 4 5 7 GWT Orig Hoi Total 131 The regression slo p es of the two scattergram s provide visual confirmation that the two types of scoring, holistic and analytic, with either type of reader, ESL or GWT, appear to be scoring the sam e writing traits in a similar manner. Figure 4.2 GWT Analytic Scores R egressed on Original GWT Holistic Scores y - 14.17* + 23.3, r 2 «.535 ‘ — 200 180 160 2 o • - 140' * re | 120 O100 re < 80 ■ O Ana GWT Raw... 60 40 2 3 6 8 9 10 11 4 5 7 GWT Orig Hoi Total Therefore, with reference to the second research question which asked whether the course criteria (represented by the analytic rubric) were different in kind or degree from the criteria used to a s s e s s the Graduation Writing T est, (a holistically-scored exam ination,) the statistical re su lts' indicate that the two types of evaluation criteria (i.e., holistic and analytic) are generally of the sam e kind; that is, they appear to a s s e s s the sam e overall writing abilities. 132 Application of Analytic Criteria However, while preliminary analyses indicate assessm en t of similar overall writing by the two scoring guides and two groups of raters, the m anner in which the raters evaluate the various analytic subskills appears to depend on group association. Stepw ise regression analyses confirm that analytic criteria w ere differentially applied by the two sets of readers. (S ee Table 14). B ecause this type of analysis allows the variable (each criterion factor, e.g.. Content, Organization, M echanics) which h as the highest correlation with the dependent variable (e.g., total analytic score) to b e entered first (cf. Hatch & Farhady, 1980), the results for the two groups provide data on contrasting approaches. The first stepw ise regression w as perform ed to explore th e relationship betw een the ESL Specialist analytic global scoring and the analytic subcom ponents. As the first step of the analysis, the results show that ESL Specialists applied Language U sage a s the criterion factor most related to their total scoring (adj. R2=.653). However, Content figures prominently in the global rating. In combination with Language Use, it accounts for a combined 88% of the shared variance (adj. R 2=.883). The second stepwise regression analysis examined the relationship b e tw ee n GW T R ater analytic global scoring and th e analytic subcom ponents. The results of the first step in this analysis show that GWT R aters applied Content a s the criterion factor m ost related to their total scoring (adj. R2 =.830). Language Use also figures into GWT R aters1 scoring for a combined 92% of the shared variance (adj. R2 =.920). The two stepwise regression analyses revealed that the two reader groups applied the analytic criteria differently. That is, the higher the Language U se score when being evaluated by ESL Specialists, the higher the e ssay 's total analytic score; in a like m anner, the higher the Content score when being evaluated by GWT Raters, the higher the total analytic score. The three remaining categories of Organization, Vocabulary and M echanics are entered in the sam e order for each group with similar correlation coefficients for each group. TABLE 14 ANALYTIC CRITERIA STEPWISE REGRESSION RESULTS ESL Specialists GWT Raters Criterion Adi. R2 F-test Criterion Adi. R2 F*test S tep 1 Lang. Use .653 376.202 Content .830 975.944 S tep 2 Content .883 748.465 Lang. Use .920 1145.733 S tep 3 M echanics .950 1267.628 M echanics .966 1881.977 S tep 4 Organize. .981 2558.181 Organiza. .986 3511.424 S tep 5 Vocab. 1.0 976310.213 Vocab. .993 5544.267 <P<,001) The m anner in which each reader group applied the criterion factors w as also exam ined through descriptive statistics (Table 15). T hese indicate ESL Specialists' u se of the full range of possible sco res for M echanics (0-20), L anguage U se (0*20), and Vocabulary (0*15). In addition, ESL readers gave higher scores (indicated by the m eans) In all categories than did GWT readers. This usage confirms that the two groups 134 did apply the subcom ponents differently. However, the ESL Specialists' use of the full range of scores suggests that they were discriminating in their scoring of each paper and not merely assigning higher scores. TABLE 15 DESCRIPTIVE STATISTICS OF ANALYTIC SCORING FACTORS ESL Specialists m ean s.d. ranae. GWT Raters m ean s.d. ranae Content 16.62 5.187 5-25 14.82 5.143 4-20 Organization 13.61 4.116 4-20 13.3 3.747 4-20 Vocabulary 9.07 2.961 0-15 8.55 2.824 2-15 LanguageUse 9.35 4.067 0-20 7.92 4.151 2-20 Mechanics JUL81 4.437 0-20 10.19 3.848 3-19 TOTAL....... 60,46 20,768 24-98 54.785 _ 19.713 25-100 Analytic versusHolistlC-Scoring Further statistical analyses examined the relationships betw een holistic scores and analytic variables. An initial multiple regression analysis was performed, regressing GWT Raters' use of analytic variables on the original GWT holistic scores, which found that only two variables significantly predict the direction of the holistic scores. T hese two are Language Use (t-value: 6.86, p<.001) and Content (t-value: 2.723, p<.0071). A subsequent stepwise regression analysis (results presented in Table 16) confirmed the sam e two GWT Raters' analytic variables as significant: Language Use (adj. R2=.441) and Content (adj. R2=.467). Therefore, nearly half of the shared variance is accounted for by these two 135 variables. T he o th er th ree v ariab les (O rganization, V ocabulary, and M echanics) did not achieve significant im provem ent to be entered into the equation. A seco n d multiple regression analysis w as perform ed to exam ine th e relationship of ESL S pecialists' analytic s c o re s and original GWT holistic scores. T he ESL Specialists' analytic variables w ere reg ressed on th e original GWT holistic sc o re s an d found two variables significantly predict th e direction of holistic sc o res: L anguage U se (t-value: 4.586, p<.0001) and Organization (t-value: 2.034, p<.0433). A confirmatory stepw ise regression analysis resulted in three steps. As the first ste p of this analysis, the results confirm L anguage U se a s the criterion factor m ost related to predicting holistic sc o re s (adj. R2 =.407). The second step show s Content a s contributing to the holistic scoring (adj. R 2=.459). And th e third and last ste p sh o w s th e criterion factor of M echanics a s contributing to the holistic GWT scoring (adj. R2 =.467).The o th er two v ariab les, O rganization an d V ocabulary, did not ach iev e sufficient improvement and were not entered into the equation. G raduation Writing T est holistic s c o re s a p p e a r to h av e both L anguage U se and Content a s significant predictors. 136 TABLE 16 GWTJHOLISTIC/ANALYTIC STEPWISE REGRESSION RESULT& ESL Specialists GWT Raters Criterion Adi. R2 F-test Criterion Adi. R2 F-test S te p 1 Lang. Use .407 137.621 Lang. Use .441 158.133 S te p 2 Content .459 85.264 Content .467 8 8 .1 6 9 S te p 3 M echanics .467 5 9 .1 0 8 Not E ntered: Organization Organization Vocabulary Vocabulary M echanics (D<.001) Seeking similar data, but with ESL Specialists' holistic re-scoring instead of original GWT scoring, a sep arate multiple regression analysis used ESL analytic variables regressed on ESL holistic scores. Language U se w as found to contribute significantly (t-value: 3.53, p<.0005). Subsequent stepw ise regression analysis found only two steps: The first w a s L anguage U se (adj. R 2= .261t F-test 71.379); th e se co n d w as Organization (adj. R2=.278, F-test 39.347). Throughout all quantitative analyses, L anguage U se w as the one criterion factor that reliably predicted holistic scores across all readers and both scoring instrum ents. T his m e a n s that holistic sc o re s c h a n g e predictably in relation to the Language Use factor but not in relation to the other four factors. In response to the second research question, the two se ts of criteria do a p p ear similar in general overall a sse ssm e n t of writing abilities but different in the d eg ree to which they a s s e s s using specific factors. The faculty questionnaire totals resulted in a 25% faculty resp o n se for the 137 overall u se of criteria to em phasize C ontent with a 20% resp o n se for Language U se in regular academ ic coursework. However, Language Use contributes significantly more than Content to GWT scores b ecau se raters basically focus on form in evaluating the G raduation Writing Test, while they resp o n d m ore to C ontent in regular academ ic c o u rse writing assignm ents. P hase III: The Re-scoring of GWT E ssays The third research question for this study asked, "How would NNS student scores, especially the pass/fail ratio, b e altered if the sam e criteria used by faculty for evaluation of coursew ork w ere used to a s s e s s student perform ance on the Graduation Writing T est (GWT)?" In order to answ er th e third re se a rc h questio n , o n e hundred previously scored GWT essay s written by eleven NS and eighty-nine NNS students w ere re-scored by four GWT R aters. The e ssa y s were randomly assigned to four sep arate batches, W, X, Y, and Z, in the original score range of 3 to 10. (Totals are presented in Table 17.) Each batch included the following specific num ber of papers in each score: TABLE 17 ORIGINAL GWT SCORED ESSAY DISTRIBUTION BY BATCH BATCH W In) X fnl Yin) Z fn) Orig. GWT Score 3 1 4 2 3 3 3 5 5 5 5 5 6 10 10 10 10 7 4 4 4 4 8 1 1 1 1 9 1 1 1 1 10 1 1 1 1 Total oer batch: 25 25 25 25 Using the analytic rubric, the GWT R aters re-scored the previously scored GWT essay s. The total pass/fail ratios for each batch and for both groups of raters are presented in Table 18. Using the sam e midpoint cut off a s is used for the G raduation Writing T est, this group of GWT R aters scored an additional 32 student e ssa y s a s passing with the analytic rubric; 4 of the previously p a sse d e ssa y s failed. Of th ese four essa y s, all w ere in the original 7 score range. The alterations in scoring involved only NNS student essay s. For the four rater pairings, interrater reliability ranged from .665 to .875. (S ee Appendix S for reliability coefficient table.) A 2-tailed paired t-test of original GWT holistic sco res with analytic GWT R ater raw sc o re s confirm ed a significant difference (t-value: -38.527, p<.0001) betw een the two types of scoring m ethods. The u se of analytic scoring resulted in a significant difference (i.e., improvement) in the pass/fail ratio of NNS student e ssa y s when scored by GWT R aters. This m ean s that an increased num ber of NNS e ssa y s received passing sc o re s w hen their writing w as a s s e s s e d by faculty read ers using criteria which the general faculty say they u se in regular academ ic coursework. (S ee Appendix T for detailed batch scores.) In a sep arate analysis, the ESL Specialists w ere ask ed to re-score the GWT e ssa y s using the original GWT holistic scoring rubric in order to verify that their scoring would conform to the GWT R aters'. All read ers w ere experienced in holistic scoring, but only one had scored the GWT. While scoring independently and lacking the typical hours of training in norming sessions, interrater reliability betw een pairs of ESL Specialists, scoring holisticalty, ranged from .623 to .661, the low end of the m oderate 139 range for acceptability. Altered scores included 13 additional passing and 10 additional failing essay s. A paired W est of original GWT scores with ESL Specialists* holistic scores found no significant difference (2 tailed, t- value: .408, p<.6842). Using the analytic rubric, however, ESL Specialists responded to student essay s with more passing scores than the GWT Rater group (51 additional passes, no additional fails). A paired t-test w as used to compare ESL analytic raw scores to ESL holistic scores, and the results confirmed a significant difference (2 tailed, t-value:-41.806, p<.0001). TABLE 18 ______________ANALYTIC SCORING PASS/FAIL (P/B .R ATI OS_________ Distributed ESL Specialists GWT Raters Orig. GWT Batch Holistic P/F C hanaes P/F Ratio C hanaes P/F Ratio W 7P:18F 10P:0F 17P:8F 13P:0F 20P: 5F X 7P:18F 12P:0F 19P:6F 8P:1F 14P:11F Y 7P:18F 13P:0F 20P:5F 4P:2F 9P:16F Z 7P;18F 16P:0F 23P:2F 7P:1F 13P:12F Total: ESL Specialists: 51 additional Pass; 0 additional Fail 51 altered scores GWT Raters: 32 additional Pass; 4 additional Fail 36 altered scores Grand Total: 83 additional Pass:^.additional Fail SLaltered S C P r< ?s_ A 2x2 chi-square analysis w as used to determ ine w hether a significant relationship existed betw een the two groups of raters a s they applied the analytic rubric. Using pass/fail scores by the two groups of 140 raters, the chi square with Yates correction found that ESL Specialists and GWT Raters grade independently (X*=12.894, p<.0003), confirming that they do not apply analytic scales in the sam e way. In response to the third research question, if criteria used to asse ss NNS writing in regular coursework were used to a sse ss the GWT, more essays authored by NNS student writers would pass the test, regardless of the rating faculty group membership. Reader Reactions In addition to the quantitative analyses of holistic and analytic scoring guide usage, an underlying concern for direct writing assessm ent has always included inconsistency in rater scoring attributed to individual interpretations of the rubrics. In regard to this issue, an unexpected, yet ultimately very important, qualitative concern w as presented and must be addressed. Once readers had completed the various stag es of the scoring, without solicitation all eight were anxious to share their insights by notes, phone calls, and hallway discussions. All readers were insecure about their scores ("I fear I am becoming unnormedl"3), were uncomfortable with their aw areness of certain scoring distribution em phases ("Were they all supposed to be 3 s? H ), and w ere puzzled and fearful about their interpretations of the new analytic scoring rubric criteria. Perhaps the most important aw areness was voiced by the senior GWT Rater, one who had been regularly involved in the GWT scoring since its inception on the cam pus. After completing the analytic scoring, he 141 insisted on discussing his "find," which w as that he w as surprised to see how strong the content and organization of th ese ESL papers were because of his new view through the separate factors. In a later separate discussion, a second GWT Rater expressed a similar comment. With an aw areness that holistic assessm ent may collapse criteria by the one score (and possibly under the weight of one criterion), ESL Specialists had previously expressed unanimous pleasure in rejecting holistic scoring of NNS student writers. They also all agreed that the analytic rubric, while more difficult to m anage due to its novelty, w as a more appropriate m easure of ESL writing. W hen scoring holisticalty, they were very concerned that they had been attracted to specific aspects of writing, (e.g., "shining examples," "correct use of the word advice," and "unclear copies"), fearing that their holistic scores probably reflected these superficial elem ents. They were, therefore, pleased to have scores for assignm ent to these individual aspects that would not disproportionately influence the overall score. One ESL Specialist wrote: T m not sure if this is the point of the two rubrics, but the Analytic Rubric seem s to be a fairer way to score ESL writers than the GWT Rubric." She also noted, however, a s did most raters (som e of whom pointed out their own erasures), that sh e felt "less consistent" with the wide range of scores (0-100) available in the analytic scoring than with the limited choice (1 to 6) in the holistic rubric. After struggling with separate elem ents in the analytic rubric, another GWT Rater said he had the most difficulty sorting through what he considered the difference betw een Vocabulary and Language Use and 142 su g g ested that th e two categories be com bined for future application. He stated that articles and prepositions sta n d out a s vocabulary item s from his perspective. Voicing a similar concern about category cross-over after her reading, an ESL Specialist asked if non-ESL-Specialist read ers would consider m isuse of a noncount noun (e.g., advices) a s a gram m atical error (L anguage U se) or a s a spelling error (M echanics). T his distinction, probably along with a num ber of others, w as never clarified for the readers b ecau se no one asked during the independent readings. T h ese e x p essio n s of co n cern s reinforced a unanim ous resp o n se from both se ts of read ers (ESL Specialists and GWT R aters) that norming through group discussions would have benefited all. Although th e holistic re-scoring by ESL S pecialists w as accom panied by calibrating e s s a y s (sam ple pap ers determ ined to be exam ples of scores), the analytic scoring offered no such opportunity. B ecause all read ers w ere experienced holistic read ers, they m ay have subconsciously expected the support of th e se sam ple p ap ers and the usual supplem ental discussion to further define the scoring guide. Ironically, this expectation of support and clarification s e e m s counterintuitive to the fact that, a s experienced classroom teach ers, they all g rad e student writing independently e ach term . Their a w a re n e ss of group scoring, how ever, p resen ted different n e e d s and expectations for rater reliability than th o se a sso ciated with individual classroom tea ch e r accountability: O nce they becam e writing a ssessm en t raters, they w ere no longer independent co u rse writing evaluators. In spite of initial, albeit limited, explanations and descriptions by th e researcher, this contact w as insufficient support for instilling the n ecessary confidence which the eight re a d e rs required for g e n e ra l a s s e s s m e n t p u rp o se s. How m uch this m e ta m o rp h o sis a lte re d th e c la ssro o m te a c h e rs a s th ey b e c a m e a sse ssm e n t rea d ers is not com pletely known, but the rea d ers them selves w ere aw are of som e degree of ch an g e an d w ere anxious to discuss it. 144 Notes: 1. T hese included Com m unications, Hotel & R estaurant M anagem ent, A erospace Engineering, Chemical & M aterials Engineering, Geology, and Physics. 2. R esp o n se s from the ten d ep artm en ts of Biology, M anagem ent & Hum an R eso u rces, Social S cience, Philosophy, English & Foreign L anguages, Health, Physical Education & Recreation, Behavioral Science, Horticulture, Foods & Nutrition, and Agricultural B usiness M anagem ent w ere divided approximately in half. 3. T hese quotations w ere drawn from notes which faculty raters submitted after their scoring. The researcher p o sse sse s the notes. CHAPTER 5 145 SUMMARY, CONCLUSIONS, IMPLICATIONS AND RECOMMENDATIONS Organization of the Chapter This chapter sum m arizes the study, draw s conclusions b ased upon the significant findings, ad d resses the implications of the research findings for both the teaching and testing of non-native speaking (NNS) student writers, and offers suggestions for change and recom m endations for future research. Summary Purpose The general purpose of the study w as to investigate the efficacy of the Graduation Writing Test (GWT), the Instrument used to implement the California State University G raduation Writing A ssessm ent Requirem ent (CSU-GWAR) at California State Polytechnic University, Pom ona (Cal Poly Pom ona), a s a m easure for assessing the writing com petency of non native s p e a k e rs of E nglish (NNS) enrolled in th e U niversity's undergraduate and graduate program s. The major focus of the study w as the investigation of the relationship betw een two se ts of criteria used for evaluation of NNS student writing at the University: 1) criteria used for regular academ ic coursework (i.e., analytic criteria) and 2) criteria used for the GWT (i.e., holistic criteria). 146 Procedures The research w as conducted in three phases. In the first phase, a questionnaire w as designed and distributed to the faculty at Cal Poly Pom ona, to ascertain the criteria by which faculty a s s e s s NNS student writing in their regular academ ic coursework. Subsequently, 392 returned q u e stio n n a ire s w ere totaled and analyzed for freq u en cies and p e rc en ta g es; in addition, contingency ta b le s provided additional relationships betw een responses and respondents. The survey responses provided data which described general faculty expectations for NNS student writing and which provided details of writing opportunities for NNS students in University coursework. The second phase of the research analyzed responses to Question 15 of the faculty questionnaire, in order to operationalize the d ata regarding coursework writing criteria a s an analytic rubric for comparison with criteria of the GWT holistic rubric. Two groups of raters (ESL Specialists and GWT Raters) re-scored previously rated essay s from Cal Poly Pom ona's GWT in order to com pare the two scales and the w ays in which the two groups of raters applied the scales. Both qualitative and quantitative m ethods were employed to create and com pare the analytic rubric devised for this study with the holistic rubric currently used to evaluate NNS student writing on the GWT. Following the re-scoring of the GWT e ssa y s with the holistic and analytic rubrics, the third phase of this research used quantitative analyses to determ ine differences in pass/fail ratios from the previously-scored 147 GWT e ssa y s of NNS students. Additionally, reader resp o n ses to the scoring procedures were summarized. findings Phase_l:_Summarvof__Questlonnaire_BesDOJses In response to the first research question, "What criteria do faculty use to m easure NNS student writing in their academ ic courses?", the questionnaire responses confirmed the variability of faculty expectations for student writing presented in previous research (e.g., Bridgman & C arlson, 1983; S antos, 1988; Vann, et at., 1991; V aughn, 1991; Janopoulos, 1993). (R esponses to Question 15, which asked for explicit writing criteria, are detailed in the sum m ary of P h a se II, the Analytic Criteria.) In general, faculty perceived the topic of writing by NNS students at the University a s a serious issue, dem onstrated by the high num ber of returned questionnaires from across the disciplines (N=392), the rapid return rate (within three weeks), the few omitted survey responses, and the involvement of senior faculty. Nearly all faculty were aw are of a high percentage of NNS students in their classes every quarter, and most faculty felt that th ese students encountered problem s in their courses. The main areas for concern were listed a s language difficulties, inadequate prior academ ic preparation, and cultural differences. A high response focused on weak writing skills as the m ost significant language problem, which confirmed faculty aw areness and concern. In fact, most faculty generally recognized th ese n eed s and 148 encouraged or offered NNS stu d en ts som e a ssista n ce with writing problems. Many faculty respondents expected writing a s part of coursework but not the sam e type, amount, or m ode a s that which is expected on the GWT, and not under the sam e conditions, such a s writing in-class under time constraints. Interestingly, while the GWT is generally taken at the beginning of th e third or junior y ear (upon com pletion of 135 undergraduate units), an increased em phasis on writing ap p eared in upper-division coursew ork. However, since NNS student writers need opportunities for developing academ ic discourse skills through guided practice with feedback (M endelsohn & Cumming, 1987; Hamp-Lyons, 1991d), the general lack of writing assigned by faculty w as significant, especially resp o n ses that indicated a lack of written work beyond short an sw ers (i.e., exceeding 250 words) and a lack of tim ed writing opportunities. Through the questionnaire's self-report method, faculty said they expected NNS students to write a s much or more than a few years ago; they also expected them to u se technology, including spelling and gram m ar checkers available with com puter usage; and they were divided over whether product or process received the greatest degree of em phasis in coursework. Specific types of writing assignm ents w ere offered by faculty with m odes they w ere able to label, especially d ata support and interpretation, yet for coursework, the types of writing necessary for the GWT were rarely expected. 149 T hree specific variables affected re sp o n se s in a predictable m anner; th ese included length of time teaching at Cal Poly Pom ona, native language background, and GWT scoring participation. Faculty with ten or more years of teaching experience at Cat Poly Pom ona indicated they were willing to change, saying they assigned more writing at the time of the questionnaire distribution than a few years ago, and they assigned more writing at all levels. Additionally, those with this length of teaching time were also more likely to score the GWT. Language background for faculty had an occasionally distinct effect on perceptions and practices. Faculty who w ere native sp eak ers of English (NS), for exam ple, m ore often believed that NNS students encountered course problems than did NNS faculty. In addition, faculty NS offered fewer opportunities for timed writing and said they graded NNS student writing more leniently than did NNS faculty. Differences in perception were also confirmed by those who had scored the GWT. They saw NNS student writing in a different light than did the vast majority (80%) who had not scored the test. In fact, in comparison with non-scoring faculty, those who had scored the test felt that NNS students more often encountered problem s in their courses. Faculty who had scored the test assigned writing more often than they had a few years before, and they assigned more writing at all levels than did non-scorers. However, GWT readers, along with the majority of faculty respondents who had never been readers, were also split in their opinions of whether NNS students should be required to m eet the sam e standards a s NS 150 students, and GWT read ers were divided in their resp o n ses to grading more leniently or with equal severity. Nearly all faculty were able to assign usage em phases to specific criteria, such a s content, organization, spelling, and grammatical control, and did so to varying degrees. B ecause of the low non-response rate to the very detailed criterion, it appeared that most faculty respondents both recognize and apply their understanding of various evaluation criteria to their students' writing for coursework. P erhaps the most revealing findings drawn from the questionnaire are those which highlighted the discrepancies betw een beliefs and practice. These are significant because they may be responsible for the rift in determining which criteria faculty actually use to a s s e s s ESL student writing. Faculty attitudes toward standards were contrasted with grading practices and show ed that faculty have not resolved this concern for them selves; that is, they do not ag ree that all students (NS and NNS) should or should not be required to m eet the sam e criteria for writing, and faculty do not agree that they should grade all student work in the sam e m anner. Confirming this variability w ere com parisons betw een attributed usage and importance of selected criteria: While respgnses to Question 16 revealed that m ost faculty respondents thought nearly all listed criteria w ere important, the resp o n ses to Question 15 countered this belief by revealing that faculty do not actually a s s e s s their own students' writing using the criteria to which they attribute importance. 151 Phase.ll;.Summary_of.Analytic Criterion Application Quantitative and qualitative m easu res w ere em ployed to answ er the second research question, "Are the course criteria different in kind or degree from the criteria used to a s s e s s the Graduation Writing Test, one version of the California State University-m andated G raduation Writing A ssessm ent Requirem ent? If so, in what way(s) a re th e se two s e ts of criteria different?" The responses to Question 15 of the survey w ere weighted, totaled and used to devise a new analytic rubric. The analytic rubric included five factors: Content 25%, Organization 20%, L anguage Use 20%, M echanics 20% , and Vocabulary 15%. R aters assig n ed points corresponding to percentages (e.g., Content 0-25 possible points). The advantages of the analytic criteria are that they are detailed and easily explained to raters; no single factor b e ars a disproportionate influence on the cumulative score; and results are available for explication to students by points assigned to each factor. The holistic rubric used on the Graduation Writing T est includes 6 points, with 4-5 listed a s high-low levels of a single hyphenated score with a single descriptor, thus, mathem atically creating a 5-point continuum of scores. T he disadvantages of this rubric are that descriptions for each level m ust be explained to raters through the u se of sam ple p ap ers and training discussions; a single descriptor may attract rater's attention and b e a r heavily on assig n ed sco re (e.g., "...m arred by m ore than a few grammatical errors"); and the results of the criterion application are difficult to explain to students afterward. 152 Further com parisons w ere m ad e possible by re-scoring 100 previously scored GWT essay s, the majority of which w ere written by NNS student writers (89 NNS; 11 NS). C om parisons of analytic and holistic m ethods w ere conducted on scoring results of two different groups of raters, ESL S pecialists and GWT R aters. Statistical a n a ly se s w ere conducted to evaluate similarities and differences betw een the two se ts of criteria. The scoring criteria w ere alike in kind; that is, both m ethods of scoring with both groups of raters w ere assessin g the sa m e underlying writing abilities. Language Use and Content w ere the two analytic criteria responsible for reliably predicting holistic scores. Differences w ere also evident, especially the m anner in which the two groups of raters applied the criteria. ESL Specialists applied the criterion factor of Language Use most, with Content also prominent, while GWT R aters applied the criterion factor of Content m ost, with Language Use, second. And while it initially appeared that ESL Specialists assigned higher sco res com pared to GWT R aters, the ESL Specialists used the entire range of scores, while GWT R aters did not apply sco res at the low end of the range. An important difference betw een th e questionnaire resp o n se s for criteria used in coursework and the criteria used for GWT scoring becam e evident in this comparison. The questionnaire resp o n ses indicated that, for coursework, Content w as the m ost important factor (i.e., 25%), but for the GWT essays, raters appeared tp focus more on Language Use in scoring. 153 P h a se III: Sum m ary of Pass/Fail Differences T he third re se a rc h question ask ed , "How would NNS stu d en t sco res, especially the pass/fail ratios, be altered, if th e sam e criteria used by faculty for evaluation of coursew ork w ere u se d to a s s e s s stu d en t p erfo rm an ce o n th e G raduation W riting T est?" In re s p o n se to this question, m athem atical and statistical analyses w ere used. Using the sa m e midpoint cut score a s th e GWT, the analytic rubric w as applied by both se ts of raters. The results indicated that, if th e criteria u se d in coursew ork w ere applied to GWT e ssa y s, m ore NNS stu d en t e s s a y s would p a s s , reg ard less of th e rater group. Specifically, analytic scoring by pairs of GWT R aters resulted in 32 additional passing e ssa y s from the original corpus of 100 e ssa y s, with an additional 4 failing e ssa y s (from th e 7 range of sco res, which included only NNS student e ssa y s). And scoring by pairs of ESL Specialists resulted in 51 additional passing e ssa y s, with no failing e ssa y s. A com parison of pass/fail ratios, however, found that th e two rater groups w ere applying th e rating scale differently. After scoring, both groups, ESL Specialists and GWT R aters, w ere anxious to d iscu ss their co n cern s and findings. Two GWT R aters w ere surprised to s e e that content and organization w ere strong in th e NNS student e s s a y s a s they applied analytic scales. T he ESL Specialists w ere unanim ous in their rejection of holistic scoring and ag reed that th e analytic rubric w as a m ore appropriate way of scoring e s s a y s written by NNS students. Several points of concern for clarity within the new scoring guide w ere e x p re sse d after th e scoring had b e e n com pleted. T he prim ary co n cern of both g ro u p s w as that th e re a d e rs felt dep riv ed of th e opportunity to discuss their scoring in a trainingfnorming session. As a consequence, they felt insecure and uncomfortable with their scores, in spite of the fact that all raters were experienced holistic readers, a s well a s experienced classroom te a c h e rs who g rad e their students' work independently every term. Conclusions Faculty Awareness of NNS Writing Problems Faculty at Cal Poly Pom ona are aw are that NNS students have writing problems in academic coursework. Awareness of the serious issue of NNS student writing at the University was confirmed by the high return rate for the faculty survey (N=392), which represented the entire University, (i.e., all forty-seven departm ents, In six colleges and one school) and which included a wide range of faculty opinions and practices focusing on NNS student writing in coursework and, consequently, on the GWT at Cal Poly Pomona. The rapid return of the questionnaires (within three weeks), and the fact that, of the 28 questions with num erous subcategories, few had omitted reponses, all indicate faculty engagem ent with the issues raised in assigning and grading NNS student writing. In addition, nearly all faculty were aw are of NNS students enrolled a s a high percentage of students In their courses every quarter. T hese data support the general figures of the NNS student population In California, and draw attention to the impact of NNS student needs in the University curriculum. 155 Faculty Participation in Scoring the GWT While faculty are aw are of NNS student writing problem s in coursework, faculty do not support the GWT by participation in the scoring of the test nor by preparation of NNS students for the writing which will be expected of them . Since over 80% of the faculty responding to the questionnaire have never scored the GWT, in spite of the fact that it is, according to the Chancellor's Executive Order, an "all cam pus test" (1987, p. 3), a serious problem is the lack of potential backwash from the GWT to the regular academ ic coursework, resulting in diminished opportunity for faculty and coursework to be affected by the sharing of standards and requirements for student writing, including NNS student writing, although White (1985) h as argued that such backwash should be an important result of holistic scoring sessions. The questionnaire responses confirmed, consequently, that the a w are n ess of NNS student problem s in coursework, the opportunities for writing lengthy products under timed conditions, and the applications of clear standards and criteria, supported by appropriate grading policies, are not effected a s often by faculty who have not shared these Ideas with other faculty across the disciplines in GWT scoring sessions a s they are by GWT raters. Faculty Pedagogical Em phases Faculty do not teach and do not test the sam e skills in regular academ ic coursew ork a s those required for the GWT. Taken at the beginning of the junior year of studies (completion of 135 quarter units), the GWT is a 75-minute essay examination, written on a topic which often 156 requires the personal expository mode, and written by hand in ink, without reference opportunities (i.e., no dictionaries). In contrast, for coursework, faculty expect students to use computers, word processors, or typewriters, which have the advantages of spelling and gram m ar checkers. Faculty require few assignm ents for courses (outside those offered by the English and Foreign L anguages Department) that use the m ode of personal exposition, a s m ost course writing involves d ata interpretation and support, or description, or comparison/contrast. And few faculty assign work for any courses, but especially for lower division courses, that requires in-class writing under a time limit. Lower division courses also have fewer opportunities for writing of any type (i.e., general, lengthy [over 250 words], or timed) than are offered In upper division coursework; however, since the GWT should be taken at the beginning of the junior year, students should have already taken and passed the test by the time these increased opportunities are available to them in upper division courses. Additionally, because an entire essay is expected for the GWT. the fact that 40% of the faculty never assign writing for lower division courses that exceeds the equivalent of one typed page (250 words), and only one-third of the faculty expect 1 to 2 assignm ents of this length, significantly few opportunities exist for NNS students to prepare for the University examination of writing competence in the GWT. Discrepancies between teaching and testing may be attributed to the purposes of each. Faculty in academ ic courses expect students to control the content of the course (e.g., the details of a revolution in a history class), which forms the basis for m ost writing assignm ents; 157 w hereas for a general test to m easure writing com petence, content appears to be a much less salient factor as long a s ideas are developed, leaving the em phasis on form to take precedence over other criteria. Additionally, differential access to materials in the two settings of coursework and GWT testing confirm a discrepancy in faculty and testing administrator expectations. The university involved in this research is a polytechnic institution, with a dominant em phasis on engineering and com puter sciences and technologies. It not only attracts and admits students for whom English writing skills may be weak or perceived to be less important than computational skills, but validates through coursework the expectation for technological assistance (i.e., spelling and grammar software) to overcome grammatical and mechanical errors. The difference in expectations for GWT writing practices and coursework writing practices is punitive for NNS students who have learned English and controlled writing for coursework through computer assistance but are not allowed the sam e tools for a test which permits or denies graduation from the University. A high-stakes educational test needs to reflect the curriculum in all aspects, including its technology. D ifferences betw een stan d ard s and grading also m arked a particular rift in faculty expectations for NNS students. The fact that faculty often said they believed that both NS and NNS students should be required to m eet the sam e criteria for writing, while many still announced that they graded NNS student writing more leniently, creates an additional cause for concern. When faculty, assign grades by one standard to a group of students in consideration of a particular exception (in this case, English 158 writing skill), yet later, on an exit exam ination, hold the sam e group accountable to stan d ard s which they w ere not required to m eet in preceding academ ic coursework, the students are doubly penalized. gaculty_A ppltcatiQ D -of.Pifferinfl-C riter.ia A lack of faculty consensus Is reflected in the differences betw een criteria used to a sse ss NNS student writing in regular academ ic courses and the criteria used to a sse ss NNS student writing on the GWT. In course writing, faculty expect Content to be the m ost important indicator of com petence, followed by Organization, Language Use, and M echanics at approximately the sam e degree of em phasis, with Vocabulary a s the least important. On the other hand, the GWT essays, scored a s final products, attract readers to Language Use (the only factor which reliably predicted holistic scores in all analyses across both analytic and holistic instruments and with both s e ts of re a d e rs); th erefo re, th e "gram m atical inconsistencies" of th e m idpoint 3 sc o re, a p p e a r to attract a disproportionate amount of attention a s faculty readers attribute more to the factor which focuses on gram m ar than to the others. The Graduation Writing Test appears to be testing gram m ar (i.e., Language Use) over all other possible factors. Consequently, a s a test of grammar, this focus has a predictable penalizing effect on NNS student writers. In addition, NNS students are affected in a practical se n se by the penalizing effect of the difference in the application of scoring criteria. Considerably more essay s were p assed using the analytic scoring guide than were passed using the GWT guide. Since 32 additional NNS essay s 159 w ere p assed with the paired results of analytic scoring by the GWT R aters and 51 additional e ssa y s w ere p a sse d with the paired results of analytic scoring by ESL Specialists (each from the original corpus of 100 e ssa y s with an original ratio of 72 fail:28 pass), the difference betw een course criteria, used in this research a s an analytic rubric, and holistic criteria, borrow ed from the GWT rubric, confirm s a se rio u s discrepancy in expectations and opportunity for NNS students facing the GWT. O ne specific exam ple, confirm ing rater variability b a se d on academ ic discipline a s group m em bership, w as the m anner in which rater groups applied th e analytic criteria to previously scored GWT e ssa y s (V aughan, 1991). D espite similar training, the ESL Specialists focused m ore on L anguage Use, with Content a s an important second factor, but GWT R aters focused more on Content. The em phasis on Content by GWT R aters, while supportive of overall faculty resp o n se to criteria applied in coursew ork, is notable in that it w as th e se sam e raters who ex p ressed surprise at the quality of content and organization in the e ssa y s which the analytic scales required th ese raters to directly a s s e s s in addition to other criteria. Their u se of the new scales may have presented novel criteria for a sse ssm e n t, contributing to a different and positive em p h asis in their scoring. R ater variability w as also confirmed in how the raters applied the scale. Although popular opinion and the higher p a ss rate for NNS student e ssa y s with ESL raters applying the analytic sc ale s (i.e., 51 additional passing papers) m ay encourage the idea that ESL tea ch e rs are m ore lenient graders, the results of this study do not b e ar this out. While the 160 m ean of factors for ESL readers su ggests that ESL Specialists assigned higher scores, this group, as a whole, m ade more distinctions within their scoring, evidenced by the descriptive statistics which show that they used the full range of possible scores, while GWT R aters w ere unwilling to assign scores at the lowest end of the range. The variables attributed to faculty differences and testing problems are considered in the significant findings which resulted from scoring. In general, both types of scoring and both groups of raters w ere assessing basically the sam e writing skills, regardless of the scoring instrument used. T hese results also help to explain the recom m endations by Gumming (1990) who, after studying the effects of both types of scoring, advocated holistic scoring over analytic scoring for large-scale assessm ent, based on expense and time requirements for holistic scoring, and the assessm ent of similar skills. However, differences betw een groups w ere found in application of analytic criteria, which required that raters apply a greater number of criteria than for holistic scoring a s each analytic factor had to be considered and assigned a score. The use of analytic rubrics com pared with holistic rubrics w as initially perceived a s problematic when differences in pass/fail ratios for analytic scales resulted, since both types of ratings appear to a sse ss the sam e underlying skills, and both groups of raters appear to be using Language Use and Content a s criteria for their assessm en ts. O ne likely explanation of the differences in additional passing p a p ers with the analytic sc a le s is created by th e original batch distributions of 25 previously scored essay s in each. Since 19 of the 25 essays in each batch 161 w ere from the midrange scores of 5, 6, and 7 (ail including the 3 score), purposely selected to include the score of 3 (which NNS students have the m ost difficulty in overcoming), the higher sco res assigned by ESL Specialists could be attributed to these readers' abilities to look beyond typical ESL surface errors (cf. Bochner et al., 1992). But b ecau se GWT Raters' results also included 32 additional passing and 4 additional failing essay s (from the entire corpus of 100 essays), the findings by Mendelsohn and Cumming (1987) which maintain that analytic scales are the best m easure for discriminating scores in the midrange, appear applicable to this research also. The concern for time a s an issue in decisions on asse ssm e n t instruments considered in previous research (e.g., Spooner-Smith, 1980; Bauer, 1981; Quellmalz, 1984) does not apply in this c a se since the readers were not reading simultaneously and collectively, and w eeks w as the tim e allotm ent instead of the day or two typically required for communal scoring. On the other hand, none of the raters involved in the use of scoring with the analytic rubric w as uncomfortable with the time which analytic scoring required; one rater even com m ented that once she w as normed, she felt it took her no longer to score analytically than it did holistically. Reliability for all pairs of read ers w as m oderate, even without training, sam ple papers, and practice scoring, but this may be attributed to the considerable prior holistic scoring experience all raters involved in this study p o ssesse d . The eight raters all expressed concerns over their reliability and norming , which again may be linked to their experiences, 162 an d thus, to their expectations (Janopoulas, 1993). V aughan found that, even with training sessio n s, raters in her study w ere uncertain about their judgm ents and concluded, "Holistic a sse ssm e n t is a lonely act" (1991; p. 120). R aters' expectations in this research confirm ed this assum ption a s they sought support through a com m unal event. S ince th e m odes, tim e limits, referen ce a c c e s s , technological a c c e ss, practice opportunities, and criterion em phasis on th e GWT all differ from th e general tren d s ex p ected for writing in regular adadem ic coursew ork, the stan d ard s and expectations of the GWT are not reflected in the curriculum of the University. As such, the GWT at Cal Poly Pom ona ap p ears to lack content and educational validity. Implications for Teaching and Testing W hen the GWAR w as assig n ed to the individual cam p u se s of the California S tate University system , it soon b ecam e evident that no single te st would b e a c c e p te d by th e differing faculty on e a c h c a m p u s. C onsequently, each cam p u s w as allowed to devise its own form of the test. The version used at Cal Poly Pom ona h as three major problem s: The generic holistic scoring guidelines used for th e GWT are discernible only to th e few faculty who h av e sc o red th e te st repeatedly (cf. B aker & Q uellm alz, 1979); th e te st lacks co n ten t validity in that it d o e s not rep resen t th e criteria and type of writing used in academ ic coursew ork; and academ ic coursew ork d o e s not offer all stu d en ts, especially NNS stu d en ts, opportunities to learn th e skills which th e GW T te sts, T he curriculum at Cal Poly P om ona cannot b e instantly revised to m atch the 163 test, but a test could be devised to represent the curriculum and faculty expectations, and to allow for the reciprocity of the test with teaching, which is required for educational validity. Suggestions for C hange Data Collection O ne of th e difficulties faced in this research w as establishing precise num bers of NNS students. R ecords are not m aintained for this segm ent of the University student population. A consequence of this lack is that only anecdotal evidence is available to alert adm inistrators and faculty to the special or unm et n e e d s of th e se stu d en ts. W hen the num bers are known, the impact may cau se these concerns to be seriously studied by researchers at the University. The T est The general underlying assum ption of the G raduation Writing A ssessm ent Requirem ent is that "...because there is no distinction in the value of deg rees students can earn -reg ard less of their native lan g u a g es- there should be no distinction in expectations for students on the GWAR" (ESL W orkgroup, 1988; p. 9). In suggesting opportunities for NNS students to satisfy th e GWAR in alternative ways, this chapter d o e s not suggest a lowering of standards. Instead, the present im balance, which is exacerbated by current GWT holistic scoring, n eed s to be corrected. The failure rate of 80% for NNS students may not be the product of only w eak writing skills. Rather, the rate may additionally be attributed to the varying 164 faculty expectations and writing opportunities to which NNS students are exposed in academ ic courses. Hamp-Lyons (1991c) contends that NNS stu d en ts, w hen p resen ted with learning opportunities, h av e "a real possibility for interlingual d ev elo p m en r at the University (p.243). The opportunities m ust be adjusted and the test itself m ust be linked to th ese opportunities. "If a test d o e s not work, it is abandoned and replaced by a better one" (White, 1986;p.76). Merely correcting the scoring guide to reflect the even-num bered scale from which it originated, so a s to avoid the midpoint aggregation of e ssa y s (as suggested in the 1985 G reene report), is not sufficient since the 6-point scale is only a statistical ploy for increasing reliability, not validity. As White suggests, a "better" test to fulfill the writing com petency examination needs to be created for this cam pus, one which reflects coursework and opportunities which students have for writing, one in which faculty believe and act on in their classroom practices, and one in which they participate a s raters. O ne type of such evaluation would consist of a writing test to represent the kind of writing necessary in a student's major, a s suggested by Jo h n s (1991a). Faculty from each student's major would be required to confirm the student's com petence In the necessary writing factors (e.g., m ode, style, and diction). The opportunities for backw ash to the specific departm ents and m ajors are evident. The dichotomy which splits teachers from raters (Rodby, 1992) would be erased by consistent stan d ard s in both settings. 165 Portfolio a sse ss m e n t h a s also b een su g g e ste d to fill th e g ap b etw een teaching and testing, and it offers m any of th e s e sa m e advantages (with num erous additional difficulties in m anaging the logistics and p ap er load), although co n cern s over holistic judgm ents of th ese m e a su re s c re a te th e possibility for the sa m e type of inappropriate assessm en t a s the current GWT holistic scoring (White, 1993). Given the fact that great num bers of NNS students are penalized by the scoring of the GWT each quarter, the need for im m ediate action in devising an alternative to the current holistically-scored e ssay examination is necessary. Recom m endations for Future R esearch This focus for this research w as predom inantly aim ed at NNS student writers at the University. While som e of the resp o n ses m ay have included faculty criteria applied to both native and NNS students, future research on the specific differences betw een the two groups a s perceived by faculty would add to this groundwork. Further, variability am ong and within raters with respect to issu es of NNS student writing h as been confirmed in this research. However, it w as beyond the scope of this study to a s s e s s a range of rater variables a s isolated factors. If raters could b e m atched for background variables, further information on how writing criteria are applied might be gained and u sed to assist in th e developm ent of writing a sse ssm e n t in order to consider the degree of influence attributed th ese variables. References 166 Allaei, S.K. & Connor, U. (1991). Using performative assessm ent instruments with ESL student writers. In L. Hamp*Lyons (Ed.), Assessing Second Language Writing in Academic Contexts (pp.227-240). Norwood, NJ: Ablex. Arthur, B. (1979). Short-term changes in EFL composition. In C.A. Yorio, K. Perkins, and J. Schacter (Eds.), On TESOL *79:T h e LearnerJn Focus (pp.330-3421. W ashington, D.C.:Teachers of English to Speakers of Other Languages. Baker, E.L.& Quellmalz, E. (1979). Study No.3, Effects of topic, sam ple and rater group membership on the stability of scoring criteria application. In Results of Pilot Studies: Effects of Variations in Writing Task Stimuli on the Analysis of Student-Writing Perform ance. Los Angeles: University of California. Bamberg, B. (1984). A ssessing coherence: A reanalysis of essay s written for the National A ssessm ent of Educational Progress, 1969-1979. R esearch in the Teaching of English. 18.305-319. Barritt, L., Stock, P., & Clark, F. (1986). Researching practice: Evaluating assessm ent essays. College Composition. 3 7 .315-327. Basham, C.S. & Kwachka, P.E. (1991). Reading the world differently: A cross-cultural approach to writing assessm ent. In L. Hamp-Lyons (Ed.), A ssessing Second Language Writing in Academic Contexts (pp.37-50). Norwood, NJ: Ablex. Bauer, B.A. (1981). A study of the reliabilities and the cost-efficiencies of three methods of assessm ents for writing ability. Champaign: University of Illinois. (ERIC Document Reproduction Service No. ED 216357). Bereiter, C. (1980). Development in writing. In L.W. Gregg & E.R. Sternberg (Eds.), Cognitive P rocesses in Writing. Hillsdale, NJ: Erlbaum. Blok, H. & de Glopper, K. (1992). Large scale writing assessm ent. In L.Verhoeven & J. H.A.L. DeJong (Eds.), The Construct of Language Proficiency: Applications of Psychological Models to Language A ssessm ent. Philadelphia, PA: John Benjamins Publishing Co. 167 Bochner, J.H., Albertini, J.A., Samar, V.J., & Metz, D.E. (1992). External and diagnostic validity of the NTID Writing Test: An investigation using direct magnitude estimation and principal components analysis, Research in the Teaching of English. 26.(3),299-314. Borowiec, E.J. (9188). Pathways to Qualitv through Diversitv:_The CSU GWAR in Transition. Long Beach, CA: Office of the Chancellor, California State University. Breland, H.N. & Jones, J.J.(1984). Perceptions of writing skills. Written Communication. 1(1), 101-119. Bridgeman, B., & Carlson.S. (1983). Survev of Academic Writing Tasks Required of Graduate and Undergraduate Foreign Students fRR 83-18, Report 15). Princeton, NJ: Educational Testing Service. Brossell, G. (1986). Rhetorical specification in essay examination topics. College Enotish. 45.165-173. Brown, R.C., Jr. (1986). Testing black student writers. In K.L. Greenberg, H.S. Weiner, & R.A. Donovan (Eds.), Writing Assessment: Issues and Strategies (pp. 45-52). New York: Longman. Brown, J.D. (1991). Do English and ESL faculty rate writing sam ples differently? TESOL Quarterly. 25 (4), 587-603. Brown, J.D. & Bailey, K.M. (1984). A categorical instrument for scoring second language writing skills. Language Learning. 34 (4), 21-42. California State University, (1980). Graduation Writing Test Rubric. Pomona, CA: California State Polytechnic University. California State University, (19931. Cal Polv Pom ona's Strategic Plan. 1993-94. Pomona, CA: California State Polytechnic University. California State University, (19941. Toward the 21st Centurv:Ca! Polv Pom ona's Five-Year Plan. Pomona, CA: California State Polytechnic University. California State University, (19941. Cal Polv Pomona Catalog. Pomona, CA: California State Polytechnic University. Carlson, S.B. (1991). Program evaluation procedures: Reporting the program publicly within the political context. In L. Hamp-Lyons (Ed.). Assessing Second Language Writing in Academic Contexts (pp.293-322), Norwood, NJ: Ablex. 168 Carlson, S.B., Bridge man, B., Camp, R., & W aanders, J. (1985). Relationship of Admission,Test Scores to Writing Performance of, Native and Nonnative Speakers oLEnollsh (RR 85*21, Report 19). Princeton, NJ: Educational Testing Service. Chancellor's Office (1987). Executive Order 514. Competency in Student Writing Skills. Long Beach, CA: Office of the Chancellor,California State University. Chancellor's Office (1982,1990) Survey of CSU Upper Division Writing Proficiency Requirement. Long Beach, CA: Office of the Chancellor,California State University. Chapa, J. (1990). Population estim ates of school age language minorities and limited English proficiency children of the United States, 1979* 1988. Proceedings of the First Research Symposium on Limited English Proficient Students' Issues. Washington, D.C.: Office of Bilingual Education and Minority Affairs, U.S. Department of Education. Charney, D. (1984). The validity of using holistic scoring to evaluate writing: A critical overview. Research in the Teaching of English. 18 (1), 65-81. C hase, C.l. (1968). The impact of som e obvious variables on essay test scores. Journal of Educational M easurement. 5_(4), 315-318. Cherry, R.D., & Meyer, P.R. (1993).Reliability issues in holistic assessm ent. In M.M. Williamson and B. A. Huot (Eds.), Validating Holistic Scoring for Writing Assessm ent: Theoretical and Empirical Foundations.fp p .1 09-141). Cresskill, NJ: Hampton Press, Inc. Coffman, W.E. (1971). On the reliability of ratings of essay examinations in English. Research in the Teaching of English. 5* 24-37. Cohen, A. D. (1994). Assessing Language Ability in the Classroom , 2nd ed. Boston, MA: Heinle & Heinle. Collier, V.P. (1994). Table discussion at 1994 TESOL Conference, Baltimore, MD. Conklin, E.L, (1982). Writing answ ers to essav questions: A naturalistic study of the writing process. Unpublished doctoral dissertation, Indiana University of Pennsylvania. 169 Connor, U.N., & Carroll, P.L. (1993). The interpretation of task s by writers and rea d ers in holistically rated direct asse ssm e n t of writing. In J.G . C arson & I. Leki (Eds.), Reading in the Composition C lassroom : S eco n d L a n o u ao e E e rsp e ctiy es.fPP. 141 >160). Boston, MA: Heinle & Heinle. Cooper, C.R. (1977). Holistic evaluation of writing. In Cooper, C. R., & Odell. L. (Eds.),. Evaluating Writing; D escribing. M easuring. Judging (pp. 3>32). U rbana, II: National Council of T eachers of English. Cortes, C. E. (1992). The education of language minority students: A contextual interaction model. In Bevond Language: Social and Cultural Factors in Schoolino L anouaoe Minoritv. S tu d en ts. Sacram ento, CA: Bilingual Education Office. Coxwell, D.L. (1991). T he clash betw een teachers* personal view sof. student writing and.view s im posed b v th e state. (ERIC Docum ent Reproduction Service No. ED 332 209). C ronbach, L.J. (1988). Five perspectives on the validity argum ent. In W. W ainer & H.I. Braun (Eds.), Test Validity (pp. 3-127). Hillsdale, NJ: Lawrence Erlbaum. Cumming, A. (1990). Expertise in evaluating second language compositions. L anguage Learning. Z.O), 31-51. Cum mins, J . (1979). Linguistic interdependence and the educational developm ent of bilingual children. Review of Educational R e se a rc h . 42, 222-251. Davidson, F. (1991). Statistical support for training in ESL com position. In L. Hamp-Lyons (Ed.). A ssessing S e cond L anguage Writing in Academ ic C ontexts (pp.155-166). Norwood, NJ: Ablex. Diedrich, P.B. (1974). M easuring Growth in ■English. U rbana, IL: National Council of T each ers of English. Diedrich, P.B., French, J.W ., & Carlton, S.T. (1961). Factors in judgm ents of writing ability. (R esearch Bulletin 61-15). Princeton, NJ: Educational Testing Service. (ERIC Docum ent Reproduction Service No. 002-172). Educational Testing Service (1986). T est of W ritten English Rubric. Princeton, NJ: Author, Elbow, P. (1993). Ranking, evaluation, and liking: Sorting out three forms of judgm ent. CollegeEnQ lish..55 (2), 187*206. ESL W orkgroup, (1988). Report of the English a s a S econd L anauaoe W orkgroup. Long Beach, CA: Office of the Chancellor,California S tate University. Fein. D.M. (1980). A com parison of English and ESL com positions. Unpublished m aster's thesis, University of California, Los Angeles. Follman, J.C ., & Anderson, J.A. (1967). An investigation of the reliability of five procedures for grading English them es. R esearch in the Ieflgbipg-pIEnglisb, l* 190-200. Freedm an, S.W . (1979). Why do teach ers give the g rad e s they do? Coileoe_Composition and Communication. 30. 161-164. Freedm an, S.W . (1981). Influences on evaluators of expository essays: Beyond the text. R esearch in the Teaching of English. 15. 245-255. Freedm an, S.W. (1984). The registers of student and professional expository writing: Influences on teach ers' resp o n ses. In R.Beach & L.S. Bridwell (Eds.). New Directions in Com position R esearch (pp.338-339) New York: The Guilford P ress. Freedm an,S.W ., & Calfee, R.C. (1984). Understanding and com prehending. Written Com m unication. 1 (4), 459-490. Freedm an, S.W., & Robinson, W .S. (1982). Testing proficiency in writing at San Francisco S tate University. College Composition and Communication. 2 3 (4), 393-398. G aschler, L. (19 9 0 ,1 9 9 2 ,1 9 9 4 ). N otes from personal interviews with the Testing C enter Officer, Cal Poly Pom ona. G ere, A. R. (1980). Written composition: Toward a theory of evaluation. College English. 4 2 (1). 44-58. Godshalk, F.I., Swineford.F., & Coffman, W.E. (1966). The M easurem ent of Writing Ability. Princeton, NJ: College Entrance Examination Board. G reen, P.S. & Hecht, K. (1985). Native and nonnative evaluation of learners' errors in written discourse. System . 13.77*97. 171 G reene, J. A. (1985). Graduation Writing Test Program Review: Final Report. Pom ona, CA: California S tate Polytechnic University. Grobe.C. (1981). Syntactic maturity, m echanics, and vocabulary a s predictors of quality ratings. R esearch in the le a c h in o of Enolish. 15. 75*86. Halpern, S., Spreitzer, E., & Givens, S. (1978). Who can evaluate writing? College Composition and Communication. 2 2 (4), 396*397. Hamp-Lyons, L. (1987). Perform ance profiles for academ ic writing. In K. Bailey, R. Clifford & E. Dale (Eds.), Language Testing R esearch (pp.78*92). Monterey, CA: D efense Language Institute. Hamp-Lyons, L. (1991a). Basic concepts. In L. Hamp-Lyons (Ed.), A ssessing Second Language Writing In Academic_Contexts (pp.5- 18). Norwood, NJ: Ablex. Hamp-Lyons, L. (1991b). Reconstructing "academ ic writing proficiency." In L. Hamp-Lyons (Ed.), A ssessing Second Language Writing in Academic Contexts (pp.127-154). Norwood, NJ: Ablex. Hamp-Lyons, L. (1991c). Issues and directions in assessin g second language writing in academ ic contexts. In L. Hamp-Lyons (Ed.), A ssessing Second Language Writing in Academic Contexts (pp.323-330). Norwood, NJ: Ablex. Hamp-Lyons, L. (1991d). Pre-text: Task-related influences on the writer. In L. Hamp-Lyons (Ed.), A ssessing Second Language Writing in Academic Contexts (pp.87-110). Norwood, NJ: Ablex. Harris, W.H. (1977). T eacher response to student writing: A study of the response patterns of high school English teachers to determ ine the basis for teacher judgm ent of student writing. R esearch in the Teaching of English. 11,175-185. Hatch, E.M. & Farhady, H. (1982). R esearch Design and Statistics for Applied Linguistics. Rowley, MA: Newbury House. Henerson, M.E., Morris, L.L, & Fitz-Gibbon, C.T. (1978). How to M easure Attitudes. Newbury Park, CA: S age Publications. Henning, G. (1991). Issues in evaluating and maintaining an ESL writing assessm ent program. In L -. Hamp-LvonsfEd.L A ssessing Second Language Writing in Academic Contexts (pp.279-292). Norwood. NJ: Ablex. 172 Hillocks, G., Jr. (1986). B esearch on Written Composition: New Directions for Teaching. Urbana, IL: National Conference on R esearch in English. Homborg, T.J. (1984). Holistic evaluation of ESL compositions: Can it be validated objectively? TESOLQuarterlv. 18,87*107. Horowitz, D. (198A ). W hat professors actually require: Academic tasks for the ESL classroom . TESOL Quarterly. 28. (3). 445*462. Horowitz, D. (1991). ESL writing assessm ents: Contradictions and resolutions. In L. Hamp-Lyons (Ed.), A ssessing Second Language Writing in Academic Contexts (pp.5-18). Norwood, NJ: Ablex. Hughes.D.C., Keeling, B., &Tuck, B.F. (1983). Effects of achievement expectations and handwriting quality on scoring essavs. Journal of Educational M easurem ent. 2Q, 65*70. Huot, B.A. (1990a). The literature of direct writing assessm ent: Major concerns and prevailing trends. Review of Educational Research. fi(U2), 237-263. Huot, B.A. (1990b). Reliability, validity, and holistic scoring: W hat we know and what we need to know. College Composition and Communication. 41 (2 ), 201*213. Huot, B.A. (1993). The influence of holistic scoring procedures on reading and rating student essays. In M.M. Williamson and B. A. Huot (Eds.), Validating Holistic Scoring for Writing Assessm ent: Theoretical and Empirical Foundations (pp.206*236). Cresskiil, NJ: Hampton Press, Inc. IRA/NCTE Joint Task Force. (1994). Standards for the A ssessm ent of Reading and Writing. International Reading Association and National Council Of T eachers of English. Urbana, IL: Author. Jacobs, H.L., Zinkgraf, S.A., Wormuth, D.R., Hartfiel, V.F., & Hughey, J.B. (1981). Testing ESL Composition: A Practical Approach. Rowley. MA: Newbury House Publisher, Inc. Janopoulos, M. (1989). R eader com prehension and holistic assessm ent of second language writing proficiency. Written Communication. 8.(2), 218*237. 173 Janopoulos, M. (1993). Comprehension, communicative com petence, and construct validity: Holistic scoring from an ESL perspective. In M.M. Williamson and B. A. Huot (Eds.), Validating Holistic Scoring. for_Wrfting A ssessm ent:Theoretical and Empirical Foundations (d p . 303-325). Cresskill, NJ: Hampton Press, Inc. Johns, A.M. (1981). N ecessary English: An academ ic survey. TESOLQuarterlv. 15, (1), 51-57. Johns, A.M. (1991a). Interpreting an English competency exam: The frustrations of an ESL science student. WrittenCommunicatlon. fL (3), 379-401. Johns, A. (1991b). Faculty assessm ent of ESL student literacy skill: Implications for writing assessm ent. In L. Hamp-Lyons (Ed.), A ssessing Second Language Writing in Academic Contexts (pp.167-180). Norwood, NJ: Ablex. Jones, B.E.W. (1978). Marking of student writing by high school teachers in Virginia during 1976. Dissertation Abstracts International, 38, 3911 A. Kaplan, R. (1990). Writing in a multilingual/multicultural context: W hat's contrastive about contrastive rhetoric. The Writing Instructor. IQ, 7- 18. Keech, C.L., & McNelly, M.E. (1982). Comparison and analysis of rater responses to the anchor papers in the writing prompt variations study. In J.R. Gray & L.P. Ruth (Eds.), Properties of Writing Tasks: A Study of Alternative Procedures for Writing A ssessm ent (p p . 260- SI 5). Berkeley:University of California Graduate School of Educational Bay Area Writing Project. Kerschner, L. R. (1992). CSU Efforts to_Meet the N eeds of ESL Students. Long Beach, CA: Office of the Chancellor,California State University. Klammer, T. (1983). Writing competencies: What should we expect from nonnative speakers? Long Beach, CA: English Council of the California State University. Krashen, S.D. (1981). Second Language Aoouisition and Second Language Learning. New York: Pergam on Press. 174 Land, R.E. Jr., & Whitley, C. (1989). Evaluating second language essay s in regular composition classes: Toward a pluralistic U.S. rhetoric. In D.M. Johnson and D.H. Roen (Eds.), R ichness in Writing; Empowering ESL Students (p p .284-293). New York: Longman. Leki, I. (1992). Understanding ESL Writers: A Guide for T eachers. Portsmouth, NH: Boynton Cook, Heinemann. Lightbown, P. (1985). Great expectations: Second language acquisition research and classroom teaching. Applied Linguistics. £, 173*189. Lloyd-Jones, R. (1977). Primary trait scoring. In Cooper, C. R., & Odell, L. (Eds.),. Evaluating Writing: Describing. Measuring. Judging (pp.33- 68). Urbana, II: National Council of T eachers of English. Markham, L.R. (1976). Influences of handwriting quality on teacher evaluation of written work. American Educational R esearch Journal. 13.277*283. M arshall, J.C . (1967). Composition errors and essay examination grades re-examined. American Educational R esearch Journal. 4.375*386. McColly, W. (1970). W hat do es educational research say about the judging of writing ability? Journal of Educational R esearch. 64,147- 156. McGirt, J.D. (1984). The effect of morphological and syntactic errors on the holistic scores of native and non-native compositions. Unpublished m aster's thesis, University of California, Los Angeles. McKendy, T. (1992). Locally developed writing tests and the validity of holistic scoring. R esearch inJhe Teaching of English. 23.(2), 149- 166. McNamara, T. (1990). Item response theory and the validation of an ESP test for health professionals. Language Testing. 2(1), 52-76. Mendelsohn, D., & Cumming, A. (1987). Professors' ratings of language use and rhetorical organization in ESL compositions. TESL C anada JiM nal. 5.(1), 9-26. Meyer, B.F.J. (1982). Reading research and the composition teacher: The importance of plans. College Composition and Communication. 33, (1). 37-49. Moye, A.J. (1983). Report on the Graduation W ritingjftssessment Reouirement. Long Beach, CA: California State University Chancellor's Office. Mullen, K.A. (1980). Evaluating writing proficiency in ESL. In J.W . Oiler, Jr. & K. Perkins (Eds.), R esearch in Language-Testing (pp.17M 76). Rowley, MA: Newbury House. Mullis, I.V.S. (1984). Scoring direct writing assessm ents: W hat are the alternatives? Educational M easurem ent: Issues and Practice. 3 (1). 16-18. Myers, M. (1980). A Procedure for Writing AssessmeflLandidolistic Scoring. Urbana, IL: National Council of T eachers of English. Najimy, N.C. (1981). M easureforM easure: A Guidebook for Evaluating Students’ Expository Writing. Urbana, IL: National Council of T eachers of English. Nold, E. & Freedm an, S. (1977). An analysis of readers' responses to essays. R esearch in the Teaching of English. 11.164-174. Obrecht, F. (1992). How to prepare for the California State University writing proficiency examination. Hauppauge, NY: Barron's Educational Series. Oiler, J.W., Jr., (1979). Language T ests at School. London: Longman. Oppenheim, A.N. (1966). Questionnaire Design and Attitude M easurem ent. New York: Basic Books. Ostler, S. E. (1980). A survey of academ ic needs for advanced ESL. TESQL Quarterly, 14.(4), 489-502. Pula, J.J., & Huot, B.A. (1993). A model of background influences on holistic raters. In M.M. Williamson and B. A. Huot (Eds.), Validating Holistic Scoring for Writing Assessm ent: Theoretical and Empirical Foundations (pp. 237-265). Cresskill, NJ: Hampton Press, Inc. Purves, A.C. (1984). In search of an internationally valid schem e for scoring compositions. College Composition and Communication. 35 (4), 426-438. Quellmalz, E. M980L Problem s in Stabilizing the Ju d g ment Process. (CSE Report No. 136). Los Angeles: University of California. 176 Quellmalz, E.S.(1984). Toward successful large-scale writing assessm ent: W here are we now? W here do we go from here? Educational M easurement: Issues and Practices. 3 (11.29-32,35. Quellmalz, E., Capell, F.J., Chou, C. (1980). Defining Writing; Effects of Discourse and Resoonse.M ode. (CSE Report No. 132).Los Angeles: University of California. Raimes, A. (1990). The TOEFL Test of Written English: C auses for concern. TESOLQuarterlv. 24.(3), 427-433. Reid, J. M. (1990). The dirty laundry of ESL survey research. TESOL Quarterly, 24.(2), 323-341. Reynolds, C.R. (1982). Methods for detecting construct and predictive bias. In R.A. Berk (Ed.), Handbookof.M ethods for Detecting B ias. Baltimore, MD: Johns Hopkins UP. Rodby, J. (1992). Appropriating Literacy: Writing and Reading in English a s a Second Language. Portsmouth, NH: Boynton/Cook, Heinemann. Ross, S., Burne, K.G., Callen.J., Eskey, D., McKay, J. (1984). Expectations and evaluations of the second language student: MatttersoLarticulation in California education. A Report to the English Liaison Committee of the Articulation Conference of California. Ruth, L., & Murphy, S. (1988L Designing Writing Tasks for the A ssessm ent of Writing. Norwood, NJ: Ablex Publishing Company. Santos, T. (1988). Professors' reactions to the academ ic writing of the nonnative-speaking students. TESOL Quarterly. 22.(1), 69-90. Scannell, D.P., & Marshall, J.C . (1966). The effect of selected composition errors on grades assigned to essay examinations. American Education R esearch Journal. 2,125-130. Schum ann, J. (1978). Second language acquisition: The pidginization hypothesis. In E. Hatch (Ed.), Second Language Acoulsitlon. Rowley, MA: Newbury House. Sloan, C.A. & McGinnis, I. (1978). The Effect of Handwriting on Teachers' Grading of High School Essays. DeKalb, IL: Northern Illinois University. (ERIC Document Reproduction Service No. ED 220 836). 177 Smith, F. (1988). Understanding Reading: A Psychollnguistic. Analvsis.of. Reading and Learning to.Read. (4th ed.) Hillsdale, NJ: Lawrence Erlbaum. Smith, W.L. (1993). Assessing the reliability and adequacy of using holistic scoring of essay s a s a college composition placem ent technique. In M.M. Williamson and B. A. Huot (Eds.), Validating Holistic Scoring for Writing Assessm ent: Theoretical and Empirical Foundations (pp. 142*205). Cresskill, NJ: Hampton Press, Inc. Spooner-Smlth, L. (1980). M easuresofH ioh School Students* Expository WritinoLPirect and Indirect Strategies. (CSE Report No. 133). Los Angeles: University of California. Stansfield, C.W., & Ross, J. (1988). A long-term research agenda for the test of written English. Language Testing. 5 (21.160*186. Steele, J.M. (19791. The a s s e s s m e n ts writing proficiency via oualitatitive ratings of the writing sam ples. Washington, D.C.: QPO. (ERIC Document Reproduction Service No. ED 175 944). Stewart, M.F., & Grobe, C.H. (1979). Syntactic maturity, m echanics of writing, and teachers' quality ratings. Research in the Teaching of English. 13,207-215. Stock, P.L., & Robinson, J.L. (1987). Taking on testing: T eachers a s tester-researchers. English Education. 13,93-121. Thompson, R.M. (1990). Writing proficiency tests and remediation: Som e cultural differences. TESOL Quarterly. 21(1). 99-102. Torres, F. (1994). Notes from personal conversation with Learning Resource Center Director, Cal Poly Pomona. Vann, R., Lorenz, F.O., & Meyer, D.M. (1991). Error gravity: Faculty response to errors in the written discourse of nonnative speakers of English. In L. Hamp-Lyons (Ed.), A ssessing Second Language Writing in Academic Contexts (pp.181-196). Norwood, NJ: Ablex Publishing Corp. Veal, R.L. and Hudson, S.A. (1983). Direct and indirect m easures for large-scale evaluations of writing. Research in the Teaching of English. X L 290-296. - 178 V aughan, C. (1991). Holistic assessm ent: W hat go es on in the reader's mind? In L. Hamp*Lyons (Ed.), A ssessing Second Language, Writing inA cadem ic Contexts (pp. 111-126). Norwood, NJ: Ablex. W ade, B. (1978). R esp o n ses to written work: The possibilities of utilizing pupils' perceptions. Educational Review. 3Q, 149-158. W eaver, B.T. (1982). Com petency testing and writing program developm ent. In K.L. G reenberg, H.S. W iener, & R.A. Donovan (Eds.), N otes from the.NationaLTestino Network in Writing. (October, 1982),13. White, E.M. (1984). Hotisticism. College Composition and Communication. 25.(4), 400-409. White, E.M. (1985). Teaching and A ssessing Writing. San Francisco: Jossey -B ass Publishers. White, E.M. (1986). Pitfalls in the testing of writing. In K.L. G reenberg, H.S. W einer, & R.A. Donovan (Eds.). Writing A ssessm ent: Issues and Strategies (pp. 53-78). New York: Longman. White, E.M., & Polin, L.G. (1986). R esearch in effective teaching of writing: Volumes I and II. Final Project. Report to California State University Foundation. (ERIC Document Reproduction Service No. ED 275 007). White, E.M. (1993). Holistic scoring: P ast triumphs, future challenges.ln M.M. Williamson and B. A. Huot (Eds.), Validating Holistic Scoring for Writing A ssessm ent: Theoretical and Empirical Foundations (p p . 79-108). Cresskill, NJ: Hampton P ress, Inc. Wiggins, G. (1993). A ssessm ent, authenticity, context, and validity. KflCDfiQ. 25 (3), 200-214. Williamson, M.M. (1993). An introduction to holistic scoring: The social, historical and theoretical context for writing assessm ent. In M.M. Williamson and B. A. Huot (Eds.), Validating Holistic Scoring for Writing A ssessm ent: Theoretical and EmpirlcaLFoundations (pp. 1- 44). Cresskill, NJ: Hampton P ress. Inc. W inters, L. (1980). The effects of differing response criteria on the assessm en t of writing com petence. (ERIC Document Reproduction Service No* ED 212 659). 179 Witte, S. (1983). Topical structure and writing quality: S om e possible explanations of readers' judgm ents of students' writing. Visible language, 1 2 ,177-205. Zamel, V. (1985). Responding to student writing. TESOL Quarterly. 10.(1), 79-101. APPENDICES APPENDIX A FACULTY QUESTIONNAIRE 182 ESL Writing Survey Department (optional) Rank: (optonatL. Native Kind tv check appropriate reaoonata. marking ontvone box unlaw othtnartM Indicated; Years otTaachlntt O Less than 1 year LJ6*i0years Q 1*5 years Q Mora than 10 years Levels J lE 1. Do you have ESL (Engtlah as a second language) students tn your classes? D a , Yes D b . No 2. If so, on the average, what percentage of .your classes Indude ESL students? □ a . Under 5% □ b , s-10% □ c , 11*25% □ d . 28-50% □ e. More than 50% 3. How often do you have ESL students In your classes? D a. Every quarter □ b. Every other quarter O c.0 n ceay ear D d . Occasionally D e . Never 4. How much are you using writing for your course assignments compared to 2*3 years ago? D a . Less D b. Same 5. How often do you expect students to use a computer, a word processor, or a typewriter for writing assignments which are to be turned in for a grade? D a . Never D b . Occasionally D c . Frequently D d. Always 6. In your classes, which do you emphasise most? D a. Process of creating project/work D b . Pinal product D c. Other (Please explain)___________ - L J Adult Education □ Vocational Education □ Professional Training O Other___________ LJ Elementary Q Secondary □ Community College Q CoHegeAJnlversity Years at Cal Polv: Q Less than t year D 6*10 years Q Lees than 1 year □ t.5 years_____ □ More than 10 years Oc.M ore 183 IH g e n e r a l how many wrttmg Assignments of various types do you require tor the following levels? 7. For towar-dhrlsion courses? □ a. None/Not Applicable O b .t-2 □ c . 3-0 O d.7-10 O e . More than to 8. For upper-dMston courses? □ a . None/Not applicable O b . 1-2 O c.3-6 □ a . 7*10 O a. More than 10 9. How many writing assignments exceeding aonx. 250 words do you give lor tower-dhrHton courses? O a. None/Not applicable □ b -1*2 O c .3 4 □ a . 7*io O e . More than to to. How many writing assignments exceeding aoox. gso wordi do you give for upoafrdivHton couiias? O a. None/Not applicable O b .t-2 O c .3 4 □ d.7-10 O e.M ore than to 11. How many writing assignments exceeding appx. so words do you give per quarter under timed Irwcfaae eondltloM for lower-dlviaton coursea? O a. None/Not applicable O b .i-2 O c.3-6 □ d.7-10 O e* More than 10 12. How many writing assignments exceeding appx. 50 words do you give per quarter under timed In-daia condition! far unoer-dMaton courses? O a. None/Not applicable □ b .1 * □ c. 3-6 □ d.7-10 O e.M ore than 10 13. In your opinion, should non-native speakers of English be required to meet the same criteria lor English writing skids as native speakers of English? O s . Yes O b . No O c . Unsure O d . Other (Please explain)__________________________________________________ 14, How do you grade ESL student writing in comparison to native speakers of English? O a. More leniently O b. More severely O c.Sam o O d. Other (Please exnlalnl 184 1 S. For your course writing assignments, how often do you CORRECTor c o m m e n t o h the following? a. content (ideas, knowledge) b. sequencing c. word deletion A /or unnecessary additions d. preposition A /or amcie (aan.me) usage e. original ideas A/or factual support f. conation (imwng togemer) g. reg iite', level c* 'o c e , e.g. fonnai o r i f t t o r * - tone) h. ounctunon I. detailed development ). lengm/brevitv k. sentence construction l spelling m. topic identification (what tope requires) n. writer's purpose/audience awareness o. grammatical control (e.g., agreemenesubj/veib/pronoun) p capitalization q, organization (inct. beginning, middle, ending) r. appropriate word/idiom usage s. verb usage (tense, modal, auxiliary, gerund, infinitive) I form/lormat/handwrtting a. originality b. order c. vocabulary range d. grammar e. mechanics/punctuation (.specific development g. logic h. register/tone I. sentence variety ). legible handwriting/format very □ □ □ □ □ □ □ □ □ a/ways a often □ sometimes □ rarely □ never P □ □ D □ P □ □ a □ P □ □ □ P P □ □ □ □ P □ □ □ □ P □ □ □ □ P □ □ □ □ P □ always p □ often □ □ sometimes □ □ P never P □ □ □ □ P □ □ □ □ P □ □ □ □ P □ □ P □ P □ □ □ □ P □ □ □ □ P □ □ □ □ P □ □ □ □ P □ □ □ □ P □ □ □ □ P Imoortant In vour students' w ritfno? somewhat not very □ □ □ □ □ □ □ □ □ □ somewhat not very □ P □ □ □ □ □ □ □ □ not at si □ □ □ □ □ . notatal □ □ □ □ P * 185 17. W hich of fho blowing is-most often the bat is for w rm ng assignments in your courses? Cheek ail met apply: D a . Description O g , Cause and effect Ob. Narration □ h. Data support/interpretation Dc. Process D I. Aroumemason D d. D efinition D j. Comparison/Contrast O e. Summary D k . Ctassiflcalon O f. Personal exposition □ L Omar tPleate exolaint___ 18. Do ESL students encounter problems m your courses? □ a . Yes Db. N o D c . Unsure IS. If ESL students do encounter problems In your courses, what Is (are) the probable cause(s)? Cheek all that apply: D a Cultural differences D a Language difficulties Db. Inattention Df. Financial concerns Dc. Emotional problems Dg. Basic intelligence D d . Inadequate prior academic preparation O h . O ffier (Please explain)__________ 20. II language Is a problem, what klnd(s) of problem(s) seems (seem) most significant? Check all that apply: n D a Siow /lnefflcient reading D b. Understanding spoken questions D c. Responding orally to questions D d . Slow/Inexact note-taking D e. Weak communication w ith peers D l Weak w riting skffis D g. Understanding lectures D h . Understanding w ritten questions D l OffierfPieaseaxolalnl 21. H ow do you encourage students to seek assistance w ith English w riting skills? Check all that apply: □ D D □ □ D D □ □ a I don't b. Suggest/require students get assistance c. Suggest Learning Resource Center tutoring d. Suggest EOP tutoring e. Personally confer with/assist students mysel t. Match students w ith more capable peers g. Suggest Reading Program tutoring in LRC h. Suggest students read more for pleasure I. Other (Please explain) ___________ 22. How often have you scored the Graduation W riting Test at Cal Poly? D a . Never D b . One lim e O c . 2-5 times D d. M ore than 6 times but not almost always Do. Almost always Please return in the attached envelope to: K. Russikoff English & Foreign Lang. Dept. Bldg 24, Rm. 205 Thank you APPENDIX B FACULTY QUESTIONNAIRE RESPONSES 187 ESL Writing Survey (N«392) (iA)Oepariment fowohan i-*7 rasrasoanfflnai (2A )R an»c foonoftan Pmt (179146% : Assoc (40110%: AstH (161 4% Led (501 13% : No rasn (107127% (3A ) Native Lanouaoe: Entrttth (2661 73%: Non-EnnHth (65117% (2B lana/lana. oms.1: W o mm («H 10% K lndhr check aooroorteto ftiBOMW. m artrtna only ana baa unless otherwise Indicated; (4A1 Years t > 1 Teaching: (4) 1% Last than 1 year (65) 17% 6-10 year* (35) 9% 1*5 years (266)73% M ora than 10 yearn (S A 1 Levels of Previous Teachlno Esnertenca fChack all that anoM: (20) 7% Elamantary (66)17% Mult Education (77) 20 ' 8acondary (16) 5 Vocational Education (112)29 Community Colaga (92)23 Prolaaalonal Training (329)64 Cotiege/Universiiy (13) 4 Omar Incl (2)Extansion Programa (1) H aaltfi Dapl L A County: (i) L A A rt Collage of Design; (1) Consulting; (1) Language School; (1) CCD; (1) Plano; (1) Bank-sponsored courses; (1) Law School; (1) Sarnmars; (1) Paaca Corps; (1) Elementary A id (6 A 1 Years at Cal Polv: (13) 3% Lass than 1 year (80)21% 6*10 years mat 23% 1-5 veers___________ C 20B 1B 3% M ora than_10 years__(311% No response 1. Do you have ESL (English as a second language) students in your classes? (369)94% a. Yes (17) 4 b. No (6) 2 No Response 2. II so, on the average, what percentage ol your classes Include ESL students? (26) 7% a. Under 5% (67) 17 b.5-10% (109)2B C . 11-25% (115)29 d. 26-50% (51) 13 e. M om than 50% (24) 6 No Response 3. How often do you have ESL students k n your classes? (353)90% a. Every quarter (10) 3 b. Every other quarter (1) .3 c. Once a year (9) 2 d. Occasionally (4) 1 e. Never (15) 4 No Response 4. How much are you using writing for your course assignments compared to 2*3 years ago? (24) 6% a. Less (247)63 b.Same (117)30 c.More (4) 1 No Response 5. How often do you expect students to use a computer, a word processor, or a typewriter for w riting assignments which are to be turned in for a grade? (39)10% a. Never (87) 22 b. Occasionally (128)33 c. Frequently (137)35 d. Always (1) .3 No Response 188 6. In your classes, which do you emphasize most? (131)33% a. Process o! creating projecV w orfc (149)38 o. Final product (71) 18 c.Bom (33) 8 d. Other fPlaaia axnlainl (8) 2 No Response 1*6 assignments 55% 7.11* ia% Total 66% IN GENERAL how many w riting assignments of various typas do you raquira for me follow ing levels? 7. For tewar-dlvmon coursas? (118)30% a. Nona/Not applicable (108)28 b.1-2 (107) 27 C . 34 (36) 9 d. 7-10 (15) 4 a. M ora man 10 (8) 2 NoRasponsa 8. For uppsf-diYtilon coursas? (51) 13% a. Nona/Not applicable (87) 22 b. 1-2 (174)44 C. 3-6 (60) 15 d .7 ‘10 (13) 3 a. M ora man 10 (7) 2 No Response 1-6 assignment* 67% Z J 1 ± ________ l f i & Total 65% 9. How many w riting assignments exceeding a p o x . 850 word* do you give for tewer-dhrislon coursas? (159) 40% a. Nona/Not applicable (130)33 b. 1 * 2 (70) 18 c. 3-6 (20) 5 d. 7-10 (4) i a. M ora man 10 (9) 2 NoRasponsa 1 * 6 assignments 51% 7. 1 1 * Total 57% 10. How many w riting assignments exceeding aomr. 250 word* do you give for uooer-dlvtelon courses? (67) 17% a. Nona^ot applicable (144)37 b.1-2 (138)35 C . 36 (32) 8 d.7-10 1*6assignments 72% (4) i a. Mora man 10 7-n* 9% (7) 2 No Response Total 81% 189 11. How many writing assignments exceeding appx. S O words do you give per quarter under timed In-dasacondltlona ter tower-divM Qn courses? (239) 6i4e a. NonaMot appficabte (S O ) 20 b. 1*2 (46) 12 C . M (12) 3 d. 7*10 1*6 assignments 3244 (6) 2 e. More than 10 7*n» 5% Total 3744 (7) 2 No Response 12. How many writing assignments exceeding appx. S O words do you give per quarter under timed In^lasi conditions lor upper-dhriltan courses? (197)5044 a. NoneMot applicable (107)27 (65) 17 (12) 3 (4) 1 (7) 2 b. 1 * 2 c. 30 a. m o e. More than 10 No Response 1 * 6 assignments 4444 Total 4844 13. In your opinion, should non-native speakers ol English be required to meet the same criteria lor English witting sktts as native speakers ol English? (263)6744 a. Yes (67) 17 b. No (46) 12 c. Unsure (12) 3 d. Other (Please explain) (4) i No Response 14. How do you grade ESL student w riting in comparison to native speakers ol English? (115)2944 a. More leniently (0) 0 b. More severely (250) 64 c. Same (21) 5 d. Other (Please explain)________________________________ (6) 2 No Response 190 IS. For your court* wrwng assignments, how often do you connect or cqmmeht.on me following? always often aomaamsa rarefy never no fvi*>nea a. content (fleas, knowledge) (252)67* (69) 24* (27) 7 * (3)1* (4) 1* rvsponN (17) b. sequencing (67) 24 (124)35 (ill) 31 (27)8 (10)3 (33) c. word deletion A/or unnecessary additions (94) 25 (130)35 (97) 26 (32)9 (16)4 (23) d. preposition U o t amcie (a.an.the) utaga (104)28 (87) 24 (106)29 (47)13 (24)6 (25) a. original idaas t / o r factual support (186) 51 (121)33 (43) 12 (8) 2 (5) 1 (29) f, cohesion (linking together) (124)34 (146)40 (80f 22 (13)4 (5) 1 (24) g. register (level of voice. e.g. formal or informal tone) (35) 10 (52) 15 <10* 31 (80)23 (67)20 (50) h, punctuation (104)28 (106)29 (9* 26 (44)12 (17) 5 (25) I. detailed development (76) 22 (127)36 (108)31 (28) 8 (10) 3 (42) J . iengtn*revity (51)14 (124)34 (111) 33 (47)13 (23) 6 (28) k. sentence construction (105)29 (125)34 (100)27 (25)7 (11) 3 (26) 1 spelling (165)44 (105) 28 (83) 17 (28)8 (12) 3 (19) m. topic identNication (what topic requires) (120)34 (121) 34 (70) 20 (28)8 (13) 4 (40) n. writers purpose/audience awareness (91) 25 (112)31 (90) 25 (51)14 (17) 5 (31) o. grammatical control (e.g.. agreement*!*] Aeib/pronoun) (127) 35 (110)30 (91) 22 (30) 8 (18) 5 (26) p. capitalization (107)29 (87) 24 (61) 22 (63)17 (25) 7 (29) q. organization (Ind. beginning, middle, ending) (130)35 (117)32 (89) 24 (22) 6 (9) 2 (25) r. appropriate word/idiom usage (88) 24 (121)33 (115)31 (29) 8 (14) 4 (25) s, verb usage (tense, modal, auxiliary, gerund, infinitive) (120)33 (106)29 (88) 24 (36)10 (18) 5 (24) t lorm/tormaVhendwriling (62)23 (90) 25 (105) 29 (55) 15 (29) 8 (31) 191 16, Which of Me totowlnfl do vnu consider important In your students'writing? wry somewfiM net wry not Men no response a. originality (180)49% (149)41% (32)9% (5) 1% (26) border (174)46 (172)47 e. vocabulary range (62) 17 (210)60 d. grammar (204)55 (136)68 e. mechamcsSpunctuailon (159)44 (155)43 (14)4 (79)21 (23) 6 (45)12 (2) 1 (6) 2 (3) 1 (5) 1 (30) (26) (24) (28) wry somewhat not wry not at all no response t. specific dewiopment (193)54 (146) 41 (13) 4 (2) 1 (38) g. logic (266) 78 (74) 20 (2) 1 (2 ) 1 (15) h. register/tone (34) 10 (181)53 (98) 28 (31)9 (48) I. sentence variety (42) 12 (184)51 (109)30 (26)7 (31) ).legt>!ehandwrttingftormat(i54)43 (139)38 (50) 14 (19)5 (30) 192 17. Which of mo following is most often me basts for w riting assignments m your courses? Check all that apply: (210) 54% a. Description (140) 36% g. Cause and effect (66) 17 b. Narration (231) 59 h. Data support/interpretation (149)38 c. Process (122)31 i. Argumentation (114)29 d. Defmnon (164) 47 ). Compartson/eontrast (165)42 e. Summary (52) 13 It CtassMcation (74) 19 t. Personal exposition (37) 9 L Omar (Please explain)____ 18. Do ESL students encounter problems M your courses? (231)59% a. Yes (49) 12 b. N O (104) 27 c. Unsure (6) 2 No Response 19. If ESL students do encoumar problems m your courses, what Is (are) the probable cause(s)? Check all that apply: (99) 25% a. Cultural differences (299)76% a. Language difficulties (28) 7 b. Inattention (16) 4 f. Financial concerns (12) 3 c. Emotional problems (10) 2 g. Basic intelligence (144)37 d. Inadequate poor academic prep. (27) 7 h. Other (Please explain)_ _ 20. If language Is a problem, what klnd(s) of problem(s) seems (seem) most significant? Check all that apply: (137) 35% a. Slowrnieffldent reading (267) 66% I. Weak writing skills (140)36 b. Understanding spoken questions (141)36 g. Understanding lectures (144)37 c. Responding oraliy to questions (161)41 h. Understanding written questions (62) 16 d. Siow A m exact note-taking (27) 7 I Other (Please explain)______ (62) 21 e. Weak communication w ith peers 21. How do you encourage students to seek assistance w tth English w riting skills? Check all that apply: (55) 14% a. I don't (201) 51 b. Suggest/require students get assistance (211)54 c. Suggest Learning Resource Center tutoring (65) 16 d. Suggest EOP tutoring (178) 45 e. Personally confer with/assist students myself (70) 18 f. Match students w tth more capable peers (55) 14 g. Suggest Reading Program tutoring m LRC (66) 17 h. Suggest students read more lor pleasure (59) 15 L Other (Please explain)__________ 22. How olten have you scored me Oraduetion W riting Test at Cel Poly? (312)60% a. Never (24) 6 b. One time (29) 7 C . 2-5 times (15) 4 d. More than 6 times but not almost always (8) 2 e. Almost always (4) 1 No Response Please return in the attached envelope to: K. Russlkoff English & Foreign Lang. Dept. Bldg 24, Rm. 205 Thank you APPENDIX C FACULTY QUESTIONNAIRE OPEN-ENDED RESPONSES 194 OPEN-ENDED QUESTION RESPONSES INCLUDES FACULTY QUESTIONNAIRE RESPONSES TO QUESTIONS 5A, 6 ,1 3 ,1 4 ,1 7 ,1 9 ,2 0 , 21 5A. Levels of Previous Teaching Experience: Other (2)Extension Program s (1) Health Dept. LA County (1) LA Art College of Design (1) Consulting (1) Language School (1) CCD (1) Piano (1) Bank-sponsored courses (1) Law School (1) Seminars (1) Peace Corps (1) Elementary Aid 6. In your classes, which do you em phasize most? (33) 8 d. Other (Please explain)_______ d. Other responses included: (1) Content (3) a very broad question, not answered by a check,not clear (l)dem onstration of coursework w/others, i.e., teaching kids (3) critical thinking, problem solving (l)com prehenslon & expression (1) written essays, oral communication skills (1) not applicable (1)both creative process& its outcom e are important, not possible to answer (1) critical reading & writing (3) daily quizzes, exam s, els discussion (1) substantive content (2)? (2) understanding/ application to real life problems (2) no lost data, appropriate engineering results (6) concepts, learning information, scientific content (1) working representative problems (1) interpretation/synthesis of material from analogous material (1) continued growth (1) numerical/algebraic calculations 195 13. In your opinion, should non-native speakers of English be required to m eet the sam e criteria for English writing skills a s native speakers of English? (n*12, 3%) d. Other (Please explain)______ d. "Other” responses included: (1) ESL "accent" okay. (1) Only if have time to edit, not in-class assignm ents. (1) My students write in French only, not English. (1) For grade I em phasize ideas, organization, effectiveness; ESL students should have the support (out of class and in class context) to m aster English writing skills; but we need a writing center (drop in) & broader tutorial support to provide this help. (1) I take into consideration linguistic differences but if their grammar detracts from the whole they have to get help. (1) I think their should be a program which allows students (ESL) to upgrade their English. (1) Frequently depends on the student's abilities. (1) The clum siness of language associated with ESL is distinct from the clumsiness of language associated with other factors (lack of understanding, lack of study, etc.) (1) Modified skills. (1) Minimum basic reading & writing, nothing fancy but stitl correct. (1) I wish I knew! I try not to grade syntax and grammar because I know its (sic) unfair to students. (1) I've always had difficulty here. I use writing as an indicator of critical thinking. When writing is poor or disadvantaged, I cannot a sse ss the critical thinking. Shouldn't I grade all students fairly?? A number of additional comm ents were added even though the respondents had not checked the "Other" box. Additional comments for a., b. and c. included those which fell into the following categories: 1) adamant; 2) uncertain and/or sympathetic; and 3) exceptions and/or excuses: 1. Adamant Yes, by graduation. Eventually. Yes, if they are to be certified a s public school teachers. How could we expect less if they are to function in this society? Yes, emphatically. Yes--Absolutely. They need to be able to communicate clearly. If their profession will require written English. Ultimately by graduation. 196 They will have to in the work situation to be successful. Neither can write decently. In Chem. classes, yes. Mow could it be otherwise? Definitely. Otherwise they don't understand or cannot express them selves to their own detriment. Absolutely-English is an international language now, much more so than Greek, Latin, German or Jap an ese (for science) is concerned. 2. Uncertain/Sympathetic With compassion. But not in practice. Support in process vital. Yes & No. To the degree possible but we must be realistic. Not sure/uneven field of evaluation. Not if they are returning to their native countries after graduation. Yes, but with compassion. 3. Exceptions/Excuses Skill level should not be substantially less; expect less from foreign students; more from U.S. residents/citizens. But close; writing may be rough but ideas must be sam e. I don't grade on English skills. Leeway on grammar, none for content. Many spell much better than som e natives. Depends, for report & technical writing, yes. D epends-yes on edited term papers, no on in-class papers which are graded for content. I try to hold for similar levels of argumentation, research, organization, coherence, but allow for som e "accent." In most ways, yes. In certain areas of grammar, such as preposition and article use, no. Out of class writing, yes; in class no, but there should be som e standard for in class writing. Certain language patterns may be different from native speakers but thought and development standards the sam e. With support services, yes for lower level courses, particularly during the first 2 years. Not necessarily, but for upper division courses we would be doing them a disservice if we did not. In terms of understanding ideas & concepts, yes!I In term s of writing grammatically correct sentences.no. But I do correct their work. It is probably unrealistic to expect ESL students to meet the sam e criteria but we should strive for it. 197 No for ideas, but allow for som e ESL differences. Ves, but not on timed In class assignm ents. Yes, provided they are given a chance to improve, rewrite. The objective is to bring them to the sam e level. Yes, though I should not expect them to b e perfectly fluent in English. Generally less. If they are international students returning hom e, I grade easier here, harder on critical thinking related to application to their writing. N eed to be understood & able to com m unicate ideas, no m atter how sim ple the words & sentences. With adequate assistance. S am e for take-hom e assignm ents. They should build up to this and show interest and purpose. Should be approaching sa m e standards. I'm in the sciences w here the details of English are not a s critical. Poor English students can do well. But my opinion is 90% y es with som e allowance for occasional m istakes. They m ust be able to properly express them selves and to correctly com m unicate the subject m atter. So long a s they are offered opportunity to develop skills. Sam e in content, slight leniency in English. They should be able to ex p ress them selves clearly but minor m istakes in spelling or gram m ar are tolerable. This is especially so in the sciences. lm possible»but ESL students m ake up "language handicap" by hard work. 14. How do you grade ESL student writing in com parison to native sp eak ers of English? (n=21,5% ) d. O ther (P lease explain)_______ d. "Other" com m ents included: (1) S am e for content, m ore opportunity to revise for English. (1) I correct all the gram m ar, spelling, etc, but.... (1) Som etim es let students com e to office to answ er test questions if grades w ere poor. (1) Have not. (1) Have individual conferences to achieve grade. (1) Probably I u se a different standard, with good conceptual developm ent I will give above a C for poor (weak) gram m ar; bad gram m ar usually limits g rad es to C or lower. (1) Give them so m e leeway, but not much. (1) I look for content & "excuse" structural errors. I also suggest they have som eone read their p ap ers before submitting them . (1) Individual analysis. (1) Spend m ore time talking to student about what he/she intends to say in papers. 198 (2) NA (1) I grade only ESL papers (1) I do not require writing assignm ents (1) G rade for content only (1) I am more interested in content and analytical ability than in grammatical perfection but if their writing does not m ake se n se or if the gram m ar is poor I insist that the student have an English speaker proofread their papers before they turn them in (1) It depends on if the em phasis is on gram m ar or ideas in given assignm ent (1) Many ESL students cannot write a simple declarative sentence. They are not the only ones, but their problem s are the most severe. (1) The critique is more complete. (1) Many English only students cannot write effectively. (1) I list scale. Additional com m ents for a., b. and c. were also offered and again these fell into categories, but this time they included: 1) need to explain the sam e grading standards; 2) need to explain their exceptions: and 3) need to explain how they assist students: 1. Sam e Yes, Try. I grade the sam e or so I would like to believe. All students are anonym ous and subject to uniform standards. Does not affect my grading. Projects are team -based. Technical problem s are graded the sam e. Usually. I grade the sam e for everyone. I have just relaxed the standards for gram m ar, syntax, etc. for everyone- 2. Exceptions All exam s & reports are take-hom e; content graded equally; gram m ar, punctuation, spelling, vocabulary forgiven for ESL’s on th ese criteria; I em phasize content. I d o n 't grade ANYONE on their spelling, grammar, etc. a s long a s their point is clear (sic). I try to grade for intended meaning, not composition or spelling. I care more about content. But expect improvement. I am not at all sure. If there is no attempt to fix spelling and/or gram m ar m istakes, I get annoyed and may grade more severely. On the other hand, if m istakes are not so blatant, I may tend to overlook 199 som e grammatical m istakes and grade more leniently for those I assum e to be ESL. More leniently but very, very slightly. More lenient on gram m ar only. Make exceptions on ESL papers for articles/prepositions. Sam e for out of class writing; a little leniently for in class. Take hom e sam e; in class, more leniently. Based on the purpose of the class, and being a 100-level class, I do not grade down for writing a s long a s the meaning is clear and good comprehension of the material exists. I will note on the paper though that the student will need to work on increasing their writing ability a s it will be expected of them at higher levels. Allow for som e ESL differences. More leniently for fomal stylistic aspects, e.g., grammar, spelling, etc; sam e for content, Ideas, substance. Place less concern on style, etc. I do not grade students' writing skill. I grade their understanding of course material reflected in their writing. I correct grammatical or spelling errors for early assignm ents and grade leniently; towards the end of the quarter I am less lenient. For the most part papers are graded for content. I do correct English but do not penalize students. Although I think they should be held at the sam e level, I find it difficult to do so. Som e allowance for grammar. No student is graded on their (sic) writing unless communication is significantly affected. Then they must rewrite it. Mostly technical writing-efforts and contents are more Important. But am probably more lenient if the substance is there and leniency gives them a better grade if they are on the boundary line betw een two grades. But more on physics content than on gram m ar (sic). I hope the sam e but perhaps a little more leniently. I do not grade for "writing/grammar." Sam e in content, slight leniency in English. 3. Assistance I encourage all students to have som eone proofread their out-of-class written papers before they turn them in. I do grade down for spelling & grammatical errors & I correct all errors on paper, usually it is my non-ESL students that have most difficulty, For in class exam s students can u se dictionary Yes, but I would spend a lot more time with them 200 I strive to understand their ideas about the assigned topic. Oftentimes I m eet with them to clarify— in oral language-w hen I can I u se their native language. I also instruct them In the writing process, using their assignm ents as exam ples. I repeatedly ask for attendance after formal class session, during office hours, etc., etc. I do m ake som e allowance for certain errors ESL students make, but I m ake sure they know where they went wrong. Som e of my ESL students have been much better writers than my native Eng. speaking students I suggest students use spell check & gram m ar output to assist in obvious errors. I edit their work m ore-give more feedback-but I don't penalize their grade score. Since on out-of-class work I'm tough on gram m ar and encourage work with Laura at LRC. On essay exam s I don't count grammar. On complex inclass tests requiring reading I give extra time based on % estim ates of comprehension, reading speed. T hese are scheduled at LRC. Anglos often resent it. But I help them edit. I require them to use the LRC for final submittals. 7. Which of the following is most often the basis for writing assignm ents n your courses? n=37, 9%)l. Other (Please explain) 3) Analysis. 3) Analytical/problem solving. 1) Developing recommendations. Problem identification & solving. 1) Notebooks, personal journals, sketch books. 1) R esearch case study 1) Curriculum development. 1) I don't understand this question. 1) Logical positivism 1) Writing assignm ents vary considerably class to class. 1) News reports. 1) Evaluation of their speech presentations; analysis & evaluation of other students' speeches. 1) Analysis of reading a text. 1) Freewriting. 1) Application of theory response to a theory. 1) R esearch essay. 201 (1) Synthesis. (1) Writing assignm ent is usually a 5*10 page term paper where student must do som e research and express them selves {sic). (1) Creative. (1) Cogency. (1) Develop communication skills. (1) Technical. (1) Critical outside research reports. (1) Hospitality scenarios, e.g., answ ers to complaints, m em os to staff, marketing letters. (1) Expectations. (1) Full reports including all. (1) Engineering conclusion. (1) Lab reports & technical reports. (1) Non assigned {sic). (1) Convincing explanation. (1) Exam questions. (1) Informative. (1) N writing asgnm ents assigned {sic). (1) Math problems. Additional com m ents included: Reaction writing limited to one page, it varies so much. T hese definitions the hardest part of the survey to deal withl I teach design studio where writing is adjunct to drawing. 19. If ESL students do encounter problems in your courses, what is (are) the probable cause(s)? (n « 2 7 ,7%) h, Other (Please explain) (1) They think differently. (1) Speak too fast. (1) Writing ability/getting ideas across. (1) Som e do not have easy access to com puters with spelling checkers and gram m ar checkers. (1) I tell them they have a special facility knowing more than one language; that's a treasure. Now perfect it. (1) Family pressure/stress. (1) Required reading. (1) People (instructors) passing them when their writing skills are unacceptable. (1) Fear of oral assignm ents. (1) As all students, not enough composition skills. (1) Not know what to attend to. 202 (1) Perhaps som e uneasiness with expectation of analytic, somewhat original interpretations of authoritative work. (2) N/A (1) Prior negative experience. (1) Cultural passivityll! Lack of critical thinking skills. (1) Will not come for a preview of work. (1) Need prerequisite skills before they begin courses. (1) I find them by & large bright and motivated. (1) They spend too much time with friends speaking in their native language. (1) Insecurity, fear of failing. (1) More drove to plagiarism. (1) Difficulty in conveying ideas. (1) Trying to remain fluent in orig. language & speaking/writing English only at school. (1) Misinterpretation w/o comm ents or questions. (1) Different reasons apply to individuals. (1) Could be reading problems. Additional comments included: Don't know that they do. The rest of the choices are stereotypic. Some of these I am not in a position to evaluate, e.g., emotional problems. They do not take their English courses before taking my course. I don't know they don't discuss these things with m e-on a percentage basis ESL students do a s well as non-ESL. This is a Rude question. Beyond language, how should I know? Sam e sorts of problems a s other students. 20. If language is a problem, what kind(s) of problems(s) seem s (seem) most significant? (n=27,7%) i. Other (Please explain) (1) Clear exposition of ideas. (1) Oral presentations required frequently in architecture classes. (1) W eak verbal presentation skills. (1) Difficulties in case, verb tense, articles, plural forms, spelling. (1) Oral communication with me. (1) To som e extent, all of the above. (1) Greatly reduced thinking, analsis {sic) skills (when exam ples are in English). (1) Unwillingness to speak up in class & fear that they will not be able to give a speech unless they read it, not being understood by their audience (actually this is the biggest problem.) (1) Unwillingness to work up to the required standards. 203 (1) Thinking in English. (1) Forming an opinion and defending a position. (1) Overall mastery of 2nd L. (1) Technical words. (1) Sometimes we assign them a mentor so they can understand & ask questions at any time. (1) Writing what they want to write. (1) Won't ask questions-shy. (1) Understanding local procedures. Knowing meaning of new words. (1) Lack of education in prerequisite skills— again the "get by" attitude. (1) Responding to essay questions. (1) English is a difficult language-m ost try very hard. (1) Hard to get them Involved in class discussions. (1) Reading comprehension is poor. (1) Very little use of English outside classes. (1) Clearly the order of difficulty is write, read, speak. (1) Overall limitations in comprehension of oral and written English. (1) Most seem s to be with terms and definitions. (1) Understanding Physics/ Engineering/Math terms. 21. How do you encourage students to seek assistance with English writing skills? (n=59,15%) i. Other (Please explain) (1) Do more writing. (1) Writing programs. (1) Help them learn to use computers for word processing. (1) Journal writing-simply write & I read & write back. (1) Suggest them (sic) to go to English intensive training school. (We had one who went to that school for a year. The result w as incredible.) (1) Give extra reading suggestions. (1) Learn English better outside of school, then come back to school. (1) Whatever I can think of that would help. (1) Suggest they find peer assistance. (1) Suggest they take writing courses. (2) Keep a journal. (1) Recommend them (sic) to bring rough drafts to me before submission for a grade-m ost don't. (1) More difficult a s more foreign students are here-sim ply don't have time for the larger volume. Also find increased use of students' native language around campus...result is they don't improve english (sic) skills. I know this. I was a foreign student. (1)ESL classes. 204 (1) I offer to read over any term paper drafts a s many times as students want/need. (1) Give grammar exercises one on one. (1) Match students with peers, not necessarily more capable. (1) Must utilize all English skills more. (1) Suggest GWT assistance in the LRC. (1) Suggest study technical methods. (1) Read the news media. (1) Have had them write their ideas in their own language first & then sit & talk with me in English & make points they wrote. Then write their ideas in English & I give feedback. (1) I make corrections & suggestions to them on paper which I return to them. (1) Rewrite their papers; som etim es they catch many of their own errors. (1) A great deal of office hour help-step by step, etc. (1) Students haven't requested assistance on this issue. (1) Elements of Style, E.B. Strunk ( a book). (1) Suggest student communicates in English with his friends rather than in native language. Learn through practice of English skills. (1) Suggest student seeks assistance from peers. (1) Students didn't come for help or disregard advise (sic). (1) Use of computer assistance-i.e., spellcheck, grammar check. (1) Edit incorrect papers. (1) Getting help from fellow students (teammates). (1) Accept drafts of papers before final paper is due and give feedback. (1) Record lectures & put in reserve book room. (1) Visit the English Dept, and obtain advice. (1) Understand the corrections I make on their work so they do not make the sam e errors again. (1) Group project wI peers. (1) Suggest they pair w/students who have stronger English skills. (1) Set consistent expectations/standards for all students. (1) Get to know and speak to American students. International students remain too closed within their own respective cultures, i.e., Koreans with Koreans, Chinese with Chinese. (1) Suggest ESL students read poetry, and anything else, even textbooks. (1) Get a tutor off campus. (1) I let all those who understand the lecture out early, then repeat for those who are unsure. These are usually ESL's, then I go over the main points. Finally I handle individual student questions after class. (1) Speak English outside of class at all times except when actually talking to parents or relatives at home, who only speak the original language. (1) Teamwork with native speakers. 205 (1) If a students is having difficulty, I'll spend time helping them to understand the material. It is a pleasure to se e the light go on when they finally get It. I've spent many nites here till 11 or midnite to help students. (1) Suggest a decoding machine be used for watching TV programs. (1) Communicate more in English. (1) Write letters, talk with peers in English, watch TV (Sesam e Street!) (1) Suggest they take their required English classes ASAP. Many students put these off! (1) In my class they write to m e (12 pages) and tell it to their peers (12 minutes). (1) Have students write practice GWTs which I grade. (1) The students who have come for help have not asked for English language help. (1) Allow them to use a dictionary if they wish. (1) Work with peer group for studying. (1) I am not aw are of any resources at Cal Poly. (1) I don't unless asked. APPENDIX D PILOT QUESTIONNAIRE AND RESPONSES 207 ESL Writing Survey (N-7) Pilot Department romionan PSL. Music. Soc. SeL CM ! Ena.. Bee. & Como. Ena. Aar. Ena..EnQll*h. Ranlc ronHonan 4 Professors: a Lecturers______________________ NativeLanguage: 5ttl%1EnaHth:2_f29%lother languagesn Spanish..1 Japanese/English) Kindly check appropriate responses. marking only one box uniats otherwise Indicated; Yearaol Teaching □ l . m than 1 year (1| L J 6*10 years Q 1*5 yean (6) O More than 10 years Laval* of Pravteu* Teaching Evnertence fCheck all thatapoM: (1) Q Elementary (2) Q Adult Education (1) Q Secondary Q Vocational Education (3) O Community College (2) O Professional Training (7) Oconege/Unhrerslty P o th e r Yeara JiC aJ Rflhc Q u aw thaniyaar 0) L J 6-10 years _____________ □ 1-5 v*a«______________ < 6 1 □ Mora than 10 vaar*_______________ 1. Do you have ESL (English a* a second language) students In your classes? (7)100% D a . Yes □ b. No 2. If so, an the averaoe. what percentage ol your classes includes ESL students? (Q ) D a . Under 5% (1)14% Db.5-10% (2)29 D c . 11*25% (3)43 D d . 20*50% (1)14 D e. More than 50% 3. How oltan do you have ESL students In your classes? (7)100% D a. Every quarter □ b. Every other quarter D c. Once a year D d. Occasionally D e . Never 4. How much are you using writing lor your course assignments compared to 2*3 years ago? (Q ) D a. Less (3)43% D b. Same (4)57 Dc.More 5. How often do you expoci students to use a computer tor w riting assignments which are to be turned In for a grade? (2)29% D a .Never (2)29 D b. Occasionally (1)14 D c . Frequently (2)29 D d. Always 6. In your classes, which do you emphasize most? (2)29% Da. Process of creating projectfwork (3)43 D b. Final product (2)29 D c. Other (Please exptain)__J2)_Both___________________________ 208 IN GENERAL how many w riting assignments of various types do you require for the follow ing levels? 7. For towef^hrttkm courses? (0) Q a . None (2)29% D b . 1 * 2 (2)29 O c .3 4 (1)14 □ d .M O (2)29 Q e . More than 10 8. For unoer-dlvtston courses? (1)14% Da.None (1)14 Db. 1 -2 (2)29 D c.3-8 (1)14 Dd.7*l0 (2)29 Q e . Mora than 10 9. How many wrttm g assignments exceeding n m . 2S 0 words do you give tor towet-dlvlslnn courses? (1)14 D a . None (1)14 □ b. 1-2 (3)43 Dc.3-6 (2)29 Dd.7-10 (0) Q e. More man 10 10. How many writing assignments exceeding enmc 250 words do you give tor uooer-dlviston courses? (1)14% □ a. None (1)14 Q b .i-2 (3)43 Dc.3-6 (2)29 Pd.7.10 (0) Q e. More man 10 11. How many w riting assignments exceeding appx. 50 words do you give per quarter under timed Irvclasx conditions tor towet-dhrlsion courses? (4)57% D a. None (1)14 Qb. 1 -2 (2)29 Qc.3-6 (0) Dd.7-10 (0) Q e. More man 10 12. How many w riting assignments exceeding appx. 50 words do you give per quarter under timed In-class conditions tor uooer-dhrigton courses? (5)57% Q a. None (0) Q b . 1-2 (1)14 Qc. 3-8 (1)14 Qd.7-10 (0) Q e . More than 10 13. In your opinion, should non-native speakers of English be required to meet me same criteria tor English w riting skills as native speakers of English? (2)29% D a. Yes (4)57 Qb.NO (1)14 Qc. Unsure (0) Q d. Other (Please explain). 14. How do you grade ESL student writing In comparison to native speakers of English? (3)43% Q a. More leniently (0) Q b . More severely (3)43 O c,Sam e (1)14 Q d , Other (Please explain) Have only ESL_________ 209 15. For your course w riting assignments, how often do you CORRECT or CO M M EN T ON the follow ing? ------ ------ rarely never <0)09* (0)0% a. content (ideas, knowledge) b. sequencing c. word deletion & /or unnecessary additions d. preposition a/or article (a.an,ihs) usage e. original ideas a/or tactual support t, cohesion (linking together) g. register (level of voice, e.g. formal or informal tone) h. punctuation I detailed development ). length/brevity k. sentence construction L spelling m. topic identification (whet topic requires) n. writer's purpose/audience awareness o. grammatical control (e.g., p. capitalization q. organization (incl. beginning, middle, ending) r. appropriate wordMiom usage s. verb usage (tense, modal, auxiliary, gerund, Infinitive) t form /lorm aVhandwriting 16. W hich of the follow ing doj a. originality b. order c. vocabulary range d. grammar e. mechanics/punctuation f. specific development g. logic h. register/tone I. sentence variety |. legible handwrinng/tormat always (5)71% often (0)0% sometimes (2)26% (3)43 (2)29 (1)14 (2)29 (2)29 (1)14 (1)14 (1)14 (1)14 (4)57 (2)29 (1)14 (3)43 (2)29 (1)14 (1)14 (0) (3)43 (1)14 (4)57 (1)14 (3)43 (1)14 (3)43 never (0) rarely (1)14 sometimes (3)43 (0 ) (1)14 (1)14 (0) (0) (3)43 (0) (1)14 (1)14 W > (1)14 (2)29 1 (0) (0) (2)29 (0) (1)14 (1)14 (P ) (1)14 (2)29 (0) (1)14 (4)57 (0) (0) (1)14 (0) (0) (4)57 Imoortant In vour students' wrttma? very (4)57 (6)66 (0 ) (4)57 (4)57 not at all (0) (0) (0) 0)14 (0) (3)43 (1)14 (5)71 (3)43 (3)43 not very (0) (0) (1)14 (0) (1)14 not very (0) (0) (2)29 (0) (0) somewhat (4)57 (2)29 (6)86 (6)71 (2)29 (1)14 (2)28 (2)29 (0) (1)14 (2)29 (1)14 (0) often (3)43 (3)43 (3)43 (2)28 (2)29 (4)57 (4)57 (1)14 (1)14 (5)71 (3)43 not at all w > (0) (0) (0) (0) very (3)43 (5)71 (0) (1)14 (4)57 (0) (0 ) (2)29 (0 ) (0) (0) 1 ( Q ) (0) always (0) (2)29 (1)14 (3)43 (2)29 (1)14 (1)14 (3)43 (1)14 (1)14 (0) 210 17. W hich of 0)0 following It most often the basis for w riting assignments in your courses? Check ell that apply: (4)57% □ a. Description (3)43% O g. Cause and effect (3)43 O b . Narration (6)33 O h . Data support/interpretation (3)43 O c . Process (5)71 O l. Argumentation (2)29 O d . Deftnltton (6)66 0 | . Comparison/Contrast (4)57 O e . Summary (2)29 O k. Classification (2)29 Of. Personal exposition (1)14 O l Other (Please explaln)_LatLBfififiitS 16. Do ESL students encounter problems In your courses? (5)71% d a . Yes (1)14 O b . No (1)14 O e . Unsure 19. it ESL students do encounter problems in your courses, what Is (are) the probable cause(s)? Check all that apply: (3)43% O a. Cultural differences (6) 86% O e. Language difficulties (2)29 O b . Inattention (1)14 O f. Financial concerns (2)29 O c . Emotional problems (1)14 D g . Basic Intelligence (2)29 O d . InadBQuate prior academic prep. (1)14 O h . Other fPlaase exolalnl Fear. jacket confidence 20. tt language is a problem, wnat W nd(s) of problem(s) seems (seem) most significant? Check all that apply; (4)57% O a. Slow/lnafflclent reading * (5)71% O f. Weak w riting skills (3)43 O b . Understanding spoken questions (4)57 O g . Understanding lectures (4)57 O c . Responding orally to questions (1)14 O h . Understanding written question (2)29 O d . Slow/inexact note-taking (1)14 O l. Other (Please explain) (1)14 O e. Weak com m unication w ith peers U n d e r s t a n d i n g reading 21. How do you encourage students to seek assistance w ith English w riting skills? Check all that apply: (0) 0% O a. I don't (5) 71 Ob.SuggesVrequire students get assistance (7)100 O c. Suggest Learning Resource Center tutoring (1) 14 O d . Suggest EOP tutoring (7)100 O e. Personalty confer with/assist students myself (2) 29 O f. Match students w ith more capable peers (1) 14 O g . Suggest Reading Program tutoring in LRC < « ) 57 O h. Suggest students read more for pleasure (2) 29 O L Other (Please explain! Suggest ESL dasses: Ask for drafts tor rewriting 22. How often have you scored the Graduation W riting Test at Cal Poly? (2)14% O a. Never (0) 0 O b .One time (2)14 D c. 2-5 limes (0) 0 O d . More than 6 times but not almost always (3)43 O e . Almost always Please return In the attached envelope to: K. Russikoff English & Foreign Lang. Dept. Bldg 24, Rm. 205 Thank you APPENDIX E GRADUATION WRITING TEST RUBRIC POSSIBLE SCORES (The current GWT rubric) A superior response will address itself to ail aspects of the question. Though H may have occasional faults, it will be well organized, detailed and generally well written. These scores will be useful for a well-handled paper that is weak in some aspects of the superior response; e.g. it may slight one of the parts of the question; it may not be as clearly organized as the superior response; it may have some minor grammatical inconsistencies. Otherwise, the paper should be competently written. This score will be useful for the following kinds of paper; -those that are only descriptive or narrative -those in which the language is overly cliched -those that are overly repetitious -those that are general and superficial This score will also be useful for papers that are developed with some specificity and detail but are marred by more than a few minor grammatical inconsistencies. This score is to be used for papers that exhibit serious weaknesses in structure, syntax, diction and/or development. This score is to be used for papers that show very little understanding of the question or suggest incompetence in structure, syntax, and diction. Non-response papers or those that argue with or avoid the question should be given to the tqbte leader. APPENDIX F COVER LETTER FOR FACULTY QUESTIONNAIRE 214 Memorandum O ? / October 20.1993 All Faculty i,„m -i* 'S id n ey A. Ribeau Vice President for Academic Affairs Request for ESL^ Writing Survey Response As the changing demographics of the state are increasingly reflected on our campus by students for whom English is a second language (ESL), research on the needs of these students, especially regarding writing and reading skills for university-level work, has become both desirable and necessary. Faculty in all disciplines have expressed concern over attention needed in this area. In response to this demand, a questionnaire has been developed to discover by what means ESL student writing is currently assessed in academic coursework. The survey is designed to take only a few minutes. Your involvement is essential so that the entire Cal Poly faculty may be represented in a comprehensive view of practices, attitudes, and concerns focusing on ESL academic writing. Thank you for your time and attention to this increasingly important matter. APPENDIX G VOICE MAIL FOLLOW-UP TEXT T ext: Q uestionnaire Follow-Up/Voice Mail 216 Hello. This is Karen Russikoff from the English and Foreign Languages Department. Several w eeks ago you should have received a questionnaire on ESL academ ic writing. If you have already returned it, thank you. If you have not yet had the chance, could you please take a few m inutes now to com plete it and return it in its attached envelope. Your response is extremely valuable. If you need another copy of the questionnaire, please call m e at extension (num ber supplied) and I will se e that you receive one right away. Again, thank you very much for your time and effort in support of this research. APPENDIX H QUESTIONNAIRE DISTRIBUTION TO DEPARTMENTS, COLLEGES, SCHOOL Question 1 Departments (6 Colleges. 1 School) (Organized in order of distribution and responses) College of Environmental Design 1 Architecture 2 Art 3 Landscape Architecture 4 Urban & Regional Planning 5 Environmental Design College of Agriculture 6 Agricultural Business Management 7 Agricultural Engineering 8 Animal & Veterinary Science 9 Applied Technology Research 10 Foods & Nutrition 11 Horticulture College of Arts 12 Teacher Education 13 Behavioral Science 14 Communication 15 Economics 16 English & Foreign Languages 17 Ethnic & Women's Studies 18 Health, Physical Education & Recreation 19 History 20 Interdisciplinary General Education 21 Liberal Studies 22 Music 23 Philosophy 24 Political Science 25 Social Science 26 Social Work 27 Theatre & Dance College of Business 28 Accounting 29 Computer Information Systems 30 Finance, Real Estate & Law 31 Management & Human Resources 32 International Business Management 33 Operations Management School of Hotel & Restaurant Management 34 School ol Hotel & Restaurant Management College of Engineering 35 Aerospace Engineering 36 Chemical & Materials Engineering 37 Civil Engineering 38 Electrical & Computer Engineering 39 Engineering Technology 40 Industrial & Manufacturing Engineering 41 Mechanical Engineering College of Science 42 Biology 43 Chemistry 44 Computer Science 45 Geology 46 Mathematics 47 Physics APPENDIX I DEPARTMENT RETURN 220 DEPARTMENT RESPONSE College of Environmental Design 1 Architecture 2 Art 3 Landscape Architecture 4 Urban & Regional Planning 5 Environmental Design TOTAL College of Agriculture 6 Agricultural Business Mngmt. 7 Agriculture! Engineering 8 Animal & Veterinary Science 9 Applied Technology Research 10 Foods & Nutrition 1 ' Horticufture TOTAL College of Arts 12 Teacher Education 13 Behavioral Science 14 Communication 15 Economics 16 English & Foreign Languages 17 Ethnic & Women's Studies 18 Health, Physical Ed. & Rec. 19 History 20 Interdisciplinary General Ed. 21 Liberal Studies 22 Music 23 Philosophy 24 Political Science 25 Social Science 26 Social Work 27 Theatre & Dance TOTAL College of Business 28 Accounting 29 Computer Infor. Systems 30 Finance, Real Est. & Law 31 Mgmt. & Human Resources 32 International Business Mgmt. 33 Operations Mgmt. TOTAL School of Hotel & Rest. Mgmt. 34 School of Hotel & Rest. Mgmt. TOTAL ’ N n % SENT RETURNED RETURNED 20 8 40% 10 3 30 14 6 43 11 7 64 5 3 60 60 27 45% 8 7 88 3 3 100 10 6 60 2 1 SO 8 4 50 14 6 43 45 27 60% 11 6 55 24 11 46 23 8 35 14 9 64 50 29 58 9 2 22 29 11 38 19 9 47 1 1 100 2 1 50 16 5 31 11 5 45 13 7 54 11 5 45 4 1 25 6 1 16 243 111 46% 25 16 64 17 6 35 27 13 48 31 18 58 19 12 63 19 7 37 138 72 52% 16 11 69 16 11 69% 221 College of Engineering 35 Aerospace Engineering 6 3 50 36 Chemical & Materials Engln. 12 6 50 37 Civil Engineering 21 8 38 38 Electrical & Computer Engin. 41 17 41 39 Engineering Technology 18 10 56 40 Industnal & Manufacturing Engin. 18 9 50 41 Mechanical Engln. 29_________ 13________ 45 TOTAL 145 66 46% College of Science 42 Biology 34 17 50 43 Chemistry 31 15 48 44 Computer Science 16 11 69 45 Geology 8 6 75 46 Mathematics 41 19 46 47 Physics Ifl_________ IQ________ 52 ___________________TOTAL_________HQ_______ 28_______52% PARTICIPATION GRAND TOTAL 796 392 49% APPENDIX J FACULTY QUESTIONNAIRE NATIVE LANGUAGE RESPONSES 223 Question 3A, Native L anguages 1. English 73% (n=286) 2. O ther 17% (n=65) LANGUAGES_____________________________________ q Chinese/M andarin/Cantonese/Taiw anese 14 Farsi/Persian 11 Spanish 8 Tagalog/llocano/Filipino 2 Armenian 2 Hindi 2 Urdu 2 G erm an 2 Russian 2 Korean 2 Arabic 2 French/English 2 French/Germ an Spanish/Portuguese C hinese/Japanese K orean/C hinese/Japanese Turkish Sinhalese Swedish Tamil Hungarian Slav Italian Ja p a n e se V ietnam ese Not English 3. No R esponse 10% (n=41) APPENDIX K FACULTY QUESTIONNAIRE CONTINGENCY TABLES Q um . 22) G W T P aitopaaon wti O u a a . 2 A ) R a n k iM U 1 C N o « i k ia « q t a a t i • i T m m 1*1 i t i i «• a* •i* 1* i * t i it 1* t I i i •a ia a a a t i i a a a i • a i a a i * • u a «a ia •a n r •a t FU/*. 1.FuBRro(aaaor;2.Aaaociaaa;3.Aaataaiit 4. Laciurar; S. No Raaponaa owr i.i M ttn a l i ’l l r a i 4.#* ttnai 5. Aknofl ANaya 0. No R a^ m at O mMoa U ) OWT Boonno OuMMn M ) NaNwo Longuago OtmttMvmtaa 1 * a Tm i *. i t i t i t i r l i t * 11 • i *« i I I i i » • « M t a 11 I r a i a T*WX i n a i 40 a n NaVw Linguae# 1. EngWi, t. NorvEnglah: 3 ) No Raaponaa awr i.i X .lttm o 91'Sfemai 4. e*bm ai 6. AfenoR Akwaya Ouaaton 22) OWT Soonng \M0i OuaaOon M ) Vaara at Cat Pttf Pomona i t » a I TMM owr I.Navar XlUma M < S a m i 4 0*amaa 8 MmotfMwaya na *i 99 i» a m Yaara at Cal M y Pomona 1. <1; J J .5,3H (> , 4. I t a l i a n 10 G u m . 2 2 ) O W T S c o rin g w ttiQ uts IB ) E S L P ro tO o m o M i l i n 41 • • •00 n * * *4 i* * 4 10 ii i 1 1* * 0 1 0 m 4* 104 *11 O W T I .N o v o r 2.1i 3.2-5* 4 . 0 ** S. Aknori Alwoyt ESL ProW*m»: 1. Yoo;2.No;3.Unoura Ouoi. 22JOWT Soonno «M 0i Oum . 4) W 4ig oomp. 2*3 yoora too iriH M M v T M ti 1 t 1 4 1* too 00 4 * 11 11 0 a 1* 10 0 i * * 0 e T 1 0 *4 *41 I lf 4 Tan I* *11 *4 t* 1 * • I t owr I.NovW 2 . 1m 3 . 2-Sm 4 . 0 »* 5. AlmoM ANrayo Writing Now Comp, to 2*3 yoora ago 1. Loos; 2. Somo. 3. M ora; 4. No fteoponoo 227 Out* 22) OW T SoortftQ w O h tMitng (lowar Dfr) 1 ie« •1 It to 11 t t i t • 4 t 1 1 1 1 t4 1 » 10 1 4 1 t to • • 0 1 t • 0 I ***** 111 101 1M to 10 1 tot owr 1.Notwt WtBngflowDtv) 1 NonoMfcO.1*2; 3.34.4.7*10. S. 10*. 0. No R «p. 2. 1m 3.2'Sx 4. 0* M 5.MmotfAftMyt *ttiOuo* 8) Wig Uppoidhr. 1 NonoMA:2.1*2,3.34:4.7-10; S. 10*;fl.No Roop Oh**»«4 Ffe^wney T *M * > » » « » » iMk. *0 oo 14] *1 11 4 t i l t 1 T 0 1 1 t4 t 1 I t 0 1 0 t o •Ul* 11 00 i n 00 11 1 I I I wHhOuot 0)WngoKdg 2Kwofd«/UMrdtv..l.NonoMA:2.1>2;3.34;4.7<10;5.10*;• NoRoap. OlHni<taw«rrtM 1 _____ » _____ » _____•_____ » _____* T * i* i* 141 100 ti i t i I 1 to t i t 0 1 It » t 0 1 0 •1*1* 111 n o 4 0 228 Gun. 22) OWT Soomg ««h Wiflno &IM 10) E*e. 2 SO worOaAjp dr*.. 1 NonoMA. 2 1*2; 2 .24; 4.7*10; S. 10*; 0. No R np. Oh a w * « r i T | M | 1 I 1 4 I P 1 *1 111 111 ■ I I 1 t 1 11 P I 1 1 1 I 11 1 P I P 4 1 4 P f 0 P 1 1 1 4 P e P T n t I t 1*4 l i t 11 4 * OW T I.Navpr 2 . 1* 2 . 1* 2 2-3* 4. ( * l 5. Alfllo* Ahwy* wmmg (Up D r*) 1. NonoMA; 2.1*2.2.24.4.7*10; 8.10*; 0. No Map Ouaa. 11) Tknod/ low O n . 1 1 0 1 IP 1 1 t 1 4 1 1 1 1 1 I a 1 IP 1 1 I 0 a 4 p 1 1 1 1 a I 1 1 1 0 1 « T a u M t i t 41 Om i 12) TV nodr up <»*.: 1. NonoMA. 2.1*2; 2.2* TtHlt tit at i i i 1 1 lit it a « a 1 0 14 ii a i i 0 a IP « 11 4 i 0 i 0 a P IP? ip? at It 4 1 III ; 4.7*10; Tm * tit 14 IP It P IIP 10*; 0. No ftaip. T * a it 229 O u M .tt ) a w r a o Q ( i n a w t f i O u M . l3 )S a m a C riM io t.TMi 2. No; 9. Unom ; 4. Ofwr, No Aoap. 1 • a pyMa* a a Taala 1 t 1 4 40 10 a i 818 • n a a i a aa a ai » i a a aa a ii a i a a ia • Tatoto M l t f aa it i aaa Qua*. 22) OWT Boormo watt Qua*. 14) lam a Criteria i*K2.i 1. m e n 1 antanoy; 2. mom aoaoraljf. 2. m im ; 4. Otfwr; No Raap i a a iyT*M* a a Tm *I* i aa a aaa i i a a n a a a ia a a aa a a e 18 i a aa a a a a a a i a a i Mat* i i a a a a a a e a a a a 230 Out*. 7)Q*n wlptow dt* « •» O un. ft) Wttp o k 2S0 wdaAowd* W M Dn« 1. NonoMA. MO; ft. 10*; ft. No N o * 1 * * 1 t 1 4 4 0 lit ft II • 1 0 1 • 4 *1 T 4 1 1 • 41 4 4 • 1 0 1 It • 1 0 0 4 0 I 1 t 1 1 4 • 4 114 104 14? 44 14 4 T «M 1 1 1 no * 0 4 • . I l l Out! 7) V totg A ow dw w O h Ou«i 11) TVnod wnotowd* 1 a a 4 4 4 1 *44 44 44 14 a a I 4 44 >4 t t i a 1 4 14 t t • 4 4 4 4 1 1 1 4 f I 4 4 4 a TM 4II 114 144 141 44 I t 4 Own. •lWrtQAjpdMMOhOuot IO )«M giR lN «iM ft4N I t I 4 4 ■ Tltlli 44 4 4 a 0 a 4 f t 41 i t a • 4 1 144 i i a i 4 4 a a t i a 4 4 4 4 4 a 4 1 1 4 4 a 1 W N II 144 IS* H 4 I II* Ouaa. 0)Gtn Mig/Up dN M0i Ouat. 1 2) TVrwd wrtg Alp dtv WrWng: 1. NonaMA; 2.1*2:3.3-6; a .7-10; S. 10*; •. No Amp* 231 TataM. « t a a a a Tattlt. a t aa ta aa a a It? 0 1* aa ia a a 10T a a a i ia a a aa t a a » a a i t a 0 t a a a a 0 i i i a a T i i IT 1T4 at I t T aaa Oim i 3A) Nattva Uno wdh Ouat.13) 8am# CMaria ^Saaa^na^l T t^tla Tata la1 i a a ta t aa at aa It a at a l a i a a 0 0 aaa aa at Tata la: ttl or at i t a ta t 3AtNaLLang, 1 . Eng.; 2 . Non-Eng.; a. No Ratp. 13)8«ma Cm 1 .Y tt 2. No 3. Unaura 4.00wr 5. NoRaap, Ouat.3A) Nat Lang vrth 14) (boding O ttawa F ia a u a n e y T ^ ta Tata l». t a t Tatala aa i t T n a 0 0 0 a ita ai aa aaa ia a a ai a a t a aaa aa ai aaa 3 A ) N a t. Lang. 1 . E n g ^2. N o n - E n g .; 3 . N o R atp . 1 4 )0 ra d ln g 1, m o o t lo n ia n tly 2 . m g r # ta v a ra ly a.um 4 .O 0 W T 5 . N o R atp. 232 Oust 3A )N«tlinQ w *)Q im 7)0«iwrtBtowdiv WHSng; I.N w W N A M 4 a . 3 -a 4.7*10 3.10* a . n o n * * p . 3AMtLL*ng. 1. Eng.: 2. Non-Eng.: 3. No fW»p, mm 1 t • • 1 1 t 17 I t 11 • 71 1« 11 « 1 • 4 1 • 1 1 I I I • 1 41 iMk Om i 3A) Ntt long wtfi O uti.3) Qm wrtgAjp dtv irnfaniTt M M iM tli 1 > t T*M M 11 1 7 11 • 1 11 I t t 7 110 t t I t 174 41 11 t t t 7 1 • 11 f 1 t 7 111 t l 41 l i t lmaO) WMg one 2 9 0 w S M M M t n f M 1 * ta fa w d w y T«4M 1 TM 4M l i t t t t l l i t • 7 I t 14 l i t 11 14 t T O I t t t t t 0 t 1 4 I 1 t t t t i l t t 41 t i t 233 Ou^SA) N* iwe w«b Ouei.16) Wig • * 2Sd wd#Up dk W W w o . 1 , N m m A M-} 1 M 4 7*10 S. 10* • .N o A M p . A M N o t-L ata 1. CnT. t . NotvCno.: X No Am p - T m« 1 * > 4 * TM 4 t t t Total* t o t • 17 • t t l 17 144 10T t t • 111 t t t t t t 1 t 1 4 7 0 • 7 t t t • 1 4* t t t tfot.11) TfcnotfW^jtow C ln m « > ll> iliin t a n i t t Tata 1 * 174 I I t t t i t 10 I I 7 • t >1 I t 41 • I I t t I I 1 4 t 1 7 t t t • I 41 t i t Oum 3AJ N ti l*rig v mo i Ouat.1I) Tlmod * n ^ 4 > dk T*M t* 144 t l t t 77 t t • 41 11 t t t • 1 1 I 4 1 t •» T m»m n r •or •» i t 4 » ••I 234 Ou m . 3A) N « Lwig w«i Qu m . 2) PmOB of EO. 3AVM.LM0 1 . Eng.;2. N o iv E n Q .: 3. N o N # * p . 2)1. Uniter 9H 2.M0H 111«H 4 .» 8 0 H SUmtimtM O.NOftoop. fM k 1 t a 1* • • I f « i* 1* i* i i »• *• it j ; 1 * i it 4 t in • * 41 T tM k . Q uo* 3)NHU ii0w N i Qu**3)ftMEB. 3)1 Ewryifr Itw y oC w rqir 3.0no*a yr 4 . O o o o o io n a R y 5.Ntvor O.NoRotp * * I 1*141 * >*• 1* II I I I • 0 * 1* 1 0 • 1 t t I • 4 0 • 4 11 f 1 1* t l * • • 41 1*1 Q uo*. 3 )N o 1 L o n g w 4 h O uo* 3 )C x p * c l C o m p u ter 5)1»«v*r Z.OoooMonolty 3. FnquonOy 4 Mwy* 5-N oftoip. 11 • • *1 14 11 • 1 11 14 101 11 I t 0 • 1 41 Tm ti 1 * • r ttt w 1 lit 235 " ■ • a aaa#* fit O t Im h O n I m k • n w • IW M ( I m m *wna 0tm tm ** O m 1 O tn tn » H» I f t • I t 1 ["“ r r r t 1 Vi* I • I ’ * 1 f * *!*j I 1 11 •' • a *'.ij I t iTn .«< 1_-V -l-l. • * ' Ill] 4..., til] • 1 III1 f I I.I. 1 • If IJ 1 1 I'D 1 • t i l 1 * (i(i 1 t III! 4 1 1 . < ' I • 1 ill' • 1 * *!1J 1 1 1 liH f 1 1 111 1 1 * III, * a 1 • < I • 3 1 • I 1 ' ’ I'U I • < > • m * 1 " . 1! 1 * 1 4 ,«j ! > a 111] 1 • 4 4 » * 1 1 ! 4 II 1 1 1 1 « til 1 • 4 1 -II III ’ ij?' • 4 » 1 «I I a 11 II aj i i ■ 1 T 11 • I I a 1 ■ *1 • ‘ III I M all. • 4 a a 1 11 • f «!• • a a 4 • I t *» ■ ii* • • till *4 in ai» a l A n M M n lAit 8 Undacapa A /t hiwm w AUibanARaaonalPtamg A D m m tw h I Daaipi BAgrtaraniBuatnaaMgmi 7 Affeuhsrt Entineertno • Aram I VMwtary 8 w m 8 Aralad Tacftnotogy Raaaaicfi 10 Feora t NuMttn 1 1 H M bIm 12 Taachar Eduoraon 1 9 KIV(b V w B D O m O l K O om um m llE o o n y n c a 10 Engbti i Fom&t Lang 17 E in i & Wbman'i Siuom 18 H aaAh. Ptiyafcal Ed A Bac IBHtotory <0 HardMCfcaary Oananl Ed SI LbaralStuOaa I l t k a c SSftmaeohy iRMkeaTse * 4 1________ . HSNUScanM H 4 A O O B m Twr* STTharaa A D a n e * SiAeeaaino SSCa^MarUonnMonSyat SO Fbiaraa. Haal Eataa A um 81 Managamart A H u m Raa 82 fc r n m k m Brawaaa Mgrt SSOpandfora Alaraoanrat HowTnaatM 84 ttah o et tt HeW A I_____ ... 88 Aaraapaoa E/nnMrru 8 8 ChaffMA Umrtafc Engin aTOMEniwdg 8 8 BaefeW A Oonnaar Engin s g f f is y f t S r £ £ d M E n * m n n g 480«m M y 44 C e n a la Samoa AAGaotagy 47Plyaca 236 *** » * 4 ' H titt' 9 ttO »tltO »— e i or C h i £ 4 * 9 1 4 P-vatf G -Sav4'M 6-£««ti*s hv*i»* C hM »0*ht)r Chf Cftmtft V »> * 1 9 « O O Q t m 4 1 3 OMtfrtafttavftcitt < e ' i* t> * » t t tiHtutncuw* * » _ r«*n 1 0 1 * | 4 > a 1 a • a 0 I M o I a o a a e I a ! a l 1 1 0 • « 0 o « a i 1 |9 * 4 » 0 1 ° l i * a i o a « ' | • j s ' a i o a » o j 6 I a I i »: a • 1 ( a : a jo • • 0 ' i t °l e t o i 10 • t “ I e i a ( o 4 11 e 1 0 1 4 1 a ! o • '» * f 0 J • 1 4 1 0 • ia e [ 0 1 • I i ' e 11 1 4 o 1 1 r a ; i < e ! a t tt • i 4 ! * ! - • 1 • n 0 a < T 1 •ei *o ■ : ' a* 0 i e i oi a : : a I * 1 a • * 9 J ii * * a i 4 • 1 01 • 0 1 o ’ 0 f l i e i 0 i * . 9 1 0 I 9 i 0 i i t ♦ I a ' e 1 9 » I S I e 1 < 0 1 • 0 1 1 I a a le i * 0 1 ■ 4 • e I e I * 0 i o i 0 «: o i 9 1 0 1 e ' 1 1 0 - 9 1 4 a : •: o 9 0 a t 4 J 0 • 9 1 i ■ t 4 < 0 13 0 1 a 4 | i>!e 10 9 0 a I 9 10 It 0 I - a I 1 1 0 T « » a « » 0 11 e • 0 a i o a 0 1 4 * j 9 9 e t 9 tjo r 1 » 9 l l ' 19 0 a 1 i i = 10 a a 4 »ia 9 t 4 9 1 a «o 1 14 4 4 9 4 t - ; IT a • a • t c 19 e a 3 9 1 9 ; • ■ 9 i • 4 5 I 9 11 »Ji « 9 19. t 9 9 t . 0 c - : 1 * 237 1iV )7 '(t Ii or C hi £«•*'• C h i t« v « i» rwhi Cfiou*i*r filtniitt Cvimt C*ht*#m» C m ) v C M M lS s fh f Cht It* 1 0 0 3 4 3 HC » i ; 0*»»i*f<rr#«yhhC<*» 1*f I t C m 13 iM O ll 13 1 1 1 3 > ■ e j >3 33 l< S3 3« IT 3* I* 3 0 33 33 3 < 3 3 33 3 7 3 3 33 40 14 < 43 44 4 3 43 10 T a i« < » 333 3 3 3 3 3 7 3 3 1 4 3 3 11 3 3 3 3 3 11. 3 1 i 3 3 7 3 1 1 13 3 13 1 3 13 7 1 1 3 3 7 13 10 3 1 4 17 13 1 . 1 3 13 1 0 333 238 ia * i ii? h i S .* w > » * y * » » * • * » ?» t-» e i '« 6 n t6*< tSk n « * » W iti^ g • o j t t C * > C » Siki'i »■¥**• 6 t l t l ' N C hllK M • V f t h » f t CftMtftfttt* CW C-ft-tft't V Iftt 3 na ail | ( f t t f e f t t ? 4 * a 4 f t T « f t > l 1 f t »i 3 1 C 3 3 o *1 I ® 3 9 f t • 1 ' 0 4 • • f t i a' s f t • • i ■ e; 3 f t a a 1 4 i a 3 * « a• e • * * 4 t i c 1 J * ’ 0la' i 'J r r * 3 1 1 1 • > 4 f t * i f ( •7 4 « ■l .0 1 1 •a f t * ¥ S3 ” < 4 : * * • i f t * ■ i • f t * - 3' * V • f t • 4 4 : « i • ? • • - c> j ■ a < 1 t 1 t 1 3 ! H • f t • 4 3' a * * * • • • 4 * V . a ■ i » • < • : * 0• i 1 ? :s « 4 c! a' i 3 3 i e' e • i 1 1 » ' 3 . ? 3 f t r «.at t 3 f t a • 3 3 I 1 3 ? • « *• : ’ • 3 f t » « 4 f t * • ' ? f t 3 » a * 4 • : 1 t a a t * «■ • ■ 1 3 a * • i ’ *1 1 i ii 3 3 4 • *0I3| ? 3 aa r »• a ! a 1 t 3 4 a ■ s A " a ft e a • eici a a* a »i D < 0 1 f t ar 3 4 • i Is 3 ft a • a i 3< 0 at a ? i «• 0 « o 7 » i e. a 4 1 4 • :: c< a * 3 T * *0 1 i •a a f t ' t : t . •* 44 ' < 0 a lii * ' 4 ft • t 0 e i * 4 ft # * • i e -» 4 3 i f t : f t * ' J 4 * D T p lf ttf t « •» 7 * f t i ' l l ^ 239 f c im < n » , y T * b i* ie' u $ttt it S W * £ « * > « # M > m M itlm g O f C hi Sauxt C hi Sdvhr* P v * n * G Sow trta CSdui'ts S -V im i C e n im g « h tv C « » * C r» m * f* V m ■ in m H I) O t t t h r M r t r o ) « ' 1A D » * t ! J G H T S t t h n g T*un i J 3 4 % • I ’ ! ' e i 4 ! a ! o I e e I o • 1 e 0 e 3 a. t Si I | * a! a 3 : - o ‘ i * 9 ’ 1 : a 1 a Q e 1 a a • a o i i e * 1 I 6 c e e 3, 0 o : e 3 ! 1 1 9 ' 1 1 1 e • : *1 r 0 t • 9 1 • < i 3 : : • ■a a 3 i 9 | «• ' 8 f t 9 3 . ' ' * % P * \ • « ' • ’ % S O 1 7 a c * • 1 1 1 9 * : 3 • 1 g 3 t ' 9 1 » 0 3 0 ' 9 3 ! i t C ' 3 • 1 1 e 0 ' 3 1 0 c 9 9 : • I 0 m • 3I9> a ' J 3 9 ' : «3 ’ 3 9 9 4 0 T 3 ‘ 9 • ! * 0 M O ! ) • ' 0 1 * * 0 1 ) 0 | « 0 m e 1 Tl 0 0 9 i 3 • i ' ’ e 1 o 3 * 9 O' 0 1 D 1 » 1 3 0 * ' 3 I » i 0 < 1 « I 3 1 13 i 0 t l : ■ r I • : i e< 919 » t ) e ! o 1 3 11 j ’ ’ ! 0 ( I • 4 * 1 * . ♦ • ■ n J « • ' • 1 )S ' r g p • , t 1 s % 3 : • t : p * * ’i c J j ’ 0 1 » > ) v : *• ;» ’4 » ji APPENDIX L WRITING CRITERIA FACTOR TOTALS 241 Analytic Rubric based on Question 15 responses: The instrument reflected the following guidelines (based on Jacobs et al. 1081 ESL Composition Profile and Diedrich et al.'s 1061 wrung factor response),as determined by survey responses: Content: 0-25 pts. possible (to include descriptors based on the following:) a. content (ideas, knowledge) (252)67% (86)24% (27)7% (3)1% (4)1% e. original Ideas A /or tactual support (166)51 (121)33 (43)12 (6) 2 (5) 1 L detailed development (76) 22 (127)36 (109)31 (28) 8 (10) 3 m. topic identfficaton (what topic requires) (120)34 (t2l) 34 (70)20 (28)8 (13) 4 Oraanizatlon: 0-20 ots. b. sequencing (67) 24 (124)35 (111)31 (27)8 (10)3 f. cohesion (linking together) (124)34 (146)40 (60)22 (13)4 (5) 1 J , length/brevity (51)14 (124)34 (119)33 (47)13 (23) 6 q. organization (Ind. beginning, middle, ending) (130)35 (117)32 (69)24 (22) 6 (9) 2 Vocabulary: 0-15 ots. c, word deletion A /or unnecessary additions * (64) 25 (130)35 (97)26 (32)9 (16)4 g. register (level of voice. e.g. formal or inform al tone) (35)10 (52)15 (108)31 (60)23 (67)20 n. writer's purpose/audience awareness (61) 25 (112)31 (90)25 (51)14 (17) 5 r. appropriate w ord/idiom usage (68)24 (121)33 (115)31 (29) 6 (14) 4 Lanauaae Use: 0-20 Dts. d. preposition A /or article (aan,the) usage (104)26 (67)24 (105)29 (47)13 (24)7 k. sentence construction (105)28 (125)34 (100)27 (25)7 (11) 3 o. grammatical control (e.g« agteemenfcsuty/vetb/pronoun) (127)35 (110)30 (61)22 (30) 8 (IB) 5 s. verb usage (tense, modal, auxiliary, gerund. Infinitive) (120)33 (106)29 (68)24 (36)10 (IB) 5 Mechanics: 0-20 ots. h. punctuation (104)28 (106)29 (96)26 (44)12 (17) 5 L spelling (165)44 (105)26 (63)17 (28)8 (12) 3 p. capitalization (107)29 (67) 24 (81)22 (63)17 (25) 7 t form /format/handwrlting (82)23 (90)25 (105)29 (55) 15 (29) 6 Total: 0-100 pts. APPENDIX M GWT ESSAY REQUEST 243 M E M O R A N D U M To: Lfnda Gaschler, Test Officer From: Karen Russikoff Date: November 1, 1993 RE: GWT R esearch Access and Exam Copies Thank you again for your time and interest last week. As we discussed, please find here a detailed listing of the material needed for my research in the required score batches and amounts. You indicated that it would be possible to acquire the amounts from a single test administration (probably Spring 1993) and a s such from a single topic, which will be most helpful. 3-3 Combination (Score 6) 120 3-4 (Score 7) 40 3-2 (Score 5) 40 In addition, Sampling of scores totaling 2,or 3,or 4 20 (altogether) Sampling of scores totaling 8.or 9 .1 0 .1 1 .1 2 20 (altogether) TOTAL # OF GWT ESSAYS 240 As agreed upon, I will be paying for the additional staff hours to duplicate the essays, and the copying will be done on Dr. Rivas' machine. Please let me know when the job is completed or if there are any complications since my time line requires the essays by mid-December. Thank you again for your assistance. APPENDIX N ESL PANEL ESSAY SELECTION iS i,.P a n e l S election for N s /N N S E ssa y s 245 Essay Code Panelists A, B, C (coded NS 1: NNS 2 ) (‘Discrepancies B Not used 1 931301 931401 931402 931403 931404 931405 931406 931407 931408 931409 931410 931411 931412 931413 931414 931415 931416 931417 931418 931419 931501 931502 931503 931504 931505 931506 931507 931508 931509 931510 931511 931512 931513 931514 931515 931516 931517 931518 931519 931520 931521 931522 931523 931524 931525 931526 931527 931528 931529 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 JUsed. 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 2 2 2 2 2 2 2 Dot adjudicated! 246 931530 2 2 931531 2 2 931532 2 2 9 3 1533 2 2 931534 2 2 931535 2 2 931530 2 2 931537 2 2 931530 2 2 931539 2 2 931540 2 2 931601 2 2 931602 2 2 931603 2 2 931604 2 2 9 3 1605 2 2 931606 2 2 931607 2 2 931608 2 2 931609 1 2 931610 2 2 931611 2 1 931612 2 2 931613 2 2 931614' 2 2 931615 1 1 931616 1 1 931617 2 2 931616 2 2 931619 2 2 931620 1 2 931621 1 1 931622 2 1 931623 2 2 931624 1 1 931625 2 2 931626 1 2 931627 2 2 931628 2 2 931629 2 2 931630 2 2 931631 2 2 931632 2 2 9 3 1633 2 2 931634 2 2 931635 2 2 931636 2 2 931637 2 2 931638 2 2 9316 3 9 1 2 931640 2 2 931641 2 2 931642 2 1 9 3 1643 2 2 931644 1 1 931645 2 2 931646 2 2 931647 2 2 931648 2 2 931649 2 2 2 1 2 1 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 2 1 2 2 1 1 2 2 2 2 2 1 2 1 1 1 2 2 2 2 2 2 1 1 1 1 1 1 2 2 1 1 2 2 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2 2 1 2 2 2 2 2 1 2 1 1 1 2 . 2 2 2 2 2 2 t 2 2 247 9316S0 2 2 931651 2 2 931652 1 1 931653 2 2 931654 2 2 93165S 2 2 931656 2 2 931657 2 2 931658 2 2 931659 2 2 931660 1 1 931661 2 2 931662 2 2 931663 2 2 931664 1 1 931665 1 1 931666 2 2 931667 2 2 931668 2 2 931669 2 2 931670 1 1 931671 2 2 931672 1 1 931673 2 2 931674 1 1 931675 1 1 931676 2 2 931677 1 1 931678 1 1 931679 1 1 931680 1 1 931681 2 2 931682 2 2 931683 2 2 931684 1 1 931685 2 2 931686 2 2 931687 1 1 931688 2 2 931689 2 2 931690 2 2 931691 1 1 931692 2 2 931693 2 2 931694 1 1 931695 1 1 931696 2 2 931697 2 2 931698 1 1 931699 1 1 930601 2 2 930602 2 2 930603 2 2 930604 1 1 930605 1 1 930606 2 2 930607 1 1 930608 1 1 930609 2 2 930610 2 2 2 2 2 2 1 2 2 2 2 2 2 2 1 2 2 2 1 1 2 2 2 2 1 2 1 2 1 1 2 1 1 1 1 2 2 2 1 2 2 1 2 2 2 1 2 2 1 t 2 2 1 1 2 2 2 1 2 1 1 2 2 248 930611 2 2 2 930612 2 2 2 930613 2 2 2 930614 2 2 2 930615 2 2 2 930616 1 1 1 930617 2 2 2 930618 2 2 2 930619 2 2 2 930620 1 1 1 930621 2 2 2 931701 2 2 2 931702 1 2 2 931703 1 1 1 931704 2 2 2 931705 1 1 1 931706 1 1 1 931707 2 2 2 931708 2 2 2 931709 2 2 2 931710 2 2 2 931711 1 1 1 931712 1 1 1 931713 931714 1 1 1 2 931715 1 1 1 931716 1 1 1 931717 1 1 1 931718 1 1 1 931719 2 2 2 931720 2 2 2 931721 2 2 2 931722 2 2 2 931723 1 2 1 931724 2 2 2 931725 2 2 2 931726 1 1 1 931727 1 1 1 931728 1 1 1 931729 1 1 1 931730 931731 1 1 1 931732 1 1 1 931733 1 1 1 931734 931735 931736 1 1 1 931737 1 1 1 931738 1 1 1 931739 1 1 1 931740 1 1 1 931801 1 1 1 931802 1 1 1 931803 1 1 1 931804 1 1 1 931805 931606 1 1 1 931807 1 1 1 931808 1 1 1 931809 1 1 1 2 2 2 2 2 2 2 2 1 2 2 2 2 2 2 249 i 931901 1 1 1 2 931902 1 1 1 2 931903 1 1 1 2 931904 1 1 1 2 931905 1 1 1 1 931906 1 1 1 1 931907 1 1 1 1 931001 1 1 1 2 931002 1 1 1 2 931003 1 1 1 2 931004 1 1 1 2 to ta l 240 essays Selected: 89 NNS essay s (in 3-8 range) 11 NS essa y s in 8 ,9 ,1 0 range) APPENDIX O DETAILED ANALYTIC RUBRIC 251 CQNTENlL s c o r e 2 5 -0 25-18 EXCELLENT TO VERY GOOD knowledgeable • thorough development of thesis • relevant to assigned topic • may be originally or factually supported 17-9 GOOD TO AVERAGE som e knowledge of subject • limited development of thesis • mostly relevant to topic, but lades detail 84} FAIR TO POOR limited knowledge of subject • little substance • inadequate development of topic___________________ ORGANIZATION;. SCORE 20-14 13-7 64} EXCELLENT TO VERY GOOD well-organized • ideas clearly stated/supported • logical sequencing' cohesive • length appropriate GOOD TO AVERAGE loosely organized but main ideas stand out * logical but incomplete sequencing FAIR TO POOR Ideas confused or disconnected * lacks logical sequencing_________ ■ VOCABULARY:___l&Q SCORE 15-11 EXCELLENT TO VERY GOOD. sophisticated range • effective word/idiom choice and usage • appropriate register for author's purpose/audience needs 10-6 GOOD TO AVERAGE adequate range • occasional errors in word/idiom form, choice, usage, including unnecessary additions/omissions • but meaning not obscured 54} FAIR TO POOR limited range • frequent errors of word/idiom form, usage, choice • ________meaning confused or obscured ______ ______________ LANGUAGE USE: 20-0 SCORE 20-14 EXCELLENT TO VERY GOOD effective complex structures • few errors of agreement, verb tense, articles, pronouns, prepositions 13-7 GOOD TO AVERAGE effective but simple constructions * several errors of agreem ent, verb tense, articles, pronouns, prepositions • but meaning seldom obscured 64} FAIR TO POOR major problems In constructions • frequent errors in agreement, tense, art id as. pronouns, fragments, run-ons« meaning c onfused or obscured M ECHANICS;, .2 M SCORE 20-14 EXCELLENT TO VERY GOOD demonstrates mastery of conventions * few errors of spelling, punctuation, capitalization 13-7 GOOD TO AVERAGE occasional errors of spelling, punctuation, capitalization • but meaning not obscured 6-0 FAIR TO POOR frequent errors of spelling, punctuation, capitalization • poor handwriting - meaning confused or obscured APPENDIX P BRIEF ANALYTIC RUBRIC CONTENT: 25-0 SCORE 25-18 EXCELLENT TO VERY GOOD 17-9 GOOD TO AVERAGE 8-0 FAIR TO POOR ORGANIZATION: 20-0 SCORE 20-14 EXCELLENT TO VERY GOOD 13-7 GOOD TO AVERAGE 6-0 PAIR TO POOR VOCABULARY: 15-0 SCORE 15-11 EXCELLENT TO VERY GOOD 10-6 GOOD TO AVERAGE 5-0 FAIR TO POOR LANGUAGE USE: 20-0 SCORE 20-14 EXCELLENT TO VERY GOOD 13-7 GOOD TO AVERAGE 6-0 FAIR TO POOR MECHANICS: 20-0 SCORE 20-14 EXCELLENT TO VERY GOOD 13-7 GOOD TO AVERAGE 6-0 FAIR TO POOR Total S co re R eader E ssa y C ode APPENDIX Q COVER LETTER/DIRECTIONS FOR HOLISTIC SCORING 255 January 12,1994 Dear (ESL Specialists), Thank you again for agreeing to participate in the scoring of native and non-native student GWT e ssa y s in order to assist m e with this research project. To simplify your schedule, I am providing you with both stacks of e ssa y s (25-25) at one time. You do not need to feel pressured to do them both immediately. As you score these papers, try to rem em ber to set aside your usual grading practices and to score only on an holistic basis. That m eans you should not count m istakes or ideas; Instead, score only for an overall general impression that corresponds to a single num ber on the enclosed GWT rubric. Do not m ake any m arks on the essays. You should read each essay only once, spending probably no m ore than one to two m inutes per paper. P lease circle the num ber you select (1-6) on the score sheet attached to each essay. Kindly place all e ssa y s back into the envelope and either put it in my box or return it to m e in person. While I know you are busy with your usual grading, I hope that you will be able to complete both stacks within two weeks, by Ja n . 27, so that we may move to the second type of scoring immediately thereafter. If you have any questions, don't hesitate to call m e (H phone included). Again, thank you for your generous assistance. The project would be impossible without your help. Sincerely, K. Russikoff APPENDIX R COVER LETTERS/DIRECTIONS FOR ANALYTIC SCORING 257 February 4,1994 Dear (ESL Specialists), Thank you again for your generous assistance. B ecause your holistic scoring w as so reliable (ETS would have been pleased), we can now move on to the final phase, that is scoring the sam e essays but this time with an analytical scoring guide. This type of scoring requires that you consider a range of points within five factors for analysis of the papers: Content (0-25 points), Organization (0- 20 pts.). Vocabulary (0-15 pts.), Language Use (0*20 pts.), and Mechanics (0-20 pts.)(Total points possible: 100). Each section is divided into three proficiency levels with detailed descriptors for each factor: Excellent to Very Good; Good to Average; and Fair to Poor. Points for each section vary (e.g., Content has a max. of 25; Vocabulary, only 15; others, 20), so vou wlllneed to use the correct range for each factor. By using your own judgment to determine level and degree, you will assign points for competency in each section. You must thus assign five sc o re s for each paper. This type of scoring is, of course, more demanding than holistic assessm ent since you are required to m ake many more judgm ents than you did in the first round of holistic assessm ent. In som e c ases, you may need to re-read a section of an essay to consider the level of a particular factor (e.g., Language Use or Vocabulary). However, attempt to spend no m ore than three to four minutes on each paper; additional or excessive time may bias your judgment and, a s such, your scoring. It is not necessary for you to total the points for each essay; this will be completed for you afterwards. Again, please set aside your own standards for your class grading and rely instead solely on the analytic rubric. If you can complete these by Tuesday, Feb. 2 2 ,1 can com pute the statistics by the end of the month. Please don't hesitate to call m e if you have any questions, concerns, puzzles, or unravelings along the way (H phone & W x included). I am looking forward to sharing my findings with you, and I very much appreciate your care, time, effort, and interest. Gratefully. K. Russikoff 258 February 16,1994 Dear (GWT Raters), Thank you again for your generous assistance with the scoring of native and ESL student essa y s for my research. Enclosed you will find two bundles (25/25) of previously-scaed GWT essays. It is not necessary that you know the original score since this time you will be using an analytical scoring guide com posed of criteria Cal Poly faculty u se in academ ic coursework. Analytic scoring requires that you consider a range of points within five factors for analysis of the papers: Content (0-25 points), Organization (0*20 pts.), Vocabulary (0-15 pts.), Language Use (0*20 pts.), and Mechanics (0-20 pts.) (Total points possible: 100). Each section is divided into three proficiency levels with detailed descriptors for each factor: Excellent to Very Good; Good to Average; and Fair to Poor. Points f a each section vary (e.g., Content has a max. of 25; Vocabulary, only 15; others, 20), so you will need to u se the correct range f a each fa c ta . By using your own Judgment to determine level and degree, you will assign points f a com petency in each section. You must thus assign five sc o re s for each p a p a . This type of sc a in g is, of course, m a e demanding than holistic assessm ent since you are required to m ake many m a e judgments. In som e cases, you may need to re-read a section of an essay to c o n sid a the level of a particular fa c ta (e.g., Language Use a Vocabulary). H ow eva, attempt to spend no m a e than three to four minutes on each p a p a ; additional a excessive time m ay bias your judgment and, as such, your scain g . It is got necessary f a you to total the points for each essay; this will b e completed f a you afterw ads. Please set aside your own sta n d a d s for your class grading and rely only on the rubric. If you can complete these by Monday, Feb. 2 8 ,1 can com pute the statistics early in M ach. (H ow eva, if you feel compelled to com plete them s o o n a , I would, of course, be happy to accept them!) P lease d o n t hesitate to call m e if you have any questions, c o n c an s, puzzles, a unravelings along the way (H phone & W x included). I am looking forw ad to s h a in g my findings with you, and I very much appreciate your c a e , time, effort, and intaest. Gratefully, K. Russikoff APPENDIX S READER SCORING CONFIGURATIONS AND RELIABILITIES 260 Reader Scoring Configurations: Essay Stacks: W, X , Y , Z Readers: (4) ESL Hofisttc 1, 2: (4) ESL Analytic 3,4; (4) GWT Analytic 5,3 Holistic Readers < 4 ESL Specialists: M. H. Q. PI Wl Vfc Reader M Reader O X1 X2 Reader N Reader O Y1 Y2 Reader N Reader P Z1 Z2 Reader M Reader P Reader M Wi Z < \ Reader N X1 Y1 Reader O W fe X2 Reader P Y2 Z2 Analytic R aiders (4 ESL Specialists: M. N. Q. PI Yfc W4 Reader M Reader O X3 X4 Reader N Reader O Y3 Y4 Reader N ReaderP Z3 Z4 Reader M Reader Reader M W3 Z3 Reader N X3 Y3 Reader O W 4 X4 Reader P Y4 Z4 Analytic Readers M QWT Raters; Q. R. S. Tl W5 V \fe Reader Q Reader S X5 Xq Reader S Reader R Y5 y 6 Reader T Reader R 25 26 Reader Q Reader T Reader Q W s Z5 Reader R Ye Xe Reader S W fe X5 Reader T Y5 Z q 261 Inter-reader Reliability Correlations (Pearson-pr xJuct moment correlations) Reader Scorlno Conflagrations and Correlattons: Holistic Readers f4 ESL Soadatets: M . n . o . P Corr.w. Ohg. GWT Wi Yfe Reader M Reader 0 .093 Xi X2 Reader N Reader O .736 Vi Y2 Reader N Reader P .764 Zi Z2 Reader M Reader P .827 Inter-reader Corr. Wi W 2 Reader M Reader O .661 Xi X2 Reader N Reader 0 .66 Yi Y2 Reader N Reader P .623 Zi Z2 Reader M Reader P .641 Analytic Readers (4 ESL SoedaJlsts: M . N. O. Pi W 4 • Reader M Reader O .716 X3 X4 Reader N Reader 0 .703 Y3 y4 Reader N ReaderP .789 Z3 Z4 Reader M Reader P .665 Analytic Readers f4 GWT Raters: Q. R. S. *n v* W B Reader Q R eaders .665 Xs X6 R eaders Reader R ,701 Y5 Ye Reader T Reader R .875 Z6 ze Reader Q Reader T .654 Analytic Inter-arouD Corr, W3W4 Total W5W6 Total .734 X3X4 Total X5X6 Total .834 Y3Y4 Total Y5Y6 Total t * .674 Z3Z4 Total Z5Z6 Total .755 APPENDIX T BATCH SC O R E S 263 W BATCH: (* indicates change from original GWT Pass/Fail score) (P indicates Pass: F indicates Fail) Holistic Scoring Analytic Scoring Total ESL Specialists Code Orifl. Scores Total Scores Tote/ GWT Raters Scores Total Readers Cum. Mean (ORIGINAL FAILING ESSAYS) 1301 3F 1/2 3F 46/31 77 F 32/59 91 F 168 42.0F 1415 4F 3/3 6F 72/48 120P* 55/69 124P* 244 61 .OP* 1419 4F 21 2 4F 48/60 108P* 47/60 107P* 215 53.8P* 1511 5F 3/3 6F 43/48 91 F* 44/65 109P** 200 50.0F 1536 5F 3/1 4F 50/40 90 F 38/56 94 F 184 46.0F 1505 5F 2/2 4F 65/55 120P* 49/65 114P* 234 58.5P* 1540 5F 2/2 4F 43/40 83 F 36/50 86 F 169 42.3F 1501 5F 2/2 4F 58/53 111P* 56/73 129P* 240 60.0P* 1601 6F 4/2 6F 69/60 129P* 41/62 103P* 232 58.0P* 1631 6F 2/2 4F 49/63 112P* 41/73 114P* 226 56.5P* 1635 6F 1/2 3F 37/40 77 F* 66/63 129P*- 206 51.5P* 1637 6F 2/2 4F 56/53 109P* 62/56 118P* 227 56.8P* 1606 6F 3/3 6F 66/55 121P* 56/60 116P* 237 59.3P* 1612 6F 2/2 4F 68/31 99 F - 42/68 110P*« 209 52.3P* 1618 6F»» 4/3 7P*« 48/38 86 F« 49/54 103P** 189 47.3F 1645 6F 3/2 5F 67/60 127P** 39/55 94 F • 221 55.3P* 1650 6F 2/2 4F 46/48 94 F 36/64 100F 194 48.5F 1656 6 F - 4/3 7P*.. 68/48 116P* 67/70 137P* 253 63.3P* (ORIG. PASSING ESSAYS) 1725 7P « 3/2 5F*** 45/60 105P 48/76 124P 229 57.3P 1720 7 P „ 3/3 6F*** 68/75 143P 54/67 121P 264 66.0P 1709 7P 4/3 7P 78/73 151P 63/71 134P 285 71.3P 1701 7 P - 3/3 • • « u . C D 71/63 134P 88/78 166P 220 55.0P 1806 8P 5/4 9P 90/81 171P 85/71 156P 327 81.8P 1901 9P 5/3 8P 94/100 194P 84/84 168P 362 90.5P 1004 10P 5/3 0 P 79/74 153P 80/86 166P 319 7 9 ,8 P Holistic(ESL) Additional Passing 2 AdditionalFaiiipg 3 Analytic(ESL) 10 ____________ Q _ Analytic(GWT) 13 0 Analytic(ALL) 12 0 10 13 12 •Analytic Discrepancies: 5 (5 additional pass; o fail) (With 4 rdrs. 3 would pass; 2 would fail) ••Holistic Discrepancies: 5 (2 additional pass; 3 fail) W BATCH TOTAL ANALYTIC CHANGES: 23 P; Q F (From 2 sets of 25 essays [n=50] a s combined scores, with 2 groups of readers.) 284 X BATCH; (* indicates change from original GWT Pass/Fait score) (P indicates Pass; F indicates Fait) Holistic Scoring Analytic Scoring Total ESL Specialists GWT Raters Readers Code OrlQ. Scores Total Scores Total Scores Total Cum. Mean. (ORIGINAL FAILING ESSAYS) 1414 4F 3/3 6F 49/55 104P*» 45/39 34 F» 188 47.0F 1418 4F 1/3 4F 34/48 82 F 69/29 98 F 180 45.0F 1401 4F 3/3 6F 37/60 97 F 65/31 96 F 193 48.3F 1512 5F 3/3 6F 44/52 96 F 62/27 89 F 185 46.3F 1535 5F 3/3 6F 65/55 12CP** 68/29 97 F* 217 54.3P* 1508 5F 3/3 6F 55/53 108P*« 60/32 92 F* 200 50.0F 1539 5F 2/2 4F 41/38 79 F 57/33 90 F 169 42.3F 1502 5F 3/3 6F 37/67 104P*« 52/31 83 F* 187 46.8F 1630 6F» 4/3 7P*« 44/53 97 F 57/26 83 F 180 45.0F 1634 6F» 5/3 8P# » * 50/60 110P* 64/38 102P* 212 53.0P* 1638 6F 3/3 6F 78/60 138P* 70/56 126P* 264 66.0P* 1625 6F 3/3 6F 47/36 82 F 62/33 95 F 178 44.5F 1610 6F 2/3 5F 64/68 132P* 69/36 105P* 236 59.3P* 1617 6F» 5/4 9P*.. 69/81 150P* 70/39 109P* 259 64.8P* 1641 6F« 4/3 7P*.. 40/61 101P* 69/43 112P* 213 53.3P* 1649 6F 3/3 6F 44/60 104P* 57/46 103P* 207 51.8P* 1655 6F 3/3 6F 48/53 101P* 60/43 103P* 204 51 .OP* 1659 6F« 4/3 7P*.. 75/60 135P* ~ 81/58 139P* 274 68.5P* (ORIG. PASSING ESSAYS) 1731 7P 4/4 8P 86/74 160P 71/45 116P 276 69.0P 1721 7 p .. 3/3 6F*»* 65/53 118P 69/44 113P 231 57.8P 1710 7P 3/4 7P 69/72 141P 69/56 125P 266 66.5P 1704 7P " 3/3 6F*« 54/67 121P* 55/31 86 F** 207 51.8P 1807 8P 5/5 10P 98/81 179P 86/67 153P 332 83.0P 1902 9P 6/5 11P 86/82 168P 70/62 132P 300 75.0P 1003. 10P 4/5 9P 89/10Q- 1B9P 90/67 157P 346 86,5P Holistic(ESL) Analytic(ESL) Analytic(GWT) Analytic(ALL) Additional Passing 5 12 8 9 Additional Failing 2______________Q _____________ j________________Q 7 12 9 9 •Analytic Discrepancies: 5 (4 additional pass; 1 fail) (With 4 rdrs. 2 would pass; 3 would fail) ••Holistic Discrepancies: 7 (5 additional pass; 2 fail) X BATCH TOTAL ANALYTIC CHANGES: 2QP, I F (From 2 sets of 25 essays [nss50] as combined scores, with 2 groups of readers.) •265 Y-BATCH: (* indicates change from original GWT Pass/Fail score) (P indicates Pass; F indicates Fail) Holistic Scoring Analytic Scoring Total ESL Specialists GWT Raters Readers Code Orlo. Scores Total Scores Total ScoresTota! Cum. Mean (ORIGINAL FAILING ESSAYS) 1413 4F 2/2 4F 39/35 74 F 42/35 77 F 151 37.8F 1417 4F 2/2 4F 46/36 82 F 39/33 72 F 154 38.5F 1402 4F 1/2 3F 25/40 65 F 24/32 56 F 121 30.3F 1513 5F 2/3 5F 48/61 109P** 44/52 96 F* 205 51.3P* 1534 5F 3/3 6F 70/58 128P** 45/42 87 F* 215 53.8P* 1509 5F 3/2 5F 32/27 59 F 31/27 58 F 117 29.3F 1538 5F” 4/3 7P*** 52/50 102P** 48/51 99 F* 201 50.3P* 1503 5F 4/2 6F 47/37 84 F 30/49 79 F 163 40.8F 1629 6F 3/3 6F 55/58 113P* 58/50 108P* 221 55.3P* 1633 6F» 6/3 gp-« 85/68 153P* 54/51 105P* 258 64.5P* 1604 6F»* 4/3 7P# » 84/77 161P** 38/50 86 F* 249 62.3P* 1603 6F 3/3 6F 57/61 118P* 56/49 105P* 223 55.8P* 1608 6F 3/2 5F 70/73 143P** 41/44 85 F* 226 57.0P* 1623 6F» 4/3 7P*“ 68/49 117P*« 34/40 74 F* 191 47.8F 1640 6F 3/3 6F 57/54 111 P*- 54/43 97 F* 208 52.0P* 1647 6 F - 4/3 7P*« 88/73 161P** 48/50 98 F- 259 64.8P* 1653 6F 3/3 6F 70/78 148P* 61/59 120P* 268 67.0P* 1661 6F 3/2 5F 54/50 104P** 30/39 69 F- 173 43.3F (ORIG. PASSING ESSAYS) 1735 7P 4/4 8P 59/80 139P 60/53 113P 252 63.0P 1722 7P» 3/3 6F*« 64/63 127P* 50/37 87 F** 214 53.5P 1714 7P*» 2/3 5F*** 63/43 106P* 25/35 60 F-* 166 41.5F* 1707 7P» 3/3 6F*« 63/57 120P 67/61 128P 266 66.5P 1808 BP 5/4 9P 87/78 165P 84/68 152P 317 79.3P 1903 9P 6/4 10P 66/79 145P 93/69 162P 307 76.8P 1002 10P 5/4 _ 9P 88/66 154P 91/75 166P 320 80.0P Holistlc(ESL) Analytic(ESL) Analytic(GWT) Analytic(ALL) Additional Passing 5 13 4 11 Additional Failino 3 O 2 1 8 13 6 12 •Analytic Discrepancies: 11 (9 additional pass;2 fail) (With 4 rdrs. 8 would pass; 3 would fail) ••Holistic Discrepancies: 8 (5 additional pass; 3 fail) Y BATCH TOTAL ANALYTIC CHANGES: 1Z P: 2 F (From 2 sets of 25 essays [ne50] as combined scores, with 2 groups of readers.) 266 2 -B A T m (* Indicates change from original GWT Pass/Fail score) (P indicates Pass; F indicates Fail) Holistic Scoring Analytic Scoring Total ESL Specialists GWT Raters Readers Code Orla. Scores Total Scores Total. Scores Total Curl Mean (ORIGINAL FAILING ESSAYS) 1412 4F 2/3 5F 40/48 88 F 38/35 73 F 161 40.3F 1416 4F 3/3 6F 61/52 113P** 41/42 83 F« 196 49.0F 1403 4F 1/2 3F 59/42 101 P*» 36/35 71 F* 172 43.0F 1514 5F 1/3 4F 50/52 102P** 36/47 83 F* 185 46.3F 1533 5F 2/2 4F 51/51 102P*« 33/54 87 F» 189 47.3F 1510 5F 3/2 5F 58/60 118P* 70/49 119P* 237 59.3P* 1537 5F 2/2 4F 66/49 115P*- 32/54 86 F* 201 50.3P* 1504 5F 2/2 4F 59/50 109P** 38/44 82 F- 191 47.8F 1627 6F 3/3 6F 75/42 117P* 78/54 132P* 249 62.3P* 1628 6 F - 4/3 7P*— 75/76 151P* 59/55 114P* 265 66.3P* 1632 6F 2/3 5F 65/48 113P* 74/59 133P* 246 61.5P* 1602 6F 2/3 5F 45/53 * 98 F 56/35 91 F 189 47.3F 1607 6F 3/3 6F 72/71 143P* 75/57 132P* 275 6B.8P* 1613 6F 2/3 5F 73/56 129P* 92/43 135P* 264 66.0P* 1619 6F 2/3 5F 77/68 145P** 50/39 89 F- 234 58.5P* 1646 6F 2/3 5F 70/54 124P* 69/50 119P* 243 60.8P* 1651 6F 2/3 5F 40/61 101P** 36/47 83 F« 184 46.0F 1662 6F 2/3 5F 62/63 125P** 39/34 73 F* 198 49.5F (ORIG. PASSING ESSAYS) 1736 7 p .. 2/3 5F*- 79/71 150P 69/62 131P 221 55.3P 1724 7P 4/4 8P 76/68 146P* 51/49 100F** 246 61.5P 1719 7 p .. 3/3 • • • u. to 65/57 122P 73/58 131P 253 63.3P 1708 7P 4/4 BP 67/67 134P 78/71 149P 283 70.8P 1609 8P 4/4 8P 87/75 162P 81/70 151P 313 78.3P 1904 9P 5/3 8P 89/83 172P 72/77 149P 321 80.3P 1001 1 0 P 5/6 11P 90/100. 1 9 0 P 9 8 /9 7 1 9 5 P ... 365.. 96.3P Holistic(ESL) Anatytic(ESL) Analytic(GWT) Analytic(ALL) Additional Passing 1 16 7 a A dditional Failina 2 0 1 0 3 16 8 9 •Analytic Discrepancies: 10 (9 additional pass;1 fail) (With 4 rdrs. 3 would pass; 7 would fail) ••Holistic Discrepancies: 3 (1 additional pass; 2 fail) Z BATCH TOTAL ANALYTIC CHANGES: 23 P; 1 F (From 2 sets of 25 essays (ns50] as combined scores, with 2 groups of readers.) APPENDIX U C R O SS TABULATION OF FACULTY TENURE TO ASSIGNED WRITING ‘ 268 TABLE U .l. DISTRIBUTION O F FACULTY TENURE TO ASSIGNED WRITING <1 yr. 1-5 vrs. 6-10 vrs. 10+vrs. G eneral Wrrtina N=3B9(3 missing) Lower division (3 missing) None/NA 4 33 27 53 1-2 A 15 22 66 3*6 2 26 19 60 7*10 2 7 7 19 11+ 0 4 4 7 NR 1 3 1 3 Upper division (3 m issing) None/NA A 13 12 22 1-2 6 11 16 53 3*6 1 47 36 69 7*10 1 12 12 35 11 + 0 3 2 8 NR 1 2 2 1 Lenqihv_WritinQ Lower division (3 missing) None/NA 7 32 36 81 1-2 4 31 23 72 3-6 0 18 14 38 7-10 1 4 5 10 11 + 0 1 0 3 NR 1 2 2 4 Upper division (3 missing) None/NA 5 13 17 30 1-2 5 26 29 84 3-6 1 36 27 72 7-10 1 8 5 18 11+ 0 1 0 3 NR 1 2 2 1 Timad-Wrltlng Lower division (3 missing) None/NA 8 49 50 130 1-2 2 23 16 38 3-6 2 6 8 27 7-10 0 3 2 7 11+ 0 1 2 5 NR 1 3 2 1 Upper division (3 missing) None/NA 8 39 38 110 1*2 4 25 22 55 3*6 0 16 18 31 7-10 0 4 1 7 11+ 0 0 0 4 NR 1 4 1 1 APPENDIX V CALIFORNIA STATE UNIVERSITY GWAR TEST OPTIONS BY CAMPUS 270 M ethods of meeting the Graduation Writing A ssessm ent Requirement at the twenty California S tate University cam puses (CSU Survey, 1990): (Required:Req'd.; Optional: O pt.) Cam pus Essay _ Exam Objective Test Course Option Choice of ExamI C ourse Bakersfield Opt. Opt. Opt. Chico Req'd. Req'd. Dominguez Hitts Opt. Opt. Opt. Fresno Opt. Req'd. Opt. Opt. Fullerton Req'd. Req'd. Req’ d. Hayward Req'd. Req'd. Req'd.* Humboldt Req’ d. Opt. Opt. Long Beach Req'd. Req'd. Los Angeles Req'd. Opt.* Opt. Northridge Req'd. Pom ona Req'd. Sacram ento Req'd. Opt.* Opt. San Bernardino Opt. Opt. Opt. San Diego Opt. Req'd. Opt.* Opt. San Francisco Req’ d. Opt.* Opt. S an Jo se Req'd. Req’ d. Req'd. S an Luis Obispo Opt. Opt. Opt. S an M arcos" Req'd. Sonom a Req'd. Opt.* Opt. Stanislaus Reo’d. * indicates special circum stances involved. " D a ta collected per phone call (4-21-94).

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

The Effect Of Explicit Instruction On The Summarization Strategies Of "Underprepared" Native Spanish-Speaking Freshmen In University-Level Adjunct Courses

PDF

Self-selected reading and interactive vocabulary instruction: knowledge and perceptions of word learning among L2 learners

PDF

Communicative rights and responsibilities in an East Los Angeles barrio: An analysis of epistemic modal use

PDF

The Nature Of Assisted Performance With Learning Handicapped Students During Language Arts

PDF

A longitudinal study into the learning style preferences of university ESL students

PDF

Integration Factors Affecting Commitment To Educational And Occupational Goals For Latino Para -Educators

PDF

The Effects Of Selected Variables On The Transitioning Of Low-Achieving Special Education Adolescents From High School

PDF

An Assessment Of The Zooreach! Program As A Model For The Development Of Informal Education Programs

PDF

An analysis of United States overseas English language policy, 1938-1990

PDF

A Descriptive Study Of Selected Historically Black Colleges And Universities

PDF

Educational development aid and the role of language: A case study of AIT -Danida and the Royal University of Agriculture in Cambodia

PDF

Improving the academic achievement of African-American males: a case study in San Diego, California

PDF

Child As Apprentice-Narrator: Socializingvoice, Face, Identity, And Self-Esteem Amid The Narrative Politics Of Family Dinner

PDF

A cross -cultural comparison of achievement motivation among college students in South Africa

PDF

Change beliefs and academic outcomes: A construct validation

PDF

Authenticity, appropriation and voice: One primary classroom's writing experiences

PDF

A syntactic analysis of the written language of selected Black elementary school children with reference to sociological variables

PDF

Implementation of literature circles in a rural high school English class: One teacher's journey of changing student attitudes toward reading

PDF

The Predictive Validity Of English Language Screening Instruments For Foreign Students Entering The University Of Southern California

PDF

A comparative analysis of factors underlying the reading performance of American and South African ESL university students

Asset Metadata

Creator Russikoff, Karen Ann (author)

Core Title A Comparison Of Selected Writing Criteria Used To Evaluate Nonnative Speakers Of English At A California State University

Degree Doctor of Philosophy

Degree Program Education

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag education, higher,education, language and literature,education, tests and measurements,OAI-PMH Harvest

Language English

Contributor Digitized by ProQuest (provenance)

Advisor Eskey, David E. (committee chair), Kaplan, Robert B. (committee member), Rueda, Robert (committee member)

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c20-579357

Unique identifier UC11226280

Identifier 9601051.pdf (filename),usctheses-c20-579357 (legacy record id)

Legacy Identifier 9601051.pdf

Dmrecord 579357

Document Type Dissertation

Rights Russikoff, Karen Ann

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the au...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus, Los Angeles, California 90089, USA