University of Southern California Dissertations and Theses

An Experimental Application Of 'Cloze' Procedure As A Diagnostic Test Of Listening Comprehension Among Foreign Students

(USC Thesis Other)

An Experimental Application Of 'Cloze' Procedure As A Diagnostic Test Of Listening Comprehension Among Foreign Students

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content This dissertation has been microiilmed exactly as received 6 7-404 GREGORY-PANOPOULOS, John Fred, 1924- AN EXPERIMENTAL APPLICATION OF "CLOZE” PROCEDURE AS A DIAGNOSTIC TEST OF LISTENING COMPREHENSION AMONG FOREIGN STUDENTS* University of Southern California, Ph.D., 1966 Speech University Microfilms, Inc., Ann Arbor, Michigan (C) Copyright by John Fred Gregory-Panopoulos 1967 AN EXPERIMENTAL APPLICATION OF "CLOZE" PROCEDURE AS DIAGNOSTIC TEST OF LISTENING COMPREHENSION AMONG FOREIGN STUDENTS fey John Fred Gregory-Panopoulos A Dissertation Presented to the FACULTY OF THE GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (Speech) June 1966 UNIVERSITY O F SOU THERN CALIFORNIA T H E G R A D U A T E S C H O O L U N IV E R S IT Y PA R K L O S A N G E L E S . C A L IF O R N IA 9 0 0 0 7 This dissertation, 'written by Jah.n.JE’ jied.Qreg.QiLy.TJPjanopjaulaa.... under the direction of his.....Dissertation Com mittee, and approved by all its members, has been presented to and accepted by the Graduate School, in partial fulfillment of requirements for the degree of D O C T O R OF P H IL O S O P H Y Dean Date June.> . . 19. 66. Chairman TABLE OP CONTENTS Page LIST OP TABLES . . . ....................................... iv Chapter I. THE PROBLEM AND REVIEW OF LITERATURE......... 1 Statement of the Problem ................... 2 Experiment I .............................. 3 Experiment I I ................. 4 Significance of the Study ................ 5 Delimitations of the study............... 7 Definitions ................................ 8 Abbreviations .............................. 10 Review of Literature ........................ 11 Preview of the Remaining Chapters........... 34 II. SUBJECTS, MATERIALS, AND PROCEDURES ......... 35 Subjects . ..................................... 36 Experiment I ................................. 36 Experiment II ........................ 36 Materials.......... 41 Tests ...............................41 Recording equipment ...................... 45 Information sheet ........................ 46 Cloze-test answer sheets.................. 46 Cloze-test number cards... ................ 48 Procedures ................................... 49 Preparation of the data . ................ 49 Processing the d a t a ........................ 63 Summary........................................67 Chapter Page III. PRESENTATION AND INTERPRETATION OF DATA . . . 69 Experiment I ................................... 69 Reliability of Cloze as compared with an outside criterion, CAR ..............73 Comparison of relationship of Cloze and B-C with selected outside criteria . 74 Comparison of improvement in language ability as measured by Cloze and CAR . . 76 Interpretation of findings in Experiment I ..........................................80 Experiment I I .................................82 Reliability of the three measures of language ability ........................ 84 Validity relationships of B-C and Cloze to C A R ..................................... 86 Relationships between Cloze and B-C . . . 88 Comparison of mean change in Cloze, B-C and CAR test scores...................... 91 Summary of experiment findings ........... 94 IV. SUMMARY, CONCLUSIONS, IMPLICATIONS ........... 96 Summary........................................96 Conclusions............. 99 Experiment I ................................. 99 Experiment I I ..............................100 Implications ................................ 101 APPENDIXES............................................... 107 Appendix A: Raw D a t a ....................... 108 Appendix B: Spoken Materials, Cloze Test . 113 Appendix C: Experimental Information Sheet............................130 Appendix D: Cloze Test Booklet ...... 133 Appendix E: Cloze Scoring Materials .... 138 BIBLIOGRAPHY............................... 140 LIST OF TABLES Table Page 1. Characteristics of the Group I Sample............ 37 I 2. Characteristics of the Group II Sample .... 39 3. Characteristics of the Cloze Measurement Recording........................................ 60 4. Correlations Among Seven Variables for Group I ...........................................72 5. Comparison of Coefficients of Reliability of Test Scores Between CAR and Cloze Tests as Used in Experiment I ............. 75 6. Correlations of 1st Cloze and 1st B-C Test Scores Compared with USC G-S, USC V, and 1st CAR Test Scores as Used in Experiment I . 77 7. Correlations of 2nd Cloze and 1st B-C Test Scores Compared with USC G-S, USC V, and 2nd CAR Test Scores as Used in Experiment I . 78 8. Comparison of Mean Score Differences Between Two Administrations of the CAR and Cloze Language Ability Measures as Used in Experiment I .................................... 79 9. Correlations Among Six Variables for Group II . 85 10. Comparisons of Reliability of Test Scores Made in the Three Measures of Language Ability as Used in Experiment I I .............. 87 iv ;Table Page 11. Comparison of Correlations Between Test_ Scores of the 1st Administration of the CAR Language Ability Test and the B-C and Cloze Language Ability Tests Respectively as Used in Experiment I I ............................... 89 12. Comparison of Correlations Between Test Scores of the 2nd Administration of the CAR Language Ability Test and the B-C and Cloze Language Ability Tests Respectively as Used in Experiment II ............................... 90 13. Correlations Among Test Scores Made on the Cloze Procedure and the Brown-Carlsen Listening Comprehension Tests as Used in Experiment I I ....................................92 14. Comparison of Mean Differences Between Test Scores of Subjects on the Three Language Ability Tests as Used in Experiment II . . . 93 v CHAPTER I THE PROBLEM AND REVIEW OF LITERATURE During the past decade and a half the influx of foreign students into the United States created many prob lems in the handling and administration of the extensive programs dealing with their assimilation and problems of communication. Taking into account all foreign citizens studying in the United States, both academic and military, it was estimated that our shores provided study opportuni ties for approximately 90,000 annually. A major difficulty lay in the area of communication, the teaching of speaking, listening, reading, and writing of the English language. Since the core of the problem, whether abroad or in the United States, was proficiency in the use of English, it was natural that language testing evoked much attention and comment. The English communica tion testing of foreign students had been under critical fire for some years, especially listening comprehension, 1 ;which appeared to be the hardest to test. There were ■ several listening tests that demonstrated statistical reliability and some operational validity. The most popular listening test at the time of this study appeared to be the ;Brown—Carlsen Listening Comprehension Test. There was, however, some firm dissatisfaction with it as with other tests, as documented later in the review of literature. Since the various tests and subtests of English as a second language had not been totally satisfactory, especially in the field of listening comprehension, the purpose of this study was to explore still another means of measurement, the relatively new Cloze Procedure. The term, "Cloze, " was derived from the Gestalt concept of "closure" by Taylor who experimented successfully with the measurement of silent reading comprehension (48). Taylor systematically deleted words from written language samples and then requested his subjects to write in the missing words. Dickens and Williams demonstrated the feasibility of applying Cloze Procedure (word deletion) to tape-recorded spoken language samples (13). Statement of the Problem The main inquiry of this study was to determine the extent to which the technique of Cloze Procedure could be applied usefully to the construction of a diagnostic test of listening comprehension among foreign students of English. The investigation with two groups as subjects was concerned with the interrelationships among various diagnos tic tests. It was believed that the experiment would indicate any differences that might have existed among the tests. This study was, therefore, exploratory rather than definitive. The design of the experiment was divided into seven subsidiary problems. Experiment I, also referred to as Group I, was an exploratory study in which three major questions were investigated. Experiment II, or Group II, was designed to test four hypotheses. Experiment I The three areas of investigation in Group I were: 1. To what degree, if any, would a diagnostic test of listening comprehension based on Cloze Procedure demonstrate reliability in terms of test-retest, in comparison with the California Reading Test, Advanced, Form X (CAR). 2. To what degree, if any, would the results of a diagnostic test of listening comprehension based on Cloze Procedure correlate with Brown-Carlsen Listening Comprehension Test (B-C) scores as measured with respect to their relationship to CAR and the University of Southern California English Placement Test: grammar-spelling (USC G-S), and vocabulary (USC V). 3. To what degree, if any, would a diagnostic test of listening comprehension based on Cloze Procedure compare with CAR in the measurement of improvement in language ability. Experiment II Group II was designed to test the following four hypotheses: 1. The scores obtained from the Cloze listening comprehension test for measuring language ability would indicate a significantly greater coefficient of reliability than scores from either the Brown-Carlsen Listening Comprehension Test (B-C) or the California Advanced Reading Test (CAR) when administered twice to Group II. 2. The relationship between Cloze Procedure for measurement of listening comprehension and the scores of the CAR measure of language ability would not be significantly different from the relationship between the scores of the B-C and CAR measures of language ability. 3. The scores of Cloze Procedure for measuring listening comprehension would show a signif icantly high degree of relationship with the B-C scores when applied to the same subjects. 4. There would be no significant difference in measuring language improvement between Cloze Procedure and B-C or CAR. Significance of the Study Earlier in this chapter, it was estimated that almost 90,000 foreign students, including officers of NATO and SEATO countries, study in the United States. In addition, according to Geiger (17:23), approximately thirty colleges and universities offer special summer programs of study and orientation, and about one-fourth of these programs are held in the Western states. Wallace Maner noted that during the ten-year period 1953-1963 the Immigration Service reported a total of 95,906 cases of aliens adjusting to permanent resident status in the United States (25:8). The total did not indicate what percentage were students, but according to Maner1s own University of Iowa figures, well over 50 percent of foreign students stay on in the United States as permanent residents (25). The results of the present study should be of interest to all teachers of English as a second language. It should also be of interest to the many schools seeking newer and more reliable means of testing foreign students abroad, and those here as aliens or immigrants. Much of the literature concerning the diagnostic testing of the comprehension of English as a second language has included criticism leveled at the various tests or subtests. Some major criticisms were the’ following: 1. Most of the tests took too long to administer and score. Dickens (56), Jones (57), and Kaplan (58) reported that a battery of tests 7 was used for all entering foreign students at the University of Southern California in order to measure reading, writing, speaking, and listening abilities. " However, the lengthy tests were a gruelling experience for the foreign students, the majority of whom had not been indoctrinated in "objective testing" prevalent in the United States. 2. Most of the tests were designed for native- born speakers of English. Consequently, much of the testing material was above the language ability level of the foreign students. 3. Standard listening tests, such as the Brown- Carlsen, appeared to measure inference or retention rather than language ability. The present study was thought to be significant * because at least in theory the Cloze technique had potenti alities for resolving or- reducing the difficulties listed above. Delimitations of the study This study was concerned only with the problem of designing a diagnostic listening comprehension test utilizing the Cloze Procedure. The study was not concerned with: (1) native-born speakers of English; (2) the teaching of listening comprehension? (3) the evolving of a final all- inclusive diagnostic test; (4) techniques of listening comprehension? or (5) cultural differences among foreign students. .Definitions The following operational definitions were appli cable within the limits and scope of the study and are not to be necessarily construed as general definitions: Comprehension.— Comprehension was indicated by the number of correct responses obtained to questions about information in certain test instruments during a limited period of time. Listening.— Listening was taken to mean all factors intrinsic and extrinsic, included in the processes of recep tion and interpretation of, and response to, spoken lan guage stimuli (50:5). Listening comprehension.— Listening comprehension was defined operationally in two ways: (1) by the scores made by subjects on the Brown-Carlsen test, and (2) by the scores made by subjects on the Cloze Procedure test. Reading comprehension.— Reading comprehension was defined operationally in two ways: as (1) the scores made by subjects on the University of Southern California English Placement Test with its subtests, grammar-spelling (USC G-S), and vocabulary (USC V), and (2) the scores made by subjects on the California Reading Test, Advanced, Form X (CAR) . Language sample.— Language sample was taken to mean the wording or tape-recordings of written and spoken selec tions used in this study. Rate.— Rate was defined as the average number of words per minute for the two language samples presented orally. Cloze Procedure.— Cloze Procedure was defined as a testing technique that requires subjects to replace missing words in language samples with the correct words. This procedure is explained in detail in Chapter II. Reliability.— Reliability of a test was taken to mean the degree of correlation of a test with itself 10 by means of test-retest. Validity.— Validity of a test was taken to mean the degree of correlation between a test and a selected indepen dent criterion. Foreign student.— Foreign student is defined as any student attending an American institute on an "F" or "J" visa and who is not a native speaker of English. Abbreviations For the purpose of brevity, several common terms are abbreviated as follows: 1. The University of Southern California English Placement Test. a. Grammar-spelling (USC G-S) b. Vocabulary (USC V) 2. Brown-Carlsen Listening Comprehension Test, Form Bm (B-C) 3. California Reading Test, Advanced, Form X (CAR) 4. Cloze Procedure Listening Comprehension Test, Experimental (Cloze) 5. University of Southern California experimen- 11 tal group (Experiment I or Group I) 6. Cambria Adult School experimental group (Experiment II or Group II) 7. Sequential Tests of Educational Progress: Listening 1A, Listening IB (STEP) Review of Literature For a comprehensive coverage of many facets of theory and research on listening, students have available Duker's 1964 annotated bibliography covering more than 800 publications (2). However, in this study the review of literature comprises selected references focused upon (1) the develop ment of Cloze Procedure as a measure of reading comprehen sion; (2) adaptation of Cloze Procedure to the measurement of listening comprehension? and (3) the development of the two most widely used listening comprehension tests, B-C and STEP. The review of literature was organized as follows: 1. Cloze Procedure a. Measurement of reading comprehension b. Measurement of listening comprehension 2. Listening comprehension tests and findings a. Brown-Carlsen and STEP b. TOEFL and others 12 Cloze Procedure Measurement of reading comprehension. — Wilson Taylor introduced Cloze Procedure at the convention of the Association of Education in Journalism in the summer of 1953. His complete review of the technique was presented in the fall of 1953 in Journalism Quarterly (32) . Taylor hypothesized that readers would sub consciously utilize the process of closure in Gestalt psychology when reading prose materials with deleted or missing words. In effect they would "cloze the gaps" by replacing the missing words (48:2-4). He called the technique "Cloze Procedure" and defined it as "a psycho logical tool for gauging the degree of total correspondence between (1) the encoding habits of transmitters, and (2) decoding habits of receivers." (48:3) Taylor's early applications involved simple textual mutilations. For example, every fifth word would be deleted and replaced with uniform blank spaces. The subjects would then endeavor to write in the missing words. A total of correctly replaced words was the Cloze score. 13 He also shifted the fifth-word deletion pattern for five different versions of each selection, gathering Cloze Procedure data for every word in the selections (48:2-3). Conclusions concerning methodology utilized in the study were: 1. No significant differences were evident when the order of the presented essays was changed. 2. No significant differences were evident in the scores among the varied deleted versions of each essay. 3. The more words used, the stronger was the normal curve of distribution of Cloze scores. 4. There was wide fluctuation of scores only within the first 20 to 40 words (4 to 8 deletions) of each passage used (48:82-83). Taylor applied Cloze to the study of reliability in Korean prose materials. He had selected passages to be preclassified into three levels of difficulty. He then deleted every fifth character and mailed the passages to Koreans residing in the United States. Ranking of the pas sages by mean Cloze score resulted in confirming pre-test classifications. Taylor also suggested other applications 14 of Cloze Procedure, including (1) cc oarison of written and spoken communication, (2) foreign language study, (3) clinical psychology, and (4) psychological aspects of music compositions (34:42-48). Taylor, in 1957, found the measure to be related to individual aptitudes. In an experiment with the United States Air Force Taylor found: For each of the three groups, correlation coefficients were computed between all 10 possible pairings of the five distributions of scores (before- and after-study Cloze, pre- and post-tests of compre hension, and AFQT). All 30 such coefficients were found to be positive and significant beyond the .001 level of confidence. Also for each group, the mean differences in scores attributable to learning during study, between paired Cloze tests and between the comprehension tests, were found significant beyond the .001 level. (33:26) At the same time, he reported that, in a series of language samples comparing Cloze with Flesch and Dale- Chall rankings, he found a positive correlation significant beyond the .001 level of confidence in estimating the readability of selected prose materials (33:19-26). Significant correlation was maintained even when (1) four distinct mutilation patterns were employed, (2) the presentation sequence of the passages was varied, and (3) only exact word matches of omissions were permitted to be scored as correct (33:19-26). These results were extensions of his earlier findings when he applied Flesch and Dale-Chall and Cloze rankings to samples of writings. The writings of Boswell, Huxley and James were rated equal by the three measures. Writing by Gertrude Stein was rated easy by Flesch and Dale-Chall. James Joyce was also rated easy by Flesch. Indications were that Cloze was a better rater since the other two ratings appeared to be distorted by messages with short words which were highly abstract, or short sentences in an atypical style (32). Hvistendahl in 1958 found in his study that when the subjects' Cloze scores were compared with their ACE (language) test, the correlation was .51. The second part of his study, which applied to the study of readability in rewriting copy, maintained the same correlation (19:353-4). Wilson Key in 1959 expanded the technique for a thorough comparative analysis of language. While Taylor reported the use of Cloze in exploring a way for simplify ing Korean (34), Key endeavored to evaluate Spanish translations, measuring "exact meanings" (24). He found that he could "pinpoint" words, phrases, sentences, and paragraphs that were questionable in their ability to 16 communicate effectively with the desired readership. Another possible application of Cloze tests involved the preparation of textual material for translation (24:17-8). Since the beginning of the development of Cloze Procedure there have been variations of the deletion patterns. While Taylor began with every fifth word, others have altered the pattern in accordance with their own investigations. Generally, however, two types of Cloze deletions are used: structural, in which every nth word is deleted, as with Taylor, and lexical, in which every nth noun, verb, or whatever may be desired, is deleted. Rankin, in reporting on the validity and utility of Cloze Procedure, stated: When the nouns and main verbs (lexical deletions) are the only words deleted, there appears to be a high correlation between the subjects 1 scores on the Cloze tests and on objective reading tests covering similar materials. Structural deletions correlate significantly more highly with vocabulary and reading comprehension sections of the Diagnostic Reading Test (DRT) than do lexical deletions? lexical deletions correlate significantly more highly with the story comprehension section of the DRT than do structural deletions. (7:134) Measurement of listening comprehension.— There were relatively few studies in the field of speech or listening utilizing the Cloze Procedure at the time of this 17 study. Actually, in 1950 a technique similar to Cloze was used by Stromer who investigated the relationship between factors measured by tests of his design and a "total listening ability" rating. His section on the "contextual clues test" included a method similar to what is now known as Cloze Procedure. Subjects heard sentences with a nonsense word. The subject was to identify the nonsense word and correctly replace it (47). Stromer expanded his version of a listening test to include reading, listening, and intelligence for his doctoral dissertation in 1952. He dropped the nonsense word and used the spoken word "blank" (46:70). His reliability was .77, high enough to warrant further inves tigations (46:33). In 1961, Weaver, following Rankin (7), found that the structural deletions had been easier to complete when presented for reading than when presented for listening, while lexical deletions are of equal difficulty whether read or listened to (49:93). Weaver and Kingston, in 1963, in a study applied a factor analysis to examine the relationship of Cloze tests to standard tests of reading, listening, and language symbolizing ability. They reported the use of structural 18 and lexical forms for both silent and oral readings of Cloze. The relationships of the Cloze tests to the standard tests in the silent readings were little, .16. The rela tionships of the Cloze tests to the standard tests in the oral readings were moderate, .61. The Cloze tests were related moderately to the verbal comprehension factor (36). The Cloze tests of various types appeared to be more close ly related to each other than to the other factors isolated. They concluded: There is a moderate connection between the Cloze tests presented by speaking and the verbal comprehension factor. This connection is apparently in the mode or channel of communication, since it is absent in the reading channel condition. . . . The central implication of this analysis is that Cloze tests of varied kinds are more related to each other than to the other two factors isolated by this analysis, verbal comprehension and "rote memory, flexible retrieval. ..." There is much specific variance connected with Cloze Procedure which is - unexplained by this analysis. Further attempts to relate this specific variance to other variances seem to offer interesting possibilities. (36:260-61) Dickens and Williams (13) in 1963 reported that (1) Cloze Procedure and the multiple-choice test had a similar relationship with language ability and this rela tionship appeared to be independent of the type of message involved; (2) a relatively high correlation, .73, existed between the subjects' scores on the two Cloze tests, while the two multiple-choice tests showed .37,- and (3) the two Cloze tests, according to split-half reliability, were more reliable (.80 in the persuasive and .70 in expository) than were the multiple-choice tests which had .45 in persuasive and .60 for expository (13:105-6). Williams, in his dissertation, found that the Cloze Procedure and multiple- choice tests adapted from STEP appeared to have had about the same relationship with language ability, with .49 and .48 respectively. Cloze Procedure was, however, more reliable and less dependent upon message type (50:100). For the reader who seeks additional information on Cloze Procedure, the following sources are cited. Williams' review of literature contained an excellent introduction to the development of Cloze (50:33-44). De Sola Pool gathered important papers on the theory of Cloze Procedure, and they were available in book form (6). The Rankin study of Cloze's validity and utility was also available and provided a comprehensive review of the development of Cloze since its introduction by Taylor (7). Osgood and Jenkins had contributions in the areas of encoding and verbal behavior, utilizing Cloze to measure stereotypy of verbal production. Osgood (8:293-306) applied his study to the comparison of suicide notes with 20 ordinary letters, while Jenkins' experiments (8:307-329) centered on the "Commonality of Association as an Indicator of More General Patterns of Behavior. 1 1 Listening comprehension tests and findings Many tests of listening comprehension have been devised and used for native speakers of English throughout the past 15 to 20 years. Eighteen tests had been recorded covering a time range from the 1937 Durrell-Sullivan Reading Capacity Test to the 1957 issue of the Sequential Tests of Educational Progress (STEP): Listening (54). Up to 1964 there had been a total of 94 experimental ad ministrations of the eighteen tests, most of them given two to four times. The B-C had been administered 34 times, and STEP 23 times with experimental applications (2:201-11). B-C and STEP were the only published tests with national distribution. Other tests directed specifically to the foreign student will be referred to following a discussion of B-C and STEP. B-C and STEP.— Since B-C and STEP were the most widely used of all the tests, the following reports will concern them. The B-C used in this experiment will receive 21 most of the attention of the critique. Described in detail in Chapter II > the B-C in its final design included five areas within each of the two forms: (1) immediate recall, (2) following directions, (3) recognizing transitions, (4) recognizing word meanings, and (5) lecture comprehension (51:2-3). The split-half reliability coefficients for three communities averaged . 86 for one form and a median estimate of .78 for the other form (51:13). The correlations between B-C and two mental ability tests, one reading test, and high school ranh, were as follows: Outside Test Listening Test Section I II California Test of Mental Maturity .78 .67 American Council on Education Test .37 .29 Nelson-Denny Silent Reading Test .31 .36 High School Record .28 .21 The correlations were small between listening comprehension and reading, high school record, and the ACE Test (51:17-18). The Sequential Tests of Educational Progress: Listening, or the STEP Listening Test, was produced by the Educational Testing Service in 1957. Two forms at each of 22 four levels are available. The levels go from the fourth through the fourteenth grades, the first listening tests to do so (54). The listening portion is part of a battery. By the end of the 1950s, B-C and STEP were to have been administered in many studies and were to be subjected to much criticism. Lorge, in 1959, in a critical review of the devising of B-C questioned its similarity to some sub tests on intelligence and administration. Its validity was difficult to judge, he- stated (1:651). Lindquist ques tioned, with Lorge, the B-C norming procedures, validity, and methods of administration (1:650). A favorable review of the STEP Listening Tests by Jackson (1:65-6) was more than equalized by two opposite views by Lorge (1:654-55) and Lindquist (1:652-54). Lorge questioned the short length of the passages used and the nature of data furnished to users. Lindquist not only questioned the method of administration, validity, and norming procedures, but also the sequential nature of the tests. Duker, in reviewing Haberland's study, stated that listening tests yield "vastly different" results when correlated with reading; that these listening tests do not measure the same thing; that the Brown-Carlsen test correlates most highly with the linguistic portions of other tests; that the Stephens College Test correlates most highly with a grade point average; and that listening ability probably is a combination of abilities working in association with each other. (2:76) Haberland had earlier concluded that listening ability could be objectively measured, and that B-C did not measure the same things as other tests of language comprehension (18). In 1958, Rose (28) and Martin (2:114) found correlations between B-C and California Reading Test (CAR) to be .55 and .60 respectively. Stark reported correla tions between B-C and speech .36, and between B-C and in telligence .68 (29). Still (30) reported three correla tions with B-C: grades .66; intelligence .54; and reading ; .44. It was interesting.to note that back in 1948 Nichols had found similar correlations between his unpublished lis-| i tening test and intelligence .53, and reading .46 (27). These figures were about the average of all the other studies. Lorge reported that STEP was substantially corre- j lated (.75 within grade levels) with intelligence as measured by the Cooperative School and College Ability Test (SCAT) (1:651). Between 1960 and 1966, some pointed articles and projects concerning testing of listening comprehension appeared- Experiments evaluating existing tests multiplied) as no new tests were designed for commercial use. Experi mental studies such as Bonner, I960, and Finkbeiner, 1962, added to already known information about the STEP and B-C tests. Bonner (10) reported a number of linear and multiple correlations using STEP in a battery of tests administered to grade children. Listening and intelligence .38 to . 59 Reading and intelligence .50 to .67 Reading and listening .53 to .65 Reading and listening with intelligence held constant . 30 Other correlations were also reported. Finkbeiner in her study utilizing ninth grade pupils found some interesting correlations. B-C and listening performance correlated .41. When intelligence was held constant, the partial correlation was .28. School grades correlated with B-C .58, with listening performance .46. The correlation of intelligence v with B-C was .63, with listening performance .33 (40). Petrie noted that not enough is known about listening to warrant the kinds of precise generalizations on which most present day programs of listening instruction in colleges and schools are based. Existing research is contradictory or at best inconclusive, and Petrie maintains that satisfactory means of testing listening skills do not exist (44) . Petrie's own research on testing showed that the B-C and the STEP correlated only moderately and may be measuring different skills or abilities (44). He further stated that if B-C and STEP claimed to measure listening comprehension, they should have correlated highly between themselves. "We do not know which, if either, is a valid test of listening ability." (15:249) Among the reports damaging to the validity of B-C was a study by Steeg, 1960. Steeg (45) noted low correlations in using B-C with grade point average (.14) and with grades in lecture courses (.16). Petrie concluded that "since our listening tests are of questionable validity, the results of research in which tests were used are also of questionable validity." (15:250) Petrie's dissertation contains an excellent bibliography and a complete review of the literature concerned with'^the testing of listening (2:130). In 1963, Kelly reported that the B-C and STEP correlated more highly with intelligence tests than with each other, and both tests correlated negatively with employee ratings of supervisory listening behavior (23). Johnson and Frandsen (20) administered to 2,400 freshmen the B-C taped, live, and on film. The tape had the 26 : greatest reliability, and yielded the highest scores. The film was the least reliable. They also reported that the last section of the test, which purported to test listening comprehension, appeared to evaluate a different set of skills from that tested by the first four parts. This would appear to be in keeping with the aim of B-C; that is, to measure various factors within each of the five parts of the test. In 1964, Bateman, Frandsen, and Dedmon probed into Part E, or the "Lecture Comprehension" section of the B-C (9) . According to Brown and Carlsen, this section "mea sures the ability to listen for details, get the central idea, draw inferences, understand the organization, and note degree of relevancy in a brief lecture presentation . . ." (51:3) Bateman, Frandsen, and Dedmon applied a factor analysis of the listening test items and reported some interesting results. They urged caution in interpret ing student scores on Part E, as the evidence indicated the difficulty of attempting to define such a complex variable as lecture comprehension (9:188). Although there appear to be dimensions of lecture comprehension, as yet no study has been able to define those dimensions. The trio of researchers hypothesized that "listening for details" is a 27 multidimensional skill requiring further exploration. Equally relevant is the fact that further experimentation is needed for "drawing inferences" (9:189). The University of Southern California, situated in an area that contains 4,500 to 5,000 foreign students, had its students take a battery of tests to determine the level of language proficiency. According to Kaplan and Jones, the University of Southern California English Place ment Test, and the Brown-Carlsen Listening Comprehen sion Test, do not appear to be highly valid instru ments. These tests indicated a lack of significant validity with the criterion variables. (21:163) Reports in 1965 included a study with college freshmen by Brilhart (11) on relationships between aspects of speaking and listening. Using a geometric figure test, she had listeners respond to a speaker's oral instructions for drawing specific geometric figures. At a following meeting they were administered the B-C. The results were inconclusive, and there was a call for further investiga tions with other than geometric figures used as messages. Some interesting tendencies did appear, however. The B-C showed a correlation with vocabulary of . 68 among women and .28 among men. For the women it was significant beyond the 1 percent level of confidence. The men had signifi cantly higher scores on level two figures: men .48, 28 women .07, as correlated with B-C (11:45). Did B-C measure vocabulary skills in women better than men? Did men have a relationship skill for figures? Both points could have suffered from the lack of a true experimental design that might have given reliability indications but not validity. Brown listed such inconsistencies in his report on the previous data from three studies of the listening of children (12:130). His experimental results utilizing STEP, quite important to the research in testing, indicated that (1) listening and intelligence were highly correlated, (2) listening and reading were highly correlated, (3) listening appeared to be more closely related to reading than intelligence although the correlation coefficients were not significantly different, and (4) listening was more closely related than reading to scholastic achievement when using teachers' grades, but not when the criterion was scores on an achievement test, as one would have expected (12:135-7). Kelly, in a challenging study utilizing college students, investigated the validity of the B-C and STEP listening tests (22). His subjects comprised three groups, of 69, 63, and 54.students respectively. The data upon which the correlations were computed included (1) the two 29 listening tests, and (2) results on file for the Selective Admission Test and the Cooperative Reading Test. The latter two tests were given to entering freshmen at New York University at Plattsburgh. Kelly's results confirmed earlier doubts, as reviewed by this researcher, about the validity and stability of B-C and STEP. Kelly reported that since these tests, with a single exception, failed to correlate significantly higher among themselves than they did with either a reading or an intelligence test, the construct validity is highly questionable. The reliability, at least in experiments comparing pretest and post-test scores, also is very doubtful because of the lower standard deviations and the lower reliabilities obtained for the forms administered secondly. (22:143) The first forms of both tests seemed to be easier although B-C, Form Bm, appeared to have possibility for adoption as an instrument after more research. TOEFL and others.— Finally, a review was made of tests of listening or oral comprehension designed especial ly for foreign students. Administered for the first time on February 17, 1964, a new test, Test of English as a Foreign Language (TOEFL) (55), carried with it the hopes and desires of the National Council on the Testing of English as a Foreign 30 Language, and its over 30 member organizations. TOEFL is a program of English proficiency testing, designed to assist in the placement of foreign stu dents applying for admission to United States institu tions of higher learning. TOEFL is a 3-hour, 270- item objective test with five independently scored subtests devoted to listening comprehension, structure, vocabulary, reading comprehension, and recognition of style, usage, and diction appropriate for written English. (5:109) At the time of this study few reports were available concerning the success of TOEFL. However, David P. Harris, the director of the program, was enthusiastic about the potentialities of TOEFL, although his early statistical analysis suggested four areas to be watched: 1) The difficulty of English lexical items seems to depend more on whether they are Latinate borrowing or native English than on their frequency in the written language, as indicated by the stan dard word counts. (Most of us concluded this long ago. ) 2) Although there are high intercorrelations among the various subtests (as we could expect in a test devoted exclusively to language), the listen ing comprehension subtest has the lowest correlations with other subtests. It correlates rather well with structure (which we should expect); it has a rather low correlation with vocabulary, reading, and— lowest of all— writing. 3) The three highest intercorrelation values are those among structure, vocabulary, and writing ability- 4) Though the vocabulary and reading comprehen sion subtests are highly correlated, they appear to 31 be measuring somewhat different things. (5:110-111) Since this experiment was concerned with listening compre hension, the remarks under point two seemed to indicate the need for more research and study. In relation to point three, Nichols stated that testers of foreign students should give careful attention to length, vocabulary, and naturalness (26). Bens observed that TOEFL would be useful but that it would have other than structural drawbacks, since it is an expensive test that the student abroad pays for in United States dollars. It is intensive and takes three hours to administer (38:13). The Lado Test of Aural Comprehension, 1957, has had limited use off the campus of the University of Michigan, where it was designed specifically to fit within the structure of their English course offered for foreign students (53). This test was not generally considered a published test in the sense B-C and STEP were, although it was available. There was little public information at hand concerning its reliability and validity. Schools with programs comparable in scope to that at the University of Southern California discontinued the use of the Lado Test as unsuitable for their particular needs in their foreign student programs. Bens stated that the Lado Test was not ia very satisfactory test for people who had not had the Michigan course (38). The second test, the Aural Comprehension Test, 1957, was modeled after the Lado Test and used at Maxwell Air Force Base on foreign military officers. The test consisted of 50 multiple-choice items, with answer booklets containing 50 blocks of 3 cartoon drawings, one of which was the cor rect answer. Data showed a Kuder-Richardson, Formula 21, reliability of .95, with the average discriminating index .53 (16:89). This test, a subtest of a battery, was used, according to reports, only at the Air Base. Jesse Villareal, 1948, reported a study directed to the testing of aural comprehension of English for native speakers of Spanish (35). He devised 35 connected passages of spoken English: 25 narratives and 10 two-person conver sations. The answer sheet contained 6 to 7 multiple-choice questions on each paragraph. The answer forms were in Span ish (35:123). He applied Flesch1s formula of level of dif ficulty. He administered the test to 161 Spanish-speaking and 104 English-speaking students who used English answer sheets. The results were inconclusive. The two test forms had reported reliabilities of .80 and .89. Scores were cor related with predictions of ability made by the students and I 33 j Iby qualified observers. The results showed that self- ratings tended to overemphasize failures in aural comprehen sion and that ratings by qualified observers tended to over emphasize successes. The test was too easy for native speakers. The Flesch level-of-difficulty formula and test scores had a small but positive correlation of .34. Flesch did not serve to predict the order of difficulty for native speakers of English (-.18) (35:130). The above three tests and the American Language Institute Test, concerning which little information has been published, comprise the only recorded listening compre hension tests designed specifically for non-native speakers. The information was limited, specialized to specific seg ments, and in two cases highly suspect as to validity. This review of literature may well be concluded by the following quotation from Kelly: The reason for viewing the best known of the existing listening tests with skepticism should be a goad to further and more inventive efforts at measurement and not a source of discouragement. Testing listening, because of the great vulnerabil ity of hearers to distractions, may be harder than most forms of measurement. Unlike the reader who can go back if his mind wanders, the listener appears to be particularly vulnerable to fatigue, lack of motivation, personal worries, antagonism to the 34 speaker, disinterest in the topic, momentary day dreaming, and outside interferences. Perhaps listening tests always will be less reliable than most other types of measuring devices, but surely renewed attempts can bring about improvements. The importance of listening is so great that scholars must not abandon the effort. (22:143) Preview of the Remaining Chapters The present chapter has presented the problem, its background, significance, delimitations, and definitions of its terms. The last section contained the review of literature involving two areas, namely, the Cloze Procedure and the measurement of listening comprehension. The remainder of this experimental study is divided into three chapters, an Appendix and Bibliography. In Chapter II is a report of the experimental subjects, materials and procedures. Chapter III presents and inter prets data. The summary, conclusions, and implications of the study are included in Chapter IV. In the Appendix will be found the various materials utilized in this study. The reference sources are listed in the Bibliography. CHAPTER II SUBJECTS, MATERIALS, AND PROCEDURES In general this study was designed to investigate the possibilities of developing an English listening com prehension test especially adapted for use with foreign students. The study was divided into two main parts. Exper iment I was exploratory. Various techniques and procedures were tested in terms of feasibility; preliminary compari sons were made between the new (Cloze) test and other measuring instruments of listening comprehension and gener al language ability. The results were encouraging? there fore, Experiment II was designed and completed. This second experiment was focused upon several direct compari sons with a widely used commercially available test of listening and an accepted test of general language ability. Specific details of the above general design are described in the remainder of this chapter. 35 Subjects The two groups of subjects used in this experiment comprised a total of 90 students from the University of Southern California and Cambria Adult School of the Los Angeles City School System. All subjects were foreign students enrolled to learn speaking, listening, reading, and writing of the English language. Experiment I Group I utilized 43 foreign students enrolled in courses in the English Communication Program for Foreign Students at the University of Southern California. A total of 45 subjects took part in the initial testing. Two stu dents were rejected for failure to complete all tests. Experiment II Forty-seven foreign students, classified as having "non-immigrant ’ f student" status, were made available for the experiment by Cambria Adult School. They were designated as Group II. These two samples formed the basic test groups: Group I (N=43), and Group II (N=47) . Characteristics of the groups are shown in Tables 1 and 2. Appendix A pre sents the individual characteristics of the subjects. TABLE 1 CHARACTERISTICS OF THE GROUP I SAMPLE Origin Male Female Grand 18-23 24-29 30-35 Total 18-23 24-29 30-35 Total Total Japan 1 5 1 7 0 7 Korea 3 2 1 6 0 6 China 3 1 4 1 1 5 Kuwait 5 5 0 5 Saudi Arabia 5 I 5 0 5 French Guinea 2 2 0 2 Hungary 1 1 1 1 2 India 1 1 2 0 2 Iran 1 1 1 1 2 I L u> -o TABLE 1— Continued Male Female Grand u n g m 18-23 1 24-29 30-35 Total 18-23 24-29 30-35 Total Total Thailand 0 1 1 2 2 Germany 1 1 0 1 Indonesia 0 1 1 1 i Mexico 1 1 0 1 i Pakistan 1 1 1 0 1 Venezuela 1 1 0 1 TOTAL 19 13 5 37 3 3 0 6 43 N 43 Age range 18-35 Male 37 or 86% Mean age 24.2 Age range, male 18-35 Female 6 or 14% Median age 23. 7 Age range, female 19-29 CO 00 TABLE 2 CHARACTERISTICS OF THE GROUP II SAMPLE Origin Male Female Grand 18-23 24-29 30-35 36-55 Total 18-23 24-29 30-35 36-55 Total Total Japan 6 6 12 2 1(42) 3 15 Iran 7 1 1 9 0 9 Mexico 2 1 3 1 1 4 Korea 1 1 2 1 1 3 Italy 1 1(54) 2 0 2 Egypt 0 1 1 1 Germany 0 1(44) 1 1 Honduras 1 1 0 1 Hong Kong 1 1 0 1 Israel 1 1 0 1 Jordan 1 1 0 1 Lebanon 1 1 0 1 Okinawa 1 1 0 1 u> 1 TABLE 2— Continued Origin Male Female Grand 18-23 24-29 30-35 36-55 Total 18-23 24-29 30-35 36-55 Total Total Panama 1 1 0 1 Peru 1(Japanese) 1 0 1 Spain 0 1 1 1 Switzerland 0 1 1 1 Turkey 1 1 0 1 Yugoslavia 0 1 1 1 TOTAL 22 10 4 1 37 1 2 5 2 10 47 N 47 Age range 19-54 Male 37 or 79% Mean age 26.0 Age range, male 19-54 Female 10 or 21% Median age 23. 7 Age range, female 19-44 41 Materials Tests In Experiment I the following five tests were administered: 1. University of Southern California English Placement Test: grammar-spelling (USC G-S): 60 responses. 2. University of Southern California English Placement Test: reading vocabulary (USC V): 60 responses. 3. California Reading Test, Advanced, Form X (CAR): 45 responses (questions 103-147). 4. Brown-Carlsen Listening Comprehension Test, Bm, Abbreviated Form, Part A and E (B-C): 38 responses, 17 in immediate recall, and 21 in lecture comprehension. 5. Cloze Procedure Listening Comprehension Test (Cloze): 126 responses in immediate recall. The first three of the above tests, limited to reading and writing, were used to explore their respective merits as outside criteria by which to estimate validity of B-C and Cloze. B-C was used in an abbreviated form since 42 ! that was in accordance with the current requirements of the USC Testing Bureau. The instruments, USC G-S, USC V, and B-C, as parts of a battery of tests, were administered in September as a normal entrance procedure by the USC Testing Bureau and the faculty members who were teaching English for foreign stu dents. This experimenter received permission from the faculty members to administer additional tests, CAR and Cloze, during the remainder of the semester. Since these additional tests were administered during class hours, the experimenter had to accomodate his testing schedule to the convenience of the faculty members. The two administrations of each CAR and Cloze were approximately one month apart. In Experiment II the following three tests were administered twice, one week apart: (1) CAR, (2) B-C, and (3) Cloze. Analysis of the correlations obtained in Experiment I (Table 4) did not reveal a clear-cut advantage for any one of the three reading tests (as a possible independent validity criterion) in comparison with the other two. For Experiment II, therefore, CAR was chosen as the outside criterion because it was a national rather than a local test, and because it seemed somewhat broader in 43 scope in the measurement of language ability. Kaplan and Jones also found that CAR, the Speech Interview, and the Larry-Ward Test of Articles and Particles appeared to be the most valid of the instruments administered by the University of Southern California (21:163). In Experiment II the entire B-C was administered (rather than the abbreviated version) because faculty mem bers at Cambria were willing to grant the requested amount of time. A general description of the B-C follows for purposes of comparison with the Cloze Procedure described later in this chapter. The B-C is an objective test con cerned with measuring, according to its authors, five impor tant listening skills: (A) Immediate Recall, which measures the ability to keep a sequence of details in mind until a ques tion is asked which requires thinking back over the sequence; (B) Following Directions, which measures the ability to follow oral directions; (C) Recogniz ing Transitions, which measures awareness of the function of transitional words and phrases within sentence contexts; (D) Recognizing Word Meanings, which measures the ability to recognize meanings of words from context; and (E) Lecture Comprehension, which measures the ability to listen for details, get the central idea, draw inferences, understand the organization, and note degree of relevancy in a brief lecture presentation read by the examiner. (51:3) As indicated earlier, the B-C was designed specifically for use in testing native-born speakers of 44 English. Consequently, some of the specific parts are above the general ability of foreign students who are non native speakers of English. The following examples are representative of questions found in the areas of Immediate Recall and Following Directions. The questions have been altered regarding details concerning numbers, names, etc., as reproduction of actual test materials is prohibited. Examples of Immediate Recall: "Question A. Listen to these directions: Nine of you will walk six blocks down Pine Street, turn left and walk four more blocks to Maple Street, then cross over on Maple for one more block to the corner house at 301 Maple. The number of persons directed to walk the distance was _____ ?" "Question B is: The total number of blocks to the house at 301 Maple is _____ ?" Following Directions includes references to a group of numbers over letters printed on the machine scored answer sheet. A sample question would be: "Question S is: Add the number above the third consonant to the number above the fifth vowel, then subtract the number above the first vowel from that sum. Three more than the resulting sum is _____ ?" The complete B-C included 76 responses: 17 in immediate recall, 20 in following directions, 8 in :recognizing transitions, 10 in recognizing word meanings, land 21 in lecture comprehension. The CAR and B-C are not reproduced in the appen dixes of this study due to copyright restrictions. How ever, copies are available to teachers and scholars (52,51). Test scores of both groups are reported in Appendix A. Recording equipment ~- Materials and equipment for recording included: 1. Recorders: AMPEX 351 (2) 2. Microphone: Altec 6398 3. Console: General Electric BC-l-A, Model 4BCla2 4. Sine-Square Generator: Heathkit, Model AG IO 5. Splicer: Editall KS-2 6. Tape: Scotch 111 7. Leader Tape: Scotch Brand, Plain White Paper, # 20-10 8. Timing Tape: Scotch Brand, Printed Plastic- White, # 24-10W 9. Splicing Tape: Scotch Brand, # 41, 7/326 10. Location: Studio A, Broadcasting Department, 46 : Los Angeles City College. Materials transcribed by tape-recording for the experiment included the Brown-Carlsen and Cloze Procedure Listening Comprehension Tests. All spoken instructions, both live and on tape, are reproduced in full in Appendix B. Information sheet Although the standard background materials were available in school files, the uniqueness of the group presented an opportunity to obtain some personal information concerning the subjects' backgrounds and individual prob lems in language. A comprehensive two-page personal infor mation sheet was designed to be completed by the foreign student. The sheets were presented to the instructors of the classes several days prior to the experiment. They called attention to the fact that the information was to be for research, as was the experiment, and would not be graded. The instructors had the subjects complete the information sheets before the first day of the experiment. The information sheet is reproduced in Appendix C. Cloze-Test answer sheets The main criteria in constructing the answer sheet 47 were clarity, simplicity, and available space. The heading was quite simple, requiring the name of the subject, in structor, and the date. This was followed by (1) a section on sample questions, and (2) a section for responses during the lecture. Each sentence of a Cloze-mutilated paragraph was given a number. The paragraph was lettered for later coding in item analysis, if needed. . The blanks for the missing words were lettered according to the sentence. For example: Section I A. 1. a. _______ 2. a. _______ In cases in which there were sentences with two or more deletions, appropriate numbers of blanks were placed side by side, as in regular sentence order. For example: B. 4. a. _______ 5. a. _________ b. ___ 6. a. _________ b. c. _______ The lecture contained 56 sentences with a total of 126 word deletions. Section I was in double space. The Cloze answer sheet is included in Appendix D. Cloze-Test number cards Williams (50:73) in his preliminary studies found it was desirable to show sentence numbers as the subjects followed the blanks on the Cloze answer sheets. This experimenter followed Williams' procedure in the use of large number cards, although he found that writing the number on the blackboard was also effective. The number cards, however, tended to stand out more and catch the eye of the student when necessary. These cards were used as a "silent" aid and were much more effective than interjecting more oral material, such as the announcing of the sentence numbers. Numbers "0" through "75" were printed on 8% by 11 inch light Manila folders. The folders were selected not only for strength, but also because they produced less noise than the rustling paper used in earlier experiments. Drawn in India ink with a guarter-inch (point) drafting pen, the numbers stood approximately six inches in height; two holes were punched at the top in such a manner as to allow the numbers to be "flipped" as the corresponding sentences were being heard by the subjects. 49 Procedures Preparation of the data The data involved in this experiment included: (1) construction of the Cloze listening comprehension test, (2) recording, (3) administration of test materials, (4) recording previous test data, and (5) reproduction. Construction of Cloze Test.— Five criteria were utilized in the preparation of the Cloze materials. The first criterion was that the test’s administra tion be limited to the duration of the normal class hour, preferably 4C to 45 minutes. It was thought that this restriction would minimize the dangers of student fatigue? it would eliminate the problem of dispersion of attention as a result of between-class student activities, one which ; might occur if the testing were extended to subsequent meetings. The second requirement was that the Cloze Procedure be applied to a brief lecture on a single topic. The rationale of this requirement was based upon the fact that a large proportion of the foreign students1 work at an American school depends -on their ability to comprehend materials presented in lecture form. Therefore, a 50 "typical" 20 minute lecture was written by the experimenter. The lecture was thought to be long enough for purposes of realism, yet short enough to permit the administration of all testing procedures during one class hour. The third criterion was that the subject matter of the lecture should not be identifiable with any specific geographic area, specialized topic, particular date, or either sex. The hope was to find a topic that would be of general interest and yet avoid introducing bias to groups of subjects of varied nationalities, college majors, etc. The fourth criterion was that the content of the lecture be as factual and nonemotional as possible. Harwood (43:23) listed this criterion as primary in a list of requirements for materials used in his studies of listening, explaining that its use was intended to avoid effects of emotionally loaded language. The a priori expectation that people calculate numbers in their native tongue, no matter how long removed from the original source, influenced the researcher to limit the use of numerals to a minimum. The fifth criterion was that materials should be spoken at an optimum rate for listening comprehension. In some previous studies, namely those of Goldstein (3), Franke (41), and Darley (39), there was evidence to indi cate that speaking rates and effectiveness could be mea sured with some validity. Goodman-Malamuth (42:84) took Franke's (41) range of 140-185 words per minute in critical listening and concluded that 150 words per minute was very significantly superior to 175 and 200 words per minute. He further concluded that listenability decreased as the rate of presentation decreased from 150 to 125 words per minute (42:84). However, evidence by Diehl, White, and Burk (14) indicated in a study that listenability could be effective at a speaking rate of 125 words per minute. Since the above findings were for native-born speakers or listen ers, and there were no studies available applicable to foreign subjects, the experimenter applied the rate of delivery of approximately 125 words per minute for this study. In the establishing of these criteria, it was the desire of the experimenter to originate a lecture as unified and plausible-sounding as one would expect to be given in university level classes. The many linguistic variables, especially with foreign students, made it impractical to design the lecture in a mathematical fashion. It was hoped that any 52 imbalances would be compensatory rather than cumulative and that the selection would be representative of ordinary English. Three points were made in setting up the goal. The first point was that the overall effect be acceptable to the subjects as a "typical" lecture. Next, it was the desire of the experimenter to avoid obvious blunders resulting from the many potential linguistic variables. Point three was to try to keep numerous details of style, form, clarity, interest, and appropriateness, etc., in representative balance. Some degree of success was evident in the distribution of test scores, which appeared to be reasonable and sensible. The topic eventually selected was "The Nature of Study," which included various practical suggestions on such matters as reading, listening, note-taking, outlining, use of library, etc. It was thought that all the subjects shared a common characteristic— they were all foreign stu dents trying to succeed in an American school. This choice of topic was apparently a successful one, since many of the subjects returned after the test and requested copies of "The Nature of Study" to assist them in preparation for their regular studies. The experimental instrument, the lecture on "The 53 Nature of Study," was administered with a modified form of the Cloze Procedure as reported in earlier studies. It was decided to select various paragraphs from the lecture and employ the technique of deleting words in such fashion that the test be neither so difficult or so easy as to result in Cloze scores which would be so restricted in range that they would fail to differentiate between high, medium, or low abilities. Two methods of deleting were considered. Taylor (34:48) described the deletion of every fifth word with substitution of ten blank typed spaces in the use of Cloze Procedure in written form. Williams (50:149-50)in a pilot study duplicating Taylor's study in oral form, found that his subjects' scores were so narrow in range as to prevent meaningful statistical analysis. In his final experiment Williams (50) employed a deletion system of omitting words at every fifth second in order to obtain higher scores and wider range. In the pilot study the Cloze scores (number of correctly replaced words) were only about 20 percent of the total number of deleted words as compared with Taylor's 70 percent in the written form (50:55). Williams' Cloze score percentage rose to 50 percent with the fifth-second application. Both methods had their drawbacks as utilized 54 by Williams. The pilot study by Williams contained a total of only 35 possible responses (50:149). This may have partially accounted for the narrow range, especially if any of the word replacements were particularly difficult. The second attempt, using a design of every fifth second, led to the danger of the subject "predicting time" in waiting for the next responses. The experimenter, to avoid the drawbacks mentioned above, initiated two actions. First of all, since it was recognized that the greater the number of responses con tained within a test, the greater the reliability (31), an effort was made to include more responses. The various paragraphs contained a total of 125 deleted words or responses. This was felt to have been quite sufficient to insure reliability. The second step was the deletion of every fifth word as in Taylor's studies. It was noted through a time- check that these words occurred with the irregular frequency of one deletion every three to five seconds. This system lowered the ability of the student to anticipate omissions, which one might do in listening to deletions at regular time intervals. The researcher of this study assumed a priori that attention, if it wanders, tends to do so well after the 55 opening of an introduction. He therefore applied a modifi cation of Taylor's original formula of keeping intact the first and last ten words in each passage (48:33). The opening and closing paragraphs were left intact, along with others, as the average listener tends to be generally more attentive at the beginning and close of an oral dis course. Nine paragraphs contained Cloze deletions or mutilations, and twenty paragraphs were left intact. Recording.— The person used to record the B-C Listening Test and the Cloze experimental lecture was a professional broadcast announcer and chairman of the broad casting department of a local college. He held an M. A. degree and had been teaching in college for twelve years. He was forty-nine years old; thirty of these years were spent in broadcasting. The individual used to record the test instructions was the experimenter. A Ph.D. Candidate in Speech, he had been teaching since 1950, including four years as a college instructor. He spent two and one-half years as an Academic Consultant to foreign governments establishing English lan guage laboratories. In addition, he had nineteen years experience in the creative and technical areas of film 56 radio, television, and theatre. All recordings were under the direction of the department chairman, who possessed a First-Class Radio Engineer's License. The recordings were operated by an erase-edit system through remote control. Random noise or a "tone level" was recorded at 15 inches per second, with the sine-sguare generator set at 60, with the frequen cy multiplier set at 10, range IV. The result was a tone with a frequency of 600 cps. This random noise or tone was the replacement for deleted words from the lecture. Williams (50:62) experimented with many sounds in order to find one that would not "jolt" his subjects as they lis tened to the deletion tone. The sound in this experiment was conceived with the same intention. Dubbing and editing operations for the B-C and Cloze are described in detail in the following sections. The B-C test tape was recorded at 7.5 inches per second (ips) according to the Manual of Directions. The recording included instructions, test materials, and ques tions. The running time was 41 minutes. The Cloze tape was divided into three sections: (1) the introduction or instructions, (2) the preliminary practice set, and (3) the lecture. All sections, excluding 57 ; the lecture, were recorded at 7-5 ips. The Cloze lecture itself was recorded at 15 ips to facilitate the editing. The various paragraphs chosen for Cloze mutilation were designated, and within them every fifth word was deleted. A word was defined as any letter, figure, or cluster of figures, set off by spaces in written context, following Taylor's methodology (48:34); e. g. , "da" in Leonardo da Vinci, or the article "a," would be words. The tape was manually transported across the recorder 1s playback head to chart the duration of each word to be deleted and marked with an editing grease pen cil. The portions of the tape to be removed were then cut out singly, and in each case spliced with the random noise or "tone" tape, equal in length to the discard. The splices were done with a standard splicing block and Scotch splicing tape. The lecture tape spliced with the "tone" in place of words at 15 ips was duplicated at 7.5 ips, in accordance with the other three sections of the experimen tal Cloze final tape. The lecture or Cloze-mutilated tape at 7.5 ips was readied to be modified to include the response periods, which were between the sentences. Using the method 58 described earlier, advancing the tape across the recorder's playback head, the midpoint in each interval between the sentences was marked with a grease pencil, as Williams had done (50:66). For each deleted word in a sentence, one five-second response period would follow. For two or more deleted words, a ten-second response period would follow that sentence. Blank leader tapes of 37.5 inches were pre pared for a five-second response, and of 75 inches for a ten-second response. The appropriate response periods were then spliced into the marked intervals between sentences. Recorded time of the Cloze test was 30 minutes and 30 seconds; characteristics of this recording are found in Table 3. Recorded sequence.— All the recorded materials were then spliced into their proper sequence for the Cloze- mutilation test. The B-C was recorded in sequence as re ported earlier. The sequences of all recorded materials were as follows: Brown-Carlsen Listening Comprehension Test, Form Bm: 1. Introduction 2. Instructions 59 ; 3. Part A, Immediate Recall and Responses 4. Instructions 5. Part B, Following Directions and Responses 6. Instructions 7. Part C, Recognizing Transitions and Responses 8- Instructions 9. Part D, Recognizing Word Meaning and Responses 10. Instructions 11. Part E, Lecture Comprehension 12. Instructions 13. Questions on Part E 14. Concluding Instructions Cloze Test: 1. Introduction 2. Instructions, with preliminary practice set 3. Lecture, immediate recall, with responses 4. Concluding instructions The complete time sequence of the various sections and intervals of the tape-recorded Cloze test items are noted in Table 3. 60 TABLE 3 CHARACTERISTICS OF THE CLOZE MEASUREMENT RECORDING Introduction Opening 4:22 Lecture Title "The Nature of Study" Number of words 2419 Delivery time: lecture 19:40 Speaking rate 122.98 wpm Number of paragraphs 28 Paragraphs intact 19 Paragraphs with Cloze deletions 9 Number of sentences 209 Sentences without deletions 153 Sentences with deletions 56 Word deletions 126 6 sentences with 4 deletions 15 sentences with 3 deletions 22 sentences with 2 deletions 13 sentences with 1 deletion Lecture Time with Deletions 19:40 Response Periods 56 Time of Response Periods 5:28 Closing Remarks 1:00 TOTAL TIME 30:30 61 Administration of test materials. — Test materials were administered at two locations: Building A, Room 108, of the Speech Department complex at the University of Southern California, and Room 205 of the Administration Building of the Cambria Adult School. The classroom condi tions-" concerning temperature, light, comfort, visibility, etc. were evaluated by the experimenter and adjudged con sonant with the requirements of a controlled experimental environment. A two-page information sheet concerning individual background, explained earlier, had been given to the sub jects one to two days earlier by their instructors. The sheet was to be returned completed by the day of the exper iment. All materials and equipment were prepared for use prior to each class time. An Ampro Model 50A tape recorder was used for the playback of all recorded materials. An auxiliary speaker was added to insure total listenability for the class. Test levels were set prior to class time. In all administrations of the test the regular instructor was at hand to introduce the experimenter to the subjects, who had been alerted to the purpose of the visit earlier in the week. Experimental Group I (N=43) received a total of ; 62 ■ !seven tests, while the Experimental Group II (N=47) under took six tests common to the first group. The administration of the materials varied slightly between the written or reading tests and the listening tests compared in the experiment. The subjects had been informed earlier of the "research" tests and appeared to have accept ed them as such. All seemed to be eager to be part of the "research team" during the experiment. The test booklet, in the case of the written exam, and the answer sheets for the listening tests were distributed while the researcher was introduced and while he gave the opening oral instruc tions. Each test purpose was identified and a general intro duction given. The text of these opening instructions is included in Appendix B. Instructions for the other tests are not included in the appendixes since they are prominent tests in general circulation throughout the nation. The Cloze test demanded more description because of its unique ness. It was quite probable that none of the subjects had ever been exposed to that type of testing. The tape of the Cloze test included an introduction with an orientation test to acquaint the subjects with the new technique of partici pation. The experimental test began with the subjects listening to the material and making their responses on the 63 answer sheet. The experimenter was present with the numbered "flip" cards which indicated the current question or response to be marked. Upon completion, and prior to their dismissal, the subjects were thanked for their partic ipation and informed that the results, when available, would be shared with them. Recording previous test data.— The Testing Bureau of the University of Southern California provided for Group I scores on the USC G-S, USC V, and B-C tests, as de scribed earlier in Chapter II. These were part of the out side criteria of verbal ability. These data are reported in Appendix A. Reproduction.— All written test materials were reproduced by mimeograph process. The master stencils were cut on an IBM electric typewriter in prestige-pica type. All reproduction was done on 20 pound bond mimeograph paper. The reproduction was under the guidance of the Los Angeles City College Secretarial Pool and performed by them. Processing the data The four specific steps in processing the data 64 were: (1) scoring of test materials, (2) tabulation of raw data, (3) application of raw data to machine computa tion, and (4) statistical procedures. Scoring of test materials.— All scoring on standard commercial tests used was tallied in accordance with the printed directions. The Cloze answer sheets described earlier were scored basically in accordance with the methodology devel oped by Taylor (48:2-3) and used by Williams (50:84-5) and others. The responses of the subjects were written in the blanks alongside the sentence numbers provided on their answer sheets. The number of responses varied from one to four blanks in a sentence depending on the number of dele tions per sentence. An answer sheet key with the precise words deleted from the original text was then employed to correct the answers. The total number of correctly re placed words constituted a Cloze score. In addition to the exact words, a few synonyms, within strict limitations, were accepted as correct. The answer sheet with correct answers is recorded in Appendix D. Tabulation of raw data.— The raw scores were re- ;corded on 3 by 5-inch cards that had been numbered serially 65 from 01 to 45 in Group I, and from 01 to 47 in the final study, Group II (see Appendix E) . Later, the data from the cards were transferred to 17x22 worksheets and laid out in a similar manner to the raw data reported in Appendix A. Previous recorded data were included on the cards and finally on the work sheets. Application of raw data to machine computation. — Two sets of IBM punch cards were prepared from the tabulated data and are reported in Appendix A. Set I contained data for Group I: 1. Subject code number 2. USC G-S test score 3. USC V test score 4. B-C test score 5. CAR test, first administration, score 6. CAR test, second administration, score 7. Cloze test, first administration, score 8. Cloze test, second administration, score Set II contained the following data for Group II, the final experimental group: 1. Subject code number 2. CAR test, first administration, score 3. CAR test, second administration, score 4. B-C test, first administration, score 5. B-C test, second administration, score 6. Cloze test, first administration, score 7. Cloze test, second administration, score The entries on the punch-card were verified by- comparing the data, worksheets with a "print-out. 1 1 Statistical procedures.— The major procedures used individually for the groups were: 1. Determining correlations among: (a) USC G-S test score, (b) USC V test score, (c) B-C test score, first application, (d) B-C test score, second application, (e) CAR test score first application, (f) CAR test score, second application, (g) Cloze test score, first ap plication, (h) Cloze test score, second appli cation. Correlation coefficients were com puted by machine data processing, using the Pearson product-moment formula for original measurements. 2. Determining the means for (a) through (h) above. Means were computed by machine data 67 processing programmed for ungrouped data. 3. Determining the standard deviations for points (a) through (h) above. Standard deviations were computed by machine data processing utilizing the formula for original measure ments. Summary Subjects used in this experiment were 90 foreign students enrolled in English for Foreign Students classes. Forty-three of this number, Group I, were enrolled at the University of Southern California? forty-seven, Group II, were from the Cambria Adult School of the Los Angeles City School System. Materials of the study basically involved written and spoken tests. Cloze, a new instrument for measuring listening comprehension, was designed for this experiment. It was compared with the Brown-Carlsen Listening- Comprehen sion Test (B-C), available on the commercial market and used throughout the United States. The California Reading Test, Advanced, (CAR) was used as an outside criterion because it was widely used in 68 colleges and universities as a measure of verbal ability. An English grammar-spelling (USC G-S) and English vocabu lary (USC V) test were also considered. The reliability of the three main tests, CAR, B-C, and Cloze was measured by a test-retest design. Validity of the B-C and Cloze in this experiment depended on their correlation with CAR. Basic statistical procedures included: (1) comput ing correlation coefficients among the measurements within each group, and (2) computing means and standard deviations for the measurements. Computations were by machine data processing. CHAPTER III PRESENTATION AND INTERPRETATION OF DATA The first section of this chapter presents the findings of Experiment I (43 subjects at USC); the second section contains the results of Experiment II (47 subjects at Cambria Adult School). Experiment I Experiment I was an exploratory study designed to test the feasibility of utilizing Cloze Procedure as an instrument for the diagnostic testing of language ability through a measurement of listening comprehension. The experiment was to be evaluated in two major areas: its reliability and its comparison with the established relia bility of an instrument measuring language ability. Two established instruments purported to measure language ability were CAR and B-C. The former is a silent reading test of comprehension, the latter a listening test of comprehension. CAR's coefficient of reliability for reading 69 70 comprehension was reported at .91 (52:8), utilizing the Kuder-Richardson Formula 21. The reliability coefficient for B-C obtained by correlating scores on alternate forms of the test was .78 (51:13). CAR was selected as the out side criterion for three reasons: (1) It was a silent reading comprehension test (most foreign students can read better than they can speak); (2) its reliability was very high, although that estimate is usually higher than the alternate form estimate as used in B-C; and (3) studies in the review of literature questioned the validity of some areas in B-C. Since the study was exploratory, and Cloze and B-C were tests of listening comprehension, it was decided to compare their relationship with other tests administered to foreign students. Another area to be observed was whether Cloze could measure improvement or change as effectively as CAR. Finally, the materials, techniques and procedures used in the administration of the Cloze test were to be explored in terms of their feasibility, eliminating "bugs," and possibly planning improvements for future research. The major items in question were: the suitability of the lecture "The Nature of Study," the success of the editing 71 technique used in deleting words from the manuscript and splicing in random noise, and the effectiveness of the pretest practice sentences. These items were described in detail in Chapter II. The above areas to be observed served as indica tions as to whether or not continued research would be appropriate and worthwhile. The raw data in Experiment I comprised the measure ments of five language comprehension tests administered to the subjects. Two tests were presented twice for reliabil ity measurement. As reported in Chapter II, the measure ments were (1) USC G-S; (2) USC V; (3) B-C; (4) CAR; (5) CAR (second application); (6) Cloze; and (7) Cloze (second application). These data are reported in Appendix A. See Table 4 for correlations. In interpreting, the data values were tahen from Guilford's Table D (4:538-39). The levels of significance are reported in their respective tables with all relation ships based on the Pearson interpreted on the basis of minimum significance of r_ at the .05 level. Relationships at the .01 level of confidence were considered to be highly (or very) significant. The interpretation of the Group I data included TABLE 4 CORRELATIONS AMONG SEVEN VARIABLES FOR GROUP I (N=43) Variables 1 2 3 4 5 6 7 1. USC G-S . 771 . 514 . 798 .614 . 511 . 547 2. USC V . 339 , 604 . 462 .489 . 560 3. B-C . 537 . 584 .484 .458 4. 1st CAR . 768 . 504 . 537 5. 2nd CAR . 586 . 576 6. 1st Cloze . 926 7. 2nd Cloze <1 t o 73 these statistical comparisons: 1. The test-retest reliability of Cloze was compared with the test-retest reliability of the selected outside criterion CAR. 2. Improvement in language ability as measured by Cloze was compared with the same variable when measured by CAR. - 3. The relationship between Cloze and B-C was compared with selected outside criteria. Reliability of Cloze as compared with an outside criterion, CAR Reliability was measured by means of test-retest procedures. The reliability of Cloze test data was com pared with the reliability of CAR test data. The Cloze instrument was administered twice, with an interval of one month, to a group of 43 students. The scores of the two administrations were correlated by the product-moment technique. The coefficient of reliability thus obtained was found to be very high (.926). In a similar manner, the CAR, which is purported to measure reading comprehension and vocabulary, was adminis tered twice to the same group of 43 subjects. The result ing scores were also correlated by the product-moment | 74 j :technique. In this instance, the coefficient of reliabil ity read .768. Table 5 presents these correlation coefficients, their z transformations, and the significance of the dif ference of these z transformations. As can be seen in the Table, the Cloze test had a significantly higher coefficient of reliability than did the CAR with the same group of sub jects. The difference was highly significant at the .01 level. Comparison of relationship of Cloze and B-C with selected outside criteria Accepting that validity is measuring what is pre sumed to be measured, it was decided to determine whether the experimental instrument Cloze related to a selected outside criterion as favorably as did the B-C with the same criterion. If it did not compare as favorably, there would be no need to extend the study further. To make this comparison, both Cloze and B-C data were correlated with the USC G-S and USC V test data. Correlations were also computed between Cloze and the first and second administrations of CAR as well as between B-C and the same first and second administrations. See TABLE 5 COMPARISON OF COEFFICIENTS OF RELIABILITY OF TEST SCORES BETWEEN CAR AND CLOZE TESTS AS USED IN EXPERIMENT I CAR ' diff. of t r z z' s ^ ratio P CAR .768 1.015 Cloze .926 1.631 .616 2. 75 <.01 U 1 Table 4. These correlations were then compared by z trans formations. As is.illustrated in Tables 6 and 7 the results of these comparisons showed no significant differences between any of the coefficients of correlations of Cloze with the above outside criteria and correlations of B-C with the same criteria. Comparison of improvement in language ability as measured by Cloze and CAR On the assumption that a good measuring instrument will register change when change has taken place, it was decided to compare the mean scores between the first and second administrations of Cloze and CAR. The means of both administrations, the differences of these means, and the significance of these differences, are presented in Table 8. The differences in means for both Cloze and CAR were highly significant (P<.001). Both instruments showed a significant mean growth on the part of 43 subjects. Since these two administrations of each test were several weeks apart, and since during these weeks the subjects were re ceiving intensive training in English communication, the reasonable expectation was that significant improvement TABLE 6 CORRELATIONS OP 1ST CLOZE AND 1ST B-C TEST SCORES COMPARED WITH USC G-S, USC V, AND 1ST CAR TEST SCORES AS USED IN EXPERIMENT I 1st r Cloze z 1st r B-C z diff. of z 1 s t ratio P 1st CAR . 504 . 554 . 537 . 600 . 046 0.21 .84 USC G-S . 511 . 564 . 514 . 569 . 005 0. 02 .93 USC V .489 . 535 . 339 . 353 . 182 0.81 .42 TABLE 7 CORRELATIONS OP 2ND CLOZE AND 1ST B-C TEST SCORES COMPARED WITH USC G-S, USC V, AND 2ND CAR TEST SCORES AS USED IN EXPERIMENT I 2nd r Cloze z 1st r B-C z diff. of z ' s t ratio P 2nd CAR . 576 .656 . 584 . 668 .012 0.05 .93 USC G-S . 547 . 614 . 514 . 569 . 045 0. 20 . 85 USC V . 560 .633 . 335 . 353 . 280 1.25 .22 <1 00 TABLE 8 COMPARISON OF MEAN SCORE DIFFERENCES BETWEEN TWO ADMINISTRATIONS OF THE CAR AND CLOZE LANGUAGE ABILITY MEASURES AS USED IN EXPERIMENT I 1st mean 2nd mean diff. of means t ratio P CAR 18. 09 22. 70 4.61 5. 71 <.001 Cloze 23.42 34. 79 11. 37 12.31 <.001 Note: Tests given one month apart 80 : would take place. The results from both CAR and Cloze sustained this expectation. ‘ i Interpretation of findings in Experiment I Experiment I was intended to be exploratory. In quiry into possible reliability and validity was made of the use of Cloze Procedure as a diagnostic listening com prehension test for foreign students. Two accepted tests, CAR and B-C, were used as the main criteria for comparison with Cloze. Other tests, mainly the USC G-S and USC V, were used as secondary measurements in Experiment I. It was first necessary to determine whether the experimental instrument Cloze was a reliable instrument. This was accomplished by administering the instrument twice ; to a group of 43 subjects. The test-retest coefficient of reliability of .926 was highly satisfactory. Although the CAR had a reported index of reliability this index was computed on native English speaking stu dents in U. S. secondary schools. In order to make a com parison with foreign students, it was decided to determine its reliability with the same 43 used as subjects in this study, in a test-retest procedure. This would take into account any variation in the characteristics of the subjects1 tested, and utilize the same test of reliability. Thus, in effect, it would establish a standard group for comparing the reliabilities of the two instruments. As reported, the reliability of CAR was found to be moderately high (.768). When this was compared with the reliability of Cloze, it was found that Cloze had a significantly higher coefficient of reliability than did CAR. The next step was to determine whether Cloze would relate as did the B-C, the standardized listening instru ment used in this study, to certain other outside criteria of language ability. As reported, it was found that there were no significant differences between Cloze and B-C with respect to their relationships with outside criteria. Finally, it was decided to compare Cloze with one of the standard language tests (CAR) in measuring improve ment in language ability of the group of subjects. As was reported, both Cloze and CAR showed significant increases in mean scores over a period of several weeks— an increase that was in accordance with reasonable expectations. The findings in Experiment I indicated many positive factors that encouraged further development of the experi ment. They seemed to point to the fact that Cloze Proce dure as an instrument for measuring language ability of foreign students appeared to be comparable with, and possibly more reliable than, other widely used tests. In addition to the high reliability of .926 posted by Cloze, a clue to its high relationship with vocabulary was noted in Table 9. The USC V correlated with B-C .339, while it registered .489 with Cloze. It was decided, upon inter preting the results, that Cloze Procedure be exposed to the full experiment. It would include the Cloze, CAR, and B-C tests, which would be administered twice, each application a week apart rather than a month. The two listening tests would be compared for reliability in a test-retest proce dure, and also compared to CAR, an outside criterion of language ability. Experiment II In this section are presented the findings of Experiment II, and the testing of the four hypotheses formulated as a result of the earlier findings of the Group I s tudy. The raw data used in this experiment comprised the six measurements of the experimental test on listening comprehension, Cloze, and the two outside criteria, one on reading comprehension, and the other on listening 83 comprehension. The measurements were: (1) CAR; (2) CAR retested; (3) B-C; (4) B-C retested; (5) Cloze; (6) Cloze retested. These data are reported in Appendix A. The levels of significance are reported in cited Tables with all relationships based on^the Pearson The basis of minimum significance is interpreted at the .05 level. Relationships were considered highly ( or very) significant at the .01 level of confidence. Guilford's Table D (4:538-39) provided the values in interpreting the experimental data. The four hypotheses formulated from the results of the Group 1 study were: Hypothesis 1.— Scores obtained from the Cloze Pro cedure for measuring language ability would indicate a significantly greater coefficient of reliability than scores from either the B-C or the CAR measures of language ability respectively, when administered twice to a group of 47 experimental subjects. Hypothesis 2.— The relationship between scores of the Cloze Procedure for measuring language ability and the scores of the CAR measure of language ability would not be significantly different from the relationship between the 84 scores of the B-C and CAR measures of language ability. Hypothesis 3.— The scores of the Cloze Procedure for measuring language ability would show a significantly high degree of relationship with the scores of the B-C test when applied to the same subjects. Hypothesis 4.— The Cloze Procedure would measure mean improvement in language ability comparably with the B-C and CAR. Reliability of the three measures of language ability To test the first hypothesis, the coefficients of reliability for all three measures of language ability used in this investigation were obtained by administering these instruments twice, one week apart, to the 47 subjects of this experiment. This test-retest procedure was performed in order that some finding could illustrate whether or not the testing devices were measuring consistently and accu rately. The scores obtained from the two administrations were correlated by means of the Pearson product-moment technique. See Table 9 for correlations among all tests administered. These reliability coefficients were then TABLE 9 CORRELATIONS AMONG SIX VARIABLES FOR GROUP II (N=47) Variables 1 2 3 4 5 6 1. 1st CAR . 730 . 665 . 645 . 691 .659 2. 2nd CAR . 611 . 679 . 720 . 713 3. 1st B-C . 774 .655 . 713 4. 2nd B-C . 726 . 793 5. 1st Cloze . 923 6. 2nd Cloze oo Ln 86 compared by means of z transformation technique. As reported in Table 10, the coefficients of reliability for CAR and B-C were moderately high (.730 and .774 respectively). CAR's reliability was consistent with the .768 reliability found in Experiment I. The B-C reliability of .774 was in keeping with its published co efficient of reliability, .78 (51:13). The coefficient of reliability for Cloze, however, was very high (.923). In ,Experiment I it was .926. As can be seen in Table 10, there was a highly significant difference (P<- 01) between the reliability coefficients of Cloze and both CAR and B-C. There was no significant difference between the reliability coefficients of CAR and B-C. Thus, since the coefficient of reliability of Cloze was significantly greater than CAR or B-C, Hypothesis 1 is tenable at the .01 level of significance. Validity relationships of B-C i and Cloze to CAR To test Hypothesis 2, the scores of the CAR were respectively correlated with the scores from Cloze and B-C. This procedure to determine the validity of the test in strument was to compare it with CAR, an accepted outside criterion of language ability testing, and to compare the TABLE 10 COMPARISONS OF RELIABILITY3 OF TEST SCORES MADE IN THE THREE MEASURES OF LANGUAGE ABILITY AS USED IN EXPERIMENT II Instruments 1 Compared 2 b r i b r2 Z1 Z2 dif f. of z ' s t ratio P CAR B-C . 730 . 774 .929 1. 030 . 101 0.48 CM in « CAR Cloze . 730 .923 . 929 1. 610 .681 3.21 <.01 B-C Cloze . 774 .923 1. 030 1.610 . 580 2. 74 \. 01 aTests given one week apart. v All correlations were significant at the .01 level. 00 •o 88 : results with B-C. The action was taken to determine if Cloze measured what it was designed to measure, namely language ability by means of listening comprehension. The significance of differences between these correlations were then computed by means of z transformations. The results of these correlations and comparisons are reported in Tables 11 and 12. In these Tables are reported the correlations between the first and second administrations of Cloze and B-C with the first and second administrations of CAR. These correlations were moderately high, ranging from .611 to .720. Once again, when the z transformation was applied, there were no significant differences between B-C and Cloze relationships with CAR, suggesting that the experimental test Cloze performed as favorably as did the accepted B-C. Although the tendency is definitely toward Cloze, no significant differences were found between the Cloze and B-C relationships to CAR. Hypothesis 2 must be retained pending further investigation. Relationships between Cloze and B-C To test Hypothesis 3, Pearson product-moment co efficients of correlation were computed between scores on TABLE 11 COMPARISON OF CORRELATIONS BETWEEN TEST SCORES OF THE 1ST ADMINISTRATION OF THE CAR LANGUAGE ABILITY TEST AND THE B-C AND CLOZE LANGUAGE ABILITY TESTS RESPECTIVELY AS USED IN EXPERIMENT II Correlation with 1st CAR N r z dif f. between z ' s t ratio P 1st B-C 47 .665 . 802 1st Cloze 47 .691 .850 .048 0.23 . 82 2nd B-C 47 .645 . 766 2nd Cloze 47 .659 . 791 .025 0.12 . 90 Note: All correlations were significant at the .01 level. oo V £ > I TABLE 12 COMPARISON OP CORRELATIONS BETWEEN TEST SCORES OF THE 2ND ADMINISTRATION OF THE CAR LANGUAGE ABILITY TEST AND THE B-C AND CLOZE LANGUAGE ABILITY TESTS RESPECTIVELY AS USED IN EXPERIMENT II Correlation with 2nd CAR N r z diff. between z' s t ratio P 1st B-C 47 .611 . 711 1st Cloze 47 . 720 .908 . 197 0. 93 . 34 2nd B-C 47 .679 .827 2nd Cloze 47 . 713 .893 . 066 0. 31 . 75 Note: All correlations were significant at the .01 level. <D o i 91 i j ithe Cloze and the B-C, for both the first and second ad- i ;ministrations. This correlation was to establish a co- :efficient of validity with an established listening compre- l-hension test of language ability. In Table 13 are reported the results of these correlations. The correlations were moderately high, :ranging from .655 between the first administrations of the two tests to .793 between the second administrations of the two tests. Since coefficients of validity above .65 are usually difficult to obtain, the results gathered in this study can be considered relatively high. In the light of these results, Hypothesis 3 was retained. Comparison of mean change in Cloze, B-C, and CAR test scores As mentioned earlier in Experiment I, the fourth hypothesis was tested on the assumption that a good measur ing instrument will register change when change takes place. The means of the two administrations of the three 'tests were respectively compared by t ratios. The results of these comparisons appear in Table 14. For all three language ability measures there was a significant increase in mean score from the first to the second administration of the instruments. These I TABLE 13 CORRELATIONS BETWEEN TEST SCORES MADE ON THE CLOZE PROCEDURE AND THE BROWN-CARLSEN LISTENING COMPREHENSION TESTS AS USED IN EXPERIMENT II Cloze Brown-Carlsen 1st Admin. 2nd Admin. 1st Admin. 2nd Admin. .655 .726 .713 .793 Note: 1. First and second test administrations one week apart. 2. All correlations were significant at the .01 level. to to TABLE 14 COMPARISON OF MEAN DIFFERENCES BETWEEN TEST SCORES OF SUBJECTS ON THE THREE LANGUAGE ABILITY TESTS AS USED IN EXPERIMENT II Tests 1st means 2nd means difference of means t ratio P CAR 15. 30 21.28 5.98 7. 57 <.001 B-C 22. 21 29.98 6. 77 6.45 <•001 Cloze 27. 06 35.81 8. 75 10. 30 <.001 Note: Tests given one week apart. yo u> 94 differences were all highly significant (jp<. 001). The amount of improvement on CAR and Cloze in Experiment II was greater than expected in light of im provement found in Experiment I. The best interpretation seemed to be that the improved scores in Experiment II reflected not only improvement in the subjects' English language ability, but also their increased “ knowledge of how to use the test instruments, i. e., a practical factor. Thus, with the Cloze Procedure measuring improve ment of the 47 subjects as comparably as either B-C or CAR, hypothesis 4 was considered tenable. Summary of Experiment II findings In this chapter were presented the findings of the experimental study. The results of the Group I study enabled the investigator to propose four hypotheses to be tested in the experimental study with Group II. These hypotheses were with respect to (1) relia bility of Cloze Procedure as compared to the outside criteria, CAR and B-C? (2) the relative validity of Cloze Procedure and the B-C when both are compared with the out side criterion CAR; (3) the validity of Cloze Procedure when compared with the outside criterion B-C; and (4) the 95 ability of the Cloze Procedure to measure improvement in language ability as compared to the outside criteria, B-C and CAR. In summary, the data showed that: (1) the Cloze Procedure appeared to be highly reliable; (2) the Cloze Procedure appeared to be significantly more reliable than the outside criteria, B-C and CAR; (3) the Cloze Procedure appeared to validate satisfactorily with both outside crite ria; and (4) the Cloze Procedure appeared to measure improvement as effectively as did the outside criteria. In the next chapter are presented the summary, conclusions, and implications of this investigation. CHAPTER IV SUMMARY, CONCLUSIONS, IMPLICATIONS Summary Ever increasing numbers of foreign students have been coming to our shores during the past decade and a half. The large number has created many problems in administering extensive programs of placement and study. One critical area has been the testing of the English language ability of the. foreign students. Much of the literature concerning testing has in cluded heavy criticism, especially in the area of listening comprehension. Whether used as a single test or a sub-test or a battery, popular tests of listening comprehension have been faulted in the areas of reliability and validity. This has been evident in the testing of foreign students as I • j well as native speakers of English. The purpose of this study was to investigate the feasibility of applying Cloze Procedure, a relatively 96 97 untried technique, as a diagnostic test of the listening comprehension of foreign students- This communication research technique, Cloze, was to be administered within the framework of an experimentally controlled design. Prob lem areas investigated in this study were: (1) determining the reliability of the Cloze listening comprehension test as compared with the California Reading Test, Advanced (CAR), and the Brown-Carlsen Listening Comprehension Test (B-C); (2) comparing the relationship of Cloze and B-C with selected outside criteria in the measurement of language ability; (3) finding the relationship of Cloze scores with B-C scores in measuring language ability of subjects; and (4) comparing the relationship of Cloze with B-C and CAR in measuring mean'growth of language ability. The experiment was developed in four main stages: (1) the selection and adaptation of the test materials; (2) the selection of the subjects; (3) the testing of the subjects; and (4) the processing of the raw data and inter pretation of findings. All instruments of measurement were standard tests currently in use, with the exception of the Cloze Procedure test which was used experimentally and was designed by the researcher. 98 In applying the Cloze Procedure, subjects attempted to replace words that had been systematically deleted from selected paragraphs of a 20-minute tape-recorded lecture entitled "The Nature of Study. 1 1 A total of 126 words were deleted. The number of correctly replaced words became the subject's Cloze score. Information and answer sheets were mimeographed for the Cloze tests. The standardized tests included their own printed answer sheets. The subjects comprised foreign students enrolled at the University of Southern California and at Cambria Adult School in Los Angeles. They were designated as Group I (N=43), and Group II (N=47). In testing, earlier scores were obtained in Group I from application of B-C and the USC English Placement Test with sub-tests on grammar-spelling (USC G-S) and vocabulary (USC V). Cloze and CAR were administered twice. Experiment II, with Group II, consisted of two applications each of Cloze, CAR, and B-C. Cloze and B-C as listening comprehension tests were recorded and played back for the subjects. In all cases comprehension was taken to mean total correct scores and responses registered in each test. Only one test was administered during any given class 99 period. The two rooms used for the experiment were well lighted by ceiling lights and large windows, and ventila tion was ample. The temperature was comfortable and no "noise" interruptions were observed. The Pearson product-moment correlation and the customary formula, for computing the significance of dif ference between means, were the principal statistical tech niques used in processing the data. All data were processed on an IBM 1401 electronic computer at the University of Southern California Data Processing Center. All work was supervised by Robert Jones, Director of the Testing Bureau at the University. Conclusions The problem in Experiment I was stated in the form of three questions; the problem in Experiment II was stated in the form of four hypotheses. Following were the conclusions reached regarding the above seven items: Experiment I 1. The test-retest reliability coefficient for Cloze Procedure was high (.926); the test- retest reliability of CAR was moderately high 100 (.768). Comparison of these two coefficients revealed that Cloze was very significantly higher than CAR «. 01) . 2. Comparison of the correlations of Cloze and B-C with CAR, USC G-S, and USC V showed no signifi cant differences between Cloze and B-C with respect to their relationship to the selected outside criteria. 3. Measurement of Cloze and CAR showed a similar and significant improvement during a month's time between test-retest. The differences in means for both the first and second Cloze and the first and second CAR were highly signifi cant (P<. 001) . Experiment II 4. The coefficients of reliability for CAR and B-C were moderately high (.730 and .774 respec tively) . The coefficient of reliability for Cloze was very high (.923). There was no sig nificant difference between the reliability co efficients of CAR and B-C. There was a highly significant difference (P<. 01) between the 101 reliability coefficients of Cloze and both CAR and B-C. 5. The correlations of Cloze to CAR in determining validity were moderately high, ranging from .611 to .720. The relationship of B-C to CAR was .611 to .679. Although no significant differ ence was found between Cloze and B-C, the ten dency favored Cloze. 6. The correlations between Cloze and B-C were moderately high, ranging from .655 to .793. 7. Cloze, B-C, and CAR showed significant in creases in mean scores from the first to the second administrations of the instruments. Each of these three differences was highly signifi cant (P <C. 001) . Implications The uniqueness of Cloze Procedure allows much speculation as to further research in many areas. This report refers basically to some recommendations in the field of oral communication. The present study, along with Williams' (50:131), may be regarded as preliminary steps in the application of Cloze Procedure to measurement of 102 language comprehension, because the testee could concentrate on a language selection without being concerned with addi tional distracting stimuli in the form of questions, multi ple-choice answers, additional reading, etc. With the above in mind, suggestions for further research were the following: 1. A study duplicating the current study where subjects would respond orally rather than in written form. The move would be a step closer to more ideal conditions in the testing of oral language ability. The use of a language labo ratory would be essential. 2. A future study duplicating the present one, utilizing native-born speakers of English. This would be a correlation study utilizing the Cloze, B-C, and STEP listening tests with sever al outside criteria, possibly reading comprehen sion and intelligence tests. 3. Further research with various homogeneous linguistic groups, such as native speakers of Spanish, German, French, Arabic, Japanese, Mandarin, etc., identifying factors peculiar to each one. 103 4. A Cloze Procedure study designed to help ele mentary school children learn English as a sec ond language. This would apply, for example, to the Mexican-American in Southern California and Texas, to the Puerto Rican in New York, to the Negro and other sub-standard dialect speakers. 5. In sequel to or in conjunction with study four, the use of Cloze in the elementary and secon dary schools as a diagnostic tool in the mea surement of vocabulary, English proficiency, etc. 6. Research of materials for aural training of foreign students, investigating the areas of phonemics, morphemics, and tactics. 7. A correlation study to establish the relation ship between standard intelligence tests, lis tening comprehension tests, and Cloze Procedure. 8- A study to compare controlled listening where three groups would be tested: (1) a group with no current work in English as a second lan guage (ESL), (2) a group with ESL courses with I no listening instruction, and (3) a group with 104 ESL courses with specific listening training. 9. A study which would identify and isolate the factors affecting Cloze Procedure used in oral testing. 10. A study similar to the present experiment in which equally valid and reliable findings might result with less than 126 responses. 11. A Cloze listening test utilizing emphases on structural and lexical design. The procedure could include experimentation with carrying word deletion to every sixth or seventh word, or, as Williams (50) did, with applying the deletion to every fifth, or even seventh, second. 12. A sophisticated Cloze approach endeavoring to measure expression as well as comprehension, in which the spontaneous speech of foreign students would be recorded, Cloze-mutilated, and then given to other subjects for filling in or oral response. 13. An extended study of Williams (50) as applied to native speakers of English, utilizing » • parallel forms with multiple-choice testing in areas of speech such as rhetoric, debate, oral . interpretation, speech pathology, etc. 14. The potential application of the Cloze Proce dure in the field of psycholinguistics, espe cially as "a technique in content analysis, in that one uses evidence in messages as a basis for making inferences about either sources or receivers." (6:78) Williams included other recommendations for further research with the technique of Cloze Procedure which bear reviewing by future investigators (50:130-134). As the conclusion of this study it seems appropriate to state that the importance of the diagnostic testing of listening comprehension as a measurement of language ability for the foreign student is greater today than it has ever been. Perhaps its international value and effects cannot truly be measured. Still, Secretary-of Defense Robert S. McNamara commented on one area of it before the Committee on Foreign Relations of the United States Senate: Possibly the greatest return on our military assistance investment dollar comes from the training of selected foreign officers and key specialists. . . . These students are handpicked by their own countries to become instructors when they return home. They are the coming leaders of their nations, the men who 106 will have the know-how and will impart it. . . . I need not dwell upon the value of having in positions of leadership men who have first-hand knowledge of how Americans do things and think. (37:61) A P P E N D I X E S 107 APPENDIX A RAW DATA 108 I GROUP I RAW DATA Subject Age Sex Origin USC G-S USC V B-C 1st CAR 2nd CAR 1st Cloze 2nd Cloze 01 25 M Japan 42. 18. 10. 23. 21. 11.0 16.0 02 21 M Kuwait 29. 16. 18. 28. 30. 50.0 55.0 03 21 M Kuwait 12. 07. 09. 10. 18. 13. 0 22. 0 04 22 M Kuwait 13. 02. 09. 1 13. 11. 05. 5 24.0 05 22 M Kuwait 26. 13. 14. 20. 20. 28. 5 45.0 06 19 M Saudi Arabia 18. 07. 13. 11. 18. 24.0 46.0 07 21 M Kuwait 32. 17. 15. 20. 26. 34. 5 39. 5 08 25 M French Guinea 27. 26. 07. 17. 24. 27. 0 52. 5 09 29 M China 45. 16. 14. 32. 34. 29. 0 44. 5 10 29 F Thailand 21. 10. 08. 11. 21. 09. 5 17.0 11 27 M China 41. 16. 13. 26. 33. 25.0 35. 5 12 29 M Saudi Arabia 15. 03. 08. 08. 17. 41.0 43.0 13 23 M Saudi Arabia 09. 09. 00. 04. 09. 01. 0 09. 0 14 25 M Saudi Arabia 08. 06. 09. 13. 13. 12.0 29. 0 15 24 M Japan 09. 04. 14. 13. 23. 12.0 25.0 16 22 M Saudi Arabia 07. 07. 11. 13. 21. 14.0 31.0 18 29 F Iran 24. 05. 04. 20. 25. 38. 5 49.0 19 26 M Korea 15. 09. 14. 14. 28. 34.0 43. 5 20 23 M Korea 44. 23. 19. 29. 35. 21. 0 38.0 21 20 M Germany 26. 16. 17. 28. 40. 52. 5 63.0 22 20 F Indonesia 35. 14. 23. 27. 28. 28. 5 48.0 23 35 M Korea 28. 16. 06. 18. 12. 03. 5 19.0 109 Age 25 25 33 26 25 24 35 22 25 21 28 30 18 22 21 18 19 18 24 23 31 GROUP I RAW DATA— Continued 3 ex Origin USC G-S USC V B-C 1st CAR 2nd CAR 1st Cloze 2nd Cloze M French Guinea 29. 20. 07. 22. 27. 20. 5 47.0 M Japan 24. 12. 12. 16. 18. 08. 5 16. 0 M Pakistan 20. 18. 10. 14. 24. 12. 5 28. 5 F China 27. 13. 17. 22. 22. 23. 5 26. 0 M Korea 15. 02. 13. 12. 15. 08. 0 18. 0 M India 11. 02. 13. 19. 33. 24. 5 33. 0 M Iran 18. ■ 08. 08. 04. 11. 34. 5 42.0 F Thailand 30. 18. 14. 27. 34. 36. 0 36. 5 M Japan 04. 01. 07. 12. 11. 12.0 19. 0 M India 41. 15. 13. 25. 18. 24. 5 45.0 M China 17. 02. 13. 24. 25. 19. 5 32.0 M China 45. 10. 16. 28. 36. 27. 5 40. 0 M Venezuela 49. 27. 21. 31. 39. 59. 5 70.0 M Korea 12. 07. 05. 19. 16. 04. 5 11.0 M Japan 19. 03. 21. 09. 25. 02. 5 11.5 M Hungary 26. 14. 13. 13. 22. 44. 5 54. 5 F Hungary 27. 20. 09. 18. 18. 28.0 38.0 M Mexico 44. 26. 25. 25. 25. 59. 5 72.0 M Japan 19. 07. 14. 13. 18. 21.0 25.0 M Korea 08. 02. 10. 08. 14. 02.0 08. 0 M Japan 24. 17. 18. 19. 18. 19.0 28. 5 110 GROUP II RAW DATA Subject Age Sex Origin 1st CAR 2nd CAR 1st B-C 2nd B-C 1st Cloze 2nd Cloze 01 22 M Iran 10. 23. 25. 34. 27. 43. 02 22 M Mexico 11. 19. 13. 30. 26. 43. 03 22 M Iran 06. 09. 15. 21. 18. 29. 04 v54 A Italy 21. 25. 13. 20. 32. 29. 05 31 M Lebanon 13. 24. 21. 30. 24. 31. 06 20 M Korea 20. 15. 19. 21. 15. 21. 07 27 M Korea 10. 25. 20. 29. 18. 22. 08 34 M Mexico 20. 34. 21. 26. 32. 41. 09 19 M Honduras 11. IS. 17. 24. 22. 34. 10 24 M Jordan 30. 32. 46. 44. 57. 64. 11 23 M Israel 25. 32. 35. 47. 33. 49. 12 25 M Japan 06. 11. 12. 17. 13. 15. 13 25 M Japan 12. 17. 18. 17. 17. 25. 14 25 M Japan 14. 18. 21. 27. 15. 25. 15 23 M Japan 16. 22. 17. 25. 34. 35. 16 20 M Mexico 32. 37. 28. 43. 45. 61. 17 21 M Peru 18. 22. 17. 26. 27. 33. 18 35 F Spain 13. 05. 11. 20. 19. 17. 19 24 M Japan 16. 26. 28. 24. 16. 25. 20 32 M Iran 07. 15. 10. 28. 26. 34. 21 27 M Japan 11. 23. 14. 16. 14. 19. 22 19 M Iran 19. 28. 33. 42. 29. 47. 23 35 F Korea 12. 14. 09. 33. 20. 32. 24 22 M Japan 13. 25. 14. 21. 25. Ill • C M 111 GROUP II RAW DATA— Continued Subject Age Sex Origin 1st CAR 2nd CAR 1st B-C 2nd B-C 1st Cloze 2nd Cloze 25 30 F Switzerland 33. 40. 48. 68. 59. 67. 26 32 M Hong Kong 18. 28. 34. 29. 29. 33. 27 19 M Panama 18. 29. 44. 41. 58. 77. 28 28 F Japan 10. 14. 13. 24. 13. 18. 29 42 F Japan 09. 16. 16. 28. 22. 23. 30 23 M Japan 14. 18. 20. 26. 23. 33. 31 22 M Japan 07. 15. 18. 24. 26. 28. 32 20 M Iran 20. 17- 32. 23. 25. 36. 33 23 M Iran 12. 20. 11. 26. 37. 41. 34 22 M Italy 21. 31. 16. 31. 37. 49. 35 33 F Yugoslavia 13. 19. 15. 16. 11. 23. 36 19 F Mexico 12. 28. 19. 27. 46. 60. 37 26 M Iran 11. 17. 33. 41. 22. 34. 38 20 M Iran 13. 15. 34. 24. 28. 35. 39 21 M Okinawa 19. 20. 28. 33. 29. 43. 40 21 M Japan 10. 13. 13. 18. 15, 18. 41 31 F Egypt 18. 27. 25. 34. 38. 39. 42 21 M Japan 15. 15. 15. 15. 18. 28. 43 44 F Germany 12. 10. 16. 15. 13. 17. 44 24 M Japan 11. 10. 18. 26. 20. 33. 45 25 i M Turkey 25. 27. 44. 50. 45. 63. 46 26 F Japan 14. 31. 34. 45. 26. 45. 47 21 M Iran 18. 20. 21. 33. 28. 42. APPENDIX B SPOKEN MATERIALS CLOZE TEST 113 Initial Instructions to Both Groups 114 (Oral instructions upon introduction) Good afternoon. As your instructor Mr. or Mrs. (instructor's name) has told you, we are going to conduct an experiment. This test is part of that experiment. The success of it depends on your full cooperation. You will not be graded on it. The test results will not be placed into your school file. With your cooperation in this re search experiment, we hope to be able to help others like yourself who are coming to the United States of America for study. When you first arrived in this country, you took many difficult tests to determine your English level. We hope to obtain enough information from these experiments taken by you and others to help formulate examinations in which you can participate freely and measure your knowledge of English with less hardship. This test is on listening comprehension, measuring your understanding of what you hear. It is a special test and quite challenging. Earlier students have enjoyed taking it, and I am certain you will too. The test is recorded, and you must listen carefully. As I have been talking, answer sheets have been passed out to you. Make certain that you have a pencil available. Do not use a pen. An information sheet on your background given to you yesterday by your instructor will be collected at the end of the test today. I shall ask for these questionnaires separately from the test answer sheets. Thank you for your attention, and remember, the future success of our Foreign Student Program depends on your participation in this experiment. Good luck. (Pause) (Start tape) 115 i Text of Recorded Instructions Used for Both Groups The Cloze Listening Comprehension Test In order that you may participate successfully, you I must be able to hear clearly. If you can hear my voice ‘clearly, please raise your hand. (Pause) i In a few moments you will be given a special test on listening comprehension. You have been given answer ‘sheets; please make certain that you have filled in clearly lyour name, the name of your instructor, and the date. ;Please be accurate. This listening test is unlike any you have taken. You have taken tests to see if you comprehend reading. This is a special listening test. In taking it, you will find out that listening is a skill and demands close atten tion. You shall hear a short and interesting lecture on | the Nature of Study. At certain intervals you will hear a ‘sound [TONE, 3-4 seconds] that indicates a word is missing. Remember that [TONE] means a word has been removed. Some sentences contain two or three missing words on your answer :sheets. I will stop at the end of each sentence and allow ;you 5 to 10 seconds to fill in the answer or answers. Remember, we will go through the lecture without linterruption until we reach.certain sentences with this !sound [TONE]. Sentence by sentence you will try to replace jthe missing words. If you lose count of the sentence that |is being read, look at the numbered cards at the front of I the room to make certain that they are the same as your lanswer sheet. This will help you follow the sentence iorder. Remember, each sentence has a number. i i Fill in the blanks as best as you can. No-one is expected to do them all correctly, but answer as many as ■you can; it will be to your advantage. Only one answer jmay be given. If you cannot answer, leave the space blank 116 land go to the next one. If you make a mistake, erase it I completely and start over. ; The test will be in two sections. Section I con sists of 56 sentences with their missing words to be re placed daring the lecture. Let us try the sample question first as a trial ■before we officially begin. Everyone look at your answer sheet. We shall answer question number 01 under sample ; questions. Get your pencil ready and mark when you think |you have the answer you feel is right. Here is the sentence. "My name [TONE] John Jones." (pause 5 sec.) Your answer should be the word "is"— My iname ip John Jones. The next sentence is number 02.. Ready? "My [TONE] is the University of [TONE] California." (pause 5 sec.) You should have "school" and "Southern"— My school is the University of Southern California. Do you !get the idea? Let's try the last practice sentence. : "Today is the [TONE] day before we [TONE] on our summer j[TONE]." (pause 10 sec.) You should have "last," "go," :and "vacations." The completed sentence would sound like Phis: "Today is the last day before we cjo on our summer ^ vacations." I'm certain that you will do well. Remember, ;only a few of the paragraphs will be in this form. By the end of the lecture, remember, you will have finished the 56 test items in Section I. i Are you ready? The test, including the lecture, will last approximately 30 minutes. Work as fast as you can without making mistakes. You will hear the material only once. It will not be repeated or slowed down. Write your answers clearly so that the instructor will be able to read them. Good luck, and listen clearly and carefully. (Pause 10 sec. before delivery of lecture) (Play Cloze lecture tape) 117 THE NATURE OF STUDY Three great men, Charlemagne, Leonardo da Vinci and Charles Darwin, who lived centuries apart, agreed that success depended on constant study. Charlemagne, the great emperor of the West during the ninth century, believed that only by constant study could he become a good ruler and a wise man. The greatest teacher of his age was Alcuin. This Englishman was brought to France to tutor Charlemagne and his nobles. The great ruler applied himself well. He even adopted a form of shorthand to help him to learn faster and more. The brilliant Leonardo da [Vinci], in the late 15th [century], strove for perfection in [painting], science, engineering, and sculpting. // [He] improved his painting not [only] by studying optics, but [also] by dissecting the human [body]. // He felt that man [could] approach perfec tion only by [continual] study. // Darwin was a [19th] century English naturalist. // He [was] a most patient and [methodical] student. // Working day and [night], he searched 20 years [for] materials. // They composed his [monumental] book on evolution, "The [Origin] of Species." // Asked about [study], he said, j 118 "When I [am] obliged to give up [observation] and experi ment, I shall [die]." The fact is that man, to be successful in a field, must seek to perfect himself by constant study. This applies not only to his own field, but many times to re lated fields, as in the case of Da Vinci. The study habits and skills of individual students are closely related to success in school work. To be successful demands knowledge and the ability to apply that knowledge to given situations. Study is a direct means of obtaining knowledge and, in turn, success. Success in school depends on three things: (1) ability, sometimes called intelligence, (2) work or effort, the amount of time spent on your assignments, and (3) efficient methods of work, the organization of materials and the arrangement of your time. It is within these areas that study helps us. Effective study depends on your know ing how to study. Let us discuss five vital study areas. They are concentration, reading, memorizing, listening, and taking notes, and lastly, preparing for examinations. Many students complain that it takes much difficul- ; ty in settling down to study. You may claim that too many ■ things prevent you from concentrating. What is ; ' i i 9 concentration? It is the ability to direct and control your attention. Concentration is a shill to be acquired. Paying close attention does not come easily. It is a habit that must be developed. First of all, plan [your] distribution of time for i [ each] day. // Stick to it. [Allow] enough time for sleep, [exercise], and a variety of [activity]. // You cannot study [effectively] for extremely long periods [of] time. // Have a time [and] place for studying. // Develop [an] attitude for studying. // Feel [confident] that the time you [set] aside will be adequate. // [Determination] is important here. // Do [not] try to complete a [long] English assignment at one [time]. // Remember, it is far [easier] to complete large assignments [in] small dosages rather than [by] large bites. // Minimize distractions. Unless you are in need of help, study alone. Make certain that your room is free from interrupting elements. Do not read in an armchair or comfortable bed. You will certainly go to sleep. Use a straight-back chair. Radios, record players, telephones and photographs should not be within eye or ear range. Have all necessary study materials near by. Make certain !that the lighting is good. Poor lighting can cause you to ' 120 |become fatigued. Some exercise after accomplishing short jgoals helps tone your body. Even deep-breathing will help iyour circulation. ! Reading works hand-in-hand with concentration. Your main purpose is to read only as fast as you can com prehend. Different materials demand various reading rates. Most students read the sports section of the newspaper much faster than they read school assignments. You must, there fore, develop a reading rate that is flexible. Understand the general framework and subject matter of the material you are reading. Go through the chapter rapidly by "skimming" to get the general arrangement of I ideas. Note the main topics or sub-ideas. It is most important to read paragraphs by concentrating on the topic sentence of each. '"Stop at the end of a chapter or section and make a mental or written summary. The written is preferred. While you read you [must] think. // You must be [able] to question, to judge, [and] to criticize. // Devel op the [ability] to read by phrases [rather] than by indi vidual words. // [Look] for the sense of [the] phrase or sentence rather [than] a word for word [definition]. // If ithere are words [unfamiliar] to you, look them [up]. // Prepare a card file [of] these words, especially in [science] and the languages. // Memorizing is a very important area. Every student has felt at one time or another that he has forgotten his lessons too soon. You may have said to yourself, "I knew it yesterday, but I can't think of it today." First, you must learn what should be known. Second, you must place it in your memory so that it will be there when you want to use it. Good remembering depends on good understanding. Do you understand the main ideas before you go on? It is most helpful to remember essentials first. Impressions must be intensified to be remembered. Recite aloud several times to help fix ideas in your mind. Whenever it is possible, form concrete images of your subject matter. Once again we repeat, "retain ideas, not just words. 1 1 When you uti lize repetition, use it with meaning. Be aware of the thoughts or ideas involved. General retention will always be aided by good organization. Remember your facts by a system or order. You may even create a "picture in your mind." Always test yourself by writing what you must mem orize. The most important principle you should remember is that understanding should always precede memorization. 122 Area four includes how to listen and take notes on lectures. Students generally take for granted the ability to listen. Most students can improve their listening abil ity up to 25 percent. Some college instructors report averages as high as 70 percent of their students having difficulty in taking notes. You might have heard a fellow student say, "I'm never sure that I am getting the most important points in a lecture," or, "I write so much that I find myself missing most of what is said." Perhaps we are confusing listening with hearing. Hearing is the act of receiving sound impulses. Listening is the understanding of these impulses. So once again understanding plays a key role in how-to-study. Effective listening is quite similar to effective reading. In listening you search for ideas and understanding as you do in reading. One is using the ear, while the other is using the eye. Some say that the [speaker] has all of the [responsibility] in communication. // This is [not] true. // The main responsibility [lies] with you, the listener. // [First] of all, sit where [you] can clearly hear the i [ speaker] . // Do not expose yourself [to] various distrac tions by sitting [next] to a window or [doorway]. // Do not I sit far [back] in the room. // Before [you] go to the 123: lecture, [prepare] yourself. // Read your textbook [assign ment] and your outside reading. [it] will also help you [to] read previous lecture notes. // Listen and think before you start to take notes. Skilled note-takers listen about 90 percent of the time and write for 10 percent. Listen for the main idea in the title. Remember, you will use fewer words than the speak er. The less you write the more time you will have to listen. If your proportion of listening to writing time is not 90 to 10, then you are probably writing too much or possibly too little. Your notes must be easy to read. Use some form of an outline with symbols and abbreviations. Be certain that you understand what you have written. Many students, in reviewing their notes,have had difficulty in reading them. Underline important words as they come along. Plan ahead by leaving space for later definitions and explanations. The speaker may openly indicate the important divisions of the lecture. Watch for the repeating of main points in your writing. Units of thought gain the important spot here. You may record them as a sentence, phrase, or even a single word. Use what is most convenient to you. Learn to watch for the speaker's key words. They are usually I important to the developing of his main points. Concen- : trate on getting them. It is quite possible that only two ;or three supporting ideas are needed. Condense his thought into a few words. Remember that your notes [are] vital to you in [preparing] for examinations. //As you [write], leave a wide margin [on] the left side of [your] paper. // In this area [write] notes to yourself. // Indicate [material] to be reread or [emphasized]. // Once the lecture is [over] you should immediately review [your] notes. // You may add [or] change while the subject [is] still fresh in your [mind]. // Read the notes again [just] before the next lecture. // [This] will help you to [learn] more efficient ly. // Psychologists claim [that] students forget about 80 [ percent] of what they learn [within] two weeks unless they [ review] during that period. // It [is] obvious then that periodic [review] of your material is [necessary] . // What does one look for in an examination? In taking an essay type of examination it is wise to note that you do not have time to write all you know about a topic. The instructor usually suggests that only specific areas be covered. It is to your advantage to stick to those areas. Look for such key words as "list the causes" or "compare the systems. 1 1 Other key words are "illustrate, " "outline, " "diagram, " and "contrast. " Once you get a question, do not start to write immediately. Read it through. Organize your thoughts. Put them in outline form in the margin of your answer sheet for a guide. Note all the main points you can remember. Then you must write hard, fast and to the point. Do not bring in irrelevant points, called "padding." Write in ink, if possible. It is easier to read and usually neater. The instructor appreciates clarity in all forms. Take time at the end to proofread not only your answers, but the numbering of the guestions. Most objective tests are [limited] to available time. // It [is] best to work through [all] the guestions. // Answer what [you] know and do not [hesitate] too long on those [that] temporarily stop you. // Once [you] have finished, return to [the] ones you have skipped [for] re reading. // True-false guestions generally are built on a pattern of briefly describing two things and their degree of relationship to each other. The clue is to look for a changed word to overstate or understate the degree of relationship. Once you find this word, you can substitute ' others to find which makes the better statement. Such key ; words as "no," "never, 1 1 "every," "all" and "entirely" usually show that the statement is false. If a true-false statement contains two independent clauses and one is false, the whole statement is false. In multiple-choice questions reduce the possibil ities to one or two and select the best answer, even if both appear correct. In matching-questions, answer the known questions first. You may later eliminate the others on "educated guesses." For completion-questions go through answering all you know. Return for a second reading and possible changes. Do not hesitate on the exploded myth that first responses are usually right. In fact there is evidence indicating a slight tendency for improved scores with later changes. What can you do to prepare for tests? It is impor tant that you review all you have had since the last test. Do this at least once a week. Do not "cram" your studying into the late hours of the night before the test. Get your regular sleep. Do write a list of important facts or ideas. Study the points emphasized in class. Your own weakness can be reexamined by going over old home-work papers and tests. Outline or diagram materials that will probably be covered in the exam. Make up sample questions 127 and take your own test to look for weak areas. Some stu dents find that their retention is aided if they briefly summarize just before going to bed and immediately upon arising the day of the test. Try to look at a test as a regular assignment. Do not worry about it. Look on the test as an opportunity to show the instructor what you know. It will also assist him in helping you later in class. If you have truly prepared yourself, you should enter with confidence that you will do well. We have merely introduced you to five ar'eas impor tant to the field of how-to-study. Would you like more information? Do you seriously want to improve your own study habits? You will find many fine books on this sub ject in the library. The main section on Study is numbered 371.3. Your success in school [depends] on three things: your [ability] or intelligence, your work [or] your effort, and your [efficient] methods. // You develop effective [study] by reviewing the areas [discussed] in today's lec ture. // You [may] utilize the area of [concentration] to direct and control [your] attention. // Through developing the [area] of reading you may [obtain] a flexible reading i 128 rate. // [it] helps you not only [to] think but also to [question], judge, and criticize. // To [memorize] is to say that [good] remembering depends on good [understand ing]. // We have learned that [if] hearing is the receiving [of] a sound stimulus, then [listening] is giving it mean ing [ or] understanding. // You must look [for] units of thought. // An [examination] is a chance to [show] what you can do. // [The] real test is in [the] preparation for :the examination, [not] the examination itself. // Charlemagne, da Vinci and Darwin, among thousands of others, found that constant study is most vital to success in any field. How about you? Are you disappointed with the lack of success in your own field? If you are, then the solution may lie in your answer to this question, "How do you study?" 129 (Wait 10 sec. after termination of lecture.) That is the end of the lecture. You should have finished sentence number 56. Stop writing. Go over your answer sheets. Make certain that all words can be read. (As majority appeared ready) That completes your Listening Comprehension Test. (Upon completion— Oral Instructions, not part of tape) Please pass your papers to the left aisle. The stu dent in the last seat will collect them. After you have turned in your test papers, pass your information question naire to the right aisle. Will the student in the last seat please collect the questionnaires. Your instructor will discuss the results of this test with you after the data have been tabulated. This test will be repeated next week for the final results. Thank you so much for your contribution to this research project. APPENDIX C EXPERIMENTAL INFORMATION SHEET 130 APPENDIX C X . INFORMATION SHEET T h e d a t e t o d a y l a 1. 2. 3 . I t . M y n a m e I s ( D a y M o n th Y e a r ) ( l a s t ) ( f i r s t ) H r a g e i s _ _ _ _ _ _ _ _ _ . I w a s b o m o n t h e M y c o u n t r y o f c i t i z e n s h i p l e _ _ _ _ _ _ ( m i d d l e ) I n ( d a y , m o n t h , y e a r ) ( c i t y , c o u n t r y ) D i d y o u l i v e m o s t o f y o u r y o u n g l i f e b n a f a r m ? S m a l l t o w n , u n d e r UoQO? C i t y o v e r £ 0 0 0 ? I I . 5 . I a r r i v e d i n t h e U n i t e d S t a t e s C i t y o v e r 2 0 . 0 0 0 7 ( a i r , l a n d , s e a ) . H a v e y o u . w o r k e d s i n c e y o u r a r r i v a l ? _ I s E n g l i s h s p o k e n t h e r e ? m o n t h s a g o , o n _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ H r ( d a y , m o n t h , y e a r ) I f y e s , w i t h w h a n ? ( e m p l o y e r ! I I I . 7 . 8. I V . 9 . 10. 11. 12. 1 3 . Hi. V . 1 ? . I w a s g r a d u a t e d f r o m I a t t e n d e d ( y e s , n o ^ s o m e t i m e s ! ' i n ( n a m e ) ( s c h o o l ) ' U n i v e r s i t y i n s u b j e c t s w e r e o n ( c i t y a n d c o u n t r y ) My m a j o r ( c i t y a n d c o u n t r y . I r e c e i v e d a d e g r e e i n ( s u b j e c t ) ( d a t e ) W h a t l a n g u a g e s d o y o u s p e a k w e l l ? _ _ _ _ _ _ _ _ _ _ I n y o u r o w n l a n g u a g e i h a v e y o u h a d a n y d i f f i c u l t y w i t h s p e e c h ^ o r d i f f i c u l t y i n r e a d i n g ? _ _ _ _ _ o r l o s s o f h e a r i n g ? ( l e a , N o ) ( M a r k w i t h a n Z ) H o t a t a l l A l i t t l e V e i l V e r y w e l l I s p e a k £ n g l i s h , I r e a d E n g l i s h I w r i t e B u l l i s h M y f a t h e r s p e a k s E n g l i s h H y f a t h e r r e a d s E n g l i s h K y f a t h e r w r i t e s E n g l i s h 1 H r m o t h e r s p e a k s E n g l i s h M y m o t h e r r e a d s E n g l i s h M y m o t h e r w r i t e s B i g l i s h H r s n e a k s E n g l i s h ( b r o t h e r r e a d s S h g l l s b o r s i s t e r ) w r i t e s E n g l i s h 16. B e f o r e I c a m s t o t h i s c o u n t r y I s p e n t a t o t a l o f _ _ _ _ _ h o u r s s t u d y i n g E n g l i s h . I h a v e s t u d i e d E n g l i s h f o r _ _ _ _ _ h o u r s i n s c h o o l i n m y c o u n t r y , a n d ■ h o u r s w i t h p r i v a t e i n s t r u c t i o n . S o y o u f e e l t h a t y o u r s t u d y o f f i o g l i s h b e f o r e y o u o a m e t o t h e U n i t e d S t a t e s i s s u f f i c i e n t t o p r e p a r e y o u f o r u n i v e r s i t y w o r k I n B i g l i s h ? _ _ _ _ _ _ _ _ W h y ? ___________ _ 131 INFORM ATION SH E ET P a g e 2 1 7 . W a s i t n e c e s s a r y f o r y o u t o t a k e a n e x a m i n a t i o n i n E n g l i s h b e f o r e y o u w a r e a l l o w e d t o c o m e t o t h i s c o u n t r y a s a s t u d e n t ? _______ I f y o u r a n s w e r i s Y E S , w h a t t y p e o f a n e x a m i n a t i o n w a e i t a n d w h o a d m i n i s t e r e d i t ? V I . 1 8 , W h a t d o y o u f i n d m o s t d i f f i c u l t i n l e a r n i n g a f o r e i g n l a n g u a g e s u c h a s E n g l i s h ? _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 19. W h a t a r e y o u r g r e a t e s t d i f f i c u l t i e s i n c o m m u n i c a t i n g i n t h e E n g l i s h l a n g u a g e ? ( M a r k p r o b l e m a r e a s w i t h a n I . U s e XXX i f p r o b l e m i s s t r o n g i n y o u r o w n c a s e , i f a n a r e a I s n o p r o E l e m . d o n o t m a r k i t . ) a * T h e s p e a k e r s p e a k s t o o f a s t . b . T h e s p e a k e r u s e s l a r g e w o r d s . c . T h e s p e a k e r s p e a k s u n c l e a r l y . d . I d o n o t l i s t e n w e l l . ® . H X v o c a b u l a r y i s l i m i t e d . f , I c a n r e a d a c c e p t a b l y , b u t c a n n o t s p e a k a s w e l l . S . T h e s p e a k e r m o v e s o n t o a n o t h e r p o i n t b e f o r e I u n d e r s t a n d t h e f i r s t , h . I m i s s s o m e w o r d s a n d c o n c e n t r a t e o n t h o s e I m i s s f o r g e t t i n g t h e r e s t . i . I e a s i l y m i s s t h e m e a n i n g o f a s e n t e n c e i f I d o n o t c l e a r l y h e a r a l l t h e w o r d s . j . M o s t o f t h e o b j e c t i v e t e s t s g i v e n t o u s e a r l i e r w e r e t o o l o n g a n d I b e c a m e t i r e d . k . M u c h o f t h e e a r l y o b j e c t i v e t e s t s c o n t a i n e d m a t e r i a l t o o d i f f i c u l t f o r m o s t f o r e i g n s t u d e n t s t o u n d e r s t a n d . 1 . T h e r e a r e t o o m a n y i d i o m s a n d t h e y a r e d i f f i c u l t t o r e m e m b e r . m . T h e w o r d o r d e r i n E n g l i s h i s s o m e t i m e s d i f f i c u l t , n . T h e S n g l l s h l a n g u a g e l a c k s r u l e s a n d o r d e r . _ _ _ _ _ o . S o m e t i m e s I d o n o t k n o w f o r w h i c h p o i n t s I s h o u l d l i s t e n . p . I n a y m i s s a n i d e a i n s u b j e c t m a t t e r u n f a m i l i a r t o m e e v e n i f I k n o w t h e v o c a b u l a r y . 2 0 . I f y o u m i s s a w o r d o f t w o i n a s e n t e n c e , d o y o u s t i l l u n d e r s t a n d t h e m a i n i d e a o r t h o u g h t ? _ _ _ _ _ _ _ _ ( y e s , n o , s o m e t i m e s , h a r d l y a t a l l ) 2 1 . I f y o u m i s s t h e m a i n t h o u g h t , w h a t d o y o u l o o k f o r i n t h e f o l l o w i n g s e n t e n c e s t o h e l p y o u u n d e r s t a n d t h e e a r l i e r i d e a ? APPENDIX D CLOZE TEST BOOKLET 133 APPENDIX D HAME Doc , ■ John Fred DATE 14 January 1963 (Last) (First) (Middle) INSTRUCTOR Dr. Smith SAMPLE Q U E ST IO N S: 01. s is 02. a school b Southern 03. a last b go c vacations 134 135 Page 2 SECTION I A. 1. a Vinci b century c painting, art 2. a He b only c also d body, being 3. a could b continual, constant, careful, hard B. A. a 19th 5. a was b methodical, good,, serious 6. a night b for, scientific 7. a monumental, great b Origin 8. a stcdy b am c observation d die C. 9. a your b each, every 10. a Allow b exercise c activity 11. a constantly, well, effectively, hard b of 12. a and 13. a an 14. a confident b set, put 15. a Determination 16. a not b long c time, sitting, period 17. a easier, better b in c by D. 18. a must, should 19. a able b and 20. a ability, habit b rather 21. a Look b the strange, rare, unknown, 22. a unfamiliar, new b____ UP than definition 23. of science 136 Page 3 E. 24. a speaker__________b responsibility 25. a_________________ 26. a llcB> la_________ 27. a First, Moat________b y°u c speaker, lecture, voice, sound 28. a to____________ b next, close e doorway____ 29. a back, a w a y _______ 30. a y°u___________ b prepare 31. a assignment 32. a It, This b to P. 33. a are b preparing, studying 34. a write b on c your, the____ 35. a write_________ 38. a material, notea b emphasised_____ 37. a over, finished b your, the_______ 38. , or b la c alnd, memory 39. a carefully 40. a This b learn 41. a that___________b per-cent c within, In d review 42. a b review, reading c necessary, Important, vital 6. 43. a United_________ 44. a__ atop, wait, 45. a you ' b hesitate, stay c that 46. a you b the___________c_____for 137 Page 4 47. a depends b ability c or 48. a study b discussed, presented 49. a nay, should, can b concentration c your 50. a habit, area, ability b obtain, develop, achieve 51. a It, This, Reading^ to c question 52. a memorize b good c understanding 53. a If b of c listening 54. a speaker 55. a examination b show, know 56. a The b the c not APPENDIX E CLOZE SCORING MATERIALS 138 139 GROUP I TALLY CARD # Score Rank use G-S use V Brown-Car1sen CAR, 1st CAR, 2nd Cloze, 1st Cloze, 2nd GROUP II TALLY CARD # Score Rank CAR, 1st CAR, 2nd Brown-CarIsen, 1st Brown-Carlsen, 2nd Cloze, 1st Cloze, 2nd B I b l i o g r a p h y 140 BIBLIOGRAPHY Books 1. Buros, Oscar K. (ed.). Fifth Mental Measurements Yearbook. Highland Park, N. J. : The Gryphon Press, 1959. 2. Duker, Sam. Listening Bibliography. New York: Scarecrow Press, 1964. 3. Goldstein, Harry. Reading and Listening Comprehension at Various Controlled Rates. New York: Bureau of Publications, Teachers College, Columbia Universi ty, 1940. 4. Guilford, J. P.. Fundamental Statistics in Psychology and Education. New York: McGraw-Hill, Inc., 1956. 5. Harris, David P. "TOEFL: A Program for the Testing of English as a Foreign Language," in On Teaching English to Speakers of Other Languages. Champaign, 111.: National Council of Teachers of English, 1965, pp. 109-111. 6. Pool, Ithiel de Sola (ed.). Trends Jin Content Analysis. Urbana, 111.: University of Illinois Press, 1959. 7. Rankin, E. F., Jr. "The Cloze Procedure— Its Validity and Utility," in Starting and Improving College Reading Programs. Edited by Oscar S. Causey and William Eller. Fort Worth, Texas: Texas Christian University Press, 1959, pp. 131-144. 8. Sebeok, Thomas A. (ed.). Style in Language. New York: Massachusetts Institute of Technology and John Wiley and Sons, 1960. 141 142 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. Articles and Periodicals Bateman, David, Prandsen, Kenneth, and Dedmon, Donald. "Dimensions of 'Lecture Comprehension': A Factor Analysis of Listening Test Items," Journal of Communication, XIV (September, 1964), 183-189. Bonner, Myrtle Clara Stoddard. "A Critical Analysis of the Relationship of Reading Ability to Listening Ability," Dissertation Abstracts, XXI (1961), 2167- 2168. Brilhart, Barbara Lie!. "The Relationship Between Some Aspects of Communicative Speaking and Communi cative Listening," Journal of Communication, XV (March, 1965), 35-46. Brown, Charles T. "Three Studies of the Listening of Children," Speech Monographs, XXXII (June, 1965), 128-138. Dickens, Milton, and Williams, Pred. "An Experimental Application of 'Cloze' Procedure and Attitude Measures to Listening Comprehension," Speech Mono graphs , XXXI (June, 1964), 103-108. Diehl, Charles F., White, Richard C., and Burk, Kenneth W. "Rate and Communication," Speech Monographs, XXVI (August, 1959) , 229-232. Duker, Sam, and Petrie, Charles R. "What We Know About Listening: Continuation of a Controversy," Journal of Communication, XIV (December, 1964), 245-252. Garner, John E. "The Language Program for Foreign Officers at Air University," Journal of Communica tion, X (June, 1960), 87-94. Geiger, Don. "Speech and the Foreign Student Program," Western Speech, XVIII (January, 1954), 23-29. Haberland, John A. "A Comparison of Listening Tests with Standardized Tests," Journal of Educational 143 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. Research, LII (April, 1959), 299-302. Hvistendahl, J. K. "Language Ability as a Factor in 'Cloze' Scores," Journalism Quarterly, XXXV (Summer, 1958), 353-354. Johnson, F. Craig, and Frandsen, Kenneth. "Administer ing the Brown-Carlsen Listening Comprehension Test, " Journal of Communication, XIII (March, 1963), 38-45. Kaplan, Robert B., and Jones, Robert A. "Evaluation of Relative Foreign Student Success," Language Learning, XIV (1964), 161-167. Kelly, Charles M. "An Investigation of the Construct Validity of Two Commercially Published Listening Tests," Speech Monographs, XXXII (June, 1965), 139-143. ________, "Mental Ability and Personality Factors in Listening," Quarterly Journal of Speech, XLIX (April, 1963), 152-156. Key, Wilson Bryan, Jr. "Cloze Procedure: A Technique for Evaluating the Quality of Language Translation," Journal of Communication, IX (March, 1959), 14-18. Maner, W. Wallace. "More Exchange or Immigration," National Association for Foreign Student Affairs News Letter, XVI (May, 1965), 8-9. Nicho'ls, Alan C. "Apparent Factors Leading to Errors in Audition Made by Foreign Students," Speech Mono graphs , XXXI (March, 1964), 85-91. Nichols, Ralph G. "Factors in Listening Comprehen sion, " Speech Monographs, XV (1948), 154-163. Rose, Ervin. "A Comparative Study of the Brown- Carlsen Listening Comprehension Test and Three Tests of Reading Comprehension," Dissertation Abstracts, XIX (1959), 20007. 29. 30. 31. 32. 33. 34. 35. 36. 37. 144 Stark, Joel. "An Investigation of the Relationship of the Vocal and Communicative Aspects of Speech Competency with_Listening Comprehension, 1 1 Speech Monographs, XXIV (June, 1957), 98-99. Still, Dana S. "The Relationship Between Listening Ability and High School Grades," Dissertation Abstracts, XV (1955), 1761-1762. Symonds, Percival M. "Factors Influencing Test Reliability," Journal of Educational Psychology, XIX (February, 1928), 73-87. Taylor, Wilson L. "Cloze Procedure, A New Tool for Measuring Readability," Journalism Quarterly, XXX (Fall, 1953), 415-433. _______ . "Cloze Readability Scores as Indices of Individual Differences in Comprehension and Apti tude, " Journal of Applied Psychology, XLI (Febru ary, 1957), 12-26. _______ . "Recent Developments in the Use of Cloze Procedure," Journalism Quarterly, XXXIII (Winter, 1956), 42-48. Villareal, Jesse. "A Test of the Aural Comprehension of English for Native Speakers of Spanish," Speech Monographs, XV (March, 1948), 121-132. Weaver, Wendell W., and Kingston, Albert J. "A Factor Analysis of the Cloze Procedure and Other Measures of Reading and Language Ability," Journal of Communication, XIII (December, 1963), 252-261. Reports U. S. Senate. Foreign Assistance Act of 1962, Hear ings Before the Committee on Foreign Relations. 87th Congress, 2nd Session, 1962. 145 38. 39. 40. 41. 42. 43. 44. 45. 46. Unpublished Materials Bens, Allis R. "Criteria for Excellence in a Program in English as a Second Language. " Address given at the American Language Institute at California State College at San Francisco, May 22, 1965. (Mimeographed.) Darley, Frederic L. "A Normative Study of Oral Read ing Rate." Unpublished Master's thesis, State University of Iowa, 1940. Finkbeiner, Edith Joy. "A Study of the Relationship Between the Listening Ability and the Listening Performance of Ninth Grade Pupils. " Unpublished Master's thesis, University of Washington, 1962. Franke, Phyllis E. "A Preliminary Study Validating the Measurement of Oral Reading Rate in Words per Minute." Unpublished Master's thesis, State University of Iowa, 1939. Goodman-Maiamuth, Leo, II. "An Experimental Study of the Effects of Rate of Speaking upon Listenabil- ity." Unpublished Ph.D. dissertation, University of Southern California, 1956. Harwood, Kenneth A. "An Experimental Comparison of Listening Comprehensibility with Reading Compre hensibility. " Unpublished Ph.D. dissertation. University of Southern California, 1951. Petrie, Charles Robert, Jr. "A Plea for Caution," Newsletter of the Business and Professional Speak ing Interest Group. New York: Speech Assentation of America, August, 1962. (Mimeographed.) Steeg, Jacquelin. "An Exploratory Study of the Relationship Between Listening and Scholastic Achievement." Unpublished Master's thesis, Ohio University, 1960. Stromer, Walter F. "An Investigation into Sortie of the Relations Between Reading, Listening, and 146 Intelligence." Unpublished Ph.D. dissertation, University of Denver, 1952. 47. _______ . "Strength of Opinion, Auditory Memory Span, and Ability to Gather Meaning from Contextual Clues as Factors in Listening. 1 1 Unpublished Master's thesis, University of Denver, 1950. 48. Taylor, Wilson L. "Application of 'Cloze' and Entropy Measures to the Study of Contextual Constraint in - Samples of Continuous Prose." Unpublished Ph.D. dissertation, University of Illinois, 1954. 49. Weaver, Wendell W. "An Examination of Some Differ ence in Oral and Written Language Using the Cloze Procedure." Unpublished Ed.D. dissertation, University of Georgia, 1961. 50. Williams, Fred D. "An Experimental Application of the Semantic Differential and 'Cloze' Procedure as Measurement Techniques in Listening Comprehension." Unpublished Ph.D. dissertation, University of Southern California, 1962. Tests 51. Brown, James I. , and Carlsen, G. Robert. Brown- Carlsen Listening Comprehension Test. Yorikers-on- Hudson, N. Y.: World Book Co. , 1955. 52. California Test Bureau Manual. California Achieve ment Tests, Advanced Forms WXYZ. Los Angeles: California Test Bureau, 1957. 53. Lado, Robert. Test of Aural Comprehension: Exam iners' Materials, Descriptions and Norms. Ann Arbor, Mich.: University of Michigan, English Language Institute, 1957. 54. Sequential Tests of Educational Progress: Listening 1A, Listening IB. Princeton, N. J. : Educational Testing Bureau, 1957. 147 55. TOEFL, A Program for the Testing of English as a For eign Language. Washington, D. C. : TOEFL. Other Sources 56. Dickens, Milton C. Personal interview, University of Southern California, April 12, 1963. 57. Jones, Robert A. Personal interview, University of Southern California, April 18, 1963. 58. Kaplan, Robert B. Personal interview, University of Southern California, April 18, 20, and 21, 1963.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

An Empirical Study Of The Behavioral Characteristics Of Sincere And Insincere Speakers

PDF

An Experimental Study Of The Influence Of Subliminal Cue Words On Audience Responses To A Filmed Speaker'S Sincerity, Effectiveness, And Subject Matter

PDF

A Historical Study Of Speech Education At The University Of Southern California (1880 Through 1950)

PDF

An Experimental Study Of The Retention And Comprehension Of Poetry Resulting From Silent Reading And From Oral Interpretation

PDF

An Experimental Study Of Effects Of Interest And Authority Upon Understanding Of Broadcast Information

PDF

An Experimental Application Of The Semantic Differential And 'Cloze' Procedure As Measurement Techniques In Listening Comprehension

PDF

An Empirical Study Of Self-Feedback During Speech Communication

PDF

Content Analysis Of Fifty-Three High School Basic Speech Texts, 1899-1969

PDF

An Experimental Study Of The Effect Of Listener Feedback On Speaker Attitude

PDF

A Historical Study Of The Characteristics Of Acting During The Restoration Period In England (1660-1710)

PDF

Critical Study Of The Nominating Speeches At The Democratic And Republican National Conventions Of 1960

PDF

Teaching Operative Dental Technics Via Television: An Experimental Inquiry

PDF

An Experimental Study Of An Application Of Game Theory To The Selection Of Arguments By College Debaters

PDF

An Experimental Comparison Of Spoken Communication Developed Individually and Interindividually

PDF

An Analytical-Historical Study Of The Factors Contributing To The Successof Mark Twain As An Oral Interpreter

PDF

An Experimental Study Of The Accuracy Of Experienced And Inexperienced Speakers In Identifying Audience Behavior As Indicative Of Feelings Of Agreement, Indecision, Or Disagreement

PDF

An Experimental Study Of Factors Influencing The Perception Of Auditory Flutter

PDF

Becket's chameleon character: an analytical study of the universal appeal of Thomas Becket's dramatic character

PDF

A Historical Study Of The Speechmaking At The Abilene Christian College Lectureship, 1918-1961

PDF

A Survey And Analysis Of Current Attitudes Toward Censorship Of The Legitimate Theatre In The United States

Asset Metadata

Creator Gregory-Panopoulos, John Fred (author)

Core Title An Experimental Application Of 'Cloze' Procedure As A Diagnostic Test Of Listening Comprehension Among Foreign Students

Degree Doctor of Philosophy

Degree Program Speech

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag OAI-PMH Harvest,Speech Communication

Language English

Contributor Digitized by ProQuest (provenance)

Advisor Dickens, Milton (committee chair), Kaplan, Robert B. (committee member), McCoard, William B. (committee member)

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c18-226777

Unique identifier UC11360085

Identifier 6700404.pdf (filename),usctheses-c18-226777 (legacy record id)

Legacy Identifier 6700404.pdf

Dmrecord 226777

Document Type Dissertation

Rights Gregory-Panopoulos, John Fred

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the au...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus, Los Angeles, California 90089, USA