Critical thinking cognitive abilities and the analytical section of the educational testing service Graduate Record Examinations(R) General Test: A conceptual study in content validity

PDF

Download

Open document

Flip pages

Download a page range

Download transcript

Copy asset link

Request this asset

Request accessible transcript

Transcript (if available)

Content INFORMATION TO USERS
This manuscript has been reproduced from the microfilm master. UMI films
the text directly from the original or copy submitted. Thus, some thesis and
dissertation copies are in typewriter face, while others may be from any type of
computer printer.
The quality of this reproduction is dependent upon the quality of the
copy submitted. Broken or indistinct print, colored or poor quality illustrations
and photographs, print bleedthrough, substandard margins, and improper
alignment can adversely affect reproduction.
In the unlikely event that the author did not send UM I a complete manuscript
and there are missing pages, these w ill be noted. Also, if unauthorized
copyright material had to be removed, a note w ill indicate the deletion.
Oversize materials (e.g., maps, drawings, charts) are reproduced by
sectioning the original, beginning at the upper left-hand comer and continuing
from left to right in equal sections with small overlaps.
Photographs included in the original manuscript have been reproduced
xerographically in this copy. Higher quality 6” x 9’ black and white
photographic prints are available for any photographs or illustrations appearing
in this copy for an additional charge. Contact UM I directly to order.
ProQuest Information and Learning
300 North Zeeb Road, Ann Arbor, M l 48106-1346 USA
800-521-0600
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CRITICAL THINKING COGNITIVE ABILITIES
AND THE
ANALYTICAL SECTION OF THE EDUCATIONAL TESTING SERVICE
GRADUATE RECORD EXAMINATIONS® GENERAL TEST:
A CONCEPTUAL STUDY IN CONTENT VALLDTY
by
George Kailiwai III
A Dissertation Presented to the
FACULTY OF THE GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTORATE OF EDUCATION
December 2000
Copyright 2000 George Kailiwai III
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
UM I Number: 3041476
Copyright 2000 by
Kailiwai, George, III
All rights reserved.
_ ___ __®
UMI
UMI Microform 3041476
Copyright 2002 by ProQuest Information and Learning Company.
All rights reserved. This microform edition is protected against
unauthorized copying under Title 17, United States Code.
ProQuest Information and Learning Company
300 North Zeeb Road
P.O. Box 1346
Ann Arbor, Ml 48106-1346
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
UNIVERSITY OF SOUTHERN CALIFORNIA
School of Education
Los Angeles, California 90089-0031
This dissertation, written by
George Kailiwai III
under the direction of h -^^Dissertation Committee, and
approved by all members o f the Committee, has been
presented to and accepted by the Faculty o f the School
of Education in partiedfulfillment of the requirements for
the degree o f
D o c to r o f E d u catio n
November 8, 2000
Dissertation Committee
Chau
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
DEDICATION
This dissertation is dedicated to Debbie, my best friend, soul mate, and loving
wife; and to my son, GK, the “best buddy” a father could ever want. Their unending
love, support and patience have made this dissertation possible.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
ACKNOWLEDGMENTS
I wish to extend my deepest appreciation to my chairperson. Dr. Dennis
Hocevar, for his guidance, encouragement, and assistance in completing this
dissertation. I also wish to thank Dr. William Michael and Dr. Gretchen Guiton for
their insightful feedback and direction. All three have helped me realize my
intellectual potential.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
TABLE OF CONTENTS
DEDICATION ii
ACKNOWLEDGMENTS iii
LIST OF TABLES vi
ABSTRACT viii
CHAPTER
I THE PROBLEM I
Purpose 5
Importance of the Study 6
Research Questions 9
Delimitations 10
Limitations 11
II LITERATURE REVIEW 15
What is Critical Thinking? 17
The Philosophy School of Thought 20
The Psychology School of Thought 22
Reaching a Consensus 25
What is Critical Thinking Not? 45
What Does the ETS Say the Analytical Section
of the GRE General Test Measures? 46
What Do Ennis, Millman and Tomko Say the Cornell
Critical Thinking Test, Level Z Measures? 52
IE METHODOLOGY 56
Development of Critical Thinking Criteria 57
Assessment of Critical Thinking Definitions 57
Assessment and Comparison of Delphi Technique
Experiments 58
iv
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Chapter Page
Measuring Critical Thinking in the Analytical
Section of the GRE General Test 66
Measuring Critical Thinking in the Cornell
Critical Thinking Test, Level Z 90
Scoring Consistently 129
IV RESULTS AND DISCUSSION 132
Does the Analytical Section of the GRE Reasonably
Measure Critical Thinking? 148
V SUMMARY, CONCLUSION, AND RECOMMENDATION 158
Summary 158
Conclusion 161
Recommendation 163
REFERENCES 165
APPENDIX 172
v
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
LIST OF TABLES
Table
1
2
3
4
5
6
7
8
9
10
11
12
Page
Graduate Record Examinations (GRE) Minimum Test
Score Requirements for a Sample of University Doctorate
Schools of Education in Educational Leadership 3
Rank Order of Critical Thinking Definitions from Delphi
Technique Experiment 26
Rank Order of Critical Thinking Abilities from Delphi
Technique Experiment 27
Comparison of Stahl & Stahl (1991) Critical Thinking
Cognitive Skills with APA-Sponsored (Stahl N. N. & Stahl
R. J., 1991) Critical Thinking Study 62
Criteria for Measuring Critical Thinking 65
GRE Practice General Test Analytical Section No. 2 Using
Facione (1990) Criteria 134
GRE Practice General Test Analytical Section No. 5 Using
Facione (1990) Criteria 136
Summary of GRE Practice General Test Analytical
Sections 2 and 5 Using Facione (1990) Criteria 137
Analytical Section #2 Using Stahl N. N. and Stahl R. J.
(1991) Criteria 1 through 7 138
Analytical Section #2 Using Stahl N. N. and Stahl R. J.
(1991) Criteria 8 through 16 140
Analytical Section #5 Using Stahl N. N. and Stahl R. J.
(1991) Criteria I through 7 141
Analytical Section #5 Using Stahl N. N. and Stahl R. J.
(1991) Criteria 8 through 16 142
vi
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table Page
13 Summary of GRE Practice General Test Analytical
Sections 2 and 5 Using Stahl N. N. and Stahl R. J. (1991)
Criteria 1 through 7 143
14 Summary of GRE Practice General Test Analytical
Sections 2 and 5 Using Stahl N. N. and Stahl R. J. (1991)
Criteria 8 through 16 144
15 Cornell Critical Thinking Test, Level Z Using Facione
(1990) Criteria 145
16 Cornell Critical Thinking Test Using Stahl and Stahl (1991)
Criteria 1 through 7 149
17 Cornell Critical Thinking Test Using Stahl and Stahl (1991)
Criteria 8 through 16 151
18 Comparison of GRE Practice General Test Analytical
Sections with Cornell Critical Thinking Test, Level Z using
Facione (1990) Criteria 153
19 Comparison of GRE Practice General Test Analytical
Sections with Cornell Critical Thinking Test, Level Z using
Stahl N. N. and Stahl R. J. (1991) Criteria 1 through 7 153
20 Comparison of GRE Practice General Test Analytical
Sections with Cornell Critical Thinking Test, Level Z using
Stahl N. N. and Stahl R. J. (1991) Criteria 8 through 16 154
vii
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
ABSTRACT
Graduate school admissions officials use several factors in accepting their
applicants and assess variations between these determinants through the use of
normalized test scores from the Graduate Record Examinations (GRE) General Test.
These officials, however, typically use the scores from the quantitative and verbal
sections and ignore the analytical section test score of the GRE General Test. This
omission may be unfortunate because this section may measure another ability
universities have enthusiastically embraced as a central outcome of higher education-
-critical thinking. This conceptual study explores the content validity of the
analytical section of the 1997 GRE Practice General Test to determine whether it
reasonably measures critical thinking. Criteria are derived from two published
Delphi Technique experiments and are applied to the analytical section of the 1997
GRE Practice General Test and the Cornell Critical Thinking Test. Level Z. The
results show the analytical section met at least one critical thinking sub-skill for each
of the four Facione (1990) core critical thinking abilities— interpretation, analysis,
inference, and evaluation~and at least 70% of the items met 6 of the 11 Facione sub
skills. The analytical section also met 8 of the 16 Stahl N. N. and Stahl R. J. (1991)
core critical thinking skills. As one would expect, the Cornell Critical Thinking Test,
Level Z out-numbered the analytical section of the 1997 GRE Practice General Test
in 6 of the 11 Facione sub-skills and 7 of the 16 Stahl N. N. and Stahl R. J. critical
thinking skills. While the content validity of the analytical section of the GRE
viii
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
General Test suggests that it reasonably measures critical thinking, further empirical
research into the construct validity of critical thinking in the analytical section of the
GRE General Test is needed. Until then, the existing content validity of the
analytical section of the GRE General Test suggests graduate school admissions
officials should consider including the analytical section test scores of the GRE
General Test in their admissions decisions.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER I
THE PROBLEM
Graduate school admissions officials use several determinants for either
accepting or rejecting their applicants. Typically, these determinants include a
completed application form, official transcripts of all undergraduate and graduate
work, three letters of recommendation, a personal statement, and current test scores
(i.e., within 5 years of the application date) for the Graduate Record Examinations
(GRE) General Test or Miller Analogies Test (MAT). The significance of their
applicants’ undergraduate and graduate grade point averages (GPAs) may vary from
university to university, suggesting the same GPA from two universities may reflect
vastly different cognitive abilities. Letters of recommendation that address the
applicant’s professional experience and scholarly potential may differ in content,
style, accuracy and effectiveness. Likewise, personal statements discussing reasons
for pursuing higher education may vary widely from one applicant to another. To
help assess these variations, graduate school admissions officials (D. Robb, personal
communication, September 21, 1995) use normalized test scores such as the scores
from the GRE General Test to assist them in differentiating the cognitive abilities of
their applicants.
Graduate school admissions officials (D. Robb, personal communication,
September 21, 1995), however, typically use the cumulative score from the
quantitative and verbal sections of the GRE General Test in their selection process,
I
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
and typically ignore the analytical test score of the GRE General Test. Some
universities also accept minimum scores from the Miller Analogies Test (MAT) in
lieu of GRE General Test scores. The MAT is a test of 100 analogies, given in a
timed, 50 minute test period. The Psychological Corporation (2000), a Harcourt
Assessment Company, develops the MAT. Table 1 lists the test scores for 10
university doctorate schools of education in educational leadership (Clemson
University College of Health, Education, and Human Development Department of
Educational Leadership and Counseling, 1997; East Carolina University School of
Education Department of Educational Leadership, 2000; East Tennessee State
University College of Education Department of Educational Leadership and Policy
Analysis, 1999; Florida State University Department of Educational Leadership,
2000; Idaho State University College of Education, 2000; University of Georgia,
Athens College of Education, 2000; University of Houston College of Education
Educational Leadership and Cultural Studies, 1994; University of Nevada Reno
College of Education Department of Educational Leadership, 2000; University of
South Carolina College of Education Educational Leadership and Policies, 1999).
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 1
Graduate Record Examinations (GRE) Minimum Test Score Requirements for a
Sample of University Doctorate Schools of Education in Educational Leadership
University GRE Test Score Admissions Requirements
Clemson University “GRE with a minimum of 1450 on three sections, including
recommended minimum scores of 500 on the verbal and
quantitative sections.”
East Carolina “An acceptable GRE (at least 1000) or MAT (at least 50)
University score is required.”
East Tennessee State “Combined score of 1000 or higher on the verbal and
University quantitative portions of the GRE.”
Florida State
University
“A minimum score of 850 on the GRE.”
Idaho State “A minimum combined score of 1000 on the Verbal and
University Quantitative sections of the Graduate Record
Examinations.”
University of “Score of 950 or higher on the combined verbal and
Georgia, Athens quantitative sections of the GRE.”
University of “Interview required to determine admission if any indicator
Houston falls below these scores:
GPA (for last 60 hrs.)--3.25
Verbal G RE-500
Quantitative GRE--515
Analytical GRE— 530
Automatic denial if two or more indicators fall below these
scores:
GPA (for last 60 hrs.)— 3.00
Verbal G RE-450
Quantitative GRE— 475
Analytical G RE-500”
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table I (continued)
University GRE Test Score Admissions Requirements
University of “Graduate Record Examinations (GRE) scores.
Nevada, Reno Educational Leadership requires test scores on the verbal
and quantitative for all applicants.”
University of South “Submission of aptitude test scores (1000 on GRE or 50 on
Carolina Millers Analogies Test preferred).”
University of Minimum cumulative GRE verbal and quantitative test
Southern California scores of 1000.
Table 1 shows that 8 of the 10 doctorate schools of education in educational
leadership use a cumulative verbal and quantitative test score in their admissions
decisions and ignore the analytical section test score of the GRE General Test. This
omission of the analytical test score in graduate school admissions decisions may be
unfortunate because the analytical section of the GRE General Test may measure
another ability universities have enthusiastically embraced as a central outcome of
higher education-critical thinking (Bransford, Sherwood, & Sturdevant, 1987;
Carpenter & Doig, 1988; Chaffee, 1992; Cromwell, 1992; Ennis, 1987; Ennis, 1993;
Facione, P. A., Sanchez, Facione, N. C., & Gainen, 1995; Paul, 1992; Paul, 1995h;
Paul & Binker, 1995a). Therefore, if critical thinking is an outcome of higher
education, and if the analytical section of the GRE General Test measures critical
thinking abilities, then graduate school admissions officials should not ignore the
significance of the analytical test score in their admissions decisions. The problem is
4
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
determining whether the analytical section of the GRE General Test reasonably
measures critical thinking.
Purpose
The purpose of this conceptual study is to determine whether the analytical
section of the GRE General Test reasonably measures critical thinking. To achieve
this purpose, this study has three goals. The first goal is to answer the question,
“What is critical thinking?” Several experts in the field of critical thinking have
proposed critical thinking definitions. Is there a consensus of what is critical
thinking? Furthermore, what is critical thinking not? For example, how does critical
thinking differ from creative thinking and problem solving? What are the cognitive
skills associated with critical thinking?
The second goal is to develop a set of criteria for measuring critical thinking.
The findings from the first goal, particularly evaluating what critical thinking experts
have identified as critical thinking cognitive skills are crucial in establishing these
criteria. I assess the critical thinking content validity of the analytical section of the
1997 GRE Practice General Test (ETS, 1997) by applying these criteria to each of
the items.
The third goal is to determine whether the analytical section of the GRE
General Test reasonably measures critical thinking. The first step in accomplishing
this goal is to apply the same criteria I developed from the second goal to a well-
accepted commercially available critical thinking test— the Cornell Critical Thinking
5
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Test, Level Z (Ennis, Millman, & Tomko, 1985b). The second step is to compare the
results from the two instruments’ scores for each critical thinking criterion. Finally, I
evaluate the analytical section of the GRE General Test’s ability to reasonably
measure critical thinking.
Importance of the Study
If the analytical section of the GRE General Test reasonably measures critical
thinking, university leaders and admissions officials may have a basis for changing
their graduate school admissions policies. This change may include adding a
minimum GRE analytical test score to minimum GRE verbal and quantitative test
scores. Table I shows that 2 of the 10 doctorate schools of education in educational
leadership include a minimum analytical section test score of the GRE General Test
in their admissions decisions. The Department of Educational Leadership and
Counseling at Clemson University (Clemson University College of Health,
Education, and Human Development Department of Educational Leadership and
Counseling, 1997) requires its doctoral applicants in educational leadership have a
minimum of 1450 on all three sections of the GRE General Test, including
recommended minimum scores of 500 on both the verbal and quantitative sections.
The Department of Educational Leadership and Cultural Studies at the University of
Houston (University of Houston College of Education Educational Leadership and
Cultural Studies, 1994) requires its doctoral applicants in educational leadership have
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
minimum verbal, quantitative, and analytical GRE test scores. Failing to meet any
criterion may require an interview to gain entry or result in an automatic denial.
Admissions officials may also consider emphasizing the analytical test score
over both the verbal and quantitative test scores. For example, the University of
Houston requires test scores of 530,515, and 500 for the analytical, quantitative, and
verbal sections, respectively. Admissions officials may be motivated to emphasize
their applicants’ analytical test scores of the GRE because of the test scores’
relationship with critical thinking and the importance university officials have placed
on critical thinking as a desired outcome of higher education.
The National Center for Education Statistics (Greenwood, 1992) conducted a
series of workshops to identify and assess the outcomes of higher education in the
United States. A central finding of these studies was the college graduate’s ability to
think critically and communicate effectively (Baron, 1987; Bransford et al„ 1987;
Carpenter & Doig, 1988; Chaffee, 1992; Cromwell, 1992; deBono, 1976; Halpem,
1992; Lewis & Smith, 1993; Lipman, 1991; Nickerson, 1987; Paul, I995d; Paul,
1995e; Paul, 1995f; Paul, I995h; Quellmalz, 1985). Several universities themselves
emphasize the importance of critical thinking as a desired outcome of their
institutions.
Harry R. Lewis (1997, February 23), Dean of Harvard University encourages
his students to rejoice in discovery and critical thought as an end to achieve the
mission of Harvard University. Like Harvard University, the University of Southern
7
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
California (USC) has also identified critical thinking as a desired outcome. The USC
Learning Center is the location of the central resource and referral agency for
learning enrichment. The Learning Center has two departments--the Department of
Academic Services (DAS) and the Department of Learning Services (DLS). The goal
of the DAS and DLS (University of Southern California [USC], 2000) is to facilitate
independent learning, critical thinking, integrative understanding and responsible
action.
USC also conducts several classes whose major focus is in critical thinking.
For example, Communication 201 places emphasis on research procedures, critical
thinking, and argumentative thinking. Communication 523 stresses critical thinking
in its classroom. In Political Science 371, the instructor promotes discussion and
critical thinking about all the facets of European political thought. Philosophy 130 is
Legal Issues and Critical Thinking. This course discusses the development of critical
thinking and reasoning skills by examination and evaluation of arguments on both
sides of controversial legal issues. The course title for Philosophy 150 is Reasoning
and Critical Thinking. This course seeks to develop each student's skill in
understanding and using carefully constructed arguments. The prerequisite for
International Relations 210 is Analytical Exercise One: The Research Puzzle: Basic
Tools and Critical Thinking. The School of Accounting emphasizes critical thinking
skills, interpersonal skills, and group interaction. The Molecular Microbiology and
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Immunology Department claims its desire is to foster independent and critical
thinking.
Critical thinking is also pervasive on the USC campus in other areas. For
instance, the Office of Student Conduct typically requires a student to attend a
critical thinking class or write a critical thinking reflective paper as a remedy to a
nonacademic violation. The vision of the USC library system is to create leaders for
the 21st century by enabling students to focus their critical thinking.
The process developed in this study is also important because the researcher
could apply it to other instruments. These instruments include the verbal and
quantitative sections of the GRE General Test as well as other normalized tests such
as the Miller Analogies Test (MAT), Medical College Assessment Test (MCAT),
Law School Admission Test (LSAT), and the Graduate Management Assessment
Test (GMAT). Applying this process to other instruments may help the researcher
determine whether they also reasonably measure critical thinking.
Research Questions
This study answers the following research questions:
1. What is critical thinking?
2. How does the philosophy school of thought view critical thinking?
3. How does the psychology school of thought view critical thinking?
4. Is there a consensus of what critical thinking is?
5. How does critical thinking compare to creative thinking?
*
9
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
6. How does critical thinking compare to problem solving?
7. What role does critical thinking play in problem solving?
8. What are some of the cognitive abilities associated with critical
thinking?
9. What set of criteria reasonably measures critical thinking?
10. What does the ETS say the analytical section of the GRE General Test
measures?
11. What do Ennis, Millman and Tomko (1985b) say the Cornell Critical
Thinking Test, Level Z measures?
Delimitations
This conceptual study has two delimitations. First, I will delimit this study to
the analytical section of the 1997 GRE Practice General Test. ETS (1997) states this
practice test is intended to help the examinee become familiar with the paper-based
versions of the General Test and contain many of the kinds of questions that are
included in currently used forms of the General Test. The computer-based versions
of the GRE General Test are not within the scope of this study. Second, I will
compare the analytical section test items from the GRE Practice General Test to one
critical thinking test-the Cornell Critical Thinking Test, Level Z. Other university
and college level critical thinking appraisals include the Watson-Glaser Critical
Thinking Appraisal, Form A (Watson & Glaser, 1980b), the Ennis-Weir Critical
Thinking Essay Test, and the New Jersey Test of Reasoning Skills (Ennis & Norris,
10
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
1989). I will not use the Watson-Glaser Critical Thinking Appraisal, Form A
primarily because the 16 test questions in the inference section assume the examinee
possesses “certain commonly accepted knowledge or information that practically
every person has” (Watson & Glaser, 1980b, p. 2). Several of these inference items
appear to be culturally biased. I will not use the Ennis-Weir Critical Thinking Essay
Test because of the absence of multiple-choice test items. Last, I will not use the
New Jersey Test of Reasoning Skills because of the difficulty in obtaining the test
and its test manual.
Limitations
This study has three significant limitations. Many experts in the field of
critical thinking (Ennis, 1987; Facione, 1990; Facione, 1995; Jones, Hoffman,
Moore, Ratcliff, Tibbetts, & Click, 1995; Paul & Nosich, 1995, p. 127) divide
critical thinking into cognitive abilities and dispositions. Dispositions differ from
cognitive abilities in that they are behavioral tendencies or affective traits of mind
that concern how one is inclined to use cognitive abilities (Jones et al., 1995, p 161).
Dispositions include curiosity, perseverance, open-mindedness, fair-mindedness,
flexibility, honesty in recognizing one’s personal prejudices, intellectual humility,
developing insight into egocentricity, and willingness to reconsider, to name a few
(Facione, 1990; Jones et al., 1995). This study’s instruments (i.e., the analytical
section of the 1997 GRE Practice General Test and the Cornell Critical Thinking
11
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Test, Level Z) do not measure critical thinking dispositions and are not included in
this study.
The second limitation is that Facione (1990), Jones, Hoffman, Moore,
Ratcliff, Tibbetts, and Click (1995), and Paul and Nosich (L995) include listening,
writing and speaking critically as critical thinking cognitive abilities. Since the GRE
General Test and the Cornell Critical Thinking Test, Level Z are multiple-choice
instruments, they do not measure ones abilities to listen critically, write critically,
and speak critically, and thus are beyond the scope of this study.
The Final limitation centers on the concept of content validity. Since the early
1950s (Messick, 1989, p. 16), test validity has been broken into five categories—
content validity, criterion-related validity, predictive validity, concurrent validity,
and construct validity. The 1954 Technical Recommendations for Psychological
Tests and Diagnostic Techniques and the 1966 Standards for Educational and
Psychological Tests and Manuals (as cited in Messick) defined the five categories of
validity as follows:
Content validity is evaluated by showing how well the
content of the test samples the class of situations or subject matter
about which conclusions are to be drawn.
Criterion-related validity is evaluated by comparing the test
scores with one or more external variables (called criteria)
considered to provide a direct measure of the characteristic or
behavior in question.
Predictive validity indicates the extent to which an
individual’s future level on the criterion is predicted from prior test
performance.
Concurrent validity indicates the extent to which the test
scores estimate an individual’s present standing on the criterion.
12
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Construct validity is evaluated by investigating what qualities
a test measures, that is, by determining the degree to which certain
explanatory concepts or constructs account for the performance on
the test, (as cited in Messick, 1989, p. 16)
In this study, I do not compare GRE analytical test scores with Cornell
Critical Thinking Test, Level Z test scores, thus criterion-related validity, predictive
validity, and concurrent validity do not apply, leaving content validity and construct
validity as the two remaining sources of validity. Messick (1989) claims that
construct validity subsumes content relevance and representativeness as well as
criterion-relatedness, because such information about the content domain of
reference and about specific criterion behaviors predicted by the test scores are
relevant to an appraisal of an instrument’s construct validity. Since there are no test
scores, this study is limited to comparing the content validity of the analytical section
of the GRE Practice General Test and the Cornell Critical Thinking Test, Level Z
against criteria established through expert opinion on the domain of interest, that is,
critical thinking. The limitation is that Messick (1989, p. 17) states construct validity
embraces all forms of validity to which empirical evidence and theoretical rationales
support the adequacy and appropriateness of inferences and actions based on test
scores or other modes of assessment. “In this generalized sense, construct validity
may ultimately be taken as the whole of validity in the final analysis” (Messick,
1989, p. 21). Thus, for the purposes of applied decision making, reliance on content
validity alone is not enough (Messick. 1989, p. 17).
13
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Messick’s (1989) assertion that reliance on content validity alone is not
enough for the purposes of applied decision making should not lessen the importance
of this study. Facione and McPeck (as cited in Kurfiss, 1988, p. 8) conducted a study
that revealed high correlations between the Watson-Glaser Critical Thinking
Appraisal scores and measures of intelligence and reading comprehension. The
correlations were 0.46 and 0.77, respectively. Likewise, Ennis et al. (1985) found
their Cornell Critical Thinking Test, Level Z correlated with scholastic aptitude tests
around 0.5 with the actual range being 0.36 to 0.71. Kurfiss (1988) claims this is the
problem that plagues tests of critical thinking to this day— the high correlation
between critical thinking tests and intelligence quotient (IQ), reading comprehension,
and scholastic aptitude tests suggests all these tests are in reality measuring a single,
general factor of intelligence, what some theorists like Spearman, Jensen, and
Hermstein call “g” (as cited in Sternberg, 1996). Conducting a content validity
analysis on the analytical section of the GRE General Test and the Cornell Critical
Thinking Test, Level Z and comparing the findings from each instrument may reveal
how well the content of the items sample the content of the domain of interest (i.e.,
critical thinking)— a feature the collection of test scores as empirical data in construct
validation may miss.
14
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER II
LITERATURE REVIEW
This chapter summarizes the results of a literature review of critical thinking,
the GRE analytical section, and the Cornell Critical Thinking Test, Level Z. This
chapter seeks to answer the following research questions: “What is critical
thinking?” “How does the philosophy school of thought view critical thinking?”
“How does the psychology school of thought view critical thinking?” “Is there a
consensus of what critical thinking is? “How does critical thinking compare to
creative thinking?” “How does critical thinking compare to problem solving?”
“What role does critical thinking play in problem solving?” “What are the cognitive
abilities associated with critical thinking?” “What does the ETS say the analytical
section of the GRE measures?” and “What do Ennis, Millman and Tomko (1985b)
say the Cornell Critical Thinking Test, Level Z measures?”
I answer the research question, “Is there a consensus of what critical thinking
is?” by presenting the results of three Delphi Technique critical thinking experiments
(Facione, 1990; Jones et al., 1995; Stahl, N. N. & Stahl, R. J., 1991).
The goal of Facione’s (1990) Delphi Technique experiment was to determine
the role of critical thinking in educational assessment and instruction. In meeting this
goal, 46 scholars, educators, and critical thinking experts addressed the cognitive
abilities dimension of critical thinking, the dispositions of critical thinking, and
15
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
specific recommendations on critical thinking instruction and assessment. The
cognitive abilities and dispositions of critical thinking are the focus of this study.
Jones et al. (1995) also conducted a Delphi Technique experiment with 600
faculty, employers, and policy makers to identify writing, speech and listening skills,
and critical thinking cognitive abilities and dispositions that college graduates should
achieve to become effective employees and citizens. This group of 600 faculty,
employers, and policy makers began their Delphi Technique experiment with the
findings from Facione (1990). This Delphi Technique experiment confirmed the
findings from the 1990 Facione study and only suggested name changes to some of
Facione’s critical thinking sub-skills. As with the Jones et al. study, the Facione
critical thinking cognitive abilities are the focus of this study.
The goal of the Stahl N. N. and Stahl R. J. Delphi Technique experiment
(1991) was to reach consensus on a set of definitions for critical thinking and
identify critical thinking cognitive abilities that could serve as criteria for a gifted
education program. I use the findings from the Facione (1990) and Stahl N. N. and
Stahl R. J. studies to develop criteria for measuring the content validity of critical
thinking in the analytical section of the 1997 GRE Practice General Test (ETS, 1997)
and the Cornell Critical Thinking Test, Level Z (Ennis et al., 1985b). These criteria
are the subject discussed in the following chapter.
16
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
What is Critical Thinking?
The contribution of critical thinking extends from the time of Socrates
(Public Broadcasting System [PBS], 2000b; 2000c). Socrates was the most renowned
philosopher of Classical Greece whose notoriety extends from his questioning
teaching method and dogged search for the truth (Paul & Binker, 1995b). Socrates
(PBS, 2000a) and later Plato, would always attempt to tease the truth out of people
through a series of questions such as “What is the evidence?” and “If this is true does
it not follow that certain other matters are true?” Since the time of Socrates, critical
thinking has been used as a moral force to promote “good” and guard against the
propensities of humans to accept fallacious arguments (i.e., sophistic logic) and draw
inappropriate conclusions (Lewis & Smith, 1993).
Paul (1995i) also traces the deepest intellectual roots of the critical thinking
movement 2,400 years ago through the vision and teaching practice of Socrates.
Socrates discovered that through a probing method of questioning the many
authorities of his day, he could not justify on rational grounds their confident claims
to knowledge. Socrates’ insights may have been variously articulated by a scattering
of intellectuals in the last three centuries: Voltaire, John Henry Newman, John Stuart
Mill, and William Graham Sumner, to name a few (Paul, 1985). Similarly, Facione
et al. (1995) tie critical thinking to the eighteenth century Enlightenment, the
Renaissance, the medieval focus on logical argumentation, and the North African
and Roman preparation of jurists and lawyers.
17
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Hawes (1990) and Kurfiss (1988), list John Dewey as the first modem
forerunner of the critical thinking movement. John Dewey in How We Think (as
cited in Hawes, 1990) described what he called “reflective thinking” as a condition
and a method. According to Dewey (as cited in Hawes), reflective thinking is both a
state of doubt, hesitation, perplexity, and mental difficulty in which thinking
originates, and an act of searching, hunting, inquiring, and finding material that will
resolve the state of doubt, hesitation, perplexity, and mental difficulty. This purpose-
oriented method includes concepts such as problem definition, elaboration of
alternatives into hypotheses, reasoning, using observations and available knowledge,
and testing hypotheses. In this sense, reflective thinking incorporates reason and
logic as essential components of problem solving.
Kurfiss (1988) lists Goodwin Watson and Edward M. Glaser as two
forerunners of the critical thinking movement who helped to further define critical
thinking through their Watson-Glaser Critical Thinking Appraisal (Watson & Glaser,
1964a, 1964b, 1980a, 1980b). Watson and Glaser (1980a) viewed critical thinking as
a composite of attitudes, knowledge, and skills. This composite includes attitudes of
inquiry, knowledge of the nature of valid inferences, abstractions, and
generalizations, and skills in employing and applying the above attitudes and
knowledge. Their appraisal contains five subtests. These subtests include inference
(i.e., discriminating among degrees of truth or falsity of inferences drawn from given
data); recognition of assumptions (i.e., recognizing unstated assumptions or
18
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
presuppositions in given statements or assertions; deduction (i.e., determining
whether certain conclusion necessarily follow from information in given statements
or premises); interpretation (i.e., weighing evidence and deciding if generalizations
or conclusions based on the given data are warranted); and evaluation of arguments
(i.e., distinguishing between arguments that are strong and relevant and those that are
weak or irrelevant to a particular question at issue).
Another early and influential leader in the critical thinking movement was
Robert H. Ennis. Ennis (1962, p. 83) developed a concept of critical thinking as “the
correct assessing of statements.” Equally important in his work (1962, p. 84) was his
delineation of 12 critical thinking aspects:
1. Grasping the meaning of statements.
2. Judging whether there is ambiguity in a line of reasoning.
3. Judging whether certain statements contradict each other.
4. Judging whether a conclusion necessarily follows.
5. Judging whether a statement is specific enough.
6. Judging whether a statement is actually the application of a
certain principle.
7. Judging whether an observation statement is reliable.
8. Judging whether an inductive conclusion is warranted.
9. Judging whether the problem has been identified.
10. Judging whether something is an assumption.
19
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
11. Judging whether a definition is adequate.
12. Judging whether a statement made by an alleged authority is
acceptable.
The literature review reveals that for the most part, critical thinking depends
on one's notion of the reasons for critical thought and what it means to think
critically (Cromwell, 1992, p. 38). Lewis and Smith (1993) and Cromwell (1992)
link the reasons for critical thought to philosophy and what it means to think
critically to psychology. The two disciplines are very different, as they have diverse
views regarding such fundamental ideas as the nature of “truth,” that is, how it is
defined and how it is found. Stated another way, philosophers explore the question,
“Why is critical thinking important?” whereas psychologists view critical thinking as
a cognitive ability (Paul, 1995c, I995g). Thus, the definition of critical thinking
depends upon whom answers the question, “What is critical thinking?” These
definitions range from “reasonable, reflective thinking that is focused on deciding
what to believe or do” (Norris & Ennis, 1989, p. 1) (i.e., the philosophy school of
thought) to the “correct assessment of statements” (Ennis, 1962, p. 81) (i.e., the
psychology school of thought).
The Philosophy School of Thought
Philosophers generally define critical thinking as disciplined, self-directed
thinking which exemplifies the perfection of thinking appropriate to a particular
mode or domain of thinking (Paul, 1990). Similarly, Norris and Ennis (1989, p. I)
20
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
define critical thinking as “reasonable and reflective thinking that is focused upon
deciding what to believe or do.” In this line of reasoning, Norris and Ennis (1989)
first define critical thinking as “reasonable” thinking. This means that it is good
thinking that relies appropriately upon the use of good reasons. People who form
beliefs or act without good reasons are acting arbitrarily and unreasonably. Good
thinking is not arbitrary because good thinking does not lead to just any conclusions,
good thinking leads to the best conclusions.
Second, Norris and Ennis (1989) define critical thinking as “reflective”
thinking. Critical thinkers must be reflective in that they examine the reasonableness
of their own and others’ thinking. Thinking does not become reasonable through
chance alone. Rather, critical thinking comes about consciously. Third, Norris and
Ennis define critical thinking as being focused. This definition suggests critical
thinking is consciously directed, that is, it is thinking with a purpose and does not
occur accidentally or without reason. Finally, Norris and Ennis claim critical
thinking is focused on a decision about what to believe or do. This characteristic
indicates that critical thinking can evaluate statements (i.e., what we believe) and
actions (i.e., what we do).
Paul and Nosich (1995) assert that a domain of critical thinking is the ability
to compare and evaluate perspectives (i.e., a cognitive ability) and to do so with fair-
mindedness (i.e., a disposition). The philosopher would judge whether that person
had made such evaluations in a relevant and consistent way, with attention to
21
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
accuracy, fairness, and completeness in describing each perspective, and with
sensitivity to the degree of precision appropriate to the topic. The philosopher would
further assess critical thinking about and in terms of the elements of thought, and
judge a person's skill at recognizing the frame of reference underlying a position,
whether that person could see relevant alternatives, whether the frame of reference
that individual identified fits the available evidence, or whether the answer was deep
or merely mechanical, clear or vague, fair or biased (Paul and Nosich, 1995, p. 131).
Finally, in Paul’s (1995b, p. 91) most recent definition of critical thinking, he
defines critical thinking as “thinking about your thinking while you're thinking in
order to make your thinking better.” There are two crucial characteristics: (a) critical
thinking is not just thinking, but thinking which entails self-improvement; and (b)
this improvement comes from skill in using standards by which one appropriately
assesses thinking. Thus, an individual achieves self-improvement through critical
thinking.
The Psychology School of Thought
Whereas the field of philosophy has grown through discourse and
argumentation, the field of psychology has evolved from a tradition of
experimentation and research (Lewis & Smith, 1993). Unlike philosophers,
psychologists are more concerned with the thinking process and how this process can
help people make sense out of their experience by constructing meaning and
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
imposing structure. Central to constructing meaning and imposing structure are other
well-accepted definitions of critical thinking from the school of psychology.
The Dictionary of Education (Good, 1973, p. 608) defines critical thinking as
thinking that proceeds on the basis of careful evaluation of premises and evidence
and comes to conclusions as objectively as possible through the consideration of all
pertinent factors and the use of valid procedures from logic. Although logic has its
basis in philosophy, psychologists also view the proper use of logic as a cognitive
skill. Similarly, Miller and Connelly (1997) directly associate critical thinking with
informal logic. They claim critical thinking is the application of logical concepts to
the analysis of everyday reasoning and problem solving (i.e., informal logic).
Guilford (1977, p. 149) states critical thinking “can prevent us from being gullible,
from believing everything we hear or read.” It involves checking on the information
we have however it came about, and making decisions or rendering judgments
regarding it. It is a way of telling us whether we are on the right track and should
continue or whether we are wrong and could do better (Guilford, 1977).
Typical critical thinking definitions are presented below. These definitions
provide examples from both schools of thought.
Critical thinking is the correct assessment of statements
(Ennis, 1962).
Critical thinking is thinking that proceeds on the basis of
careful evaluation of premises and evidence and comes to
conclusions as objectively as possible through the consideration of
all pertinent factors and the use of valid procedures from logic
(Good, 1973).
23
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Critical thinking is a composite of attitudes, knowledge, and
skills. This composite includes: (a) attitudes of inquiry that involve
an ability to recognize the existence of problems and an acceptance
of the general need for evidence in support of what is asserted to be
true; (b) knowledge of the nature of valid inferences, abstractions,
and generalizations in which the weight or accuracy of different
kinds of evidence are logically determined; and (c) skills in
employing and applying the above attitudes and knowledge (Watson
& Glaser, 1980a).
Critical thinking is concerned with decisions about the
goodness of items of information, checking on the information that
we have, however it came about, and making decisions or rendering
judgments regarding it (Guilford, 1977).
Critical thinking is deciding rationally what to do or what not
to believe (Norris, 1989).
Critical thinking is thinking about your thinking while you’re
thinking in order to make your thinking better. Two things are
crucial: (a) critical thinking is not just thinking, but thinking which
entails self-improvement; and (b) this improvement comes from skill
in using standards by which one appropriately assesses thinking. To
put it briefly, it is self-improvement (in thinking) through standards
that assess thinking (Paul, 1995b).
Critical thinking is reasonable, reflective thinking that is
focused on deciding what to believe or do (Norris & Ennis, 1989).
Critical thinking is the process of purposeful, self-regulatory
judgment which results in interpretation, analysis, evaluation, and
inference, as well as explanation of the evidential, conceptual,
methodological, criteriological, or contextual considerations upon
which that judgment is based (Facione, 1990).
Critical thinking is the development of cohesive and logical
reasoning patterns (Stahl, N. N., & Stahl, R. J., 1991).
The purpose of critical thinking is, therefore, to achieve
understanding, evaluate viewpoints, and solve problems. Since all
three areas involve the asking of questions, we can say that critical
thinking is the questioning or inquiry we engage in when we seek to
understand, evaluate, or resolve (Maiorana, 1992, as cited in Fowler,
1997).
Critical thinking is careful and deliberate determination of
whether to accept, reject, or suspend judgment (Moore & Parker,
1994, as cited in Fowler, 1997).
Broadly speaking, critical thinking is concerned with reason,
intellectual honesty, and open-mindedness, as opposed to
24
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
emotionalism, intellectual laziness, and closed-mindedness. Thus,
critical thinking involves: following evidence where it leads,
considering all possibilities, relying on reason rather than emotion,
being precise, considering a variety of possible viewpoints and
explanations, weighing the effects of motives and biases, being
concerned more with finding the truth than being right, not rejecting
unpopular views out of hand, being aware of one’s own prejudices
and biases, and not allowing them to sway one’s judgment (Kurland,
1995, as cited in Fowler, 1997).
Critical thinking is the use of those cognitive skills or
strategies that increase the probability of a desirable outcome. It is
used to describe thinking that is purposeful, reasoned and goal
directed--the kind of thinking involved in solving problems,
formulating inferences, calculating likelihoods, and making
decisions when the thinker is using skills that are thoughtful and
effective for the particular context and type of thinking task. Critical
thinking also involves evaluating the thinking process— the reasoning
that went into the conclusion we’ve arrived and the kinds of factors
considered in making a decision. Critical thinking is sometimes
called directed thinking because it focuses on a desired outcome
(Halpem, 1996, as cited in Fowler, 1997).
Critical thinking is the application of logical concepts to the
analysis of everyday reasoning and problem solving (Miller &
Connelly, 1997).
Reaching a Consensus
Stahl N. N. and Stahl R. J. (1991) conducted a Delphi Technique experiment
whose focus was determining the criteria for selecting and implementing a critical
thinking program for their school district’s gifted education program. A set of critical
thinking definitions and a list of specific critical thinking abilities from the literature
were generated as the basis for the criteria. Stahl N. N. and Stahl R. J. then applied
these criteria to four nationally known critical thinking programs to determine which
program would meet the most criteria. According to Stahl N. N. and Stahl R. J.. 40
teachers and 24 administrators would be directly responsible for implementing the
25
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
gifted education critical thinking program. Of these 64 teachers and administrators,
16 volunteered for this experiment. Fourteen teachers and administrators participated
in the experiment to its completion.
The panelists rank ordered 10 critical thinking definitions and 25 critical
thinking abilities in order of importance to their gifted education program. After
three iterations, the panel members reduced the 10 critical thinking definitions to 7
and the 25 critical thinking abilities to 16. Table 2 lists the rank order of the panel's
critical thinking definitions, and Table 3 lists the panels 16 most important critical
thinking abilities.
Table 2
Rank Order of Critical Thinking Definitions from Delphi Technique Experiment
Rank_____________________ Critical Thinking Definition___________________
1 Critical thinking is to develop cohesive, logical reasoning patterns
and understanding assumptions and biases underlying particular
positions.
2 Critical thinking is the process of determining the authenticity,
accuracy and worth of information or knowledge claims.
3 Critical thinking is the ability to reach sound conclusions based on
observations and information.
4 Critical thinking is taken to be the correct assessment of statements as
well as reflective and reasonable thinking that is focused on deciding
what to believe or do.
5 Critical thinking is that aspect of thinking that can be discursively
formulated, can be subjected to evaluation criteria, and can be taught.
26
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 2 (continued)
Rank_____________________ Critical Thinking Definition________________
6 Critical thinking is rationally deciding what to do or believe.
7 Critical thinking is an attitude of being disposed to consider in a
thoughtful, perceptive manner the problems and subjects that come
__________ within the range of one’s experiences._________________________
Table 3
Rank Order of Critical Thinking Abilities from Delphi Technique Experiment
Rank Critical Thinking Abilities
I Critical thinking distinguishes between verifiable facts and value
claims.
2
Critical thinking determines the accuracy of a statement.
3 Critical thinking seeks a clear statement of the issue.
4 Critical thinking tries to separate emotional thinking from logical
thinking.
5 Critical thinking detects bias.
6 Critical thinking recognizes logical inconsistencies in a line of
reasoning.
7 Critical thinking identifies stated and unstated assumptions.
8 Critical thinking distinguishes between warranted and unwarranted
claims.
9 Critical thinking judges whether there is ambiguity in a line of
reasoning.
10 Critical thinking judges whether certain statements contradict each
other.
27
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 3 (continued)
Rank Critical Thinking Abilities
11 Critical thinking determines the strength of an argument.
12 Critical thinking judges whether the problem has been identified.
13 Critical thinking determines the reliability of a claim or source.
14 Critical thinking takes a position when the evidence and reasons are
sufficient to do so.
15 Critical thinking keeps in mind the original and/or basic concern.
16 Critical thinking understands the difference between a conclusion
which might be true and one which must be true.
A chi-square analysis on the results of the Delphi Technique experiment
showed that agreement or consensus on the rank order of the 7 critical thinking
definitions was statistically significant (Stahl, N. N. & Stahl, R. J., 1991). The
statistical data, however, did not result in statistical significance for the highest
ranked definition of critical thinking. Instead, the largest break between composite
rankings occurred between the first definition, “Critical thinking is to develop
cohesive, logical reasoning patterns and understanding assumptions and biases
underlying particular positions” and the second definition, “Critical thinking is the
process of determining the authenticity, accuracy and worth of information or
knowledge claims.” Thus, only the first definition was used as the basis for
determining the school district’s choice for a gifted education critical thinking
program.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Agreement or consensus among the 14 teachers and administrators
concerning critical thinking abilities was also found to be statistically significant
(Stahl, N. N„ & Stahl, R. J., 1991). Like the critical thinking definitions, there was
no single statistically significant critical thinking ability. Unlike the critical thinking
definitions, an analysis of the composite rankings did not reveal a large break that
could serve as an acceptable division point between abilities to be selected from
those not selected as criteria. Therefore, all 16 critical thinking abilities were selected
as criteria for evaluating the school district’s choice for a gifted education critical
thinking program.
The American Philosophical Association (APA) completed a two-year
international Delphi Technique experiment (Facione, 1990) conducted by a cross-
disciplinary panel in an effort to define, teach, and measure critical thinking. The
panel members included 46 men and women from throughout the United States and
Canada. They represented many different scholarly disciplines in the humanities,
sciences, social sciences, and education. The panel of experts (Facione, 1990) was
asked to try to form a consensus about the meaning of critical thinking and began by
asking themselves the question: Who are the best critical thinkers we know and what
is it about them that leads us to consider them the best? In the end, the panel of
experts agreed to this statement regarding critical thinking and the ideal critical
thinker.
We understand critical thinking to be purposeful, self-regulatory
judgment which results in interpretation, analysis, evaluation, and
29
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
inference, as well as explanation of the evidential, conceptual,
methodological, criteriological, or contextual considerations upon
which that judgment is based. CT is essential as a tool of inquiry. As
such, CT is a liberating force in education and a powerful resource in
one’s personal and civic life. While not synonymous with good
thinking, CT is a pervasive and self-rectifying human phenomenon.
The ideal critical thinker is habitually inquisitive, well-informed,
trustful of reason, open-minded, flexible, fair-minded in evaluation,
honest in facing personal biases, prudent in making judgments,
willing to reconsider, clear about issues, orderly in complex matters,
diligent in seeking relevant information, reasonable in the selection of
criteria, focused in inquiry, and persistent in seeking results which are
as precise as the subject and the circumstances of inquiry permit.
Thus, educating good critical thinkers means working toward this
ideal. It combines developing CT skills with nurturing those
dispositions which consistently yield useful insights and which are the
basis of a rational and democratic society. (Facione, 1990, p. 3)
In addition to defining critical thinking and describing the ideal critical
thinker, the panel of experts sought to identify the core critical thinking cognitive
abilities and affective dispositions. The panelists (Facione, 1990) identified
interpretation, analysis, evaluation, inference, explanation, and self-regulation as
core critical thinking cognitive abilities. The definitions for each critical thinking
cognitive ability and their respective sub-skills follow. Explanations from Jones et al.
(1995) for the first four core critical thinking cognitive abilities and sub-skills are
subsequently presented.
Interpretation (Facione, 1990) was defined as “to comprehend and express
the meaning or significance of a wide variety of experiences, situations, data, events,
judgments, conventions, beliefs, rules, procedures, or criteria” (p. 13). This critical
thinking cognitive ability (Jones et al., 1995) involves understanding and expressing
30
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
the meaning and significance of a variety of communications. The sub-skills of
interpretation include categorization, decoding significance, and clarifying meaning.
The expert panel (Facione, 1990) defined the sub-skill, categorization, as “to
apprehend or appropriately formulate categories, distinctions, or frameworks for
understanding, describing or characterizing information,” and “to describe
experiences, situations, beliefs, events, etc., so that they take on comprehensible
meanings in terms of appropriate categorizations, distinctions, or frameworks” (p.
13). Examples of categorization (Facione) include recognizing a problem and
defining its character without prejudice to inquiry; determining a useful way of
sorting and sub-classifying information; making an understandable report of what
one experienced in a given situation; and classifying data, findings or opinions using
a given classification schema. The three stakeholder groups (i.e., faculty, employers
and policy makers) in the Jones et al. study (1995) concluded the categorization sub
skill was very important for college graduates because they should be able to make
comparisons by noting similarities and differences between or among informational
items. Jones et al. also claim the categorization sub-skill includes the ability to
formulate categories, distinctions, or frameworks to organize information in such a
manner to aid comprehension, classify and group data, findings, and opinions on the
basis of their attributes or a given criterion, and translating information from one
medium to another to aid comprehension without altering the intended meaning.
31
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Decoding significance (Facione, 1990) includes “to detect, attend to, and
describe the informational content, affective purport, directive functions, intentions,
motives, purposes, social significance, values, views, rules, procedures, criteria, or
inferential relationships expressed in convention-based communication systems,
such as in language, social behaviors, drawings, numbers, graphs, tables, charts,
signs and symbols” (p. 14). Examples of decoding significance (Facione) include
detecting and describing a person’s purposes in asking a given question; appreciating
the significance of a particular facial expression or gesture used in a given social
situation; discerning the use of irony or rhetorical questions in debate; and
interpreting the data displayed or presented using a particular form of
instrumentation. Jones et al. (1995) felt the term, “decoding significance” was an
academic term they did not understand, and renamed this sub-skill, “detecting
indirect persuasion.” ‘T he ideal goal is that students with this [sub-skill] can detect
and describe the content and affective intentions and purposes expressed in
communication” (Jones et al., p. 124).
Clarifying meaning (Facione, 1990) is “to paraphrase or make explicit,
through stipulation, description, analogy or figurative expression, the contextual,
conventional or intended meanings of words, ideas, concepts, statements, behaviors,
drawings, numbers, signs, charts, graphs, symbols, rules, events or ceremonies” and
“to use stipulation, description, analogy or figurative expression to remove
confusing, unintended vagueness or ambiguity, or to design a reasonable procedure
32
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
for so doing” (p. 14). Examples include restating what a person said using different
words or expressions while preserving that person’s intended meanings: finding an
example which helps explain something to someone; and developing a distinction
which makes clear a conceptual difference or removes a troublesome ambiguity.
This sub-skill enables college graduates to recognize confusing or vague language
that requires clarification to increase comprehension; ask relevant and penetrating
questions to clarify facts, concepts, and relationships; identify and seek additional
resources, such as resources in print, that can help clarify communication; develop
analogies and other forms of comparisons to clarify meaning; and provide an
example that helps to explain something or remove a troublesome ambiguity (Jones
et al.. 1995).
The panel of experts identified analysis as the second critical thinking
cognitive ability. Analysis (Facione, 1990) was defined as “to identify the intended
and actual inferential relationships among statements, questions, concepts,
descriptions, or other forms of representation intended to express belief, judgment,
experiences, reasons, information, or opinions” (p. 14). Analysis sub-skills include
examining ideas, detecting arguments, and analyzing arguments.
Examining ideas (Facione, 1990) was defined as “to determine the role
various expressions play or are intended to play in the context o f argument,
reasoning or persuasion; to define terms; to compare or contrast ideas, concepts, or
statements; and to identify issues or problems and determine their component parts,
33
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
and also to identify the conceptual relationships of those parts to each other and to
the whole” (p. 14). Examples of examining ideas include identifying a phrase
intended to trigger a sympathetic emotional response that might induce an audience
to agree with an opinion; examining closely related proposals regarding a given
problem and determining their points of similarity and divergence; determining how
to break up a complicated assignment into smaller, more manageable tasks; and
defining an abstract concept. Jones et al. (1995) considered this an extremely
important sub-skill.
Detecting arguments (Facione, 1990) is “given a set of statements,
descriptions, questions or graphic representations, to determine whether or not the set
expresses, or is intended to express, a reason or reasons in support of or contesting
some claim, opinion, or point of view” (p. 15). Examples include determining
whether a paragraph presents a claim as well as a reason or reasons in support of that
claim; determining whether the author of a newspaper editorial sufficiently presents
reasons for or against a given claim or opinion; or identifying any claims in a
commercial announcement and the claims being advanced as support in the
commercial announcement.
Analyzing arguments (Facione, 1990) is “given the expression of a reason or
reasons intended to support or contest some claim, opinion or point of view, identify
and differentiate; (a) the intended main conclusion, (b) the premises and reasons
advanced in support of the main conclusion, (c) further premises and reasons
34
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
advanced as backup or support for those premises and reasons intended as supporting
the main conclusion, (d) additional unexpressed elements of that reasoning, such as
intermediary conclusions, unstated assumptions or presuppositions, (e) the overall
structure of the argument or intended chain of reasoning, and (f) any items contained
in the body of expressions being examined which are not intended to be taken as part
of the reasoning being expressed or its intended background” (p. 15). Examples
include, given a brief argument, paragraph-sized argument, or a position paper on a
controversial social issue, identify the author’s chief claim, the reasons and premises
the author advances on behalf of that claim, the background information used to
support those reasons or premises, and crucial assumptions implicit in the author’s
reasoning. Some Facione examples include, given several reasons or chains of
reasons in support of a particular claim, develop a graphic representation which
usefully characterizes the inferential flow of that reasoning.
In the detecting and analyzing arguments sub-skills, Jones et al. (1995)
included the ability to identify the ideas presented and assess the interests, attitudes
or views contained in those ideas; the ability to recognize the relationship between
the purpose of a communication and the problem issues that must be resolved in
achieving that purpose; the ability to identify the unstated assumptions of an
argument and the background information provided to explain reasons which support
a conclusion; and the ability to assess the constraints on the practical applications of
an idea.
35
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Evaluation (Facione, 1990) was defined as “to assess the credibility of
statements or other representations which are accounts or descriptions of a person’s
perception, experience, situation, judgment, belief, or opinion; and to assess the
logical strength of the actual or intended inferential relationships among statements,
descriptions, questions, or other forms of representation” (p. 15). Evaluation’s sub
skills include assessing claims and assessing arguments.
Assessing claims (Facione, 1990) means “to recognize the factors relevant to
assessing the degree of credibility; to ascribe to a source of information or opinion;
to assess the contextual relevance of questions, information, principles, rules or
procedural directions; and to assess the acceptability, the level of confidence to place
in the probability of truth of any given representation of an experience, situation,
judgment, belief or opinion” (p. 15-16). Examples include recognizing the factors
which make a person a credible witness regarding a given event or credible authority
on a given topic; determining whether a given principle of conduct is applicable to
deciding what to do in a given situation; and determining whether a given claim is
likely to be true or false based on what one knows or can reasonably find out.
The panel (Facione, 1990) defined assessing arguments as “to judge whether
the assumed acceptability of the premises of a given argument justify one’s
accepting as true (deductively certain), or very probably true (inductively justified),
the expressed conclusion of that argument; to anticipate or to raise questions or
objections, and to assess whether these point to significant weakness in the argument
36
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
being evaluated; to determine whether an argument relies on false or doubtful
assumptions or presuppositions and then to determine how crucially these affect its
strength; to judge between reasonable and fallacious inferences; to judge the
probative strength of an argument’s intended or unintended consequences with a
view toward judging the acceptability of the argument; and to determine the extent to
which possible additional information might strengthen or weaken an argument” (p.
16).
Examples of assessing arguments (Facione, 1990) include given an argument,
judge if its conclusion follows either with certainty or with a high level of confidence
from its premises, and given an objection to an argument, evaluate the logical force
of that objection. Other examples include checking for identifiable formal and
informal fallacies, evaluating the quality and applicability of analogical arguments,
judging the logical strength of arguments based on hypothetical situations or causal
reasoning, judging if a given argument is relevant or applicable or has implications
for the situation at hand, and determining how new data might lead logically to the
further confirmation or disconfirmation of a given opinion.
The Jones et al. (1995) Delphi Technique experiment found that the most
important aspect of the evaluation critical thinking cognitive ability is determining
whether an argument rests on false, biased, or doubtful assumptions; evaluating the
credibility, accuracy, and reliability of sources of information; assessing the
importance of an argument and determining whether it merits attention; assessing
37
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
statistical information used as evidence to support an argument; determining whether
conclusions based on empirical observations were derived from a sufficiently large
and representative sample; and determining and evaluating the strength of an analogy
used to warrant a claim or conclusion.
Inference (Facione, 1990) was defined as “to identify and secure elements
needed to draw reasonable conclusions; to form conjectures and hypotheses; to
consider relevant information and to educe the consequences flowing from data,
statements, principles, evidence, judgments, beliefs, opinions, concepts, descriptions,
questions, or other forms of representation” (p. 16). Inference sub-skills include
querying evidence, conjecturing alternatives, and drawing conclusions.
Querying evidence (Facione, 1990) means “to recognize premises which
require support and to formulate a strategy for seeking and gathering information
which might supply that support; and to judge that information relevant to deciding
the acceptability, plausibility or relative merits of a given alternative, question, issue,
theory, hypothesis, or statement is required; and to determine plausible investigatory
strategies for acquiring that information” (p. 17). For example, when attempting to
develop a persuasive argument in support of one’s opinion, one should be able to
judge what background information would be useful to have and develop a plan
which might yield a clear answer as to whether or not such information would be
germane in determining whether a given opinion is more or less reasonable than a
competing opinion; or plan a search which will reveal if that information is available.
38
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Jones et al. (1995) renamed the “querying evidence” sub-skill to “collecting
and questioning evidence.” This sub-skill (Jones et al.) includes determining the
most significant aspect of a problem or issue that needs to be addressed, prior to
collecting evidence; combining disparate pieces of information whose connection is
not obvious; determining whether one has sufficient evidence to form a conclusion;
and judging what background information would be useful to develop a persuasive
argument in support of one’s opinion.
Conjecturing alternatives (Facione, 1990) is “to formulate multiple
alternatives for resolving a problem, to postulate a series of suppositions regarding a
question, to project alternative hypotheses regarding an event, to develop a variety of
different plans to achieve some goal; and to draw out suppositions and project the
range of possible consequences of decisions, positions, policies, theories, or beliefs”
(p. 17). For example, given a problem with technical, ethnical or budgetary
ramifications, one should be able to develop a set of options for addressing and
resolving that problem; or given a set of priorities with which one may or may not
agree, project the difficulties and the benefits which are likely to result if those
priorities are adopted in decision making.
Jones et al. (1995) renamed “conjecturing alternatives” to “developing
alternatives and hypotheses.” This sub-skill (Jones et al.) includes the consideration
of both pros and cons of each alternative when making decisions; seeking evidence
to confirm or disconfirm alternatives; seeking the opinion of others in identifying
39
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
and considering alternatives; assessing the risks and benefits of each alternative in
deciding between them; and after evaluating the alternatives generated, developing
when appropriate, a new alternative that combines the best qualities and avoids the
disadvantages of previous alternatives.
Drawing conclusions (Facione, 1990) means “to apply appropriate modes of
inference in determining what position, opinion or point of view one should take on a
given matter or issue; given a set of statements, descriptions, questions or other
forms of representation, to educe, with the proper level of logical strength, their
inferential relationship and the consequences or the presuppositions which they
support, warrant, imply or entail; to employ successfully various sub-species of
reasoning, such as reasoning analogically, arithmetically, dialectically, scientifically,
etc.; and to determine which of several possible conclusions is most strongly
warranted or supported by the evidence at hand, or which should be rejected or
regarded as less plausible by the information given” (p. 17). Examples include
carrying out experiments and applying appropriate statistical inference techniques in
order to confirm or disconfirm an empirical hypothesis; considering various
opposing views of a controversial issue and the reasons advanced for them; gathering
relevant information, and formulating one’s own considered opinion regarding that
issue; and deducing a theorem from axioms using prescribed rules of inference.
Jones et al. (1995) include seeking various independent sources of evidence,
rather than a single source of evidence to provide support for a conclusion; and
40
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
developing and using criteria for making judgments that are reliable, intellectually
strong, and relevant to the situation at hand as critical aspects of drawing
conclusions.
Explanation (Facione, 1990) was defined as “to state the results of one’s
reasoning; to justify that reasoning in terms of the evidential, conceptual,
methodological, criteriological, and contextual considerations upon which one’s
results were based; and to present one’s reasoning in the form of cogent arguments”
(p. 18). Explanation sub-skills include stating results, justifying procedures, and
presenting arguments.
Stating results (Facione, 1990) means “to produce accurate statements,
descriptions or representations of the results of one’s reasoning activities so as to
analyze, evaluate, infer from, or monitor those results” (p. 18). Examples of stating
results include stating one’s own reasons for holding a given view; writing down for
one’s own future use one’s current thinking about an important or complex matter;
stating one’s research findings; conveying one’s analysis and judgment regarding a
work of art; and stating one’s considered opinion on a matter of practical urgency.
Justifying procedures (Facione, 1990) means “to present the evidential,
conceptual, methodological, criteriological, and contextual considerations which one
used in forming one’s interpretations, analyses, evaluation or inferences, so that one
might accurately record, evaluate, describe or justify those processes to one’s self or
to others, or so as to remedy perceived deficiencies in the general way one executes
41
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
those processes” (p. 18). Examples of justifying procedures include keeping a log of
the steps followed in working through a long or difficult problem or scientific
procedure; explaining one’s choice of a particular statistical test for purposes of data
analysis; stating the standards one used in evaluating a piece of literature; explaining
how one understands a key concept when conceptual clarity is crucial for further
progress on a given problem; showing that the prerequisites for the use of a given
technical methodology have been satisfied; reporting the strategy used in attempting
to make a decision in a reasonable way; and designing a graphic display which
represents the quantitative or spatial information used as evidence.
Presenting arguments (Facione, 1990) means * ‘to give reasons for accepting
some claim; or to meet objections to the method, conceptualizations, evidence,
criteria or contextual appropriateness of inferential, analytical or evaluative
judgments” (p. 18). Examples include writing a paper in which one argues for a
given position or policy; anticipating and responding to reasonable criticisms one
might expect to be raised against one’s political views; and identifying and
expressing evidence and counter-evidence intended as a dialectical contribution to
one’s own or another person’s thinking on a matter of deep personal concern.
Self-regulation (Facione, 1990) was defined as “self-consciously monitoring
one’s cognitive activities, the elements used in those activities, and the results
educed, particularly by applying skills in analysis, and evaluation to one’s own
inferential judgments with a view toward questioning, confirming, validation, or
42
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
correcting either one’s reasoning or one’s results” (p. 19). Self-regulation sub-skills
include self-examination and self-correction.
Self-examination (Facione, 1990) means “to reflect on one’s own reasoning
and to verify both the results produced and the correct application and execution of
the cognitive skills involved; to make an objective and thoughtful meta-cognitive
self-assessment of one’s opinions and reasons for holding them; to judge the extent
to which one’s thinking is influenced by deficiencies in one’s knowledge, or by
stereotypes, prejudices, emotions or any other factors which constrain one’s
objectivity or rationality; and to reflect on one’s motivations, values, attitudes and
interests with a view toward determining that one has endeavored to be unbiased,
fair-minded, thorough, objective, respectful of the truth, reasonable and rational in
coming to one’s analyses, interpretations, evaluations, inferences, or expressions” (p.
19). Examples of self-examination include examining one’s views on a controversial
issue with the sensitivity to the possible influences of one’s personal bias or self-
interest; reviewing one’s methodology or calculations with a view to detecting
mistaken applications or inadvertent errors; rereading sources to ensure that one has
not overlooked important information; identifying and reviewing the acceptability of
the facts, opinions or assumptions one relied on in coming to a given point of view;
and identifying and reviewing one’s reasons and reasoning processes in coming to a
given conclusion.
43
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Self-correction (Facione, 1990) is defined as “where self-examination reveals
errors or deficiencies, to design reasonable procedures to remedy or correct, if
possible, those mistakes and their causes” (p. 19). Examples of self-correction
include, given a methodological mistake or factual deficiency in one’s work, revising
that work so as to correct the problem, and then determining whether the revisions
warrant changes in any position, findings, or opinions based thereon.
The panel of experts (Facione, 1990) also listed their most important critical
thinking dispositions. These affective dispositions (Facione) include inquisitiveness
with regard to a wide range of issues; concern to become and remain well-informed;
alertness to opportunities to use critical thinking; trust in the processes of reasoned
inquiry; self-confidence in one’s own abilities to reason; open-mindedness regarding
divergent world views; flexibility in considering alternatives and opinions;
understanding the opinions of other people; fair-mindedness in appraising reasoning;
honesty in facing one’s own biases, prejudices, stereotypes, egocentric or
sociocentric tendencies; prudence in suspending, making or altering judgments; and
willingness to reconsider and revise views where honest reflection suggests that
change is warranted. As mentioned previously in the Limitations section of Chapter
I, this study’s instruments (i.e., the analytical section of the 1997 GRE Practice
General Test and the Cornell Critical Thinking Test, Level Z) cannot measure these
affective dispositions, and thus, are not included in this study.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
What is Critical Thinking Not?
How does critical thinking vary from other modes of thinking? Critical
thinking should not be confused with creative thinking or problem solving. The
Dictionary of Education (Good, 1973, p. 608) defines creative thinking as thinking
that is inventive, thinking that explores novel situations, and thinking that reaches
new solutions to old problems, or thinking that results in thoughts original with the
thinker. Problem solving (Good, 1973, p. 439) is a process employed by all people at
all levels of maturity of discovering or educing new relationships among things
observed or sensed. This process includes a conscious or subconscious assumption,
or hypothesis of a possible relationship within a simple or complex system of
thought and understanding, and a means to test through experience the acceptability
of the assumption. The process is equivalent to research when there is a refinement
of the system and of the process in stating and investigating the hypothesis.
The Department of Defense (DoD) Defense Systems Management College
(DSMC) teaches its students to use critical thinking as a tool and process to deal with
complicated problems, situations, or issues requiring analysis, synthesis, or
evaluation. DSMC (McDaniel, 2000) thus considers critical thinking as a problem
solving tool in which the end results are based on reason, logic, and evidence. The
DoD uses this critical thinking model to resolve complex weapon systems
acquisition problems.
45
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
What Does the ETS Say the Analytical Section of the GRE General Test Measures?
ETS (1999, p. 8) states the GRE General Test “measures verbal, quantitative,
and analytical skills that have been acquired over a long period of time and that are
not related to any specific field of study.” The GRE General Test is administered at
designated testing centers in accordance with an annual schedule (Freed, 1989). The
analytical section (ETS, 1999) measures one’s ability to understand structured sets of
relationships, deduce new information from sets of relationships, analyze and
evaluate arguments, identify central issues and hypotheses, draw sound inferences,
and identify plausible causal explanations. Duran, Powers and Swinton (1987, p. 3)
state the GRE analytical test is intended to assess one’s developed ability to reason
with complex information as might be required in any area of graduate study.
Questions in the analytical section measure reasoning skills developed in virtually all
fields of study, thus ETS states no formal training in logic or methods of analysis is
needed to do well in these sections.
To facilitate a broader definition of academic talent, ETS began
experimenting with an analytical section in 1977 to extend graduate admissions
testing beyond strictly verbal and quantitative domains (Enright & Powers, 1991).
Beginning in 1985, the analytical section became an operational part of the General
Test yielding a separate analytical ability score (Briel, O’Neill & Scheuneman,
1993). The latest change to the GRE has been the addition of a writing assessment.
Since October 1999, ETS (1999, p. 12) has offered a GRE writing assessment to
46
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
further expand the range of skills assessed by the GRE General Test. ETS claims the
writing assessment gives the examinee the opportunity to display one’s critical
thinking and analytical writing skills. These skills include the ability to articulate and
support complex ideas, analyze an argument, and sustain a focused and coherent
discussion (ETS).
The analytical measure includes analytical reasoning and logical reasoning
items (ETS, 1995). Analytical reasoning questions test one’s ability to understand a
given structure of arbitrary relationships among fictitious persons, places, things, or
events and to deduce new information from the relationships given (ETS). There are
two principal kinds of analytical reasoning questions: (a) the first tests the ability to
infer what is necessarily true, and (b) the second tests the ability to determine what is
or is not possible (ETS). Logical reasoning questions test the ability to understand,
analyze, and evaluate arguments— recognizing the point of an argument, recognizing
the assumptions on which an argument is based, drawing conclusions and forming
hypotheses, identifying the method of argumentation, evaluating arguments and
counter-arguments, and analyzing evidence (ETS). For each of these measures,
special subject matter knowledge is not required (ETS, p. 11).
The Technical Manual (Briel et al., 1993) states there are two analytical
sections in the GRE General Test. Each section contains nineteen analytical
reasoning and six logical reasoning items. Within each section, items of each type
are arranged roughly in their order of difficulty. Each analytical reasoning group
47
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
consists of (a) a set of approximately three to seven related statements or conditions
and sometimes other explanatory material describing a structure of relationships, and
(b) three or more items that test understanding of that structure and its implications
by requiring examinees to infer what is necessarily true or to determine what is or is
not possible. The relationships are common ones, such as temporal or spatial order,
assignment to a group or set, transformations, and cause and effect. Some
relationships are fixed or constant, other relationships are variable. Although each
item in a group is based on the same set of conditions, the items are independent of
one another; answering one item in a group correctly does not depend on the answer
in any other item. Briel et al. state knowledge of formal logic or mathematics is not
required for solving analytical reasoning problems.
Logical reasoning items (Briel et al., 1993) test the ability to understand,
analyze, and evaluate arguments. These items test the examinee’s ability to
comprehend and assess logical relationships among either statements in a prose
passage or to answer options referring to a passage. Several questions may be
associated with a prose passage, but for a given passage, questions are independent
of each other. Understanding passages and questions does not require specialized
knowledge in any academic field, including logic (Duran et al., 1987, p. 6). Each
item or group of logical reasoning items is based on a short argument or on a simple
graph or table, generally an excerpt from the kind of material graduate students
encounter in their academic and personal reading. Although materials may be drawn
48
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
from many specific fields of study, such as the humanities, social sciences, and the
physical sciences, more familiar sources such as political speeches, advertisements,
and informal discussions or dialogues form the basis for some of the items.
Duran et al. (1987, p. 7) claim logical reasoning items would not be out of
place in textbooks or course materials on rhetoric, critical thinking, or formal logic
since logical reasoning questions generally test one’s ability to understand and
analyze arguments, recognize the point of an argument, recognize assumptions on
which an argument is based, draw conclusions from given premises, evaluate
arguments and counter-arguments, and analyze evidence.
Chalifour and Powers (1988) coded the content characteristics of about 1,400
GRE analytical reasoning items. These characteristics were correlated with indices of
item difficulty, discrimination, and independence from the verbal and quantitative
measures. The authors found that several item characteristics were predictive of the
difficulty of analytical reasoning items. These same variables also predicted item
discrimination, but to a lesser degree. Chalifour and Powers found that independence
from the GRE verbal and quantitative measures was not strong, although the
analytical reasoning items were generally more highly correlated with the GRE
quantitative measure than with the verbal measure. According to Duran et al. (1987),
p. 7), GRE reading comprehension questions are not designed to systematically tap
the same specific set of skills as logical reasoning questions.
49
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The predictive ability of the GRE General Test analytical section to graduate
school success has been the topic of much discussion. Enright and Powers (1991)
developed a criterion for graduate school success as an alternative to first-year grade
point average (GPA). This criterion was faculty ratings of their students’ analytical
abilities. The ancillary goal was to develop this criterion as a way to validate the
GRE analytical measure and future modifications to it. The rating instrument
consisted of a wide variety of reasoning skills. These six separate rating skills
included: analyzing arguments, drawing inferences, defining problems, reasoning
inductively, generating alternatives, and overall analytical style. The study revealed
that faculty raters were not able to distinguish among students on the six individual
scales, which exhibited very high intercorrelations. Although faculty raters were not
able to distinguish among the students through the ratings, the ratings and first-year
grades were highly correlated, indicating both criteria reflect success in graduate
school. Furthermore, each of the three GRE General Test measures--verbal,
quantitative, and analytical— was more highly correlated, on the average, with the
ratings than with first-year grade point averages. This high correlation suggests the
analytical section of the GRE General Test also measures analytical abilities, as do
the verbal and quantitative sections of the GRE General Test. Undergraduate grades,
on the other hand, correlated better with first-year grades than with the ratings. When
the three GRE measures were ranked with respect to their predictive effectiveness
for each of the 24 graduate departments that were studied (Enright & Powers, 1991),
50
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
the analytical measure was significantly more often the best or second best predictor
of faculty ratings than for first-year grades. On the other hand, the verbal and
quantitative measures tended to be the best predictors for ratings and grades. This
suggests that the ratings may be more reflective of analytical ability than of verbal or
quantitative ability. However, the verbal measure was, on average, more highly
correlated with faculty ratings of students’ analytical skills than were the analytical
measures. This suggests that faculty ratings of students’ analytical skills may have
been influenced by their students’ verbal reasoning skills. This failure to find
unequivocal evidence of discriminate validity of the ratings may reflect problems
with the ratings, with the way in which faculty rated students, or with the
discriminate validity of the analytical measures.
Sternberg (1996) conducted an I l-year study to address the question of what
the tests predict by validating the GRE as a predictor of various kinds of
performance in the Yale psychology graduate program. Sternberg had the students’
grade point averages (GPAs) for their first and second years of graduate school, and
the students’ scores on the GRE verbal, quantitative, and analytical sections for 73 of
the 167 students. Sternberg found that the GRE provided some prediction of first-
year grades (i.e., 0.18 for the verbal section, 0.14 for the quantitative section, and
0.17 for the analytical section). All the correlations were statistically significant
except for the quantitative score. Most significantly, the only consistent predictor of
graduate school performance (Sternberg), as measured by faculty ratings for
51
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
analytical, creative, practical, research, and teaching abilities, as well as for
dissertation quality, was the GRE analytical section, and this prediction was for men
only. Five of the six correlations were statistically significant for men with a median
correlation of 0.30, whereas none of the correlations was statistically significant for
women, with a median correlation of 0.02.
What Do Ennis, Millman and Tomko Say the Cornell Critical Thinking Test, Level Z
Measures?
The Cornell Critical Thinking Test, Level Z is a 52-item, 50-minute multiple-
choice instrument. As a predictor of graduate school success, Ennis et al. ( 1985a, p.
21) state the Cornell Critical Thinking Test, Level Z does as well as the GRE
General Test and MAT, with correlations between 0.2 and 0.4. The intended target
audiences are college students and adults. Ennis et al. ( 1985a) state a critical thinking
test might cover induction, deduction, evaluation, observation, credibility (i.e.,
statements made by others), assumption identification, and meaning. Ideally, it
would also measure dispositions of a critical thinker such as open-mindedness,
caution, and valuing being well-informed (Ennis et al.), but testing for such
dispositions is difficult, particularly through a multiple-choice test. Thus, critical
thinking dispositions are not deliberately tested for in the Cornell Critical Thinking
Test, Level Z.
The Cornell Critical Thinking Test, Level Z seeks to measure deduction in
items I through 10, semantics in items 11 through 21, credibility in items 22 through
25, induction (i.e., judging conclusions) in items 26 through 38, induction (i.e.,
52
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
planning experiments) in items 39 through 42, definition and assumption
identification in items 43 through 46, and assumption identification in items 47
through 52. Although these aspects of critical thinking are listed separately, Ennis et
al. (1985b) recognize there is considerable interdependence among these aspects in
the critical thinking process.
The examinee’s Cornell Critical Thinking Test, Level Z test score is
computed by using the formula “rights minus one-half the number wrong” (Ennis et
al., 1985a, p. 6) to help reduce the effects of haphazard guessing. Prior to taking the
test, the administrator cautions the examinees to not make wild guesses. It is also
possible to compute subtest scores using groups of items assigned to six different
critical thinking components: induction through items 17 and 26 through 42,
deduction through items I through 10 and 39 through 52, observation through items
22 through 25, credibility through items 22 through 25, assumptions through items
43 through 52. and meaning through items 1L through 21 and 43 through 46.
Ennis et al. (1985a) checked the Cornell Critical Thinking Test, Level Z for
reliability and internal consistency by comparing the responses between odd and
even numbered responses for 12 different groups. The reliability estimate values
using odd-even intercorrelations range from 0.50 to 0.77, suggesting high reliability
and internal consistency.
Ennis et al. (1985a) performed an item analysis to produce discrimination and
difficulty indices. Item discrimination is the extent to which an item correlates with
53
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
the total score. A difficulty index indicates the proportion of a tested population that
answered an item correctly, so that a difficulty index of 0.95 means that 95% of the
population answered the item correctly. Discrimination and difficulty indices can be
averaged over all the items in a test or part of a test. Ennis et al. found that the
discrimination indices ranged from 0.20 to 0.24, and the difficulty indices ranged
from 0.55 to 0.61. These indices suggest the Cornell Critical Thinking Test, Level Z
items correctly discriminate the range of the examinees’ critical thinking cognitive
abilities.
Ennis et al. (1985a) assessed the content validity of the Cornell Critical
Thinking Test, Level Z. Members of the Illinois Critical Thinking Project (Ennis et
al.) intensively discussed each item. Using their expert judgment, these panelists
concluded “there was universal agreement on the correctness” of the items (p. 15).
With respect to assessing the construct validity of the Cornell Critical Thinking Test.
Level Z, Ennis et al. were less successful. Ennis et al. claimed “there is still not
enough information available to make a construct validity judgment with justified
confidence” (p. 16). Messick (1989, p. 41) states that relying on content as the sole
validity basis is flawed-content validity “resides” in a test, whereas construct
validity is a property of the test responses and of the inferences based thereon— the
problem is that content validity is focused upon test forms rather than test scores,
upon instruments rather than measurements.
54
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The many definitions of critical thinking and the three Delphi technique
experiments identifying critical thinking cognitive abilities provide a methodology
for developing a set of criteria for reasonably measuring critical thinking. I develop
these criteria in Chapter III.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER HI
METHODOLOGY
This chapter describes the research methodology and procedures I used in
this study. The first section describes the methodology I employed to develop the
criteria for critical thinking and answers the research question, “What set of criteria
reasonably measures critical thinking?” This methodology consists of three
subsections. In the first subsection, I assess critical thinking definitions and identify
the most recurring critical thinking characteristics or attributes. In the second
subsection, I assess and compare the findings of two Delphi Technique experiments
(Facione, 1990; Stahl N. N., & Stahl, R. J., 1991) I described in Chapter II. I do not
use the Delphi Technique experiment performed by Jones et al. (1995) because they
did not change the findings from the Facione (1990) study. In the third subsection, I
derive my set of critical thinking criteria from the definitions described in the two
previous subsections. The second section explains the procedures I used to evaluate
each GRE analytical section item against my set of critical thinking criteria. I
provide several examples to show how I scored each GRE analytical section test item
for critical thinking. The third section describes how I evaluated each Cornell
Critical Thinking Test, Level Z item against this same set of criteria. I also provide
several examples to explain the rationale in my scoring. My findings, analysis and
interpretation of the findings follow in Chapter IV.
56
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Development of Critical Thinking Criteria
Assessment of Critical Thinking Definitions
The critical thinking definitions presented in Chapter II contain several
recurring and salient critical thinking characteristics or attributes. From these
definitions, critical thinking entails performing a “good” assessment or evaluation.
The “good” implies thinking that is purposeful or goal-directed; disciplined;
structured; objective (i.e., open-minded and suspends judgment until all the evidence
and all the alternatives have been carefully tested (if possible) and evaluated);
cohesive; and based on the sound concepts of logic, logical reasoning patterns and
inferences (i.e., inductive and deductive reasoning). There is no place for sophistic
logic in critical thinking. “Good” also means that critical thinking is rational,
reasonable, and reflective— the critical thinker is aware of his or her own biases,
prejudices, and limitations. The critical thinker is also thinking about his or her
thinking while he or she is thinking (i.e., an affective trait or disposition that strives
for intellectual honesty and self-awareness). Since the critical thinker is cognizant of
his or her own weaknesses, the critical thinker seeks to improve his or her own
thinking (i.e., another affective trait or disposition).
The critical thinking definitions also suggest critical thinkers possess certain
cognitive abilities. These skills include the ability to interpret statements or data (i.e.,
accurately sort, organize, classify, and correlate statements or data so as to present
the statements and data in meaningful and understandable terms); analyze the
57
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
statements or data for the main argument and supporting or detracting arguments;
make inferences about the parts of the statements and data (e.g., through induction
and deduction) to arrive at some conclusion once the statements and data have been
carefully studied; and use logic and logical reasoning to evaluate statements or data
for credibility and logical strength.
Several definitions (Kurfiss, 1988; Maiorana, 1992, as cited in Fowler, 1997)
seem to erroneously equate critical thinking with problem solving. For example,
Maiorana states the purpose of critical thinking is, therefore, to achieve
understanding, evaluate viewpoints, and solve problems. Kurfiss states critical
thinking is problem solving in situations where solutions cannot be verified
empirically. Most critical thinking experts (Halpem, 1996,as cited in Fowler. 1997;
Miller & Connelly, 1997; McDaniel, 2000; & Watson & Glaser, 1980a) view critical
thinking as a tool in the problem solving process. Thus, the majority of critical
thinking definitions identify interpretation, analysis, inference, and evaluation as
critical thinking cognitive abilities. The cognitive abilities described in various
critical thinking definitions are consistent with the findings from the three critical
thinking Delphi Technique experiments (Facione, 1990; Jones et al., 1995; and Stahl,
N .N ..& Stahl, R. J„ 1991).
Assessment and Comparison of Delphi Technique Experiments
As described in Chapter II, Stahl, N. N. and Stahl, R. J. (1991) derived one
critical thinking definition and sixteen critical thinking cognitive skills through the
58
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
use of the Delphi Technique. A panel of fourteen teachers and administrators reached
a consensus on their definition of critical thinking: “Critical thinking is to develop
cohesive, logical reasoning patterns and understanding assumptions and biases
underlying particular positions” (Stahl, N. N., & Stahl, R. J., 1991, p. 84). The panel
also reached a consensus on identifying the top critical thinking cognitive abilities.
The APA-sponsored two-year Delphi Technique experiment (Facione, 1990)
reached a consensus in defining critical thinking as purposeful, self-regulatory
judgment which results in interpretation, analysis, evaluation, and inference, as well
as explanation of the evidential, conceptual, methodological, criteriological, or
contextual considerations upon which that judgment is based. This definition is
consistent with other critical thinking definitions and the critical thinking definition
from the Stahl, N. N. and Stahl R. J. (1991) Delphi Technique experiment.
The Stahl N. N. and Stahl R. J. (1991, p. 84) definition, “Critical thinking is
to develop cohesive, logical reasoning patterns and understanding assumptions and
biases underlying particular positions” is consistent with the APA-sponsored
cognitive skill, evaluation. In the Facione study (1990), the panel of experts included
“logical strength” in their definition of evaluation (p. 15) and “to determine whether
an argument relies on false or doubtful assumptions or presuppositions and then to
determine how crucially these affect its strength” (p. 16) within assessing arguments
(i.e., a sub-skill of evaluation).
59
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Since the Stahl N. N. and Stahl R. J. (1991) critical thinking definition
stresses evaluation, the majority of their critical thinking cognitive skills are related
to the APA’s cognitive skill, evaluation. For example, Stahl N. N. and Stahl R. J.’s
(1991, p. 85) cognitive skills, “Critical thinking distinguishes between verifiable
facts and value claims,” “Critical thinking determines the accuracy of a statement,”
“Critical thinking detects bias,” “Critical thinking distinguishes between warranted
and unwarranted claims,” “Critical thinking judges whether certain statements
contradict each other,” and “Critical thinking determines the reliability of a claim or
source” are comparable to the Facione’s (1990) “assessing claims” and “assessing
arguments” sub-skills within the critical thinking cognitive skill, evaluation.
Likewise, the Stahl N. N. and Stahl R. J. cognitive skills, “Critical thinking tries to
separate emotional thinking from logical thinking,” “Critical thinking recognizes
logical inconsistencies in a line of reasoning,” and “Critical thinking judges whether
there is ambiguity in a line of reasoning" are related to Facione’s “assess the logical
strength” and “judge between reasonable and fallacious inferences” characteristics
within evaluation.
The Facione (1990) critical thinking skill, analysis, accounts for five of the
seven remaining Stahl N. N. and Stahl R. J. (1991) cognitive skills. These skills
include “Critical thinking seeks a clear statement of the issue,” Critical thinking
identifies stated and unstated assumptions,” “Critical thinking determines the
strength of an argument,” “Critical thinking judges whether the problem has been
60
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
identified,” and “Critical thinking keeps in mind the original and/or basic concern”
(p. 85). These Stahl N. N. and Stahl R. I. critical thinking skills are similar to the
“detecting arguments” and “analyzing arguments” Facione (1990) sub-skills within
analysis. Facione defines these sub-skills as identifying and differentiating the
intended main conclusion, additional unexpressed elements such as intermediary
conclusions, and unstated assumptions or presuppositions.
There are two Stahl N. N. and Stahl R. J. (1991) critical thinking skills the
Facione (1990) study also identifies in their critical thinking cognitive skill,
inference. These critical thinking skills include “Critical thinking takes a position
when the evidence and reasons are sufficient to do so” and “Critical thinking
understands the difference between a conclusion which might be true and one which
must be true” (Stahl N. N. & Stahl, 1991, p. 85). Facione (1990) defines inference as
identifying and securing elements needed to draw reasonable conclusions and lists
“drawing conclusions” as one of the inference sub-skills (p. 16-17).
There are no Stahl N. N. and Stahl R. J. (1991) critical thinking skills
included in Facione’s (1990) interpretation critical thinking cognitive ability.
Table 4 summarizes and compares the Stahl N. N. and Stahl R. J. (1991)
critical thinking cognitive abilities with the critical thinking cognitive abilities
derived from the Facione (1990) study— interpretation, analysis, inference, and
evaluation. In Table 4, INT is interpretation, ANA is analysis, EVA is evaluation,
and INF is inference.
61
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 4
Comparison of Stahl & Stahl (1991) Critical Thinking Cognitive Skills with APA-
Sponsored Critical Thinking Study (Facione. 1990)
Critical Thinking Cognitive Skill INT ANA INF EVA
Critical thinking distinguishes between verifiable
facts and value claims.
X
Critical thinking determines the accuracy of a
statement.
X
Critical thinking seeks a clear statement of the issue. X
Critical thinking tries to separate emotional thinking
from logical thinking.
X
Critical thinking detects bias. X
Critical thinking recognizes logical inconsistencies
in a line of reasoning.
X
Critical thinking identifies stated and unstated
assumptions.
X
Critical thinking distinguishes between warranted
and unwarranted claims.
X
Critical thinking judges whether there is ambiguity
in a line of reasoning.
X
62
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 4 (continued)
Critical Thinking Cognitive Skill INT ANA INF EVA
Critical thinking judges whether certain statements
contradict each other.
X
Critical thinking determines the strength of an
argument.
X
Critical thinking judges whether the problem has
been identified.
X
Critical thinking determines the reliability of a claim
or source.
X
Critical thinking takes a position when the evidence
and reasons are sufficient to do so.
X
Critical thinking keeps in mind the original and/or
basic concern.
X
Critical thinking understands the difference between
a conclusion which might be true and one which
must be true.
X
In addition to interpretation, analysis, evaluation, and inference, the APA-
expert panel identified two additional critical thinking skills— “explanation” and
“self-regulation.” Facione (1990) describes “explanation” as ‘‘to state the results of
one’s reasoning; to justify that reasoning in terms of the evidential, conceptual,
methodological, criteriological, and contextual considerations upon which one’s
results were based; and to present one’s reasoning in the form of cogent arguments”
(p. 18). The panel also described the explanation sub-skills as stating results,
justifying procedures, and presenting arguments. Unlike the Stahl N. N. and Stahl R.
J. (1991) Delphi Technique experiment, the APA panel of experts also considered
63
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
“self-regulation” as a critical thinking ability (p. 19). Facione describes “self
regulation” as “self-consciously monitoring one’s cognitive activities, the elements
used in those activities, and the results educed, particularly by applying skills in
analysis, and evaluation to one’s own inferential judgments with a view toward
questioning, confirming, validation, or correcting either one’s reasoning or one’s
results” (p. 19). Self-regulation sub-skills include self-examination and self-
correction.
Explanation and self-regulation, as defined by the Facione (1990) do not lend
themselves well to multiple-choice examinations like the GRE General Test and the
Cornell Critical Thinking Test, Level Z. Measuring explanation as a critical thinking
skill is more appropriate for essay-type examinations in which the author states the
reasons for defending a position on a certain issue based on the evidence, method,
criteria, or context. Measuring self-regulation is also impossible to do with the GRE
General Test and the Cornell Critical Thinking Test, Level Z. Rather than cognitive
thinking abilities, explanation and self-regulation are more affective traits or
dispositions. Thus, I have not included explanation and self-regulation as critical
thinking criteria. Table 5 summarizes my criteria and definitions for measuring
critical thinking. I apply these criteria to every item in the 1997 GRE Practice
General Test and the Cornell Critical Thinking Test, Level Z.
64
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 5
Criteria for Measuring Critical Thinking
Criterion___________________________ Description_________________________
1 INTERPRETATION is “to comprehend and express the meaning or
significance of a wide variety of experiences, situations, data,
events, judgments, conventions, beliefs, rules, procedures, or
criteria.” Sub-skills include categorization, decoding significance,
and clarifying meaning.
2 ANALYSIS is “to identify the intended and actual inferential
relationships among statements, questions, concepts, descriptions, or
other forms of representation intended to express belief, judgment,
experiences, reasons, information, or opinions.” Sub-skills include
examining ideas, detecting arguments, and analyzing arguments.
Critical thinking seeks a clear statement of the issue.
Critical thinking identifies stated and unstated assumptions.
Critical thinking determines the strength of an argument.
Critical thinking judges whether the problem has been identified.
Critical thinking keeps in mind the original and/or basic concern.
3 INFERENCE is “to identify and secure elements needed to draw
reasonable conclusions; to form conjectures and hypotheses; to
consider relevant information and to educe the consequences
flowing from data, statements, principles, evidence, judgments,
beliefs, opinions, concepts, descriptions, questions, or other forms of
representation.” Sub-skills of inference include querying evidence,
conjecturing alternatives, and drawing conclusions.
Critical thinking takes a position when the evidence and reasons are
sufficient to do so.
Critical thinking understands the difference between a conclusion
which might be true and one which must be true.
65
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 5 (continued)
Criterion___________________________ Description_______________________
4 EVALUATION is “to assess the credibility of statements or other
representations which are accounts or descriptions of a person’s
perception, experience, situation, judgment, belief, or opinion; and
to assess the logical strength of the actual or intended inferential
relationships among statements, descriptions, questions, or other
forms of representation.” Sub-skills of evaluation include assessing
claims and assessing arguments.
Critical thinking distinguishes between verifiable facts and value
claims.
Critical thinking determines the accuracy of a statement.
Critical thinking tries to separate emotional thinking from logical
thinking.
Critical thinking detects bias.
Critical thinking recognizes logical inconsistencies in a line of
reasoning.
Critical thinking distinguished between warranted and unwarranted
claims.
Critical thinking judges whether there is ambiguity in a line of
reasoning.
Critical thinking judges whether certain statements contradict each
other.
__________ Critical thinking determines the reliability of a claim or source
Measuring Critical Thinking in the Analytical Section of the GRE General Test
Each criterion in Table 5 provides a means for determining the extent to
which an item measures critical thinking. For each item in the 1997 GRE Practice
General Test, if the item measures a critical thinking cognitive ability (i.e.,
interpretation, analysis, inference or evaluation), I assign a “ I” or “X” to that
criterion. Within each criterion, I also assign a “ 1” or “X” to all Facione (1990) sub
skills the item measures as well as to each of the skills derived by Stahl N. N. and
66
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Stahl R. J. (1991). The Appendix contains an example score sheet I used for each
item.
I provide several GRE analytical and logical reasoning examples (ETS, 1997)
to explain the process of “scoring” each item. The first example contains three
analytical reasoning items (i.e., question 20 from section 2, question 14 from section
5, and question 12 from Section 2) from the 1997 GRE Practice General Test (ETS).
The second example contains three logical reasoning items (i.e., question 23 from
section 2 and question 25 from section 5).1
Example 1--GRE Analytical Reasoning Items That Measure Critical
Thinking Abilities.
Question 20. Section 2.
A developer is planning to build a housing complex on an empty tract of
land. Exactly seven different styles of houses~Q, R, S, T, W, X, and Z— will be built
in the complex. The complex will contain several blocks, and the developer plans to
put houses of at least three different styles on each block. The developer will build
the complex according to the following rules:
Any block that has style Z on it must also have style W on it.
Any block adjacent to one that has on it both style S and style X must have
on it style T and style Z.
‘From “GRE Practice General Test,” by Educational Testing Service (ETS), 1997,
pp. 11, 12, 23, 24. Copyright 1997 by the ETS. Reprinted by permission of
Educational Testing Service, the copyright owner.
67
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
No block adjacent to one that has on it both style R and style Z can have on it
either style T or style W.
No block can have on it both style S and style Q.
Which of the following can be the complete selection of house styles on a block?
(A) Q, R, S
(B) Q, S, X
(C) R, T, Z
(D) S,W,Z
(E) T, X, Z
One can use deduction to eliminate answers (A) and (B) because they violate
the rule that “No block can have on it both style S and style Q.” Likewise, one can
eliminate answers (C) and (E) because they violate the rule that “Any block that has
style Z on it must also have style W on it.” By process of elimination, the answer is
(D). Furthermore, having house styles S, W, and Z violate no rules.
This item asks one to use interpretation as a critical thinking cognitive skill
because in interpretation, one must “comprehend and express the meaning or
significance o f ... situations, data... rules, procedures or criteria" (Facione, 1990,
p. 13). This item provides one with a situation (i.e., a developer who is planning to
build a housing complex on an empty track of land), data (e.g., seven different styles
of houses), rules, procedures, and criteria (i.e., the four rules as described in the
item). One must also use the categorization sub-skill to formulate categories such as
68
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
acceptable and unacceptable combinations of houses, and the decoding significance
sub-skill since one must “detect, attend to, and describe the informational content
[of] inferential relationships” (e.g., house styles that must be in the same blocks)
(Facione, 1990, p. 13). The clarifying meaning sub-skill is required because this sub
skill is needed “to paraphrase or make explicit, through stipulation, description,
analogy or figurative expression, the contextual conventional, or intended meaning
of words” (Facione, 1990, p. 14). To assist in answering this item, one typically
draws a diagram, using letters for names (i.e., analogy), and codes to stipulate the
various conditions. This constitutes “clarifying meaning” (Facione, 1990, p. 14).
One must also use the analysis cognitive skill “to identify the intended and
actual inferential relationships among statements” (Facione, 1990, p. 14) as
delineated by the rules in the scenario. Likewise, one must examine ideas (i.e., an
analysis sub-skill) to “identify issues or problems and determine their component
parts, and also to identify the conceptual relationships of those parts to each other
and to the whole” (Facione, 1990, p. 14). The item’s rules identify several inferential
relationships, the parts or components to those relationships to each other and the
composition of the house styles in the housing tract.
Inference is applicable because one must “identify and secure elements
needed to draw reasonable conclusions” (Facione, 1990, p. 16). One reaches a
reasonable conclusion when enough information is given and applied to eventually
answer the question correctly. Furthermore, the sub-skill drawing conclusions, is
69
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
applicable because it states, “given a set of statements . . . educe, with the proper
level of logical strength, their inferential relationships and the consequences or the
presuppositions which they support, warrant, imply or entail” (Facione, 1990, p. 17).
The response that meets all the rules and criteria supports the final answer. For this
item, the examinee draws a conclusion that response “D” violates no rules and must
be the correct answer.
Finally, one must evaluate or “assess the logical strength of the actual or
intended inferential relationships among statements” (Facione, 1990, p. 15). This
item asks, “Which of the following can be the complete selection of house styles on
a block.” Thus, one is told that there are several acceptable answers that can be true.
The evaluation sub-skill, assessing claims, is also relevant because the item asks one
“to assess the contextual relevance of questions, information, principles, rules or
procedural directions” (Facione, 1990, p. 15). In this item, two of the four rules (i.e.,
the two rules governing house styles in adjacent blocks) are not relevant.
This item uses several Stahl N. N. and Stahl R. J. (1991) critical thinking
cognitive skills. These skills include: critical thinking seeks a clear statement of the
issue (i.e., what house styles are acceptable for a given block); critical thinking
judges whether the problem has been identified (i.e., the problem is determining the
significance of the rules in the item); critical thinking keeps in mind the original
and/or basic concern (i.e., what house styles are valid combinations); critical
thinking takes a position when the evidence and reasons are sufficient to do so (i.e.,
70
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
what is the correct answer); critical thinking understands the difference between a
conclusion which might be true and one which must be true (i.e., “which of the
following can be the complete selection” versus “which of the following must be the
complete selection”); critical thinking determines the accuracy of a statement (i.e.,
one must carefully read the item’s stem to ensure the conditions are translated
correctly); critical thinking recognizes logical inconsistencies in a line of reasoning
(i.e., why are answers (A), (B), (C), and (E) incorrect or invalid); and critical
thinking judges whether certain statements contradict each other (i.e., incorrect
responses contradict certain rules).
Question 14. Section 5.
At the start of a two-week hiking trip, eight women— Fiona. Gabriela, Judith,
Karen, Michiko, Sharita, Teresa, and Yuriko— will divide into a River Group and a
Hill Group of four members each. After following different trails for one week, the
groups will meet and the women will again divide into a River Group and a Hill
Group of four members each, which will again follow different trails for a week. The
groups must be formed with the following restrictions:
For the first week, Teresa cannot be in the same group as Yuriko.
For the second week, both Teresa and Yuriko must be in the River Group.
For each of the two weeks, if Fiona is in the Hill Group, Karen must also be
in the Hill Group.
For each of the two weeks, Judith must be in the same group as Michiko.
71
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Which of the following could be members of the River Group for the first week?
(A) Fiona, Gabriela, Karen, and Yuriko
(B) Fiona, Karen, Michiko, and Sharita
(C) Gabriela, Judith, Sharita, and Teresa
(D) Gabriela, Karen, Teresa, and Yuriko
(E) Gabriela, Sharita, Teresa, and Yuriko
The rule, “For the first week, Teresa cannot be in the same group as Yuriko”
eliminates responses (D) and (E) because Teresa and Yuriko are in the same group.
The rule, “For each of the two weeks, Judith must be in the same group as Michiko”
eliminates response (C) because Michiko is missing and response (B) because Judith
is missing. The correct response is therefore response (A).
For this item, one could also use the rule, “For each of the two weeks, if
Fiona is in the Hill Group, Karen must also be in the Hill Group” to eliminate
response (D). Response (D) includes Gabriela, Karen, Teresa, and Yuriko. Since
Fiona is not in the River Group, the examinee can assume Fiona is in the Hill Group
with Karen, since response (D) puts Karen in the River Group.
This item asks one to use interpretation as a critical thinking cognitive skill
because in interpretation, one must “comprehend and express the meaning or
significance o f... situations, data... rules, procedures or criteria” (Facione, 1990,
p. 13). This item provides one with a situation (i.e., determining who will be in the
River and Hill Groups for week one and week two), data (e.g., eight women who will
72
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
be divided into two groups for the first week and the second week), rules,
procedures, and criteria (i.e., who may or must be grouped or not grouped with
whom). One must also use the categorization sub-skill to formulate categories such
as acceptable and unacceptable groupings, and the decoding significance sub-skill
since one must “detect, attend to, and describe the informational content [of]
inferential relationships” (e.g., why certain groupings violate inferential
relationships) (Facione, 1990, p. 13). The clarifying meaning sub-skill is required
because one uses this sub-skill to draw a diagram of the Hill and River Groupings
over the two-week period.
One must also use the analysis cognitive skill “to identify the intended and
actual inferential relationships among statements” (Facione, 1990, p. 14). Likewise,
one must examine ideas (i.e., an analysis sub-skill) to “identify issues or problems
and determine their component parts, and also to identify the conceptual
relationships of those parts to each other and to the whole” (Facione, 1990, p. 14).
The problem is determining which groupings will not violate any rules or criteria.
Inference is applicable because one must “identify and secure elements
needed to draw reasonable conclusions” (Facione, 1990, p. 16). One reaches a
reasonable conclusion when enough information is given and applied to eventually
answer the question correctly. Furthermore, the sub-skill drawing conclusions, is
applicable because it states, “given a set of statements . . . educe, with the proper
level of logical strength, their inferential relationships and the consequences or the
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
presuppositions which they support, warrant, imply or entail” (Facione, 1990, p. 17).
The response that meets all the rules and criteria supports the final answer. For this
item, the examinee draws a conclusion that response “A” violates no rules and must
be the correct answer.
Finally, one must evaluate or “assess the logical strength of the actual or
intended inferential relationships among statements” (Facione, 1990, p. 15). This
item asks, “Which of the following could be the members of the River Group for the
first week?” The phrase, “could be” implies there are several right answers, thus the
examinee must use deduction to successfully eliminate incorrect responses according
to the given rules. The evaluation sub-skill, assessing claims, is also relevant because
the item asks one “to assess the contextual relevance of questions, information,
principles, rules or procedural directions” (Facione, 1990, p. 15). In this item, one of
the four rules (i.e., “For the second week, both Teresa and Yuriko must be in the
River Group") is not relevant because the stem of the item asks the examinee what
could be members of the River Group for the first week only.
This item uses several Stahl N. N. and Stahl R. J. (1991) critical thinking
cognitive abilities. These skills include: critical thinking seeks a clear statement of
the issue (i.e., what is the only acceptable grouping for the River Group for the first
week); critical thinking identifies stated and unstated assumptions (i.e., in response
(D) one can assume that Fiona is in the Hill Group); critical thinking judges whether
the problem has been identified (i.e., the problem is determining which is the correct
74
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
grouping according to the rules in the scenario); critical thinking keeps in mind the
original and/or basic concern (i.e., what groupings are valid combinations); critical
thinking takes a position when the evidence and reasons are sufficient to do so (i.e.,
what is the correct answer); critical thinking understands the difference between a
conclusion which might be true and one which must be true (i.e., “which of the
following could be members of the River Group” versus “which of the following
must be members of the River Group”); critical thinking determines the accuracy of
a statement (i.e., accurately comprehending the rules and conditions within the
item’s stem is essential); critical thinking recognizes logical inconsistencies in a line
of reasoning (i.e., why are answers (B), (C), (D) and (E) incorrect or invalid); and
critical thinking judges whether certain statements contradict each other (i.e.,
incorrect responses contradict certain rules).
Question 12. Section 2.
A government is assigning each of six embassy office workers— Farr, Golden,
Hayakawa, Inserra, Jones, and Kovacs— to embassies. There are four embassies.
Embassies L and M are located in countries with dry climates, whereas embassies P
and T are located in countries with humid climates. The office workers must be
assigned according to the following rules:
Each embassy must have at least one of the workers assigned to it.
A least one embassy in a humid climate must have at least two workers
assigned to it.
75
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Golden cannot be assigned to an embassy in a dry climate.
Inserra must be assigned to an embassy in a dry climate.
Jones must be assigned to an embassy in a humid climate.
Which of the following must be assigned either to embassy L or to embassy M?
(A) Farr
(B) Golden
(C) Hayakawa
(D) Inserra
(E) Kovacs
The correct response is (D). Embassies L and M are located in countries with
dry climates, and Inserra must be assigned to an embassy in a dry climate.
Greenwood (1992, p. 29) states “Critical thinking skills are often referred to
as ‘higher order cognitive skills’ to differentiate them from simpler (i.e., lower order)
thinking skills.” Higher order skills are relatively complex, require judgment,
analysis, and synthesis, and are not applied in a rote or mechanical manner
(Greenwood). Paul (1995a) further distinguishes critical thinking from recall. This
item does not require many critical thinking cognitive abilities because the item stem
asks a question one can answer with rote memory or recall.
This item asks one to use interpretation as a critical thinking cognitive skill
because in interpretation, one must “comprehend and express the meaning or
significance o f... situations, data . . . rules, procedures or criteria” (Facione, 1990,
76
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
p. 13). This item provides one with a situation (i.e., where do we assign embassy
personnel), data (e.g., six office workers and four embassies), rules, procedures, and
criteria (i.e., the five rules as described in the item). One must also use the
categorization sub-skill to formulate categories such as which embassies are in
humid and dry climates, and the decoding significance sub-skill since one must
“detect, attend to, and describe the informational content [of] inferential
relationships” (e.g., which office workers can be assigned to humid or dry climates)
(Facione, 1990, p. 14). The clarifying meaning sub-skill is not required.
The examinee must also use the analysis cognitive skill “to identify the
intended and actual inferential relationships among statements” (Facione, 1990, p.
14). The analysis sub-skill, examining ideas enables one to link Inserra with either
embassy L or M, because Inserra must be in a dry climate. Since one can answer this
question with recall, no other core critical thinking skills or sub-skills are required.
This item uses only a few Stahl N. N. and Stahl R. J. (1991) critical thinking
cognitive skills. These skills include: critical thinking seeks a clear statement of the
issue (i.e., who must be assigned to embassy L or M); critical thinking judges
whether the problem has been identified (i.e., where to put Inserra); and critical
thinking keeps in mind the original and/or basic concern (i.e., what limitations do the
office workers have to limit them to either embassy L, M, P, or T).
77
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Example 2— GRE Logical Reasoning Items That Measure Critical Thinking
Abilities.
Question 23. Section 2
When an osprey (a fish-eating hawk) returns from fishing to its nesting area
with a fish like an alewife, a pollack, or a smelt, other ospreys will retrace its flight
path in hopes of good fishing. There is seldom such a response if the bird brings back
a winter founder. Yet ospreys feed on winter flounder just as readily as on any other
fish.
Which of the following, if true, contributes most to an explanation of the
fishing behavior of ospreys as it is described above?
(A) Ospreys are seldom able to catch alewives, pollack, or smelt.
(B) Alewives, pollack, and smelt move in schools but winter flounder do not.
(C) Winter flounder prefer shallower waters than do alewives, pollack, or smelt.
(D) Winter flounder and pollack exhibit protective coloration, but alewives and smelt
do not.
(E) Ospreys that live in nesting areas are especially successful fishers.
This item begins with providing the examinee with three premises: (a) when
an osprey returns from fishing to its nesting area with a fish like an alewife, a
pollack, or a smelt, other ospreys will retrace its flight path in hopes of good fishing;
(b) there is seldom such a response if the bird brings back a winter founder, and (c)
ospreys feed on winter flounder just as readily as on any other fish. One is then asked
78
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
to infer an explanation for the fishing behavior of ospreys based on the relationship
of the three premises stated above.
Response (A), “Ospreys are seldom able to catch alewives, pollack or smelt”
is incorrect because it contradicts the first premise that other ospreys will retrace
another osprey’s flight path when it returns from fishing area with a fish like an
alewife, a pollack, or a smelt. Thus, the conclusion does not follow. Response (C) is
incorrect because it does not explain the behavior of ospreys based on the premises
previously stated. Instead, response (C) introduces a new premise that winter
flounder prefer shallower waters than do alewives, pollack, or smelt. This new
premise is not relevant. Response (E), “Ospreys that live in nesting areas are
especially successful fishers” does not explain the behavior of ospreys stated in the
second premise. This premise states that ospreys will not retrace another osprey’s
flight path when it returns from fishing with a winter flounder. Response (E) further
contradicts the second premise, because if ospreys were especially successful fishers,
they would not exhibit the behavior in the second premise.
The examinee is left with two probable answers, responses (B) and (D).
Response (D), “Winter flounder and pollack exhibit protective coloration, but
alewives and smelt do not” could explain why ospreys will not retrace another
osprey’s flight path if that osprey brings back a winter flounder. If the winter
flounder exhibits protective coloration, fishing for winter flounder may be more
difficult, and thus other ospreys may not be motivated to retrace another osprey’s
79
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
flight path when it brings back a winter flounder. However, response (D) contradicts
the first premise. The first premise states that when an osprey returns from fishing to
its nesting area with a fish like an alewife, a pollack, or a smelt, other ospreys will
retrace its flight path in hopes of good fishing. Response (D) contradicts the first
premise because it includes pollack in explaining the osprey’s behavior. As stated in
the first premise, ospreys will retrace another osprey’s flight path if it returns with a
pollack.
Response (B) best explains the behavior of opreys. Ospreys may not retrace
another osprey’s flight path when it returns with a winter flounder because response
(B) states alewives, pollack, and smelt move in schools, and winter founder do not. If
alewives, pollack and smelt move in schools, retracing another oprey’s flight path
when it returns with one of these fish makes sense-there may still be other alewives,
pollack or smelt in the vicinity. If winter flounder do not move in schools, then it
makes no sense to retrace the flight path of an osprey that returns with a winter
flounder, as there will be no other winter flounder to be caught.
This item requires one to interpret or “comprehend. . . the meaning or
significance o f... situations .. .judgments, conventions, [and] beliefs” (Facione.
1990, p. 13). The examinee is presented with several premises, and the item asks
why osprey would behave a certain way. One is presented several judgments,
conventions and beliefs as to why osprey would behave in such a way. One does not
need to categorize the data nor clarify meaning as defined by Facione, but the
80
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
examinee does use the decoding significance sub-skill because the examinee must
“describe the informational content, affective purport... or inferential relationships”
of the premises in the item and possible conclusions as expressed as possible
responses in the item (Facione, 1990, p. 14).
The analysis critical thinking cognitive ability applies as well, since one must
“identify the intended and actual inferential relationships among statements . . .
intended to express beliefs, judgments . . . [or] reasons” (Facione, 1990, p. 14). The
critical thinker also uses the examining ideas sub-skill “to compare and contrast
ideas, concepts, or statements” and “identify issues . . . and determine their
component parts, and also to identify the conceptual relationships of those parts to
each other and to the whole” (Facione, 1990, p. 14). The detecting arguments and
analyzing arguments are not applicable.
The critical thinker uses inference to “identify and secure elements needed to
draw reasonable conclusions . . . flowing from data... judgments, beliefs, [or]
opinions” (Facione, 1990, p. 16). The objective of the item is to test one’s abilities to
identify and secure those inferential relationships to draw reasonable conclusions.
Thus, the querying evidence sub-skill (i.e., “to judge . . . the acceptability,
plausibility, or relative merits of a given alternative, question, theory, hypothesis, or
statement” (Facione, 1990, p. 17)) and the drawing conclusions sub-skill (i.e., “given
a set of statements, descriptions, [and] questions . . . educe, with the proper level of
logical strength, their inferential relationships” and “determine which of several
81
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
possible conclusions is most strongly warranted or supported by the evidence at
hand” (Facione, 1990, p. 17) apply.
The examinee uses the evaluation critical thinking cognitive skill to “assess
the logical strength of the actual or intended inferential relationships among
statements” (Facione, 1990, p. 15). For example, the item asks one to assess the
strength of the inferential relationships between the premises and several
conclusions. Although response (D) could explain the behavior of osprey, upon
further examination, response (D) is a distracter to the correct response (i.e., answer
(B)). Both the assessing claims and assessing arguments apply. In assessing claims,
one must “assess the contextual relevance of questions, information, principles, rules
or procedural directions” (Facione, 1990, p. 15). In assessing arguments, one must
“judge whether the assumed acceptability of the premises of a given argument justify
one’s accepting as true (deductively certain), or very probably true (inductively
justified), the expressed conclusion of that argument” (Facione, 1990, p. 16). The
assessing arguments sub-skill also includes the abilities to “determine whether an
argument relies on false or doubtful assumptions or presuppositions and then to
determine how crucially these affect its strength” and “to judge the probative
strength of an argument’s premises and assumptions with a view toward determining
the acceptability of the argument” (Facione, 1990, p. 16).
Several Stahl N. N. and Stahl R. J. (1991) critical thinking skills equally
apply. This item tests several analysis critical thinking skills such as critical thinking
82
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
seeks a clear statement of the issue (i.e., why would ospreys exhibit a certain
behavior?); critical thinking determines the strength of an argument (i.e., several
reasons are presented, of which one presents the strongest rationale); critical thinking
judges whether the problem has been identified; and critical thinking keeps in mind
the original and/or basic concern (Stahl, N. N. & Stahl R. J.). Other Stahl N. N. and
Stahl R. J. critical thinking skills include, critical thinking takes a position when the
evidence and reasons are sufficient to do so; and critical thinking understands the
difference between a conclusion which might be true and one which must be true
(e.g., the difference between response (D) and (B)). These critical thinking skills
generally fall into the inference core critical thinking skill. Finally, this item requires
one to use these evaluation critical thinking skills (Stahl N. N. & Stahl, R. J.): critical
thinking determines the accuracy of a statement; critical thinking recognizes logical
inconsistencies in a line of reasoning; critical thinking judges whether there is
ambiguity in a line of reasoning; and critical thinking judges whether certain
statements contradict each other.
Question 25. Section 5
Government department head: We already have a code of ethics that
companies doing business with the department are urged to abide by. The fact that
virtually all of the companies have agreed to abide by it indicates that it is successful.
Therefore, neither stronger ethics regulations nor enforcement mechanisms are
83
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
necessary to ensure ethical behavior by companies doing business with the
department.
Which of the following, if true, casts most doubt on the department head’s
conclusion?
(A) The code of ethics applies only to companies that do business with the
department.
(B) The code of ethics was instituted only after it was discovered that several
companies had committed serious violations of ethics in their business with the
department.
(C) A government investigation found that most of the companies that agreed to
abide by the department’s code of ethics are not complying with it.
(D) A survey of major companies found that several companies stopped doing
business with the department because they did not want to agree to abide by the code
of ethics.
(E) A study of codes of ethics for companies found that the codes are most effective
when the top executives of companies that agree to abide by them are fully
committed to following them.
This logical reasoning item presents the examinee with two premises and a
conclusion. The premises are: (a) We already have a code of ethics that companies
doing business with the department are urged to abide by: and (b) The fact that
virtually all of the companies have agreed to abide by it indicates that it is successful.
84
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The conclusion is, ‘Therefore, neither stronger ethics regulations nor enforcement
mechanisms are necessary to ensure ethical behavior by companies doing business
with the department.” The examinee is then asked to determine what additional
premise would cast the most doubt on the conclusion. The item does not provide any
additional information as to whether the companies who agreed to abide by the code
of ethics are in fact abiding by it.
Response (A), ‘The code of ethics applies only to companies that do business
with the department” does not cast much doubt on the conclusion. The department
head's conclusion is already limited to companies doing business with the
department. If the department head’s conclusion included all companies, then
response (A) would cast some doubt on the veracity of the conclusion.
Response (E), “A study of codes of ethics for companies found that the codes
are most effective when the top executives of companies that agree to abide by them
are fully committed to following them” does not cast much doubt on the conclusion
either. If any of the two premises or conclusion made reference to the commitment of
the top executives of the companies to abide by the code of ethics, then this response
would be relevant. However, since the two premises and conclusion do not consider
the commitment of the companies’ top executives, this response is immaterial.
The remaining three responses (i.e., responses (D), (B), and (C)) cast some
doubt on the department head’s conclusion. Response (D), “A study of major
companies found that several companies stopped doing business with the department
85
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
because they did not want to agree to abide by the code of ethics” suggests that there
may be companies still doing business with the department that are not abiding by
the code of ethics. The critical word in response (D) is “several,” since “several” is
not all inclusive. Response (B), ‘The code of ethics was instituted only after it was
discovered that several companies had committed serious violations of ethics in their
business with the department” also casts doubt since this premise suggests that
serious violations of ethics have been a problem in the past. These violations could
still be a problem, despite companies claiming they have agreed to the code of ethics.
Response (C), “A government investigation found that most of the companies
that agreed to abide by the department’s code of ethics are not complying with it”
best casts doubt on the department head’s conclusion that neither stronger ethics
regulations nor enforcement mechanisms are necessary to ensure ethical behavior by
companies doing business with the department. Unlike responses (B) and (D) that
suggests companies are not complying with the code of ethics, response (C) clearly
states that companies are not complying with it. Response (C) is definitive and thus
casts the most doubt.
This item requires one to interpret or “comprehend . . . the meaning or
significance of... situations .. .judgments, conventions, [and] beliefs” (Facione,
1990, p. 13). The examinee is presented with two premises and a conclusion, and the
item asks which additional premise casts the most doubt on the conclusion. The
examinee does not need to categorize the data as with analytical reasoning items nor
86
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
clarify meaning as defined by Facione, but the examinee does use the decoding
significance sub-skill because the examinee must “describe the informational
content, affective purport... or inferential relationships” of an additional premise in
the item (p. 14).
The analysis critical thinking cognitive ability applies as well, since one must
“identify the intended and actual inferential relationships among statements . . .
intended to express beliefs, judgments . . . [or] reasons” (Facione, 1990, p. 14). In
this item, the addition of one premise changes the inferential relationships
considerably. The critical thinker also uses the examining ideas sub-skill “to
compare and contrast ideas, concepts, or statements” and “identify issues . . . and
determine their component parts, and also to identify the conceptual relationships of
those parts to each other and to the whole” (Facione, 1990, p. 14). Unlike the
previous item (question 23, section 2), question 25, section 5 centers on an argument.
Thus, the detecting arguments sub-skill (i.e., “given a set of statements . . . determine
whether or not the set expresses, or is intended to express, a reason or reasons in
support of or contesting some claim, opinion, or point of view” (Facione, 1990, p.
15)) and analyzing arguments sub-skill (i.e., “given the expression of a reason or
reasons intended to support or contest some claim, opinion or point of view, identify
and differentiate . . . premises and reasons advanced in support of the main
conclusion . . . [and] further premises and reasons intended as supporting the main
conclusion” (Facione, 1990, p. 15)) apply.
87
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The critical thinker uses inference or the ability to identify and secure
elements needed to draw reasonable conclusions (Facione, 1990, p. 16). The
objective of the item is to test one’s abilities to weigh the strength of several new
premises and conclude which one is best. Thus, the querying evidence sub-skill (i.e.,
“to judge . . . the acceptability, plausibility, or relative merits of a given alternative,
question, theory, hypothesis, or statement” (Facione, 1990, p. 17)) and the drawing
conclusions sub-skill (i.e., “given a set of statements, descriptions, [and] questions ..
. educe, with the proper level of logical strength, their inferential relationships” and
“determine which of several possible conclusions is most strongly warranted or
supported by the evidence at hand” (Facione, 1990, p. 17) apply.
The examinee uses the evaluation critical thinking cognitive skill to “assess
the logical strength of the actual or intended inferential relationships among
statements” (Facione, 1990, p. 15). For example, the item asks one to assess the
strength of the inferential relationships between several premises and a conclusion.
Responses (B), (C), and (D) all cast some doubt on the department head’s
conclusion, but the examinee must determine which premise is the strongest. Thus,
response (B) and (D) are distracters, and the correct response is (C). Both the
assessing claims and assessing arguments apply. In assessing claims, one must
“assess the contextual relevance of questions, information, principles, rules or
procedural directions” (Facione, 1990, p. 15). In assessing arguments, one must
“judge whether the assumed acceptability of the premises of a given argument justify
88
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
one’s accepting as true (deductively certain), or very probably true (inductively
justified), the expressed conclusion of that argument” (Facione, 1990, p. 16). The
assessing arguments sub-skill also includes the abilities to “determine whether an
argument relies on false or doubtful assumptions or presuppositions and then to
determine how crucially these affect its strength” and “to judge the probative
strength of an argument’s premises and assumptions with a view toward determining
the acceptability of the argument” (Facione, 1990, p. 16).
This item requires the use of several Stahl N. N. and Stahl R. J. (1991)
critical thinking cognitive abilities. This item tests several analysis critical thinking
skills (Stahl N. N. & Stahl R. J., 1991) such as critical thinking seeks a clear
statement of the issue (i.e., do companies abide by the code of ethics?); critical
thinking determines the strength of an argument (i.e., several premises are presented,
of which one casts the most doubt); critical thinking judges whether the problem has
been identified; and critical thinking keeps in mind the original and/or basic
concern.” Other Stahl N. N. and Stahl R. J. critical thinking skills include, critical
thinking takes a position when the evidence and reasons are sufficient to do so; and
critical thinking understands the difference between a conclusion which might be
true and one which must be true (e.g., the department head’s conclusion might be
false, given a new premise). These critical thinking skills generally fall into the
inference core critical thinking skill. Finally, this item requires one to use these
evaluation critical thinking skills: critical thinking determines the accuracy of a
89
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
statement, critical thinking recognises logical inconsistencies in a line of reasoning;
critical thinking distinguishes between warranted and unwarranted claims (e.g., the
department head’s claim becomes unwarranted with the addition of a new premise);
critical thinking judges whether there is ambiguity in a line of reasoning; and critical
thinking judges whether certain statements contradict each other.
I apply these criteria to the remaining items in sections 2 and 5 in the GRE
Practice General Test and analyze and discuss the results in Chapter IV.
Measuring Critical Thinking in the Cornell Critical Thinking Test. Level Z
I also apply each criterion in Table 5 to each item in the Cornell Critical
Thinking Test, Level Z. As with the previous GRE examples, if the item measures a
critical thinking cognitive ability (i.e., interpretation, analysis, inference or
evaluation), I assign a “ 1 ” or “X” to that criterion. Within each criterion, I also
assign a “ 1” or “X” to all Facione (1990) sub-skills the item measures as well as to
each of the critical thinking skills derived by Stahl N. N. and Stahl R. J. (1991). The
item score sheet is in the Appendix.
I provide several Cornell Critical Thinking Test, Level Z examples to explain
the process of “scoring” each item. The first example describes the scoring for two
deduction items (i.e., questions I and 7). The second example describes the scoring
for two semantics items (i.e., questions 11 and 16). The third example presents the
scoring for two credibility items (i.e., questions 22 and 24). The fourth example
shows the scoring for two induction or judging conclusions items (i.e., questions 27
90
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
and 35). The fifth example shows the scoring for one induction or planning
experiments item (i.e., question 42). The sixth example shows the scoring for one
definition and assumption identification item (i.e., question 43), and the seventh
example shows the scoring for one assumption identification item (i.e., question 52).2
Example 1-Cornell Critical Thinking Test. Level Z deduction items.
In the following item, two men, Mr. Wilstings and Mr. Pinder, are debating
about voting by 18-year olds. Mr. Pinder is the speaker. The item presents a set of
statements and an underlined conclusion. If the conclusion follows necessarily from
the statements given (i.e., a person who accepts the statements is unavoidably
committed to accepting the conclusion), the answer is “A.” If the conclusion
contradicts the statements given (i.e., the statements and the conclusion cannot both
be correct), the answer is “B.” If the conclusion neither follows necessarily nor
contradicts the statements given, the answer is “C.”
Question 1.
“Mr. Wilstings says that eighteen-year-olds haven’t faced the problems of the world,
and that anyone who hasn’t faced these problems should not be able to vote. What he
says is correct, but eighteen-vear-olds still should be able to vote. They’re mature
human beings, aren’t they?”
2 From “Cornell Critical Thinking Test, Level Z,” by Ennis, R. H., Millman, J., &
Tomko, T. N., 1985, Pacific Grove: Critical Thinking Books and Software, pp. 1 ,2,
3, 5,8,9, 10, 1 1 ,12, 14. Copyright 1985 by Critical Thinking Books and Software
(800) 458-4849. Reprinted with permission.
91
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Mr. Pinder’s argument is that 18-year olds haven’t faced the problems of the
world, and that anyone who hasn’t faced these problems should not be able to vote.
If this argument is true, anyone, aged 18 or otherwise who have not faced the
problems of the world should not be able to vote. Thus, the conclusion that 18-year
olds still should be able to vote does not follow from the statements given, and the
answer is “B.”
The significance of this item is determining under what conditions (i.e.,
according to Mr. Pinder) should 18-year olds be given the right to vote? Thus, this
question tests one’s ability to interpret or “comprehend the meaning or significance
o f .. .judgments . . . [or] beliefs” (Facione, 1990, p. 13). The interpretation sub-skill,
decoding significance, also applies because the item asks one “to detect, attend to,
and describe the . . . affective purport... [or] social significance [i.e., of allowing
18-year olds the right to vote]... in language” (Facione, 1990, p. 14). The other two
interpretation sub-skills, categorization and clarifying meaning, do not apply.
This item also tests for the critical thinking skill, analysis, since the
statements and conclusion asks one “to identify . . . inferential relationships among
statements . . . intended to express belief, judgment... or opinions” (Facione, 1990,
p. 14). In this case, the components of the inferential relationships include age,
problems of the world, and maturity. The sub-skill, examining ideas, applies because
the item asks one “to compare or contrast ideas, concepts, or statements” (Facione,
1990, p. 14). The sub-skill, analyzing arguments, is also applicable because “given
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
the expression of a reason or reasons intended to support or contest some claim,
opinion or point of view,” the item asks one “to identify and differentiate the
intended main conclusion [and] the premises and reasons advanced in support of the
main conclusion” (Facione, p. 15). As defined by Facione (1990, p. 15), the sub-
skill, detecting arguments, applies because “given the expression of a reason or
reasons intended to support or contest some claim,” (i.e., the right to vote), one must
“identify and differentiate . . . the premises and reasons advanced in support of the
main conclusion” (i.e., facing the problems of the world versus maturity).
Inference is applicable as well, since the item tests one’s ability “to identify
and secure elements needed to draw reasonable conclusions . . . [and] to consider
relevant information and educe the consequences flowing from statements . . .
beliefs, [or] opinions” (Facione, 1990, p. 16). This item tests two sub-skills: querying
evidence (i.e., “recognize premises which require support” (Facione, 1990, p. 17))
and drawing conclusions (i.e., “given a set of statements . . . educe, with the proper
level of logical strength, their inferential relationships and the consequences . . .
which they support, warrant, imply, or entail” (Facione, 1990, p. 17)). In this item,
there is sufficient evidence to conclude that the statements and conclusion are
contradictory, vice supportive or does neither. Conjecturing alternatives (i.e., the last
remaining sub-skill of inference) does not apply.
This item further tests one’s ability to evaluate or “assess the logical strength
o f ... inferential relationships among statements” (Facione, 1990, p. 15). Evaluation
93
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
has two sub-skills: assessing claims and assessing arguments. This item tests one’s
ability to assess claims and assess arguments because it asks one “to judge whether
the assumed acceptability of the premises of a given argument justifies one’s
accepting [it] as true (deductively certain)” (Facione, 1990, pp. 16-17).
When compared to Stahl N. N. and Stahl R. J.’s (1991) criteria for critical
thinking cognitive abilities, this item tests for several skills. These skills include:
critical thinking seeks a clear statement of the issue (i.e., under what conditions
should 18-year olds be allowed to vote); critical thinking determines the strength of
an argument (i.e., having faced the problems of the world vice maturity which comes
with age); critical thinking judges whether the problem has been identified (i.e., are
the statements enough to support or not support the conclusion); critical thinking
keeps in mind the original and/or basic concern (i.e., the sufficiency required to
support an argument); critical thinking takes a position when the evidence and
reasons are sufficient to do so (i.e., answers “A” or * ‘B” or ”C”); critical thinking
understands the difference between a conclusion which might be true or must be
true; critical thinking determines the accuracy of a statement; critical thinking
recognizes logical inconsistencies in a line of reasoning; critical thinking judges
whether there is ambiguity in a line of reasoning; and critical thinking judges
whether certain statements contradict each other.
In the next deductive thinking item, the same two men, Mr. Wilstings and
Mr. Pinder are debating about immigration. Mr. Pinder is the speaker. Similar to the
94
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
previous question, this item presents a set of statements and an underlined
conclusion. If the conclusion follows necessarily from the statements given (i.e., a
person who accepts the statements is unavoidably committed to accepting the
conclusion), the answer is “A.” If the conclusion contradicts the statements given
(i.e., the statements and the conclusion cannot both be correct), the answer is “B.” If
the conclusion neither follows necessarily nor contradicts the statements given, the
answer is “C.”
Question 7.
“You may not know it, but for the past ten years the Communists in our country have
been supporting a policy of unrestricted immigration. It is obvious why they support
this policy of opening our doors to foreigners. Now I hate to say this, but Mr.
Wilstings’ support of this policy leaves us but one conclusion. Mr. Wilstings is a
Communist.”
Mr. Pinder’s argument is that for the past ten years the Communists have
supported unrestricted immigration, and since Mr. Wilstings supports this policy, he
concludes Mr. Wilstings is also a Communist. Mr. Pinder’s argument neither
supports nor contradicts the conclusion, because there is a transitive flaw in Mr.
Pinder’s logic. Mr. Pinder’s logic is that Communists (i.e., variable “X”) support
unrestricted immigration (i.e., variable “Y”). Mr. Wilstings (i.e., variable “Z”)
supports unrestricted immigration (i.e., variable “Y”). Therefore Mr. Wilstings is a
Communist (i.e., “Z” is “X”).
95
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
A transitive relationship holds if there is a property between the first and
second elements and between the second and third elements. Then one can logically
conclude that the property between the first and third elements also holds. For
example, the transitive relationship, “whales have lungs, all mammals have lungs,
thus a whale is a mammal” holds. Mr. Pinder’s argument violates this logical
transitive relationship, thus his answer neither supports nor contradicts the
conclusion, and the correct answer is “C.”
Similar to the previous deduction item (i.e., question I above), this question
tests similar critical thinking cognitive skills. As defined by Facione (1990, p. 13),
this question tests one’s ability to interpret or “comprehend the meaning or
significance of.. .judgments . . . [or) beliefs” (i.e., is Mr. Wilstings a Communist).
The interpretation sub-skill, decoding significance, also applies because the item
asks one “to detect, attend to, and describe the . . . affective purport. . . [or] social
significance [i.e., what makes Mr. Wilstings a Communist]. . . in language”
(Facione, 1990, p. 14). The other two interpretation sub-skills, categorization and
clarifying meaning, do not apply.
This item also tests for the critical thinking skill, analysis, since the
statements and conclusion asks one “to identify . . . inferential relationships among
statements . . . intended to express belief, judgm ent. . . or opinions” (Facione, 1990,
p. 14). In this case, the components of the inferential relationships include
Communists, a policy of unrestricted immigration, and Mr. Wilstings’ support of that
96
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
policy. The sub-skill, examining ideas, applies because the item asks one “to
compare or contrast ideas, concepts, or statements” (Facione, 1990, p. 14). The sub
skills, detecting arguments and analyzing arguments, are also applicable because
“given the expression of a reason or reasons intended to support or contest some
claim, opinion or point of view,” the item asks one “to identify and differentiate the
intended main conclusion [and] the premises and reasons advanced in support of the
main conclusion” (Facione. 1990, p. 15).
Inference is applicable as well, since the item tests one’s ability “to identify
and secure elements needed to draw reasonable conclusions . . . [and] to consider
relevant information and educe the consequences flowing from statements . . .
beliefs, [or] opinions” (Facione, 1990, p. 16). This item tests two inference sub
skills: querying evidence and drawing conclusions (i.e., “given a set of statements ..
. educe, with the proper level of logical strength, their inferential relationships and
the consequences . . . which they support, warrant, imply, or entail”) (Facione, 1990,
p. 17). In this item, there is sufficient evidence to conclude that the statements
neither support nor contradict the conclusion because the thinking is flawed. In this
item, conjecturing alternatives does not apply.
This item further tests one’s ability to evaluate or “assess the logical strength
of... inferential relationships among statements” (Facione, 1990, p. 15). Evaluation
has two sub-skills: assessing claims and assessing arguments. This item uses both
sub-skills because it asks one “to judge whether the assumed acceptability of the
97
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
premises of a given argument justifies one’s accepting [it] as true (deductively
certain)” (Facione, 1990, pp. 15-16). The fallacious logical construct in the argument
also asks one “to judge between reasonable and fallacious inferences” (Facione,
1990, p. 16). Recognizing this fallacious logical construct thus affects which Stahl N.
N. and Stahl R. J. (1990) critical thinking cognitive skills apply.
These Stahl N. N. and Stahl R. J. (1991) critical thinking cognitive skills
include: critical thinking seeks a clear statement of the issue (i.e., is Mr. Wilstings a
Communist); critical thinking determines the strength of an argument (i.e., why
should one consider Mr. Wilstings a Communist); critical thinking judges whether
the problem has been identified (i.e., are the statements enough to support or not
support the conclusion); critical thinking keeps in mind the original and/or basic
concern (i.e., under what conditions is Mr. Wilstings a Communist and does the
evidence support this conclusion); critical thinking takes a position when the
evidence and reasons are sufficient to do so (i.e., answers “A” or “B” or “C”);
critical thinking understands the difference between a conclusion which might be
true and one which must be true; critical thinking determines the accuracy of a
statement (i.e., “Mr. Wilstings is a Communist”); critical thinking tries to separate
emotional thinking from logical thinking (i.e., Mr. Pinder’s attack on Mr. Wilstings);
critical thinking detects bias (i.e., Mr. Pinder’s bias against Communists); critical
thinking recognizes logical inconsistencies in a line of reasoning (i.e., is the property,
“Communist” transitive to Mr. Wilstings); critical thinking judges whether there is
98
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
ambiguity in a line of reasoning; and critical thinking judges whether certain
statements contradict each other.
Example 2--Comell Critical Thinking Test. Level Z semantics items.
The following two items measure one’s ability to discern meaning from
semantics. In these questions, semantics means understanding the significance of
words in the verbal and linguistic sense. These items present faulty thinking in a
conversation between Dobert and Algan. The task is to pick the one best reason why
the thinking is faulty.
Question 11.
DOBERT: I hear that you and some other crackpots are trying to get Gallton to
chlorinate its water supply. You seem to think that this will do some good. There can
be no doubt that either we should chlorinate or we shouldn’t. Only a fool would be in
favor of chlorinating the water, so we ought not do it.
ALGAN: You are correct at least in saying that we are trying to get the water
chlorinated.
Pick the one best reason why some of this thinking is faulty.
A. Dobert is mistakenly assuming that there are only two alternatives.
B. Dobert is using a word in two ways.
C. Dobert is using emotional language that doesn't help to make his argument
reasonable.
99
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Dobert’s main argument is that only a crackpot and fool would want to
chlorinate Gallton’s water supply. He does not provide any evidence or logical
explanation as to why chlorinating Gallton’s water supply would be a bad idea.
Algan diverts Dobert’s emotional charge and states “You are correct at least in
saying that we are trying to get the water chlorinated.” Dobert uses emotional
language (e.g., “crackpots” and “fool”) to attack Algan as a basis for his reasoning
(i.e., ad hominem). Although there may be more than two alternatives, Algan does
not refute that he is trying to get Gallton to chlorinate its water supply. Algan’s
response suggests that there are only two alternatives (i.e., either chlorinating or not
chlorinating the Gallton water supply). Thus, “C” is the one best answer.
This item tests one’s critical thinking cognitive ability to interpret or
“comprehend. . . the meaning or significance of... situations .. .judgments . . . [or]
beliefs” (i.e., why Gallton should or should not chlorinate its water supply) (Facione,
1990, p. 13). The question also asks one to decode the significance or “detect. . .
[Dobert’s] affective purport” (i.e., for what reason does Dobert oppose the
chlorination of Gallton’s water supply) (Facione, 1990, p. 14). The two other
interpretation sub-skills, categorization and clarifying meaning, are not applicable.
This item also asks one to analyze or “identify . . . the intended and actual
inferential relationships among statements” (i.e., the conversation between Dobert
and Algan) and examine ideas (i.e., a sub-skill of analysis) (Facione, 1990, p. 14). In
examining ideas, one must “determine the role various expressions play or are
100
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
intended to play in the context of argument, reasoning, or persuasion” (i.e., the role
of Dobert’s emotional language) (Facione, 1990, p. 14). This item meets the sub
skill, detecting arguments, or “given a set of statements, determine whether or not
the set expresses, or is intended to express, a reason or reasons in support of or
contesting some claim” (Facione, 1990, p. 15). This item also contains the sub-skill,
analyzing arguments, or “given the expression of a reason or reasons to support or
contest a claim . . . identify and differentiate . . . the intended main conclusion . . .
[or] the premises and reasons advanced in support of the main conclusion” (Facione,
1990, p. 15). Dobert’s emotional language is not enough to support his point of view.
The cognitive skill, inference, applies equally in this question because one is
asked “to identify and secure elements needed to draw reasonable conclusions . . . to
consider relevant information and to educe the consequences flowing from . . .
statements” (i.e., is Dobert’s position substantiated) (Facione, 1990, p. 16). In this
item, one must query the evidence or “recognize premises which require support”
(Facione, 1990, p. 17). Since Dobert relies on sophistic logic (i.e., ad hominem), his
premise requires additional support. One must also draw some conclusion from the
argument. According to the Facione criteria (1990), this item does not test the
conjecturing alternatives sub-skill.
In this argument, Dobert provides a claim and an argument, however, so the
critical thinking cognitive skill, evaluation or assessing “the credibility of... a
101
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
person’s . . . belief, or opinion” is applicable (Facione, 1990, p. 15). The sub-skills,
assessing claims, and assessing arguments, as stated earlier, apply.
This item tests several Stahl N. N. and Stahl R. J. (1991) critical thinking
skills. These skills include: critical thinking seeks a clear statement of the issue (i.e.,
should Gallton chlorinate its water supply); critical thinking determines the strength
of an argument (i.e., can Dobert defend his argument); critical thinking judges
whether the problem has been identified; critical thinking keeps in mind the original
or basic concern (i.e., is Dobert correct in his thinking); critical thinking takes a
position when the evidence and reasons are sufficient to do so (i.e., is the answer
“A,” ”B,” or “C”); critical thinking determines the accuracy of a statement; critical
thinking tries to separate emotional thinking from logical thinking (i.e., Dobert’s
emotionally charged language); critical thinking judges whether there is ambiguity in
a line of reasoning, and critical thinking judges whether certain statements contradict
each other (i.e., Algan’s response to Dobert’s claim).
Question 16.
DOBERT: Laying aside the question of whether medication is bad or good, wouldn’t
you say that you are proposing a plan for medication?
ALGAN: Not at all. Is killing germs in the water supply the same as treating a
disease of the human body? Certainly not. Therefore, my plan cannot be called a
plan for medication.
102
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
DOBERT: Oh, but it is medication. Isn’t one of your stated goals the prevention of
disease? Medication is the process of trying to restore or preserve health in any
manner whatsoever. Whether your plan actually would result in preserving or
restoring health doesn’t matter. The point is that you would be trying to do so and
thus would be medicating people.
Pick the one best reason why some of this thinking is faulty.
A. There is a serious mistake in the thinking in this part.
B. Dobert’s conclusion doesn’t necessarily follow from the reasons he gives.
C. Dobert and Algan are using the same word differently.
In this conversation, Dobert and Algan are using the word, “medication”
differently. Algan defines medication as something one uses to treat a disease of the
human body. Dobert’s definition of medication is more encompassing. Dobert
defines medication as the process of trying to restore or preserve health in any
manner whatsoever, and thus concludes that since chlorinating Gallton’s water will
preserve health, it is also a plan for medication. Answer “C” is the best answer.
Answer “A” is circular reasoning, because the item directs one to pick the one best
answer why some of this thinking is faulty. Dobert’s conclusion does follow from
the definition he gives for medication, thus “B” is also incorrect.
This item tests several critical thinking skills such as interpretation or to
“comprehend. . . the meaning or significance of... situations . . . judgments . . . [or]
beliefs” (i.e., is chlorinating Gallton’s water supply commensurate with the concept
103
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
of medication) (Facione, 1990, p. 13). The question also asks one to decode the
significance or “describe the informational content” (i.e., the differing interpretations
of medication) (Facione, 1990, p. 14). The two other interpretation sub-skills,
categorization and clarifying meaning, are not applicable.
This question asks one to analyze or “identify . . . the intended and actual
inferential relationships among statements” (i.e., chlorinating the Gallton water
supply and the concept of medication) (Facione, 1990, p. 14) and examine ideas (i.e.,
a sub-skill of analysis) (Facione, 1990, p. 15). In examining ideas, one is asked “to
determine the role various expressions play or are intended to play in the context of
argument, reasoning, or persuasion” (i.e., the role of medication in supporting or not
supporting chlorinating the Gallton water supply) (Facione, 1990, p. 15). This item
also asks one to detect and analyze the arguments presented by both Dobert and
Algan, because one must identify the “intended main conclusion” once “given the
expression of a reason or reasons intended to support or contest some claim, opinion
or point of view” (Facione, 1990, p. 15).
The cognitive skill, inference, applies equally in this question because one is
asked “to identify and secure elements needed to draw reasonable conclusions . . . to
consider relevant information and to educe the consequences flowing from . . .
statements” (i.e., Dobert’s argument that Algan is proposing a medical plan for
Gallton) (Facione, 1990, p. 16). In this item, one must query the evidence or
104
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
“recognize premises which require support” (i.e., the definition of medication)
(Facione, 1990, p. 17) and draw some conclusion.
Evaluation is another cognitive skill this item tests, because it asks one “to
assess the logical strength of the actual or intended inferential relationships among
statements” (Facione, 1990, p. 15). In this item, the intended inferential relationship
is the one Dobert suggests (i.e., between chlorinating and medicating). Furthermore,
the sub-skills assessing claims and assessing arguments are applicable because one
must “judge whether the assumed acceptability of the premises of a given argument
justify one’s accepting as true (deductively certain), or very probably true
(inductively justified), the expressed conclusion of that argument” (i.e., chlorinating
is equivalent to medicating) (Facione, 1990, pp. 15-16).
This item tests several Stahl N. N. and Stahl R. J. (1991) critical thinking
skills. These skills include: critical thinking seeks a clear statement of the issue (i.e.,
is chlorinating Gallton’s water supply equivalent to medicating its people); critical
thinking determines the strength of an argument (i.e., does Dobert’s definition of
medication hold); critical thinking keeps in mind the original or basic concern (i.e.,
the role of semantics in argumentation); critical thinking takes a position when the
evidence and reasons are sufficient to do so (i.e., is the answer “A,” “B,” or “C”);
critical thinking determines the accuracy of a statement (i.e., the term “medication”
as defined by Dobert and Algan); critical thinking tries to separate emotional from
logical thinking (i.e., Algan’s use of the word, “killing”); critical thinking judges
105
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
whether there is ambiguity in a line of reasoning; and critical thinking judges
whether certain statements contradict each other (i.e., Dobert and Algan have
differing interpretations of “medication”).
The Cornell Critical Thinking Test, Level Z also tests for credibility,
induction in judging conclusions, and induction in planning experiments. To test for
this wide range of cognitive skills, the Cornell Critical Thinking Test, Level Z
presents the results of an experiment performed by Drs. E. E. Brown and M. R.
Kolter in the veterinary laboratory of the British Ministry of Agriculture and
Fisheries (Ennis et al., 1985a). In this experiment, the doctors were interested in what
happens to ducklings that eat cabbage worms. The doctors secured three types of
ducklings, and split each brood into two groups. Each brood was given the same diet,
except haif of each brood was provided two cabbage worms daily per duckling.
Ennis et al. (1985b) present the results of this experiment in a table and ask the
examinee 21 questions. These questions include credibility items, induction (i.e,
judging conclusion) items, and induction (i.e., planning experiments) items.
Example 3— Cornell Critical Thinking Test, Level Z credibility items.
Results of Duckling and Cabbage Worm Experiment
Type of
Duckling
Original
Number in
Brood Regular Diet Regular Diet Plus Worms
Healthy 1 1 1 Dead Healthy 1 1 1 Dead
Mallard 8 3 I
2 2
6 3 3
106
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Results of Duckling and Cabbage Worm Experiment (continued)
Type of
Duckling
Original
Number in
Brood Regular Diet Regular Diet Plus Worms
Healthy 1 1 1 Dead Healthy 1 1 1 Dead
Pintail 6
2
I 3
8 3 1 3
Canvasback 8 4 I 3
8 3 I 1 3
Total 44 18 3 1 4 17
The doctors drew the conclusion that cabbage worms are poisonous to ducklings.
Item 22 asks one to interpret the data in the above table. Item 24 asks one to
compare which subsequent report on the experiment is more believable. Each
question contains a pair of statements (A&B) that are underlined. In each item, if the
first statement is more believable, the answer is “A,” if the second statement is more
believable, the answer is “B,” and if neither statement is more believable that the
other, the answer is “C.”
Question 22.
A. Cabbage worms are poisonous to ducklings (said by Dr. Kolter).
B. Six Canvasbacks died during the week of the experiment (said by Dr. Kolter).
C. Neither statement is more believable.
“A” and “B” are statements from Dr. Kolter, thus the source is credible.
Since both statements are from a first-hand source, on the surface, both appear to be
equally believable. However, statement “A” is an inference from the data, and
107
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
statement “B” is an observation from the data. Since observations are more
believable than inferences, the correct answer is “B.”
This item asks one to interpret or “comprehend. . . the meaning or
significance of... data” (Facione, 1990, p. 13) in the table. Interpreting these data
requires the use of the categorization sub-skill or “apprehend or appropriately
formulate categories [or] distinctions . . . for understanding, describing or
characterizing information” (e.g., apprehending the significance of the type of
duckling, sample size, and the three different categories within the control group and
the experimental group) (Facione, 1990, p. 13). Likewise, the sub-skill, decoding
significance, is equally applicable because the item asks one “to . . . describe the
informational content. . . [in] numbers [or] tables” such as in the table (Facione,
1990, p. 14). The use and interpretation of data from tables requires the use of the
clarifying meaning sub-skill.
The table also asks one to analyze or “identify the intended and actual
inferential relationships among... other forms of representation intended to express
beliefs [or] judgments” (Facione, 1990, p. 14). Based on the number of ducklings
that died during the experiment, one can justifiably infer that cabbage worms are
poisonous to ducklings. The data also ask one to examine ideas and detect and
analyze the two claims presented in the item because “given a set of... graphic
representations . . . determine whether or not the set expresses . . . or [supports] some
108
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
claim, opinion, or point of view” (Facione, 1990, p. 14). The data seem to support
the doctors' claim or conclusion, but it is nonetheless still inconclusive.
This item also asks one to infer from the data “reasonable conclusions.” Six
canvasbacks dying during the experiment is a fact; the affect cabbage worms have on
ducklings is an inference. One can reasonably conclude that the former is more
believable. Likewise, the inference sub-skills, querying evidence and drawing
conclusions apply because the item asks one to “apply appropriate modes of
inference in determining what position . . . one should take on a given matter” (i.e.,
are cabbage worms poisonous to ducklings) (Facione, 1990, p. 17).
The credibility of Dr. Kolter is important in this item. In the critical thinking
skill, evaluation, one is asked “to assess the credibility of statements . . . and to
assess the logical strength of the actual or intended inferential relationships among ..
. other forms of representations” (e.g., data from the table and the statements from
Dr. Kolter) (Facione, 1990, p. 15). Similarly, the sub-skill, assessing claims, applies
because it asks one "to recognize the factors relevant to assessing the degree of
credibility to ascribe to a source of information or opinion” (Facione, 1990, p. 16).
The sub-skill, assessing arguments, applies because one must determine whether the
argument relies on false or doubtful assumptions or presuppositions.
The Stahl N. N. and Stahl R. J. (1991) cognitive skills for this item include:
critical thinking seeks a clear statement of the issue (i.e., which statement is more
believable); critical thinking determines the strength of an argument (i.e., is an
109
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
observation more believable); critical thinking judges whether the problem has been
identified (i.e., the problem is which should I believe); critical thinking keeps in
mind the original and/or basic concern (i.e., why is an observation more believable);
critical thinking takes a position when the evidence and reasons are sufficient to do
so (i.e., is the answer to the item, “A,” “B,” or “C”); critical thinking understands the
difference between a conclusion which might be true and one which must be true
(e.g., cabbage worms may be poisonous to ducklings and six canvasbacks died
during the experiment); critical thinking distinguishes between verifiable facts and
value claims (i.e., the fact that six canvasbacks died is verifiable); critical thinking
determines the accuracy of a statement; critical thinking distinguishes between
warranted and unwarranted claims; and critical thinking determines the reliability of
a claim or source (i.e.. Dr. Kolter).
Question 24.
A. During the week following the experiment, all of the ill ducklings died. (From an
article in a magazine that can be found on almost every newsstand. The author, a
popular international writer, stated that he obtained his information from Drs. Brown
and Kolter.)
B. During the week following the experiment, the rest of the worm-fed ducklings
died (from the report written by Drs. Brown and Kolter).
C. Neither statement is more believable.
110
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Unlike item 22, in this item, both “A” and “B” are observations vice
inferences. The difference is that “B” is more believable because the report was
written by Drs. Brown and Kolter (i.e., the scientists who conducted the experiment
and originally reported the results). The author in “A” is a secondary source, and
thus data, meaning, or significance may be lost in the translation.
This item requires the use of several critical thinking skills. This item asks
one to interpret or “com prehend. . . the meaning or significance of... judgments”
(i.e., primary versus secondary sources) (Facione, 1990, p. 13). One uses the data
from the table to answer this item, thus the sub-skills, categorization and clarifying
meaning, apply. One also needs to use the sub-skill decoding significance or “detect,
attend to, and describe the informational content [or] views” (i.e., the significance of
the two reports) to answer the item (Facione, 1990, p. 14).
Question 24 further asks one to analyze or “identify the intended and actual
inferential relationships among . . . concepts” (i.e., the validity of a secondary source
versus a primary source) (Facione, 1990, p. 14). All sub-skills are applicable because
this item asks one to examine ideas, detect arguments, and “compare or contrast”
them (Facione, 1990, pp. 14-15).
This item also asks one to infer or query the evidence from the statements
and draw “reasonable conclusions” that one report is more believable than the other.
In this item, one must draw conclusions because the item asks one to “apply
111
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
appropriate modes of inference in determining what position . . . one should take on
a given matter” (i.e., answer “B”) (Facione, 1990, p. 16).
The credibility of the doctors is critical in this item. One has no reason to not
believe the data in the table, thus, one has no reason to not believe the subsequent
Brown and Kolter report. In the critical thinking skill, evaluation, one is asked “to
assess the credibility of statements . . . and to assess the logical strength of the actual
or intended inferential relationships among . . . other forms of representations” (e.g.,
the strength of the magazine article versus the strength of the findings in the report
by the doctors) (Facione, 1990, p. 15). Likewise, the sub-skills, assessing claims and
assessing arguments, applies because it asks one “to recognize the factors relevant to
assessing the degree of credibility to ascribe to a source of information or opinion”
(Facione, 1990, pp. 15-16).
The Stahl N. N. and Stahl R. J. (1991) cognitive skills for this item include:
critical thinking seeks a clear statement of the issue (i.e., which report is more
believable); critical thinking determines the strength of an argument; critical thinking
judges whether the problem has been identified; critical thinking keeps in mind the
original and/or basic concern (i.e., why is a primary source more believable); critical
thinking takes a position when the evidence and reasons are sufficient to do so (i.e.,
is the answer to the item, “A,” “B,” or “C”); critical thinking distinguishes between
verifiable facts and value claims; critical thinking determines the accuracy of a
statement; critical thinking distinguishes between warranted and unwarranted claims
112
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
(i.e., the report from the original experimenters are warranted, whereas the report
from a secondary source may not be warranted); and critical thinking determines the
reliability of a claim or source.
Example 4— Cornell Critical Thinking Test. Level Z induction (i.e.. judging
conclusions) items.
Item 27 presents the results of a subsequent experiment, and one is asked to
assess whether the item supports the original conclusion (i.e., cabbage worms are
poisonous to ducklings), goes against the conclusion or does neither. If the item
supports the original conclusion, the answer is “A,” if the item goes against the
conclusion, the answer is “B,” and if the item does neither, the answer is “C.”
Question 27.
The experiment is repeated with three different varieties of ducklings that are
younger than the ones used in the original experiment. At the end of the week, two of
the regular-diet ducklings are dead, and twenty of the worm-diet ducklings are dead.
This experiment is better than replication because three different varieties of
younger ducklings are used, and the results are similar. This experiment supports the
original conclusion that cabbage worms are poisonous to ducklings because it
eliminates other factors which may have accounted for the variation in the first
experiment. For example, this second experiment suggests that cabbage worms are
poisonous to all varieties of ducklings, thus supporting the conclusion that cabbage
worms are poisonous to ducklings. Furthermore, since the ducklings in the second
113
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
experiment were younger than in the first, this experiment also suggests that cabbage
worms are dangerous to ducklings, regardless of the age of the ducklings.
This item requires the use of several critical thinking skills. This item asks
one to interpret or “comprehend . . . the meaning or significance of... situations or
data” (i.e., the significance of a second experiment in which some of the variables
have changed) (Facione, 1990, p. 13). Interpreting these data requires the use of the
categorization and clarifying meaning sub-skills or “apprehend or appropriately
formulate categories [or] distinctions . . . for understanding, describing or
characterizing information” (e.g., apprehending the significance of changing the
varieties of the ducklings in the second experiment) (Facione, 1990, p. 14). The sub
skill, decoding significance, is equally applicable because the item asks one “to
detect, attend to, and describe the informational content, affective purport” of the
second experiment (Facione, 1990, p. 14).
The second experiment further asks one “to identify the intended and actual
inferential relationships among . . . concepts” (e.g., the relationship between the first
and second experiments) and to compare and contrast them (Facione, 1990, p. 14).
The sub-skills, examining ideas, detecting arguments and analyzing arguments are
also applicable, because “given a set of statements [e.g., the results of the first and
second experiments, do the data suggest a] reason or reasons in support of or
contesting some claim, opinion or point o f view” (Facione, 1990, p. 15). The second
experiment supports the results in the first experiment.
114
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
This item also asks one to query the evidence and draw “ ‘reasonable
conclusions.” Since the second experiment has similar outcomes as the first
experiment, one can reasonably conclude that the doctors’ conclusion is valid.
Likewise, the inference sub-skill, drawing conclusions applies because the item asks
one to “apply appropriate modes of inference in determining what position . . . one
should take on a given matter” (i.e., is the conclusion valid) (Facione, 1990, p. 17).
In the critical thinking skill, evaluation, one is asked “to assess the credibility
of statements . . . and to assess the logical strength of the actual or intended
inferential relationships among . . . other forms of representations” (e.g.. the results
from the two experiments) (Facione, 1990, p. 15). Similarly, the sub-skills, assessing
claims and assessing arguments, apply because the item asks one “to recognize the
factors relevant to assessing the degree of credibility or to ascribe to a source of
information or opinion” (Facione, 1990, pp. 15-16). One does not know who
conducted the second experiment, but there is no reason to refute the findings from
the second experiment.
The Stahl N. N. and Stahl R. J. (1991) cognitive skills for this item include:
critical thinking seeks a clear statement of the issue (i.e., is the conclusion from the
first experiment valid); critical thinking determines the strength of an argument;
critical thinking keeps in mind the original and/or basic concern (i.e., why is the
original conclusion substantiated); critical thinking takes a position when the
evidence and reasons are sufficient to do so (i.e., is the answer to the item, “A,” “B,”
115
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
or “C”); critical thinking understands the difference between a conclusion which
might be true and one which must be true; critical thinking distinguishes between
verifiable facts and value claims; critical thinking recognizes logical inconsistencies
in a line of reasoning; and critical thinking judges whether certain statements
contradict each other.
Question 35.
The experiment is repeated in Canada with three different varieties of ducklings. All
of the ducklings die, whether worm-fed or not.
Question 35 is very similar to the previous item (i.e., question 27), with one
importance difference-all the ducklings die. Notwithstanding the reason why the
ducklings die, one is asked if the results of this second experiment supports or goes
against the original conclusion (i.e., cabbage worms are poisonous to ducklings) or
does neither. Since all the ducklings die, one can surmise that the ducklings died
from some other cause (e.g., all the ducklings’ food may have been tainted with a
poisonous substance such as a pesticide). Since this experiment neither supports nor
goes against the first experiment, the correct answer is “C.”
This item tests one’s ability to interpret or “comprehend. . . the meaning or
significance of... situations or data” (i.e., the significance of the results of the
second experiment on the first experiment’s conclusion) (Facione, 1990, p. 13). All
sub-skills apply since one uses the data from the table and must decode the
116
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
significance, or “detect, attend to, and describe the informational content, affective
purport” of the second experiment (Facione, 1990, pp. 13-14).
The second experiment further asks one to analyze or “identify the intended
and actual inferential relationships among . . . concepts” (e.g., the relationship
between the first and second experiments) and to compare and contrast them
(Facione, 1990, p. 14). Like in interpretation, all analysis sub-skills apply--
examining ideas, detecting and analyzing arguments-because “given a set of
statements [e.g., the results of the first and second experiments, do the data suggest
a] reason or reasons in support of or contesting some claim, opinion or point of
view” (Facione, 1990, p. 14). The second experiment neither supports nor goes
against the first experiment’s conclusion.
This item also asks one to infer and query from the data reasonable
conclusions. Since the second experiment has a confounding variable that may
account for the deaths of all the ducklings, one can reasonably conclude that the
results of the second experiment are irrelevant. Likewise, the inference sub-skill,
drawing conclusions applies because the item asks one to “apply appropriate modes
of inference in determining what position . . . one should take on a given matter”
(i.e., is the first experiment’s conclusion still valid) (Facione, 1990, p. 17).
Evaluation is also applicable because the item asks one “to assess the
credibility of statements . . . and to assess the logical strength of the actual or
intended inferential relationships among . . . other forms of representations” (e.g., the
117
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
results from the two experiments) (Facione, 1990, p. 15). Similarly, the sub-skills,
assessing claims and assessing arguments, apply because the item asks one '"to
recognize the factors relevant to assessing the degree of credibility or to ascribe to a
source of information or opinion” (i.e., the results of the second experiment)
(Facione, 1990, p. 16).
The Stahl N. N. and Stahl R. J. (1991) cognitive skills for this item are the
same as in question 27: critical thinking seeks a clear statement of the issue (i.e.,
does the conclusion from the first experiment still valid); critical thinking determines
the strength of an argument; critical thinking keeps in mind the original and/or basic
concern (i.e., why is the second experiment irrelevant); critical thinking takes a
position when the evidence and reasons are sufficient to do so (i.e., is the answer to
the item, “A,” “B,” or “C”); critical thinking understands the difference between a
conclusion that might be true or one which must be true; critical thinking
distinguishes between verifiable facts and value claims; critical thinking recognizes
logical inconsistencies in a line of reasoning; and critical thinking judges whether
certain statements contradict each other. In this item, there is a logical inconsistency
between the results of the first and second experiments.
Example 5— Cornell Critical Thinking Test. Level Z induction (i.e.. planning
experiment) item.
In the following item, a research worker sets out to test the truth of the
statement: if any duckling eats a cabbage worm, the duckling will die within six
hours. In planning the experiment, the researcher needs to make some predictions
118
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
from the statement that if any duckling eats a cabbage worm, the duckling will die
within six hours. There are two rules for the predictions: predictions tell what would
be true, if the statement were true, and predictions should be useful in guiding an
actual experiment. Keeping in mind the two rules about predictions, one is asked to
decide which experiment would provide the best prediction.
Question 42.
O f n, o, and p, which is the best prediction? Mark A for n; mark B for o; mark C for
P-
Possible predictions:
n. If one Mallard duckling is selected at random from each of six different broods,
and each selected duckling is fed a cabbage worm, all six ducklings will be dead
within six hours.
o. Suppose twelve hungry, randomly selected Canvasback ducklings are turned
loose for one hour in a cabbage patch containing cabbage worms and then put in a
clean cage for six hours. If each dies during that period, the results of the stomach
tests will show that each has eaten a cabbage worm.
p. If a group of ten healthy Canvasback ducklings that would probably live if not fed
cabbage worms is randomly split in half, and each half is treated the same except that
one group of five eats cabbage worms, then the worm-fed ducklings will die within
six hours and the other ducklings probably will not.
119
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
This item asks one to make predictions about the results of three different
types of experiments to test the statement that if any duckling eats a cabbage worm,
the duckling will die within six hours. One is then asked to select the best experiment
that will prove the statement.
Experiment “n” is a weak predictor for several reasons. Although one
Mallard duckling is selected at random from each of six different broods, nothing is
known about the health of the ducklings, and they may begin the experiment already
sick and ready to die. Even if all the ducklings were healthy, they could all die
because of a different reason, such as from drinking a contaminated water supply.
Compared to the other experiments, the sample size for experiment “n” is also very
small.
Experiment “o” is a weaker predictor than experiment “n.” Although the
sample size is larger in experiment “o” than in “n,” the twelve Canvasback ducklings
are turned loose in a cabbage patch containing cabbage worms for one hour then put
in a clean cage for six hours. This experiment lacks control. There is no guarantee
that any of the Canvasback ducklings will eat any cabbage worms. Once the
ducklings are placed in a clean cage, they could die from eating something else in the
cabbage patch that may be poisonous to ducklings, or die from lack of water.
Experiment “p” is the best of the three. Ten healthy Canvasback ducklings
are chosen, and then the ten ducklings are randomly split into two groups. One group
becomes the control group, and the other group becomes the experimental group.
120
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Since some of the ducklings in the control group may die for other reasons,
experiment “p” states that “the other ducklings probably will not” die, thus leaving
open the possibility.
This induction item tests all of Facione’s (1990) critical thinking skills and
sub-skills. One must interpret or “comprehend and express the meaning or
significance o f... rules and procedures” (i.e., how each experiment is conducted)
(Facione, 1990, p. 13). The sub-skill, categorization, is also applicable because
experiment “p” asks one to “formulate categories [and] distinctions” (e.g., control
and experimental groups) (Facione, 1990, p. 13). Likewise, one must decode the
significance of inferential relationships within and between each of the experiments.
The sub-skill, clarifying meaning applies because one must use data presented in the
table.
The critical thinking skill, analysis, also applies because one must “identify
the intended and actual inferential relationships among statements” (Facione, 1990,
p. 14), such as whether each experiment meets the two rules of predictions and
whether the experiments are good predictors of the statement (i.e., if any duckling
eats a cabbage worm, the duckling will die within six hours). One must also examine
ideas or “compare and contrast ideas, concepts, or statements” (i.e., the ideas and
concepts behind the three experiments); detect arguments to determine whether each
experiment will satisfy the prediction; and analyze arguments (i.e., “given the
expression of a reason or reasons intended to support or contest some claim . . .
121
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
identify and differentiate . . . unstated assumptions or presuppositions”) (Facione,
1990, p. 14).
This item asks one to infer, or “identify and secure elements needed to draw
reasonable conclusions; to form conjectures and hypotheses; to consider relevant
information and to educe the consequences flowing from data [or] statements”
(Facione, 1990, p. 16). Likewise, one must query evidence, conjecture alternatives,
and draw conclusions (i.e., determine which of several possible conclusions is most
strongly warranted or supported by the evidence at hand, or which should be rejected
or regarded as less plausible by the information given).
Comparing the strengths and weaknesses of the three experiments tests ones
ability to evaluate or “assess the credibility of statements [and] the logical strength of
the actual or intended inferential relationships among statements, descriptions,
questions or other forms of representation” (Facione. 1990, p. 15). This item also
tests evaluation’s two sub-skills— assessing claims (i.e., the credibility of the
experiments to meet their predictions) and assessing arguments (i.e., recognizing
weaknesses in arguments) (Facione, 1990, pp. 15-16).
This item also tests several Stahl N. N. and Stahl R. J. (1991) critical thinking
cognitive skills: critical thinking seeks a clear statement of the issue (i.e, what is the
best experiment); critical thinking identifies stated and unstated assumptions (i.e.,
what can one expect from each of the experiments); critical thinking determines the
strength of an argument (i.e., which experiment is the strongest); critical thinking
122
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
keeps in mind the original and/or basic concern (i.e., which experiment will best
prove the conclusion that if any duckling eats a cabbage worm, the duckling will die
within six hours); critical thinking takes a position when the evidence and reasons
are sufficient to do so (i.e., choosing between “n,” “o,” and “p”); critical thinking
recognizes logical inconsistencies in a line of reasoning; and critical thinking judges
whether there is ambiguity in a line of reasoning (i.e., will the experiments provide
data they predict).
Example 6— Cornell Critical Thinking Test. Level Z definition and
assumption identification item.
The following item provides a situation in which a definition is called for.
One must select the definition that gives the best meaning based upon an assumption
that fills the gap in the reasoning.
Question 43.
‘T hat’s a nice stock car you have there, Bill,” his mother remarked.
“Stock car!” exclaimed Bill. ‘T hat’s no stock car. Did you ever see a car in a
dealer’s showroom with bumpers made out of heavy pipe? Do the automobile
manufacturers turn out cars with no fenders? Of course not.”
Bill’s mother then asked, “Just what do you mean by ‘stock car’?”
Of the following, which is the best way to state Bill’s notion of a stock car?
A. A stock car is an automobile that is for the most part made of standard parts put
out by automobile manufacturers, but which might have missing fenders and special
bumpers.
123
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
B. A stock car is an automobile that has fenders and does not have bumpers made
out of pipe.
C. A stock car is a standard automobile, as turned out by the factory and sold to the
public.
This item asks one to fill the gap in reasoning. Based upon Bill’s response to
his mother’s question, Bill’s car has bumpers made of heavy pipe and no fenders,
and he does not consider his car a stock car. Bill’s mother asks him his notion of
“stock car.” Answer “A” is not Bill’s notion of “stock car” because stock cars should
have fenders and non-special bumpers. Answer “B” is incorrect because it implies
the bumpers can be made of anything (e.g., foam rubber), as long as it is not made
out of pipe. Answer “C” is the best definition because according to Bill, “stock”
implies standard, as turned out by the factory (i.e., without modifications) and sold to
the public.
This item asks one to interpret the significance and meaning of the
conversation between Bill and his mother. To do so, one must clarify the meaning of
the term, “stock car” according to their “frameworks for understanding” (Facione,
1990, p. 13) or what D. N. Perkins (1987) calls “thinking frames” (i.e.. Bill’s
mother’s definition versus Bill’s definition). One must also decode the significance
or “describe the informational content” (Facione, 1990, p. 14) of the conversation
and clarify the meaning of what Bill means by “stock car.”
124
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
This item also asks one to analyze “the intended and actual inferential
relationships among statements" (i.e., the analysis cognitive skill), and "determine
the role various expressions play or are intended to play” (i.e., the examining ideas
sub-skill) (Facione, 1990, p. 14). This item does not test for detecting arguments or
analyzing arguments.
Inference is tested in this item as well because it asks one to query the
conversation and "identify and secure elements needed to draw reasonable
conclusions” that answer “C” is what Bill defines as "stock car” (Facione, 1990, p.
16). Thus, drawing conclusions or “given a set of statements . . . educe with the
proper level of logical strength, their inferential relationships and the consequences
or the presuppositions which they support, warrant, imply or entail” applies
(Facione, 1990, p. 17).
The evaluation critical thinking cognitive skill is applicable because the item
asks one "to assess the logical strength . . . of intended inferential relationships” (i.e.,
what is the definition of “stock car”) (Facione, 1990, p. 15). The item also asks one
to "assess the contextual relevance of... information” (i.e., assessing claims, an
evaluation sub-skill) (Facione, 1990, p. 15).
The Stahl N. N. and Stahl R. J. (1991) critical thinking skills relevant to this
item include: critical thinking seeks a clear statement of the issue (i.e., correctly
defining what Bill says is a "stock car”); critical thinking identifies stated and
unstated assumptions (i.e., Bill’s definition); critical thinking judges whether the
125
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
problem has been identified (i.e., differences in the meaning of “stock car”); critical
thinking keeps in mind the original and/or basic concern (i.e., filling the gap); critical
thinking takes a position when the evidence and reasons are sufficient to do so (i.e.,
answer “A,” “B,” or “C”); critical thinking determines the accuracy of a statement
(i.e., the miscommunication in the conversation between Bill and his mother); and
critical thinking judges whether there is ambiguity in a line of reasoning.
Example 7-Com ell Critical Thinking Test, Level Z assumption identification
item.
In the following item, people are having a conversation and there is an
unstated assumption (i.e., a statement that is taken for granted). The item asks one to
select the one (i.e., “A,” “B,” or “C”) answer that is most probably the unstated
assumption.
Question 52.
Mr. Dobert: What we should do is never punish them. That would take care of
things.
A. Children who behave badly have been punished at some time.
B. Children who are punished will misbehave.
C. Children who behave properly have never been punished.
This item asks one to fill the gap in the reasoning. Mr. Dobert states that we
should never punish children, and that that would take care of children behaving
badly. Answer “B” is incorrect because Mr. Dobert is saying we should never punish
children. Answer “C” appears to be correct, but the logical form of the reasoning is
126
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
reversed from what Mr. Dobert is saying (i.e., the cause and effect are reversed).
Answer “A” is the correct answer, because it is roughly equivalent to the statement,
“Children who are not punished will behave well,” which is the more obvious gap
filler.
This item tests all the critical thinking cognitive skills-interpretation,
analysis, inference, and evaluation. Within the interpretation cognitive skill, this item
asks one to “comprehend and express the meaning or significance” of the statement
(i.e., the relationship between punishment and children who misbehave) (Facione,
1990, p. 13). It also asks one to decode the significance or the “affective purport” of
Mr. Dobert’s statements (Facione, 1990, p. 14).
Analysis is applicable as well because the item asks one “to identify the
intended and actual inferential relationships among statements” (Facione, 1990, p.
14). Likewise, the sub-skill, examining ideas, applies because one must “determine
the role various expressions play or are intended to play in the context of argument,
reasoning, or persuasion” (Facione, 1990, p. 15).
One must also query the conversation and “identify and secure elements
needed to draw reasonable conclusions . . . and form conjectures and hypotheses” to
Mr. Dobert’s argument (Facione, 1990, p. 16). Thus, the sub-skills, querying
evidence and drawing conclusions apply as well.
Finally, this item asks one to “assess the logical strength of the actual or
intended inferential relationships among statements” (i.e., the evaluation critical
127
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
thinking cognitive skill) (Facione, 1990, p. 15). That includes the assessing claims
sub-skill or assessing the contextual relevance of information (i.e., as stated by Mr.
Dobert) and assessing the argument within the conversation.
For this item, the Stahl N. N. and Stahl R. J. (1991) critical thinking cognitive
skills include: critical thinking seeks a clear statement of the issue (i.e., is there a
relationship between punishing children and children misbehaving); critical thinking
identifies stated and unstated assumptions (i.e., the gap Filler); critical thinking keeps
in mind the original and/or basic concern (i.e., unstated assumptions in dialogue);
critical thinking takes a position when the evidence and reasons are sufficient to do
so (i.e., answer “A,” “B,” or “C”); critical thinking understands the difference
between a conclusion that might be true or one that must be true; critical thinking
determines the accuracy of Mr. Dobert’s statement; critical thinking tries to separate
emotional from logical thinking; critical thinking detects bias; critical thinking
recognizes logical inconsistencies in a line of reasoning (e.g., the inconsistency in
answer “C”); critical thinking judges whether there is ambiguity in a line of
reasoning; and critical thinking judges whether certain statements contradict each
other (e.g., “B” contradicts Mr. Dobert’s statement).
I apply these criteria to the remaining items in the Cornell Critical Thinking
Test, Level Z and discuss the results in Chapter IV.
128
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Scoring Consistently
A special note should be made on scoring consistently within and between
raters. Consistent scoring is essential in evaluating the content validity of an
instrument because of the subjective nature and interpretation one could apply to
each item in the instrument. To achieve consistency within and between raters, each
rater should adhere to the same process. In assessing the content validity of the
analytical section in the 1997 GRE Practice General Test and the Cornell Critical
Thinking Test, Level Z, the process one should adhere to for each item includes the
following:
1. Carefully read through the stem and each response.
2. Ask yourself the question, “What exactly does the stem ask me to
answer?”
3. Take the necessary steps to solve the question. These steps may entail
drawing a diagram according to conditions, criteria or premises; evaluating data;
evaluating inferential relationships; or assessing premises and possible conclusions.
4. Take notes on all your thinking related to interpreting the stem and
evaluating each response.
5. Evaluate each response, even if the obvious correct answer is the first
response. This step is important because one may use critical thinking abilities in not
choosing a response as well as choosing a response (e.g., drawing conclusions).
129
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
6. After answering the question, use the score sheet in the appendix and look
for the key words and phrases that apply. There should be a one for one match
between what the rater used to answer the question and the language from the score
sheet. For example, does the question ask one to comprehend and express the
significance of situations, data, beliefs, rules, procedures or criteria (i.e.,
interpretation)? Or does the question ask one to formulate categories or different
frameworks for understanding (i.e., categorization as a sub-skill to interpretation)?
Do this for each Facione (1990) core critical thinking skill and sub-skill. Repeat the
process for each Stahl N. N. and Stahl R. J. (1991) critical thinking ability.
7. Before marking an “X” for any critical thinking skill, one must be able to
elaborate why that critical thinking skill is used. Likewise, if a core critical thinking
skill is not being used, one must be able to explain why (e.g., recall was used, vice a
critical thinking skill).
8. Repeat steps I through 7 for each item in the instrument.
A minor experiment in inter-judge reliability was conducted to evaluate
scoring consistency using this process. The experiment resulted in high inter-judge
reliability. Two items were scored— one logical reasoning item, and one analytical
reasoning item. The logical reasoning item resulted in agreement with 9 of the 11
Facione (1990) critical thinking sub-skills and the analytical reasoning item resulted
in agreement with all 11 of the 11 Facione critical thinking sub-skills. Using the
Stahl N. N. and Stahl R. J. (1991) critical thinking abilities, there was agreement in
130
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
all 16 of the 16 critical thinking abilities for both the logical reasoning and analytical
reasoning items.
131
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER IV
RESULTS AND DISCUSSION
This chapter presents the results from applying the critical thinking criteria
developed in Chapter III to the 1997 GRE Practice General Test and the Cornell
Critical Thinking Test, Level Z.
Table 6 shows the results when applying Facione’s criteria (1990) to
analytical section 2 of the 1997 GRE Practice General Test. Section 2 scored high in
all interpretation sub-skills, in the examining ideas sub-skill for analysis, in the
drawing conclusions sub-skill for inference, and in the assessing claims sub-skill for
evaluation. Several of the logical reasoning items also met the sub-skills for
argumentation-detecting arguments, analyzing arguments, querying evidence, and
assessing arguments.
The analytical reasoning items met Facione’s (1990) categorization sub-skill,
because these items typically required the examinee to comprehend and formulate
categories. These categories included, for example, assigning embassy workers to
four different embassies, two of which are located in countries with dry climates and
two of which are located in countries with humid climates according to certain rules,
procedures, criteria, or conditions. These rules, procedures, criteria or conditions also
helped to satisfy the decoding significance sub-skill, because in decoding
significance, one must detect, attend to and describe the informational content,
affective purport, rules, procedures, criteria or inferential relationships.
132
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The instructions (ETS, 1997, p. 9) for the analytical section state, “In
answering some of the questions, it may be useful to draw a rough diagram in each
of the analytical reasoning items.” In drawing a diagram, table, or figure, the
examinee meets the clarifying meaning sub-skill, as Facione (1990, p. 14) claims this
sub-skill is “to paraphrase or make explicit, through . . . figurative expression, the
contextual, conventional or intended meaning of words . . . charts, graphs, symbols
[or] rules.”
Both analytical and logical reasoning items met the examining ideas sub
skill. In examining ideas, one determines the role various expressions play or are
intended to play in argumentation, reasoning, or persuasion. This sub-skill also asks
one to identify the conceptual relationships of those parts to each other and to the
whole. Since analytical and logical reasoning items test the examinee’s ability to
make inferential relationships, these items overwhelmingly satisfied the examining
ideas sub-skill.
This section also met the drawing conclusions sub-skill within the inference
core critical thinking ability. Facione (1990) states that in this sub-skill, when
presented a set of statements, one can educe the proper level of logical strength and
inferential relationships between them to determine what position one should take or
determine which conclusion is most strongly supported by the evidence at hand.
133
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 6
GRE Practice General Test Analytical Section No. 2 Using Facione (1990) Criteria
Item INTb ANAf INF EVA"
No. Type* C DSd CMe EIB DAh A1 QEk CA1 DCm AC° AAP
1 AR X X X X X X
2
AR X X X X X X
3 AR X X X X X X
4 AR X X X X X X
5 AR X X X X X X
6 AR X X X X X X
7 AR X X X X X X
8 LR X X X X X X X X
9 LR X X X X X X X X
10 LR X X X X X X X X
1 1 AR X X X X X X
12 AR X X X
13 A R .' X X X X X X
14 AR X X X X X X
15 AR X X X X X X
16 AR X X X X X
17 AR X X X X X
18 AR X X X X X
19 AR X X X X X
20 AR X X X X X X
21 AR X X X X X X
22 AR X X X X X X
23 LR X X X X X X
24 LR X X X X X X X X
25 LR X X X X X X X X
I q 19 25 18 25 5 5 6 0 24 20 6
Notes.
JAR means analytical reasoning; LR means logical reasoning
b INT means Interpretation
d DS means Decoding Significance
fANA means Analysis
h DA means Detecting Arguments
'INF means Inference
'CA means Conjecturing Alternatives
"EVA means Evaluation
P AA means Assessing Arguments
cC means Categorization
e CM means Clarifying Meaning
s“E r means Examining Ideas
‘ A means Analyzing Arguments
k QE means Querying Evidence
“DC means Drawing Conclusions
“AC means Assessing Claims
q £ means Summation or Total
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Finally, several of the items met the assessing claims sub-skill. Facione
(1990) claims this sub-skill is one in which the examinee assesses the credibility of a
source or statement or assesses the contextual relevance of questions, information,
principles, rules or procedural directions. The analytical reasoning items meet the
latter criterion. To note, however, item 12 did not meet this sub-skill because recall
was the only skill required to answer it correctly. Likewise, questions 16 through 19
did not satisfy this sub-skill because these questions only require the examinee to
draw locations and one or two-way streets between them. To answer these questions
correctly, one need only trace the paths between the locations.
Table 7 shows the results when applying the Facione (1990) criteria to
analytical section 5 of the GRE Practice General Test. Similar to section 2, section 5
scored well in the interpretation core critical thinking ability, examining ideas sub
skill, drawing conclusions sub-skill, and the assessing claims sub-skill. The logical
reasoning items met the majority of the sub-skills common in argumentation. Table 8
compares and adds the results between the two sections. The results from Table 8
show that the analytical section of the GRE met at least one critical thinking sub-skill
within each of the four Facione core critical thinking abilities. Furthermore, at least
70% of the items met 6 of the 11 sub-skills. This summed score is later compared
with the Cornell Critical Thinking Test, Level Z scores since the latter contains 52
items to the 1997 GRE Practice General Test analytical sections’ 50 items.
135
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 7
GRE Practice General Test Analytical Section No. 5 Using Facione (1990) Criteria
Item INTb ANAf INF EVA"
No. TypeJ C6 DSd CMC El8 DAh A' QEk CA1 DCm AC° AAP
1 AR X X X X X
2
AR X X X X X X
3 AR X X X X X X
4 AR X X X X X X
5 LR X X X X X X X
6 LR X X X X X X X X
7 LR X X X X X
8 AR X X X X X X
9 AR X X X X X X
10 AR X X X X X X
1 1 AR X X X X X X
12 AR X X X X X X
13 AR X X X X X X
14 AR X X X X X X
15 AR X X X X X X
16 AR X X X X X X
17 AR X X X X X X
18 AR X X X X X X
19 AR X X X X X
20 AR X X X X X
21 AR X X X X X
22 AR X X X X X
23 LR X X X X X X X X
24 LR X X X X X X X X
25 LR X X X X X X X X
I q 19 25 19 25 5 5 5 0 25 20 6
Notes.
JAR means analytical reasoning; LR means logical reasoning
b INT means Interpretation
d DS means Decoding Significance
f ANA means Analysis
h DA means Detecting Arguments
’INF means Inference
'CA means Conjecturing Alternatives
"EVA means Evaluation
P AA means Assessing Arguments
c C means Categorization
e CM means Clarifying Meaning
g EI means Examining Ideas
'A means Analyzing Arguments
k QE means Querying Evidence
"DC means Drawing Conclusions
“AC means Assessing Claims
q £ means Summation or Total
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 8
Summary of GRE Practice General Test Analytical Sections 2 and 5 Using Facione
(1990) Criteria
Analyical INT ANAe INF1 EVAm
Section Cb DSC CMd El' DA8 Ah QEJ CAk DC1 AC" AA°
Section 2 19 25 18 25 5 5 6 0 24 20 6
Section 5 19 25 19 25 5 5 5 0 25 20 6
I p 38 50 37 50 10 10 1 1 0 49 40 12
Notes.
TNT means Interpretation b C means Categorization
C DS means Decoding Significance d CM means Clarifying Meaning
eANA means Analysis f “El” means Examining Ideas
g DA means Detecting Arguments h A means Analyzing Arguments
'INF means Inference ’ QE means Querying Evidence
k CA means Conjecturing Alternatives 'DC means Drawing Conclusions
“EVA means Evaluation "AC means Assessing Claims
°AA means Assessing Arguments P E means Summation or Total
Table 9 shows the results when applying the first seven Stahl N. N. and Stahl
R. J. (1991) criteria to analytical section 2. At least 70% of the items in section 2 met
5 of the 7 criteria. These criteria include: (a) critical thinking seeks a clear statement
of the issue; (b) critical thinking judges whether the problem has been identified; (c)
critical thinking keeps in mind the original and/or basic concern; (d) critical thinking
lakes a position when the evidence and reasons are sufficient to do so; and (e) critical
thinking understands the difference between a conclusion that might be true or must
be true. In several of the analytical reasoning items, the examinee had to distinguish
between answers that could be true or must be true. Only the logical reasoning items
asked one to identify stated and unstated assumptions and determine the strength of
an argument.
137
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 9
Analytical Section #2 Using Stahl N. N. and Stahl R. J. (1991) Criteria I through 7
Item3 Stahl N. N. & Stahl R. J. (1991) Criteria1 1
No. Type 1
2
3 4 5 6 7
1 AR X X X X X
2
AR X X X X X
3 AR X X X X X
4 AR X X X X X
5 AR X X X X X
6 AR X X X X X
7 AR X X X X X
8 LR X X X X X X X
9 LR X X X X X X
10 LR X X X X X X X
11 AR X X X X X
12 AR X X X
13 AR X X X X X
14 AR X X X X X
15 AR X X X X X
16 AR X X X X
17 AR X X X X
18 AR X X X X
19 AR X X X X
20 AR X X X X X
21 AR X X X X X
22 AR X X X X X
23 LR X X X X X X
24 LR X X X X X X X
25 LR X X X X X X X
I 25 4 6 25 25 24 20
Notes.
JAR means Analytical Reasoning; LR means Logical Reasoning; £ means Summation or Total
bl. Critical thinking seeks a clear statement of the issue.
2. Critical thinking identifies stated and unstated assumptions.
3. Critical thinking determines the strength of an argument.
4. Critical thinking judges whether the problem has been identified.
5. Critical thinking keeps in mind the original and/or basic concern.
6. Critical thinking takes a position when the evidence and reasons are sufficient to do so.
7. Critical thinking understands the difference between a conclusion that might or must be true.
138
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 10 presents the results of the remaining nine Stahl N. N. and Stahl R. J.
(199L) critical thinking skills. At least 70% of the items met 3 of the 9 categories: (a)
critical thinking determines the accuracy of a statement; (b) critical thinking
recognizes logical inconsistencies in a line of reasoning; and (c) critical thinking
judges whether certain statements contradict each other. The examinee uses the last
skill particularly in eliminating incorrect responses. The criteria, “critical thinking
judges whether there is ambiguity in a line of reasoning” scored high (i.e,. 17 of 25),
but did not exceed the 70% threshold.
Tables 11 and 12 depict the results when applying the Stahl N. N. and Stahl
R. J. (1991) critical thinking skills to section 5. These findings are consistent with the
results in section 2. Table 13 summarizes the results for the GRE Practice General
Test analytical sections using the Stahl N. N. and Stahl R. J. criteria I through 7.
139
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 10
Analytical Section #2 Using Stahl N. N. and Stahl R. J. (1991) Criteria 8 through 16
Itema Stahl N. N. & Stahl R. J. (1991) Criteria6
No. Type 8 9 10 1 1 12 13 14 15 16
1 AR X X X X
2
AR X X X X
3 AR X X X X
4 AR X X X X
5 AR X X X X
6 AR X X X X
7 AR X X X X
8 LR X X X X X
9 LR X X X X X
10 LR X X X X
1 1 AR X X X X
12 AR
13 AR X X X X
14 AR X X X X
15 AR X X X X
16 AR
17 AR
18 AR
19 AR
20 AR X X X
21 AR X X X
22 AR X X X
23 LR X X X X
24 LR X X X X X
25 LR X X X X X
I 0 20 0 0 20 4 17 20 0
Notes.
JAR means Analytical Reasoning; LR means Logical Reasoning; I means Summation or Total
b8. Critical thinking distinguishes between verifiable facts and value claims.
9. Critical thinking determines the accuracy of a statement.
10. Critical thinking tries to separate emotional thinking from logical thinking.
11. Critical thinking detects bias.
12. Critical thinking recognizes logical inconsistencies in a line of reasoning.
13. Critical thinking distinguishes between warranted and unwarranted claims.
14. Critical thinking judges whether there is ambiguity in a line of reasoning.
15. Critical thinking judges whether certain statements contradict each other.
16. Critical thinking determines the reliability of a claim or source.
140
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table L L
Analytical Section #5 Using Stahl N. N. and Stahl R. J. (1991) Criteria 1 through 7
Item* Stahl N. N. & Stahl R. J. (1991) Criteriab
No. Type 1
2
3 4 5 6 7
1 AR X X X X X
2
AR X X X X X
3 AR X X X X X
4 AR X X X X X
5 LR X X X X X X
6 LR X X X X X X X
7 LR X X X X X
8 AR X X X X X
9 AR X X X X X
10 AR X X X X X
11 AR X X X X X
12 AR X X X X X
13 AR X X X X X
14 AR X X X X X X
15 AR X X X X X X
16 AR X X X X X X
17 AR X X X X X X
18 AR X X X X X X
19 AR X X X X X
20 AR X X X X X
21 AR X X X X X
22 AR X X X X X
23 LR X X X X X
24 LR X X X X X X X
25 LR X X X X X X
I
25 7 5 25 25 25 24
Notes.
JAR means Analytical Reasoning; LR means Logical Reasoning; £ means Summation or Total
bl. Critical thinking seeks a clear statement of the issue.
2. Critical thinking identifies stated and unstated assumptions.
3. Critical thinking determines the strength of an argument.
4. Critical thinking judges whether the problem has been identified.
3. Critical thinking keeps in mind the original and/or basic concern.
6. Critical thinking takes a position when the evidence and reasons are sufficient to do so.
7. Critical thinking understands the difference between a conclusion that might or must be true.
141
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 12
Analytical Section #5 Using Stahl N. N. and Stahl R. J. (1991) Criteria 8 through 16
Iten V * _____________ Stahl N. N. & Stahl R. J. (1991) Criteria6
No. Type 8 9 10 1 1 12 13 14 15 16
1 AR
2
AR X X X X
3 AR X X X X
4 AR X X X X
5 LR X X X X X
6 LR X X X X X
7 LR X X X X
8 AR X X X X
9 AR X X X X
10 AR X X X X
1 1 AR X X X X
12 AR X X X X
13 AR X X X X
14 AR X X X
15 AR X X X
16 AR X X X
17 AR X X X
18 AR X X X
19 AR
20 AR
21 AR
22 AR
23 LR X X X X
24 LR X X X X
25 LR X X X X X
I 0 20 0 0 20 3 15 20 0
Notes.
JAR means Analytical Reasoning; LR means Logical Reasoning; S means Summation or Total
b8. Critical thinking distinguishes between verifiable facts and value claims.
9. Critical thinking determines the accuracy o f a statement.
10. Critical thinking tries to separate emotional thinking from logical thinking.
11. Critical thinking detects bias.
12. Critical thinking recognizes logical inconsistencies in a line of reasoning.
13. Critical thinking distinguishes between warranted and unwarranted claims.
14. Critical thinking judges whether there is ambiguity in a line of reasoning.
15. Critical thinking judges whether certain statements contradict each other.
16. Critical thinking determines the reliability of a claim or source.
142
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 13
Summary of GRE Practice General Test Analytical Sections 2 and 5 Using Stahl N.
N. and Stahl R. J. (1991) Criteria 1 through 7
Analytical ___________ Stahl N. N. & Stahl R. J. (1991) Criteria1 1
Section 1
2
3 4 5 6 7
Section 2 25 4 6 25 25 24 20
Section 5 25 7 5 25 25 25 24
Sb
50 11 11 50 50 49 44
Notes.
*1. Critical thinking seeks a clear statement of the issue.
2. Critical thinking identifies stated and unstated assumptions.
3. Critical thinking determines the strength of an argument.
4. Critical thinking judges whether the problem has been identified.
5. Critical thinking keeps in mind the original and/or basic concern.
6. Critical thinking takes a position when the evidence and reasons are sufficient to do so.
7. Critical thinking understands the difference between a conclusion that might or must be true.
‘ ’ S means Summation or Total
Table 14 finishes the results for the GRE Practice General Test analytical
sections using the remaining Stahl N. N. and Stahl R. J. nine critical thinking skills.
I repeated the above process for measuring critical thinking in the Cornell
Critical Thinking Test, Level Z. Table 15 shows the results when applying the
Facione (1990) criteria to the Cornell Critical Thinking Test, Level Z, in the same
manner as they were applied to the analytical sections of the GRE Practice General
Test. At least 70% of the Cornell Critical Thinking Test, Level Z items met 8 of the
11 Facione (1990) critical thinking sub-skills. These sub-skills include decoding
significance, examining ideas, detecting arguments, analyzing arguments, querying
evidence, drawing conclusions, assessing claims, and assessing arguments.
143
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 14
Summary of GRE Practice General Test Analytical Sections 2 and 5 Using Stahl N.
N. and Stahl R. J. (1991) Criteria 8 through 16
Analytical ______________ Stahl N. N. & Stahl R. J. (1991) Criteria*
Section 8 9 10 1 1 12 13 14 15 16
Section 2 0 20 0 0 20 4 17 20 0
Section 5 0 20 0 0 20 3 15 20 0
Sb 0 40 0 0 40 7 32 40 0
Notes.
*8. Critical thinking distinguishes between verifiable facts and value claims.
9. Critical thinking determines the accuracy of a statement.
10. Critical thinking tries to separate emotional thinking from logical thinking.
11. Critical thinking detects bias.
12. Critical thinking recognizes logical inconsistencies in a line of reasoning.
13. Critical thinking distinguishes between warranted and unwarranted claims.
14. Critical thinking judges whether there is ambiguity in a line of reasoning.
15. Critical thinking judges whether certain statements contradict each other.
16. Critical thinking determines the reliability of a claim or source,
means Summation or Total
144
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 15
Cornell Critical Thinking Test. Level Z Using Facione (1990) Criteria
Item INTb ANAf INF EVA"
No. Type* c* DSd CMe EIg DAh A' QEk CA1 DCm AC° AAP
1 D X X X X X X X X
2
D X X X X X X X X
3 D X X X X X X X X
4 D X X X X X X X X
5 D X X X X X X X X
6 D X X X X X X X X
7 D X X X X X X X X
8 D X X X X X X X X
9 D X X X X X X X X
10 D X X X X X X X X
1 1 S X X X X X X X X
12 S X X X X X X X X
13 S X X X X X X X X
14 S X X X X X X X X
15 S X X X X X X X X
16 S X X X X X X X X
17 S X X X X X X X X
18 S X X X X X X X X
19 S X X X X X X X X
20 S X X X X X X X X
21 S X X X X X X X
22 C X X X X X X X X X X
23 C X X X X X X X X X X
24 C X X X X X X X X X X
25 C X X X X X X X X X X
Notes.
a D means Deduction; S means Semantics; C means Credibility
bINT means Interpretation
d DS means Decoding Significance
fANA means Analysis
h DA means Detecting Arguments
JINF means Inference
'CA means Conjecturing Alternatives
"EVA means Evaluation
P AA means Assessing Arguments
c C means Categorization
e CM means Clarifying Meaning
g “E r means Examining Ideas
‘ A means Analyzing Arguments
k QE means Querying Evidence
m DC means Drawing Conclusions
“AC means Assessing Claims
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 15 (continued)
Cornell Critical Thinking Test. Level Z Using Facione (1990) Criteria
Item INT ANAf INF EVA"
No. Type* C“ DSd CMe El8 DAh A' QEk CA1 DCm AC° AAP
26 I(IC) X X X X X X X X
27 I(JC) X X X X X X X X X X
28 I(JC) X X X X X X X X X X
29 I(JC) X X X X X X X X X X
30 KJC) X X X X X X X X X X
31 I(JC) X X X X X X X X X X
32 KJC) X X X X X X X X X X
33 I(JC) X X X X X X X X X X
34 I(JC) X X X X X X X X X X
35 KJC) X X X X X X X X X X
36 I(JC) X X X X X X X X X X
37 KJC) X X X X X X X X X X
38 KJC) X X X X X X X X X X
39 I(PE) X X X X X X X X X X X
40 KPE) X X X X X X X X X X X
41 KPE) X X X X X X X X X X X
42 KPE) X X X X X X X X X X X
43 DAI X X X X X X
44 DAI X X X X X X
45 DAI X X X X X X
46 DAI X X X X X X
47 AI X X X X X X
48 AI X X X X X X
49 AI X X X X X X
50 AI X X X X X X
51 AI X X X X X X
52 AI X X X X X X
Sq 21 52 25 52 41 41 52 6 49 52 48
Notes.
T(JC) means Induction (Judging Conclusions); I(PC) means Induction (Planning Experiments); DAI
means Definition and Assumption Identification; & AI means Assumption Identification
b INT means Interpretation
dDS means Decoding Significance
rANA means Analysis
h DA means Detecting Arguments
'INF means Inference
'CA means Conjecturing Alternatives
"EVA means Evaluation
P AA means Assessing Arguments
146
c C means Categorization
eCM means Clarifying Meaning
* “E r means Examining Ideas
'A means Analyzing Arguments
k QE means Querying Evidence
'"DC means Drawing Conclusions
“AC means Assessing Claims
q Z means Summation or Total
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Tables 16 and 17 present the results when applying the Stahl N. N. and Stahl
R. J. criteria to the Cornell Critical Thinking Test, Level Z. At least 70% of the items
met 5 of the 16 criteria. These criteria include: (a) critical thinking seeks a clear
statement of the issue; (b) critical thinking determines the strength of an argument;
(c) critical thinking keeps in mind the original and/or basic concern; (d) critical
thinking takes a position when the evidence and reasons are sufficient to do so, and
(e) critical thinking judges whether certain statements contradict each other.
However, four criteria just missed the 70% cut-off by two items (i.e., they scored 35
out of 52 vice the required 37 out of 52). These criteria include: (a) critical thinking
understands the difference between a conclusion that might be true and one that must
be true: (b) critical thinking determines the accuracy of a statement; (c) critical
thinking recognizes logical inconsistencies in a line of reasoning; and (d) critical
thinking judges whether there is ambiguity in a line of reasoning.
Table 18 compares the 1997 GRE Practice General Test analytical section
with the Cornell Critical Thinking Test, Level Z using the Facione (1990) criteria.
The Cornell Critical Thinking Test, Level Z scored decisively better than the 1997
GRE Practice General Test analytical section in 6 of the 11 sub-skills. Tables 19 and
20 compare the scores between the 1997 GRE Practice General Test analytical
sections with the Cornell Critical Thinking Test, Level Z using the Stahl N. N. and
Stahl R. J. (1991) criteria. The Cornell Critical Thinking Test, Level Z scored
147
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
decisively better than the GRE Practice General Test analytical sections in 7 of the
16 criteria.
Does the Analytical Section of the GRE Reasonably Measure Critical Thinking?
This content validity study suggests that the analytical section of the GRE
reasonably measures critical thinking. The analytical section of the GRE met at least
one critical thinking sub-skill for each of the four Facione (1990) core critical
thinking abilities— interpretation, analysis, inference, and evaluation. Furthermore,
70% of the items in the 1997 GRE General Test analytical section met 6 of the 11
Facione critical thinking sub-skills compared to 8 of the 11 Facione critical thinking
sub-skills for the Cornell Critical Thinking Test, Level Z. If these core critical
thinking abilities and sub-skills were used as a standard, the content validity suggests
the analytical section of the GRE reasonably measures critical thinking. The Stahl N.
N. and Stahl R. J. (1991) criteria lead one to a similar conclusion.
When compared to the Stahl N. N. and Stahl R. J. (1991) criteria, at least
70% of the analytical section items in the 1997 GRE Practice General Test met 8 of
the 16 criteria. When using the same 70% standard, the Cornell Critical Thinking
Test, Level Z met 5 of the 16 criteria. Sixty-seven percent of the items in the Cornell
148
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 16
Cornell Critical Thinking Test Using Stahl and Stahl (1991) Criteria 1 through 7
Item1 Stahl N. N. & Stahl R. J. (1991) Criteria*1
No. Type I
2
3 4 5 6 7
1 D X X X X X X
2
D X X X X X X
3 D X X X X X X
4 D X X X X X X
5 D X X X X X X
6 D X X X X X X
7 D X X X X X X
8 D X X X X X X
9 D X X X X X X
10 D X X X X X X
11 S X X X X X
12 S X X X X X
13 S X X X X X
14 S X X X X X
15 S X X X X X
16 S X X X X
17 S X X X X X
18 s X X X X
1 9 s X X X X
20 s X X X X
21 s X X X X
22 c X X X X X X
23 c X X X X X X
24 c X X X X X
25 c X X X X X
Notes.
JD means Deduction; S means Semantics; C means Credibility
bl. Critical thinking seeks a clear statement of the issue.
2. Critical thinking identifies stated and unstated assumptions.
3. Critical thinking determines the strength o f an argument.
4. Critical thinking judges whether the problem has been identified.
5. Critical thinking keeps in mind the original and/or basic concern.
6. Critical thinking takes a position when the evidence and reasons are sufficient to do so.
7. Critical thinking understands the difference between a conclusion that might or must be true.
149
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 16 (continued)
Cornell Critical Thinking Test Using Stahl and Stahl (1991) Criteria 1 through 7
Item4 Stahl N. N. & Stahl R. J. (1991) Criteria6
No. Type 1
2
3 4 5 6 7
26 I(JC) X X X X
27 KJC) X X X X X
28 KJC) X X X X X
29 I(JC) X X X X X
30 KJC) X X X X X
31 I(JC) X X X X X
32 I(JC) X X X X X
33 KJC) X X X X X
34 I(JC) X X X X X
35 I(JC) X X X X X
36 KJC) X X X X X
37 KJC) X X X X X
38 KJC) X X X X X
39 KPE) X X X X X X X
40 KPE) X X X X X X X
41 KPE) X X X X X X X
42 KPE) X X X X X
43 DAI X X X X X
44 DAI X X X X X
45 DAI X X X X X
46 DAI X X X X X
47 AI X X X X X
48 AI X X X X X
49 AI X X X X X
50 AI X X X X X
51 AI X X X X X
52 AI X X X X X
V
52 15 41 26 52 51 35
Notes.
JI(JC) means Induction (Judging Conclusions); l(PE) means Induction (Planning Experiments); DAI
means Definition and Assumption Identification; and AI means Assumption Identification; £ means
Summation or Total
bl. Critical thinking seeks a clear statement of the issue.
2. Critical thinking identifies stated and unstated assumptions.
3. Critical thinking determines the strength of an argument.
4. Critical thinking judges whether the problem has been identified.
5. Critical thinking keeps in mind the original and/or basic concern.
6. Critical thinking takes a position when the evidence and reasons are sufficient to do so.
7. Critical thinking understands the difference between a conclusion that might or must be true.
150
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 17
Cornell Critical Thinking Test Using Stahl and Stahl (1991) Criteria 8 through 16
Item* Stahl N. N. & Stahl R. J. (1991) Criteria5
No. Type 8 9 10 1 1 12 13 14 1 5 16
1 D X X X X
2
D X X X X
3 D X X X X
4 D X X X X
5 D X X X X
6 D X X X X X X
7 D X X X X X X
8 D X X X X X X
9 D X X X X X
10 D X X X X
11 S X X X X
12 S X X X X
13 s X X X X
14 s X X X X
15 s X X X X
16 s X X X X
17 s X X X
18 s X X X X
19 s X X X X
20 s X X X X
21 s X X X X X
22 c X X X X
23 c X X X X
24 c X X X X
25 c X X X X
Notes.
JD means Deduction; S means Semantics; C means Credibility
b8. Critical thinking distinguishes between verifiable facts and value claims.
9. Critical thinking determines the accuracy of a statement.
10. Critical thinking tries to separate emotional thinking from logical thinking.
11. Critical thinking detects bias.
12. Critical thinking recognizes logical inconsistencies in a line of reasoning.
13. Critical thinking distinguishes between warranted and unwarranted claims.
14. Critical thinking judges whether there is ambiguity in a line of reasoning.
15. Critical thinking judges whether certain statements contradict each other.
16. Critical thinking determines the reliability of a claim or source.
151
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 17 (continued)
Cornell Critical Thinking Test Using Stahl and Stahl (1991) Criteria 8 through 16
Item" Stahl N. N. & Stahl R. J. (1991) Criteria1 1
No. Type 8 9 10 11 12 13 14 15
26 KJC) X X
27 KJC) X X X
28 KJC) X X X
29 KJC) X X X
30 KJC) X X X
31 KJC) X X X X
32 KJC) X X X
33 KJC) X X X X
34 KJC) X X X
35 KJC) X X X
36 KJC) X X X
37 KJC) X X X
38 KJC) X X X
39 KPE) X X
40 KPE) X X
41 KPE) X X
42 KPE) X X
43 DAI X X
44 DAI X X
45 DAI X X
46 DAI X X
47 AI X X X X X X
48 AI X X X X X X
49 AI X X X X X X
50 AI X X X X X X
51 AI X X X X X X
52 AI X X X X X X
T 17 35 19 9 35 6 35 39
Notes.
JI(JC) means Induction (Judging Conclusions): I(PE) means Induction (Planning Experiments); DAI
means Definition and Assumption Identification; AI means Assumption Identification; I means
Summation or Total
b8. Critical thinking distinguishes between verifiable facts and value claims.
9. Critical thinking determines the accuracy o f a statement.
10. Critical thinking tries to separate emotional thinking from logical thinking.
11. Critical thinking detects bias.
12. Critical thinking recognizes logical inconsistencies in a line of reasoning.
13. Critical thinking distinguishes between warranted and unwarranted claims.
14. Critical thinking judges whether there is ambiguity in a line of reasoning.
15. Critical thinking judges whether certain statements contradict each other.
16. Critical thinking determines the reliability of a claim or source.
152
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 18
Comparison of GRE Practice General Test Analytical Sections with Cornell Critical
Thinking Test. Level Z using Facione (1990) Criteria
INT ANA* INF1 EVA"
Instrument Cb DSC CMd El' DAg Ah QB CA DC' AC" AA°
GRE4 38 50 37 50 10 10 1 1 0 49 40 12
CCTTr 21 52 25 52 41 41 52 6 49 52 48
Ap 17 -2 12
_2
-31 -31 -41 -6 0 -12 -36
Notes.
aINT means Interpretation
C DS means Decoding Significance
*ANA means Analysis
8 DA means Detecting Arguments
'INF means Inference
k CA means Conjecturing Alternatives
"’ EYA means Evaluation
°AA means Assessing Arguments
q GRE means 1997 GRE Practice General Test. Analytical Sections 2 and 5
rCCTT means Cornell Critical Thinking Test. Level Z
b C means Categorization
d CM means Clarifying Meaning
f " ‘EI” means Examining Ideas
h A means Analyzing Arguments
'QE means Querying Evidence
'DC means Drawing Conclusions
"AC means Assessing Claims
P A means Difference
Table 19
Comparison of GRE Practice General Test Analytical Sections with Cornell Critical
Thinking Test. Level Z using Stahl N. N. and Stahl R. J. (1991) Criteria 1 through 7
Stahl N. N. & Stahl R. J. (1991) Criteria1 1
Instrument i
2
3 4 5 6 7
GREa 50 1 1 1 1 50 50 49 44
CCTTb 52 15 41 26 52 51 35
r
-2
-4 -30 24 -2
.2
9
Notes.
“GRE is 1997 GRE Practice General Test Analytical Sections 2 and 5
b CCTT is Cornell Critical Thinking Test, Level Z
C A is Difference
d 1. Critical thinking seeks a clear statement o f the issue.
2. Critical thinking identifies stated and unstated assumptions.
3. Critical thinking determines the strength o f an argument.
4. Critical thinking judges whether the problem has been identified.
3. Critical thinking keeps in mind the original and/or basic concern.
6. Critical thinking takes a position when the evidence and reasons are sufficient to do so.
7. Critical thinking understands the difference between a conclusion that might or must be true.
153
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Table 20
Comparison of GRE Practice General Test Analytical Sections with Cornell Critical
Thinking Test. Level Z using Stahl N. N. and Stahl R. J. (1991) Criteria 8 through 16
Stahl N. N. & Stahl R. J. (1991) Criteria*1
Instrument 8 9 10 1 1 12 13 14 15 16
GRE 0 40 0 0 40 7 32 40 0
CCTT 17 35 19 9 35 6 35 39 6
Eb -17 5 -19 -9 5 1 -3 I -6
Notes.
'*GRE means 1997 GRE Practice General Test Analytical Sections 2 and 5
b CCTT means Cornell Critical Thinking Test. Level Z
C A means Difference
d8. Critical thinking distinguishes between verifiable facts and value claims.
9. Critical thinking determines the accuracy o f a statement.
10. Critical thinking tries to separate emotional thinking from logical thinking.
11. Critical thinking detects bias.
12. Critical thinking recognizes logical inconsistencies in a line of reasoning.
13. Critical thinking distinguishes between warranted and unwarranted claims.
14. Critical thinking judges whether there is ambiguity in a line of reasoning.
13. Critical thinking judges whether certain statements contradict each other.
16. Critical thinking determines the reliability of a claim or source.
Critical Thinking Test, Level Z met four additional Stahl N. N. and Stahl R. J.
critical thinking abilities. If one were to add these four additional criteria, the total
number of critical thinking abilities increases to 9 of 16 criteria for the Cornell
Critical Thinking Test, Level Z, which equals about the same amount for the
analytical section of the 1997 GRE Practice General Test. Using the number of Stahl
N. N. and Stahl R. J. critical thinking criteria as a standard, this content validity
study further suggests the analytical section of the GRE measures critical thinking.
In a “head-to-head” comparison between the analytical section of the 1997
GRE Practice General Test and the Cornell Critical Thinking Test, Level Z, the
Cornell Critical Thinking Test, Level Z out-numbered the analytical section of the
154
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
1997 GRE Practice General Test in 6 of the 11 Facione (1990) sub-skills. These sub
skills were primarily in the area of argumentation-detecting arguments, analyzing
arguments, querying evidence, conjecturing alternatives, assessing claims, and
assessing arguments. Of the remaining Five sub-skills, the analytical section of the
1997 GRE Practice General Test out-numbered the Cornell Critical Thinking Test,
Level Z, in two sub-skills, and the remaining three sub-skills were generally equal
between the two instruments.
In a “head-to-head” comparison using the Stahl N. N. and Stahl R. J. (1991)
criteria, the Cornell Critical Thinking Test, Level Z out-numbered the analytical
section of the 1997 GRE Practice General Test in 7 of the 16 criteria, the analytical
section of the 1997 GRE Practice General Test out-numbered the Cornell Critical
Thinking Test, Level Z in 4 of the 16 criteria, and the remaining 5 criteria were
generally equal between the two instruments. One would expect better performance
from the Cornell Critical Thinking Test, Level Z, as it is specifically designed to
measure critical thinking.
One should also note, however, that five critical thinking content validity
weaknesses were uncovered in the analytical section of the 1997 GRE Practice
General Test. There were no items which measure one’s ability to conjecture
alternative hypotheses (i.e., the conjecturing alternatives sub-skill of inference
(Facione, 1990)). The Cornell Critical Thinking Test, Level Z met this criterion
primarily in the induction (i.e., planning experiments) questions. Also missing were
155
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
items that test these Stahl N. N. and Stahl R. J. (1991) critical thinking skills: (a)
critical thinking distinguishes between verifiable facts and value claims; (b) critical
thinking tries to separate emotional thinking from logical thinking; (c) critical
thinking detects bias; and (d) critical thinking determines the reliability of a source.
The Cornell Critical Thinking Test, Level Z met these criteria with items that
addressed contentious issues (e.g., voting rights or immigration) and with items that
addressed the credibility of various types of data and sources.
The final caution comes from Messick (1989). According to Messick, content
validity does not by itself answer the question, “Does the analytical section of the
GRE reasonably measure critical thinking?” Content validity resides in a test,
whereas construct validity is a property of test responses and of the inferences based
thereon (Messick, p. 41). ‘T he major problem here is that so-called content validity
is focused upon test forms rather than test scores, upon instruments rather than
measurements (Messick).” Messick goes on to state that “Strickly speaking, even
from a content viewpoint, it would be more apropos to conceptualize content validity
as residing not in the test, but in the judgment of experts about domain relevance and
representativeness.” As in this study, the criteria I used were developed from critical
thinking experts (Facione, 1990; Stahl N. N. & Stahl R. J., 1991). Thus, the most
accurate response to the original question is, ‘The content validity of the analytical
section of the GRE General Test suggests it reasonably measures critical thinking.”
156
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Accepting this study’s conclusion should lead one to conduct further research
into the construct validity of critical thinking in the analytical section of the GRE
General Test. This construct validity may include comparing examinees’ GRE
analytical section test scores with their Cornell Critical Thinking, Level Z test scores.
This study would be a major undertaking, however, since no database currently
exists in which the researcher could make such a comparison. In the absence of such
data, if the content validity of the analytical section of the GRE General Test
suggests it reasonably measures critical thinking, then graduate school admissions
officials should consider including the analytical section test scores of the GRE
General Test in their admissions decisions.
157
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
CHAPTER V
SUMMARY, CONCLUSION, AND RECOMMENDATION
Summary
Graduate school admissions officials use several determinants for either
accepting or rejecting their applicants. To help assess variations between these
determinants, graduate school admissions officials use normalized test scores such as
the scores from the GRE General Test to assist them in differentiating the cognitive
abilities of their applicants. Graduate school admissions officials, however, typically
use the cumulative score from the quantitative and verbal sections of the GRE
General Test in their selection process, and typically ignore the analytical test score
of the GRE General Test. This omission of the analytical test score in graduate
school admissions decisions may be unfortunate because the analytical section of the
GRE General Test may measure another ability universities have enthusiastically
embraced as a central outcome of higher education-critical thinking. If critical
thinking is an outcome of higher education, and if the analytical section of the GRE
General Test measures critical thinking abilities, then graduate school admissions
officials should not ignore the significance of the analytical test score in their
admissions decisions. The problem is determining whether the analytical section of
the GRE General Test reasonably measures critical thinking.
The purpose of this conceptual study is to assess the content validity of the
analytical section of the GRE General Test to determine whether the analytical
158
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
section of the GRE General Test reasonably measures critical thinking. To achieve
this purpose, this study answers the question, “What is critical thinking,” develops a
set of criteria for measuring critical thinking, and determines if the analytical section
of the GRE General Test reasonably measures critical thinking. The criteria are
derived from two published Delphi Technique experiments conducted by Facione
(1990) and Stahl N. N. and Stahl R. J. (1991). This study applies the set of critical
thinking criteria to the analytical sections of the 1997 GRE Practice General Test and
the Cornell Critical Thinking Test. Level Z. This study answers the following
research questions.
1. What is critical thinking?
2. How does the philosophy school of thought view critical thinking?
3. How does the psychology school of thought view critical thinking?
4. Is there a consensus of what critical thinking is?
5. How does critical thinking compare to creative thinking?
6. How does critical thinking compare to problem solving?
7. What role does critical thinking play in problem solving?
8. What are some of the cognitive abilities associated with critical
thinking?
9. What set of criteria reasonably measures critical thinking?
10. What does the ETS say the analytical section of the GRE General Test
measures?
159
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
11. What do Ennis, Millman and Tomko (1985b) say the Cornell Critical
Thinking Test, Level Z measures?
This study has three significant limitations. Many experts in the field of
critical thinking divide critical thinking into cognitive abilities and dispositions.
Dispositions differ from cognitive abilities in that they are behavioral tendencies or
affective traits of mind that concern how one is inclined to use cognitive abilities.
This study’s instruments do not measure critical thinking dispositions and are not
included in this study. The second limitation is that critical thinking cognitive
abilities also include the ability to listen, write, and speak critically. Since the GRE
General Test and the Cornell Critical Thinking Test, Level Z are multiple-choice
instruments, they do not measure ones abilities to listen critically, write critically,
and speak critically, and thus are beyond the scope of this study.
The final limitation centers on the concept of content validity. Content
validity is evaluated by showing how well the content of the test samples the class of
situations or subject matter about which conclusions are to be drawn. Construct
validity embraces all forms of validity to which empirical evidence and theoretical
rationales support the adequacy and appropriateness of inferences and actions based
on test scores or other modes of assessment. Thus, for the purposes of applied
decision making, reliance on content validity alone is not enough.
160
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Conclusion
This content validity study suggests that the analytical section of the GRE
reasonably measures critical thinking. The analytical section of the GRE met at least
one critical thinking sub-skill for each of the four Facione (1990) core critical
thinking abilities— interpretation, analysis, inference, and evaluation. Furthermore,
70% of the items in the 1997 GRE General Test analytical section meet 6 of the 11
Facione critical thinking sub-skills compared to 8 of the 11 Facione critical thinking
sub-skills for the Cornell Critical Thinking Test, Level Z. When compared to the
Stahl N. N. and Stahl R. J. (1991) criteria, at least 70% of the analytical section items
in the 1997 GRE Practice General Test meet 8 of the 16 criteria, whereas the Cornell
Critical Thinking Test, Level Z only met 5 of the 16 criteria. Sixty-seven percent of
the items in the Cornell Critical Thinking Test, Level Z met four additional Stahl N.
N. and Stahl R. J. critical thinking abilities. Even if one were to add these four
additional criteria, the total number of critical thinking abilities increases to 9 of the
16 criteria for the Cornell Critical Thinking Test, Level Z, which equals about the
same amount for the analytical section of the 1997 GRE Practice General Test.
In a “head-to-head” comparison between the analytical section of the 1997
GRE Practice General Test and the Cornell Critical Thinking Test, Level Z, the
Cornell Critical Thinking Test, Level Z out-numbered the analytical section of the
1997 GRE Practice General Test in 6 of the 11 Facione (1990) sub-skills. These sub
skills were primarily in the area of argumentation— detecting arguments, analyzing
161
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
arguments, assessing claims and assessing arguments— as well as in querying
evidence and conjecturing alternatives. Of the remaining five sub-skills, the
analytical section of the 1997 GRE Practice General Test out-numbered the Cornell
Critical Thinking Test, Level Z, in two sub-skills, and the remaining three sub-skills
were generally equal between the two instruments.
In a “head-to-head” comparison using the Stahl N. N. and Stahl R. J. (1991)
criteria, the Cornell Critical Thinking Test, Level Z out-numbered the analytical
section of the 1997 GRE Practice General Test in 7 of the 16 criteria, the analytical
section of the 1997 GRE Practice General Test out-numbered the Cornell Critical
Thinking Test, Level Z in 4 of the 16 criteria, and the remaining 5 criteria were
generally equal between the two instruments. One would expect better performance
from the Cornell Critical Thinking Test, Level Z, as it is designed to specifically
measure critical thinking.
One should also note, however, that five critical thinking content validity
weaknesses were uncovered in the analytical section of the 1997 GRE Practice
General Test. There were no items which measure one’s ability to conjecture
alternative hypotheses (i.e., the conjecturing alternatives sub-skill of inference
(Facione, 1990)). The Cornell Critical Thinking Test, Level Z met this criterion
primarily in the induction (i.e., planning experiments) questions. Also missing were
items that test these Stahl N. N. and Stahl R. J. (1991) critical thinking skills: (a)
critical thinking distinguishes between verifiable facts and value claims; (b) critical
162
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
thinking tries to separate emotional thinking from logical thinking; (c) critical
thinking detects bias; and (d) critical thinking determines the reliability of a source.
The Cornell Critical Thinking Test, Level Z met these criteria with items that
addressed contentious issues (e.g., voting rights or immigration) and with items that
addressed the credibility of various types of data, and sources.
Recommendation
Messick (1989) would claim content validity does not by itself answer the
question, “Does the analytical section of the GRE reasonably measure critical
thinking?” Content validity resides in a test, whereas construct validity is a property
of test responses and of the inferences based thereon, and that content validity is
focused upon test forms rather than test scores, upon instruments rather than
measurements. The content validity of the analytical section of the GRE General
Test suggests, however, that it reasonably measures critical thinking. This conclusion
should lead one to conduct further research into the construct validity of critical
thinking in the analytical section of the GRE General Test. This construct validity
may include comparing examinees’ GRE analytical section test scores with their
Cornell Critical Thinking, Level Z test scores. This study would be a major
undertaking, however, since no database currently exists in which the researcher
could make such a comparison. In the absence of such data, if the content validity of
the analytical section of the GRE General Test suggests it reasonably measures
critical thinking, then graduate school admissions officials should consider including
163
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
the analytical section test scores of the GRE General Test in their admissions
decisions.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
REFERENCES
Baron, J. B. (1987). Evaluating thinking skills in the classroom. In J. B.
Baron & R. J. Sternberg (Eds.). Teaching thinking skills (dp. 221-247). New York:
W. H. Freeman and Company.
Bransford, J. D., Sherwood, R. D., & Sturdevant, T. (1987). Teaching
thinking and problem solving. In J. B. Baron & R. J. Sternberg (Eds.), Teaching
thinking skills (pp. 162-181). New York: W. H. Freeman and Company.
Briel, J. B., O’ Neill, K. A., & Scheuneman, J. D. (Eds.). (1993). Graduate
Record Examinations Technical Manual. Princeton, NJ: Educational Testing Service.
Carpenter, C. B., & Doig, J. C. (1988). Assessing critical thinking across the
curriculum. New Directions for Teaching and Learning. 34, 33-46.
Chaffee, J. (1992). Teaching critical thinking across the curriculum. New
Directions for Community Colleges. 77.25-35.
Chalifour, C., & Powers, D. E. (1988, May). Content characteristics of GRE
analytical reasoning items. (GRE Board Professional Report No. 84- 14P, ETS
Research Report 88-7). Princeton, NJ: Educational Testing Service.
Clemson University College of Health, Education, and Human Development
Department of Educational Leadership and Counseling. (1997, January 1). Doctor of
philosophy educational leadership [On-line]. Available:
http://www.hehd.clemson.edu/c&edl/PHdHndBk.htm [2000, July 14]
Cromwell, L. S. (1992). Assessing critical thinking. New Directions for
Community Colleges. 77. 37-50.
de Bono, E. (1976). Teaching thinking. New York: Penguin Books.
Duran, R., Powers, D. E., & Swinton, S. (1987, April). Construct validity of
the GRE analytical test: A resource document. (GRE Board Professional Report No.
81-6P, ETS Research Report 87-11). Princeton, NJ: Educational Testing Service.
East Carolina University School of Education Department of Educational
Leadership. (2000, March 6). Doctoral degree in educational leadership [On-line].
Available: http://www.soe ecu.edu/leed/edd.htm [2000, July 14]
165
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
East Tennessee State University College of Education Department of
Educational Leadership and Policy Analysis. (1999, September 26). Educational
leadership doctor of education [On-line]. Available:
http://www.fau.edu/divdept/coe/leadedd.htm [2000, July 14]
Educational Testing Service. (1995). GRE 1995-96 guide to the use of the
Graduate Record Examinations program. Princeton, NJ: Educational Testing Service.
Educational Testing Service. (1997). The Graduate Record Examinations
Practice General Test. Princeton, NJ: Educational Testing Service.
Educational Testing Service. (1999). GRE 1999-2000 information and
registration bulletin. Princeton, NJ: Educational Testing Service.
Ennis, R. H. (1962). A concept of critical thinking. Harvard Educational
Review 32 (1). 81-111.
Ennis, R. H. (1987). A taxonomy of critical thinking dispositions and
abilities. In J. B. Baron & R. J. Sternberg (Eds.), Teaching thinking skills (pp. 9-26).
New York: W. H. Freeman and Company.
Ennis, R. H. (1993). Critical thinking assessment. Theory Into Practice.
32(3), 179-186.
Ennis, R. H., Millman, J., & Tomko, T. N. (1985a). Cornell Critical Thinking
Test, Level X and Level Z Manual. Pacific Grove: Critical Thinking Books &
Software.
Ennis, R. H., Millman, J.. & Tomko, T. N. (1985b). Cornell Critical Thinking
Test, Level Z. Pacific Grove: Critical Thinking Books & Software.
Enright, M. K., & Powers, D. E. (1991, January). Validating the GRE
analytical ability measure against faculty ratings of analytical reasoning skills. (GRE
Board Professional Rep. No. 86-06P, ETS Research Rep. 90-22). Princeton, NJ:
Educational Testing Service.
Facione, P. A. (1990). Critical thinking: A statement of expert consensus for
purposes of educational assessment and instruction. Newark, DE: American
Philosophical Association. (ERIC Document Reproduction Service No. ED 315 423)
Facione, P. A., Facione, N. C., Gainen, J., & Sanchez, C. A. (1995). The
disposition toward critical thinking. The Journal of General Education. 44 (1). 1-25.
166
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Florida State University Department of Educational Leadership. (2000, April
22). Admission procedures and requirements [On-line]. Available:
http://www.fsu.edu/~edleadr/adm.html [2000, July 14]
Fowler, B. (1997). Critical thinking definitions [Online]. Available:
http://www.kcmetro.cc.mo.us/longview/ctac/definitions.htm [1999. September 25]
Freed, M. N., Hess, R. K., & Ryan, J. M. (1989). The educator’s desk
reference (EDR): A sourcebook of educational information and research. New York:
Macmillan Publishing Company.
Good, C. V. (Ed.). (1973). Dictionary of education (3rd ed.). New York:
McGraw-Hill Book Company.
Greenwood, A. (1992, November). The national assessment of college
student learning: Identification of the skills to be taught, learned, and assessed
(Report No. NCES-94-286). Washington, DC: National Center for Education
Statistics. (ERIC Document Reproduction Service No. ED 372 717).
Guilford, J. P. (1977). Wav beyond the 10. Buffalo, New York: Creative
Education Foundation, Inc.
Halpem, D. F. (1992, November). A national assessment of critical thinking
skills in adults: Taking steps toward the goal (Report No. NCES-94-286).
Washington, DC: National Center for Education Statistics. (ERIC Document
Reproduction Services No. ED 372 717)
Hawes, K. (1990). Understanding critical thinking. In V. A. Howard (Ed.),
Varieties of thinking: Essays from Harvard’s philosophy of education research center
(pp. 47-61). New York: Routledge, Chapman and Hall, Inc.
Idaho State University College of Education. (2000). Doctor of education in
educational leadership [On-line]. Available:
http://www.isu.edu/departments/coe/grad-doctorate.html [2000, July 14]
Jones, E. A., Hoffman, S., Moore L. M., Ratcliff, G., Tibbetts, S., & Click, B.
A. L. (1995, May). National assessment of college student learning: Identifying
college graduates’ essential skills in writing, speech and listening, and critical
thinking (Report No. NCES-95-001). Washington, DC: National Center for
Education Statistics. (ERIC Document Reproduction Service No. ED 383 255)
167
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Kurfiss, J. G. (1988, April 2). Critical thinking: Theory, research, practice,
and possibilities (ASHE-ERIC Higher Education Report No. 2, 1988). Washington,
DC: Association for the Study of Higher Education, ERIC Clearinghouse on Higher
Education. (ERIC Document Reproduction Service No. ED 304 041)
Lewis, A., & Smith, D. (1993). Defining higher order thinking. Theory Into
Practice 32 (3). 131-137.
Lewis, H. R. (1997, February 23). The mission of Harvard College [Online].
Available: http://www.harvard.edu/help/noframes/faql 10 nf.html [2000, May 19].
Lipman, M. (1991). Thinking in education. Cambridge: Cambridge
University Press.
McDaniel, N.A. (2000). Critical thinking. Unpublished manuscript. Defense
Systems Management College Program Management and Leadership Department
Teaching Note.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement
(3rd ed.). New York: Macmillan Publishing Company.
Miller, L. & Connelly, M. (1997). Critical thinking core concepts [Online].
Available: http://www.kcmetro.ee.mo.us/longview/ctac/corenotes.htm#statements
[1999, September 23].
Nickerson, R. S. (1987). Why teach thinking? In J. B. Baron & R. J.
Sternberg (Eds.), Teaching thinking skills (pp. 27-38). New York: W. H. Freeman
and Company.
Norris, S. P., & Ennis, R. H. (1989). Evaluating critical thinking. Pacific
Grove, CA: Critical Thinking Press and Software.
Paul, R. W. (1985). The critical-thinking movement: a historical perspective.
Phi Kappa Journal. 2-3.
Paul, R. W. (1990). Critical thinking in North America. In A. J. A. Binker
(Ed.), Critical thinking: What every person needs to survive in a rapidly changing
world (pp. 18-43). Rohnert Park, CA: Center for Critical Thinking and Moral
Critique.
Paul, R. W. (1992). Critical thinking: what, why, and how. New Directions
for Community Colleges, 7 7 .3-24.
168
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Paul, R. W. (1995a). Bloom's taxonomy and critical thinking instruction:
Recall is not knowledge. In J. Willsen & A. J. A. Binker (Eds.), Critical thinking (pp.
217-225). Santa Rosa, CA: Foundation for Critical Thinking.
Paul, R. W. (1995b). Critical thinking: Basic questions and answers. In J.
Willsen & A. J. A. Binker (Eds.), Critical thinking (pp. 91-100). Santa Rosa, CA:
Foundation for Critical Thinking.
Paul, R. W. (1995c). Ethics without indoctrination. In J. Willsen & A. J. A.
Binker (Eds.), Critical thinking (pp. 239-254). Santa Rosa, CA: Foundation for
Critical Thinking.
Paul, R. W. (1995d). Pseudo critical thinking in the educational
establishment. In J. Willsen & A. J. A. Binker (Eds.), Critical thinking (pp. 47-90).
Santa Rosa, CA: Foundation for Critical Thinking.
Paul, R. W. (I995e). Teaching critical thinking in the strong sense: A focus
on self-deception, world views, and a dialectical mode of analysis. In J. Willsen & A.
J. A. Binker (Eds.), Critical thinking (pp. 383-393). Santa Rosa, CA: Foundation for
Critical Thinking.
Paul, R. W. (1995f). The art of redesigning instruction. In J. Willsen & A. J.
A. Binker (Eds.). Critical thinking (pp. 303-334). Santa Rosa, CA: Foundation for
Critical Thinking.
Paul, R. W. (1995g). The contribution of philosophy to thinking. In J.
Willsen & A. J. A. Binker (Eds.), Critical thinking (pp. 435-473). Santa Rosa, CA:
Foundation for Critical Thinking.
Paul, R. W. ( I995h). The critical connection: Higher order thinking that
unifies curriculum, instruction, and learning. In J. Willsen & A. J. A. Binker (Eds.),
Critical thinking (pp. 273-289). Santa Rosa, CA: Foundation for Critical Thinking.
Paul, R. W. (1995i). The critical thinking movement in historical perspective.
In J. Willsen & A. J. A. Binker (Eds.), Critical thinking (pp. 37-46). Santa Rosa, CA:
Foundation for Critical Thinking.
Paul, R. W„ & Binker, A. J. A. (1995a). Critical thinking staff development:
The lesson plan remodeling approach. In J. Willsen & A. J. A. Binker (Eds.), Critical
thinking (pp. 397-408). Santa Rosa, CA: Foundation for Critical Thinking.
169
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Paul, R. W. & Binker, A. J. A. (1995b). Socratic questioning. In J. Willsen &
A. J. A. Binker (Eds.), Critical thinking (pp. 335-365). Santa Rosa, CA: Foundation
for Critical Thinking.
Paul, R. W. & Nosich, G. M. (1995). A model for the national assessment of
higher order thinking. Critical Thinking. Santa Rosa: Foundation for Critical
Thinking.
Perkins, D. N. (1987). Thinking frames: An integrative perspective on
teaching cognitive skills. In J. B. Baron & R. J. Sternberg (Eds.), Teaching thinking
skills (pp. 41-61). New York: W. H. Freeman and Company.
Public Broadcasting System (2000a). Circa 420s: Socrates and his methods
[On-line]. Available:
http://www.pbs.org/empires/thegrecks/characters/socrates p5.html [2000, August 5]
Public Broadcasting System. (2000b). Circa 420s: Socrates and the oracle of
delphi [On-line]. Available:
http://www.pbs.org/empires/thegreeks/characters/socrates p4.html [2000, August 5]
Public Broadcasting System (2000c). Socrates 469 BC-399BC [On-line].
Available: http://www.pbs.org/empires/thegreeks/characters/socrates p 1 .html [2000,
August 5]
Quellmalz, E. S. (1985, October). Needed: Better methods for testing higher-
order thinking skills. Educational Leadership. 29-35.
Stahl, N. N., & Stahl, R. J. (1991). We can agree afterall! Achieving
consensus for a critical thinking component of a gifted program using the delphi
technique. Roeper Review. 14 (2), 79-88.
Sternberg, R. J. (1996). What should we ask about intelligence? American
Scholar. 65(2). 205-217.
The Psychological Corporation. (2000). Miller analogies test [On-line].
Available: http://www.tpcweb.com/mat/g-matinO.htm [2000, September 27]
University of Georgia, Athens College of Education. (2000). Requirements
for doctor of education degree [On-line]. Available:
http://www.coe.uga.edu/leadership/doctoral.html [2000, July 14]
170
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
University of Houston College of Education Educational Leadership and
Cultural Studies. (1994, September). Doctor of education degree in educational
leadership [On-line]. Available:
http://www.coe.uh.edu/coe kiosk/elcs/doctoral/souree/edas allinfo.html [2000, July
14]
University of Nevada Reno College of Education Department of Educational
Leadership. (2000). Department of educational leadership applications for
admissions [On-line]. Available: http://www.unr.edu/educ/el/elapp.html [2000, July
14]
University of South Carolina College of Education Educational Leadership
and Policies. (1999, November 2). College of education EDLP doctor of education
degree with emphasis in curriculum studies [On-line]. Available:
http://www.ed.sc.edu/lpdred.htm [2000, July 14]
University of Southern California. (2000). Student life [On-line]. Available:
http://www.usc.edu/dept/publications/cat95/geninfo/stulife.html [2000, July 15]
Watson, G., & Glaser, E. M. (1964a). Critical thinking appraisal form YM.
New York: Harcourt, Brace & World, Inc.
Waston, G., & Glaser, E. M. (1964b). Critical thinking appraisal manual for
forms YM and ZM. New York: Harcourt, Brace & World, Inc.
Watson, G., & Glaser, E. M. (1980a). Critical thinking appraisal manual.
New York: Harcourt, Brace, & World, Inc.
Watson, G., & Glaser, E. M. (1980b). Form A: critical thinking appraisal.
New York: Harcourt, Brace, & World, Inc.
171
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
APPENDIX
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
GRE Practice General T est Section 2 Section 5 Cornell Critical Thinking Test, Level Z __
Question # :___
Score Sheet for Measuring Critical Thinking
Criterion I-INTERPRETATION is "to comprehend and express the meaning or significance of a
wide variety of experiences, situations, data, events, judgments, conventions, beliefs, rules,
procedures, or criteria.” Sub-skills include:
Categorization
Decoding Significance
Clarifying Meaning.
Criterion 2-ANALYSIS is "to identify the intended and actual inferential relationships among
statements, questions, concepts, descriptions, or other forms of representation intended to express
belief, judgment, experiences, reasons, information, or opinions.” Sub-skills include:
Examining Ideas
Detecting Arguments
Analyzing Arguments.
Stahl & Stahl CT Cognitive Skills:
Critical thinking seeks a clear statement of the issue.
Critical thinking identifies stated and unstated assumptions.
Critical thinking determines the strength of an argument.
Critical thinking judges whether the problem has been identified.
Critical thinking keeps in mind the original and/or basic concern.
Criterion 3-INFERENCE is “to identify and secure elements needed to draw reasonable
conclusions; to form conjectures and hypotheses; to consider relevant information and to educe the
consequences flowing from data, statements, principles, evidence, judgments, beliefs, opinions,
concepts, descriptions, questions, or other forms of representation.” Sub-skills of inference include:
Querying Evidence
Conjecturing Alternatives
Drawing Conclusions
Stahl & Stahl CT Cognitive Skills
Critical thinking takes a position when the evidence and reasons are sufficient to do so.
Critical thinking understands the difference between a conclusion which might be true and
one which must be true.
Criterion 4— EVALUATION is "to assess the credibility of statements or other representations
which are accounts or descriptions of a person's perception, experience, situation, judgment, belief, or
opinion; and to assess the logical strength o f the actual or intended inferential relationships among
statements, descriptions, questions, or other forms o f representation.” Sub-skills of evaluation include:
Assessing Claims
Assessing Arguments
Stahl and Stahl CT Cognitive Skills
Critical thinking distinguishes between verifiable facts and value claims.
Critical thinking determines the accuracy o f a statement.
Critical thinking tries to separate emotional thinking from logical thinking.
Critical thinking detects bias.
Critical thinking recognizes logical inconsistencies in a line of reasoning.
Critical thinking distinguishes between warranted and unwarranted claims.
Critical thinking judges whether there is ambiguity in a line o f reasoning.
Critical thinking judges whether certain statements contradict each other.
Critical thinking determines the reliability o f a claim or source.
173
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.

Asset Metadata

Creator Kailiwai, George, III (author)

Core Title Critical thinking cognitive abilities and the analytical section of the educational testing service Graduate Record Examinations(R) General Test: A conceptual study in content validity

Contributor Digitized by ProQuest (provenance)

School Graduate School

Degree Doctor of Education

Degree Program Education

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag education, educational psychology,education, higher,education, tests and measurements,OAI-PMH Harvest

Language English

Advisor Hocevar, Dennis (committee chair), Guiton, Gretchen (committee member), Michael, William B. (committee member)

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c16-126972

Unique identifier UC11330012

Identifier 3041476.pdf (filename),usctheses-c16-126972 (legacy record id)

Legacy Identifier 3041476-0.pdf

Dmrecord 126972

Document Type Dissertation

Rights Kailiwai, George, III

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the au...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus, Los Angeles, California 90089, USA