Close
About
FAQ
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
University of Southern California Dissertations and Theses
/
The TOEFL exam gateway: understanding performance factors among community college students
(USC Thesis Other)
The TOEFL exam gateway: understanding performance factors among community college students
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
Running head: THE TOEFL EXAM GATEWAY: UNDERSTANDING
PERFORMANCE FACTORS AMONG COMMUNITY COLLEGE STUDENTS
The TOEFL Exam Gateway: Understanding Performance Factors Among Community
College Students
by
Christopher Wendel Bankston
A Dissertation Proposal Presented to the
FACULTY OF THE USC ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF EDUCATION
July 4, 2017
Copyright © 2017 Christopher Wendel Bankston
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
i
ACKNOWLEDGEMENTS
Thank you, Xia, my most wonderful life partner, for your understanding and support
throughout this entire endeavor; your patience and “common-sense-logic-theory” helped me
through this amazing journey. I would like to extend further thanks to my Chair, Professor
Shafiqa Ahmadi for your incredible support and guidance throughout the entire program. Many
further thanks to Professors Ruth Chung and Rob Filback for agreeing to be my committee
members and assisting in my efforts throughout this entire process. Further thanks to my
University of Southern California Rossier School of Education Professors Mark Robison, Helena
Seli, Tracy Tambascia, Larry Picus, Monique Datta, Cathy Krop, Melora Sundt, and Kenneth
Yates.
Further acknowledgement and thanks go to Michele Dubarry, Diana Alves de Lima, and
Kanako Valencia-Suda, at De Anza College for their assistance and cooperation throughout my
entire data collection process and for your individual support throughout the entire development
of this dissertation.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
ii
TABLE OF CONTENTS
ACKNOWLEDGMENTS …………………………………………………………………… i
LIST OF TABLES ………………………………………………………………………...… vii
LIST OF FIGURES ……………………………………………………………………….… viii
ABSTRACT ………………………………………………………………………………… ix
CHAPTER 1: INTRODUCTION TO THE PROBLEM ……………………………………… 1
Background of the Problem …………………………………………………………… 1
Importance of Addressing the Problem …………………………………………………. 4
Organizational Context and Mission ……………………………………………………. 5
Organizational Goal and Current Status ………………………………………………… 6
Description of Stakeholder Groups ……………………………………………………… 7
Stakeholders Performance Goals ……………………………………………………… 7
Stakeholder Group for the Study ……………………………………………………… 8
Purpose of the Project and Questions …………………………………………………… 9
Conceptual and Methodological Framework ……………………………………………. 9
Acronyms ……………………………………………………………………………… 10
Organization of the Project …………………………………………………………… 10
Limitations and Delimitations …………………………………………………………. 11
CHAPTER 2: REVIEW OF THE LITERATURE …………………………………………… 13
The IELTS and TOEFL Exams ……………………………………………………… 13
Purpose of Exams……………………………………………………………… 13
Significance of Exams ………………………………………………………… 14
International Administration & Fees ………………………………………… 14
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
iii
United States Administration & Fees ……………………………………..…… 15
Test Scores …………………………………………………………………… 15
Institutions …………...………………………………………………………… 15
The Debate Over the Purpose and Significance of the Exams ………………… 16
Students’ First Attempt at Passing the Exams ………………………………………… 16
Research Data about Student’s First Attempts ………………………………… 16
Factors That Support First Attempt Success …………………………………… 17
Unsuccessful First Attempts and Possible Contributing Factors ………………. 18
Learning and Motivation Theory …………………………………….………………… 19
Knowledge and Skills ………………………………………………………… 19
Motivation ……………………………………………………………………… 20
Organization ……………………………….…………………………………… 20
Student Knowledge, Motivation, and Organization Influences ……………………… 20
Student Knowledge and Skills………………………………………………… 21
Student Motivation …...………………………………………………………… 23
Organizational Factors ………………………………………………………… 24
Teaching English to Speakers of Other Languages (TESOL) ………………………… 26
The Differences between ESL and EFL ……………………………………… 26
An Increasing Demand for TESOL …………………………………………… 27
The Four Areas of English Language Learning Tested in the TOEFL Exam … 27
International Schools, Colleges, Universities, and Instructor Credentials …… 29
Summary …………………………………………………………………… 31
CHAPTER 3: METHODOLOGY …………………………………………………………… 32
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
iv
Purpose of the Project and Questions ………………………………………………… 32
Methodological Framework ……………………………………………………………. 32
Assumed Performance-Based Influences ……………………………………………… 34
Preliminary Scanning Data …………………………………………………… 35
Participating Stakeholders …………………………………………………………… 38
Data Collection & Analysis …………………………………………………………… 39
Survey ………………………………………………………………………….. 40
Interviews ………………………………………………………………………. 40
Document Analysis …………………………………………………………….. 41
Observation Notes ……………………………………………………………… 42
Validation of the Performance …………………………………………………………. 43
Trustworthiness of Data ……………………………………………………………… 43
Role of Investigator …………………………………………………………………… 44
CHAPTER 4: RESULTS AND FINDINGS ....……………………………………………… 46
Results and Findings for Knowledge Causes ………………………………………… 47
Survey Results ……………………………………………………………….… 47
Findings from Interviews …………………………………………………….… 49
Synthesis of Results and Findings for Knowledge Causes …………………… 52
Results and Findings for Motivation Causes ………………………………………… 52
Survey Results …………………………………………………………………. 52
Findings from Interviews ………………………………………………………. 53
Synthesis of Results and Findings for Motivation Causes …………………… 56
Results and Findings for Organization Causes ………………………………………… 57
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
v
Survey Results ………………………………………………………………… 57
Findings from Interviews ……………………………………………………… 59
Synthesis of Results and Findings for Organization Causes ………………… 60
Chapter Summary & Validated Causes………………………………………………… 60
CHAPTER 5: RECOMMENDATIONS …………………….………………………………… 61
Validated Needs: Selection and Rationale …………………………………………… 62
Validated Knowledge Gaps and New Causes ………………………………… 62
Validated Motivation Gaps and New Causes ………………………………… 64
Validated Organization Gaps and New Causes ……………………………… 66
Solutions ……………………………………………………………………………… 70
Elective TOEFL Preparation Courses ………………………………………… 70
Implementation Plan …………………………………………………………………… 70
Proposed Solution and Environment for Implementation …………………… 71
Implementation Action Steps ………………………………………………… 73
Building Capacity ……………………………………………………………… 76
Timeline ……………………………………………………………………… 77
Evaluation Plan ………………………………………………………………………… 78
Expected Observable Results ………………………………………………… 78
Level 1 Reactions …………………………………………………………….… 79
Level 2 Learning ……………………………………………………………….. 80
Level 3 Behavior ……………………………………………………………….. 80
Level 4 Results …………………………………………………………………. 81
Future Research ………………………………………………………………………... 81
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
vi
Conclusion ……………………………………………………………………………... 82
REFERENCES ………………………………………………………………………………… 85
APPENDICIES ………………………………………………………………………………… 89
Appendix A: Conceptual Framework for Addressing the Inquiry Questions: Assumed
Knowledge …………………………………...………………………………… 89
Appendix B: Conceptual Framework for Addressing the Inquiry Questions: Assumed
Motivation ……………………………………………………………………. 93
Appendix C: Conceptual Framework for Addressing the Inquiry Questions: Assumed
Organization ……………………………………………………...…………….. 96
Appendix D: Survey Instrument ……………………………………………………….. 99
Appendix E: Interview Instrument …………………………………………………..... 105
Appendix F: TOEFL Evaluation Observation Notes …………………………………. 106
Appendix G: De Anza College ESL Courses and Sequences ………………………… 108
Appendix H: Validated Assumed Causes …………………………………………… 109
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
vii
LIST OF TABLES
Table 1. Organizational Mission, Global Goal and Stakeholder Performance Goals..................... 7
Table 2. Summary of Sources about Assumed Causes of Knowledge, Motivation, and
Organizational Issues ............................................................................................................ 37
Table 3. Linking the Instruments to the Inquiry Questions .......................................................... 43
Table 4. Survey Respondent Age Range ...................................................................................... 46
Table 5. Knowledge Surveys ........................................................................................................ 49
Table 6. Motivation Surveys ......................................................................................................... 53
Table 7. Organizational Surveys ................................................................................................... 58
Table 8. Validated Assumed Factual Knowledge and New Cause ............................................... 63
Table 9. Validated Metacognitive Knowledge and New Cause ................................................... 64
Table 10. Validated Active Choice Causes................................................................................... 65
Table 11. Invalidated Assumed Cause and New Cause ................................................................ 65
Table 12. Validated Assumed Cause for Mental Effort ................................................................ 66
Table 13. Validated Assumed Cause for Goal Setting and New Cause ....................................... 67
Table 14. Validated Cultural Model Cause and New Cause ........................................................ 68
Table 15. Validated Cultural Settings and New Cause ................................................................. 68
Table 16. Validated Organizational Resources and New Cause ................................................... 69
Table 17. ESL Course Sequence (Level, Required Course, Elective Courses) ............................ 76
Table 18. Implementation Plan Timeline...................................................................................... 77
Table 19. Level 1 Reactions.......................................................................................................... 80
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
viii
LIST OF FIGURES
Figure 1. The GAP Analysis Process Flow Chart......................................................................... 33
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
ix
ABSTRACT
Research has shown that first attempt performance on the TOEFL exam can have an extremely
low rate of success (Esmail, 2013). The TOEFL and IELTS exams have become gatekeepers into
English language institutions and society. This case study utilizes a gap analysis framework to
try and understand why students have such low performance rates on such an instrumental exam.
The primary goal was to identify knowledge, motivation, and organizational barriers that are
preventing students from successfully achieving their needed passing rates. Data were collected
by use of surveys, interviews, data collection, and observations in a community college
environment with a significant international student population. Findings showed that students
rely on additional, costly, external private institutions to prepare them adequately for the TOEFL
exam. A key factor of success then, has to do with a student’s economic ability to utilize these
services. Offering an elective TOEFL preparation course at community colleges may be a more
equitable way to assist ESL students with TOEFL exam preparation.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
1
CHAPTER 1: INTRODUCTION
International students are seeking higher education (HE) degrees in the United States and
other English-speaking countries such as Australia, Canada, and England in record numbers as
globalization continues to gain momentum (Haynie, 2014). According to the Institute of
International Education, international student enrollment in these countries is growing by 10%
annually (2015). Entry into HE institutions is dependent on obtaining satisfactory scores on the
International English Language Testing System (IELTS) or Testing of English as a Foreign
Language (TOEFL) exams for students whose first language is not English. First time pass rates,
however, often range from 64% to as low as 36% (Esmail, 2013). Different institutions require
varying score levels for student entry. If students do not achieve the minimal scores for their
chosen institution, then their academic career may be delayed and additional costs incurred. The
purpose of this study was to evaluate the reasons why a large percentage of students do not
achieve their desired scores when taking the TOEFL exam for entry into a U.S. institution of
higher education on their first attempt.
Background of the Problem
The relationship between the area of Teaching English to Speakers of Other Languages
(TESOL) and the IELTS and TOEFL exams, until recently garnered little attention from
policymakers and researchers (Institute of International Education, 2015). There are no formal
policies ensuring alignment between ESL/EFL programs and the exams to ensure that students
are prepared to pass the exams on their first attempt (United States Department of Education,
2015). The policy most closely addressing this problem would be the Common Core State
Standards (CCSS), which started with K-12 but is now being adopted by many community
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
2
colleges and have been an effective start in standardizing educational consistency across the
United States (Common Core State Standards Initiative, 2016).
Haynie (2014) citing from a report by the Institute of International Education, states that
the United States enrolled, “a record-breaking number of international students during the 2013-
2014 school year” (para. 1), admitting 886,052 undergraduate and graduate students.
International students make up nearly four-percent of all university students and make the United
States the largest host of international students in the world, with the two largest groups coming
from China (31%) and India (8.6%). Further, a study conducted by the Institute of International
Education in partnership with the U.S. Department of State’s Bureau of Educational and Cultural
Affairs (2015), shows that those numbers increased by nearly 100,000 in 2014-2015 to 974,926.
This significant increase should be putting more pressure on school administrators to ensure that
HE entrance rates and the needs of the international students are being met (Institute of
International Education, 2015). These international students pay significantly higher fees than
citizens, by as much as ten-times, which in turn allows colleges and universities to keep fees
lower for their citizens. Losing international students may reduce HE revenue by hundreds of
millions of dollars (Institute of International Education, 2015).
Linking the IELTS and TOEFL test performance directly to ESL instruction and
academic performance has been examined in small studies dating back to 1983. A quantitative
study done by Ayers and Quattlebaum (1992) explored the correlation between the Graduate
Record Examination (GRE), the TOEFL, and student Grade Point Average (GPA). The sample
participants were 67 Asian students at the Tennessee Technological University who received
their Master of Science degree in engineering between 1986 and 1990. The results of the study
“indicated that the TOEFL score was not an effective predictor of academic success, as measured
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
3
by total GPA based on all courses required in the program of study” (pp. 974-975) and that even
students with a high GPA do not always score well on the TOEFL exam.
English is an international language and as such, this issue is not uniquely American but
multinational too. Universities in Australia, Canada, and England are also starting to recognize
this problem and their researchers are focusing on how these gaps in scores can be improved. A
study by Esmail and Roberts (2013) was commissioned to determine if evidence existed of
discrimination against International Medical Graduate or British Black and Minority Ethnic
graduates. The results of their quantitative analysis revealed that 64.26% of persons did not reach
a minimal passing score set by the institution on their first attempt in all four components of the
test. A study conducted by Carol Floyd in 2014 in Southern Australia, also looking at the
correlation between GPA and testing success, revealed that the GPA gap between students who
achieved the desired score or not, on their first attempt had a small effect size, indicating that the
difference, though statistically significant, is trivial. In her conclusion, Floyd infers that this
narrow gap has more to do with prior learning in an academic English Language Learning (ELL)
environment than student preparation for the exams themselves and that “academic acculturation
could be valuable for all international students who come from markedly different academic
culture backgrounds, regardless of their English Language Proficiency (ELP) levels” (p. A-14).
Expanding beyond English speaking nations to a more global perspective reveals that
many countries now require ELL as part of their standard curriculum. A qualitative study
conducted by Gan, Humphreys and Hamp-Lyons (2004) to understand the differences between
successful and unsuccessful ELLs in Chinese universities revealed how seriously governments
take the learning of English with their students. Chinese undergraduate students are required to
take the College English Test (CET) before they can receive their bachelor’s degree and they
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
4
must pass level-4 (CET-4) with 60% or better just to graduate (Gan, Humphreys and Hamp-
Lyons, 2004). There are higher levels of the exam that can be taken by students voluntarily to
prepare them for HE and the IELTS & TOEFL exams.
The IELTS and TOEFL exams are gatekeepers into institutions of higher education. A
majority of international students come from China, India, and other developed countries. The
perception is that these are the most socioeconomically privileged, driving onto campus parking
lots in Ferraris and Maserati’s, as witnessed daily by the investigator of this study. But what goes
undetected are the “shadow” students who are the socioeconomically underprivileged yet most
determined to come to the U.S. for higher education (Chapter 4). Statistics provided by the
United States Central Intelligence Agency World Factbook (2015), show that there were 67
sovereign states and 27 non-sovereign entities, many economically underdeveloped, where
English was an official language. Although designated as an official language, students from
these countries are still required to take the IELTS or TOEFL exam for entry into HE institutions.
Importance of Addressing the Problem
The problem of an unsuccessful first attempt on the IELTS and TOEFL exams is
important to solve for a variety of reasons. The number of seats and teachers available for
ESL/EFL classrooms cannot keep up with the demand, as measured by the number of
international students, which has increased 72% in the United States alone since 2000 (Haynie,
2014). The organization of evaluation for the study has a maximum classroom capacity of 25
students per class section and a limited number of sections. These classes tend to fill before all
students have had an opportunity to register. Students who must return for additional instruction
displace other students who are trying to receive first-time instruction because they are higher up
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
5
on the date to register list. Further, the costs to students having to return for additional instruction
is unduly necessary and puts a burden on themselves and their families financially.
United States Federal student loan debt for citizens is nearing $1 trillion dollars
(Schlesinger, 2012). International students pay approximately 10-times more for courses than
citizens of the United States at community colleges. When “shadow” students transfer to a
University, they must take on student loan debt. A recent study published by the Journal of
Student Financial Aid used data from six large, public research universities in 2012 and found
that over 4% of financial aid was sought by international students (Soria, K. M., Weiner, B., &
Lu, E. C., 2014). Students who are required to retake courses will take on additional instructional
costs, which may in turn drive them away from obtaining HE in English language countries
(Esmail & Roberts, 2013). Societal demands and needs in areas such as engineering, medicine,
and science require that we train and graduate students in a cost-effective and timely manner.
Adding to this complexity, the passing scores on these exams can vary from institution to
institution, which makes it difficult to create a standardized curriculum that prepares students to
take these standardized exams.
Organizational Context and Mission
The organization being evaluated, De Anza College, is considered the top large
community college in the State of California based upon graduation, transfer, and retention rates
(EdSmart, 2015). With respect to international students (ISs), De Anza College is number three
among community colleges in the nation in IS quarterly enrollment. (New York Times, October
30, 2015). The mission statement of De Anza College states:
De Anza College provides an academically rich, multicultural learning environment that
challenges student of every background to develop their intellect, character and abilities;
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
6
to realize their goals; and to be socially responsible leaders in their communities, the
nation and the world. The college engages students in creative work that demonstrates the
knowledge, skill and attitudes contained within the college’s Institutional Core
Competencies: Communication and expression; Information literacy; Physical/mental
wellness and personal responsibility; Civic capacity for global, cultural, social and
environmental justice; Critical thinking.
Located in Cupertino, California, De Anza College is situated on a 112-acre campus with
61 buildings that house approximately 116 classrooms, lecture halls, and 43 labs. De Anza
College had approximately 34,980 students in 2013-2014 with an average 24,000 students each
quarter, of which nearly 2,800 (12%) are international students and 39% of those coming from
Asia. De Anza College employs approximately 800 full- and part-time faculty, offers 63 degree
programs and 97 certificate programs.
Organizational Goal and Current Status
De Anza’s goal can be that by Spring 2020, all registered ESL students will successfully
graduate or transfer to a 4-year university. This goal is measured by the annual overall transfer
rate of students to four-year institutions as measured by the California Community Colleges
Chancellor’s Office. The overall transfer rate for all students is currently 67.3%, tracked over six
years and ending in 2013-14. A related goal of the organization is to have 100% of ISs who take
the TOEFL exam pass on their first attempt by creating a preparation program. According to the
Institute of International Education (2015) De Anza College is third in the nation for number of
international students enrolled, behind Houston Community College and Santa Monica College
respectively, which leaves room for improvement.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
7
Description of Stakeholder Groups
There are three central stakeholder groups at De Anza College: administration, faculty,
and international students. Each group has their own needs and objectives however; there is
overlap between the three groups. Although there are many levels of stakeholders involved at De
Anza College, these three groups are most directly related to this problem of practice. The
administration at De Anza College is responsible for planning and ensuring that sufficient
courses and resources are available to meet student demand. International students come here
with the expectation that they will learn English and be able to transfer without having to retake
courses. Additionally, the ESL instructors at this institution are some of the best trained and
many hold TESOL degrees from top-tier accredited universities.
Stakeholders’ Performance Goals
De Anza College is a publicly funded community college with private donors and as such,
it has many stakeholders. For the purpose of this evaluation, three central stakeholders have been
identified and broken down in more detail in Table 1 below. The organization is in itself a
stakeholder, followed by international students, administration, and faculty as the most important
and interdependent stakeholder groups.
Table 1.
Organizational Mission, Global Goal and Stakeholder Performance Goals
Organizational Mission
De Anza College provides an academically rich, multicultural learning environment that
challenges student of every background to develop their intellect, character and abilities; to
realize their goals; and to be socially responsible leaders in their communities, the nation and the
world. The college engages students in creative work that demonstrates the knowledge, skill and
attitudes contained within the college’s Institutional Core Competencies: Communication and
expression; Information literacy; Physical/mental wellness and personal responsibility; Civic
capacity for global, cultural, social and environmental justice; Critical thinking
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
8
Organizational Performance Goal
By Fall 2020, 100% of De Anza College ESL students will complete a certificate, associate’s
degree, or successful transfer to a 4-year college or university within a three-year period.
De Anza College International Students Administration Faculty
By Fall 2020, De
Anza College will
graduate 100% of
ESL students with a
certificate, associate’s
degree, or successful
transfer to a 4-year
college or university.
By Fall 2019, 100%
of ESL students will
be able to pass the
IELTS/TOEFL exam
on their first attempt.
Administration will
create an
action/strategic plan
to improve exam
passage rates.
Administration will
provide clear and
achievable goals to
the faculty with
accountability and
oversight.
By Spring 2018, 100%
of ESL instructors will
implement 100% of the
content using 100% of
the pedagogy
recommended for
effective ESL instruction
100% of the time.
Stakeholder Group for the Study
Although three stakeholder groups were identified as key to the area of focus for the
evaluation, it was important to concentrate on the one key group that is affected the most by
taking the exams. Therefore, the stakeholder group of study was a subset of international
students; those who have taken or plan to take the TOEFL exams. This stakeholder group is
important to the organization because they come with the expectation that their ESL study will
allow them to pass the exam successfully on their first or next attempt. Their inability to do so
may damage the reputation of the organization and the students may begin to search for and
attend other institutions for their English language learning. These stakeholders have a high level
of investment as they have left their home country and will pay more than non-international
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
9
students for their education. They therefore do not want to repeat courses or take longer than
necessary to complete their goals.
Purpose of the Project and Research Questions
The purpose of this project is to evaluate the degree to which De Anza College is
achieving its ability to adequately train ESL students to pass the TOEFL exams on their first
attempt relative to other institutions globally. The analysis focused on knowledge, motivation,
and organizational influences related to achieving this organizational goal. While a complete
evaluation project would focus on all De Anza College stakeholders, for practical purposes the
stakeholders to be focused on in this analysis was international ESL students who are going to
take or have taken the TOEFL exams.
As such, the questions that guided the evaluation were the following:
1. To what extent is De Anza College achieving its implicit goal for international
students to pass the TOEFL exam with their necessary score on their first attempt?
2. What is the status of international students’ knowledge, motivation, and
organizational (KMO) influences needed to pass the exam on their first attempt?
3. What are the recommendations for organizational practice in the areas of KMO
resources?
Conceptual and Methodological Framework
The conceptual framework was implemented from an adaption of the evaluation model as
discussed by Clark and Estes’ (2008) gap analysis, which is an analytical method that helps to
clarify organizational goals and KMO influences. The methodological framework is a mix of
qualitative and quantitative case study with descriptive statistics. Assumed KMO influences on
De Anza College and its ability to provide student achievement was generated based on personal
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
10
knowledge and related literature. Various influences were assessed by using surveys, interviews,
literature review, document analysis, and content analysis. Recommendations are made based
upon research-based solutions and evaluated in a comprehensive manner.
Acronyms
CCSS: Common Core State Standards
CET: College English Test
CET-4: College English Test – Level 4
EdTech: Educational Technology
EFL: English as a Foreign Language
ELL: English Language Learning/Learner
ELP: English Language Proficiency
ESL: English as a Second Language
GPA: Grade Point Average
GRE: Graduate Record Examination
HE: Higher Education
IELTS: International English Language Testing System
ISs: International Students
KMO: Knowledge, Motivation, Organization
TESOL: Teaching English to Speakers of Other Languages
TOEFL: Testing of English as a Foreign Language
Organization of the Project
The study is organized into five chapters. This chapter provided the reader with the key
concepts and terminology commonly found in a discussion about the IELTS and TOEFL
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
11
entrance exams and their impact on students and society. The mission, goals, and stakeholders of
the organization and the framework for the project were introduced. Chapter Two provides a
review of current and past literature surrounding the scope of the study. Topics of the IELTS
exam, TOEFL exam, GRE exam, scoring, ESL listening, pronunciation, speaking preparation,
and HE entrance requirements are addressed. Chapter Three details the KMO influences as well
as the methodology to be examined when it comes to the choice of analysis, data collection, and
participants. The data and results are assessed and analyzed in Chapter Four. Finally, Chapter
Five provides recommendations for an implementation and evaluation plan for solutions, based
on the data and literature, for closing the perceived gaps.
Limitations and Delimitations
There are some limitations to the design of the project. The project relied on the honesty
and integrity of students to provide truthful and honest answers; not project what they may want
or what they thought I may want. This applies to all aspects and instruments within the project
inquiry. In addition, the project assumed that the participants clearly understood the questions
asked and that their responses correctly interpreted. Finally, all statistical analysis is correlational
and not based on a causational relationship.
The project is focused on gap analysis to determine the causes of low first-time passing
scores by students attempting the IELTS or TOEFL exams. The primary delimitation of the
project is that the context is global and not possible to fully categorize now. However, just as the
study is drawing upon previous research, future research may benefit from this project and its
application of Estes and Clark’s (2008) gap analysis model.
Finally, the project is delimited to a stakeholder group that may or may not represent the
overall global group. While these groups are important to the overall scope of the project, it
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
12
would be impracticable for this study to cover such a broad range. Future studies may be able to
include these missing stakeholders and correlate combined data to derive future results.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
13
CHAPTER 2: REVIEW OF THE LITERATURE
Reviewed in this chapter are the three interrelated areas tied to the problem being
evaluated: The IELTS and TOEFL exams, EFL and ESL instruction, and the seemingly high
number of students who are unsuccessful at their first attempt taking these exams. Being
unsuccessful, for the purpose of the study, is being defined as not achieving the scores the
student needed to satisfy the organizational goal that they have chosen. Implications of resources,
adequate school preparation for the exams, if any, teacher preparation, and countering the
stereotype that all international students are socioeconomically privileged are examined. This
chapter also reviews learning and motivation theory, in particular, attention to knowledge skills,
motivation, and organizational issues. Lastly, EFL/ESL student knowledge, motivation, and
organizational factor specifics are reviewed.
The IELTS and TOEFL Exams
The IELTS and TOEFL exams and their purpose, significance, administration, scoring,
the institutions that require them, and the debate over their purpose and significance, are
presented in this section.
Purpose of Exams
The IELTS and TOEFL exams are identical in their missions but the IELTS is used
primarily by institutions in British colonies (although it is becoming as ubiquitous as the
TOEFL). The purpose of the exams is to measure a student’s ability to use and understand
English in a college graduate or undergraduate environment, and in some countries as a means to
obtaining a Visa or residency (Educational Testing Service, 2016; British Council, 2016).
Institutions use the scores from the exams to determine admission into a program or when
granting a Visa.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
14
Significance of Exams
The global economy has a shortage of doctors and engineers, among other professions,
that our institutions need to educate and graduate in a timely manner. If students do not pass with
the minimum required scores, their entrance and training are delayed, which affects the student
and society (Esmail & Roberts, 2013). The exams are gatekeepers between students wanting HE
in countries and their institutions where English the first language. The stakes are high for these
students and can create a high level of anxiety and stress. Although the exams can be taken
repeatedly, the cost may be considered quite high (Educational Testing Service, 2016) in
addition to the expense and time for additional ESL/EFL instruction and preparation.
International Administration & Fees
The TOEFL exam can be taken in one of 4500 test centers in 165 countries (Education
Testing Service, 2016) and the IELTS can be taken in one of more than 1000 test centers in over
140 countries (British Council, 2016). Although the number of centers seems to give a good first
impression of availability, there are issues that are not so obvious on the surface. The centers are
generally located in more populous areas therefore, students located in more rural areas may
have to travel a substantial distance to reach a testing center. Further, students from rural areas
tend to have less money and cannot afford to spend an evening in a hotel so they will travel and
take the exams and return home all in the same day. This process can be mentally and physically
exhausting (Chapter 4).
Fees can also vary between countries and are not standardized. For example, the fee to
take the IELTS exam in Vietnam is approximately USD$210, whereas in the USA it is only
USD$160 (British Council, 2016). The TOEFL exam in Vietnam is only USD$160, whereas it is
USD$190 in the USA (Education Testing Service, 2016).
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
15
United States Administration & Fees
There are 127 TOEFL test centers within the United States (Education Testing Service,
2016) and the IELTS can be taken at testing centers in over 48 locations throughout the US
(British Council, 2016). Testing conditions in these centers are much more consistent but again,
they tend to be located in more populous areas and require special planning and stress for poorer
urban students. The costs to take the TOEFL exam is USD$190 and USD$160 for the IELTS.
Test Scores
Colleges, agencies, and other institutions accept IELTS and TOEFL scores and each
institution has different minimum acceptance levels. To understand the magnitude of the
importance of these scores, the IELTS and TOEFL are accepted by more than 9000 institutions
in over 130 countries (British Council, 2016; Education Testing Service, 2016). There is no
magic number with test scores. Each institution determines the minimum score for entrance: A
total score of 60-70 (TOEFL) for community colleges, 80-90 (TOEFL) for California State
Universities, 90-100 (TOEFL) for University of California colleges, and 100-120 (TOEFL) for
private institutions. Further, some institutions may only look at the overall total score while
others may look at the individual scores for each of the four sections. Again, each section score
may have a minimum set by the institution. The breakdown of scores tends to be more difficult
for students since they need to be equally strong in all four areas and ESL coursework tends to be
unequal (Appendix G).
Institutions
The majority of institutions requiring test scores from these exams are of the area of
higher education. Private adult schools and even some government institutions can require these
scores as well. For example, the IELTS score can be part of the determination for a Visa and
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
16
immigration in British territories (British Council, 2016). The most common use of the IELTS
exam however, is for entry into higher education institutions in British territories. Similarly, the
TOEFL exam is used by more than 9000 institutions for college entry, medical and other
licensing, and immigration (Education Testing Services, 2016).
The Debate Over the Purpose and Significance of the Exams
Ever since the introduction of standardized testing, there has been spirited debate on their
place and usefulness in the educational system. McKay (2002) examines the debate over English
standards, how they are defined, and the status of English as an international language so it is
requisite to ask if standardized tests are even justified. Studies have been done comparing IELTS
and TOEFL test scores to GPA and in each study, there has been no correlation found between
them, bringing into question their usefulness as gatekeepers (Ayers & Quattlebaum, 1992). The
debate then goes that these standardized exams provide additional cost, stress, and in many
instances, undo delay in the ability of students to complete their HE degrees in a timely manner.
While the studies do not deny the usefulness of the exams in determining the relative level of
English skills that a student may have, they question the amount of weight given to the exams in
terms of acceptance and placement.
Students’ First Attempt at Passing the Exams
There are studies from the 1990’s to date that have produced research data and factors
associated with a student’s first attempt when taking the exams.
Research Data about Student’s First Attempts
During the 1990’s, a number of narrowly defined studies were done regarding the exams,
and those research studies have begun to increase over the past dozen years as international
student enrollment in English institutions has risen exponentially (Haynie, 2014). Research has
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
17
been done within the United States, China, Australia, and Great Britain trying to understand the
factors that underlie the success and failure on the first attempt taking these exams (Ayers &
Quattlebaum, 1992; Esmail & Roberts, 2013; Floyd, 2014, Gan, Humphrey, & Hamp-Lyons,
2004).
Factors That Support First Attempt Success
Research data found two key factors related to student’s success: instructor training and
extracurricular English use. Success is defined as the student achieving the scores they needed
for their particular institution.
Instructor Training. Studies have revealed that successful students have cited having
well trained instructors who were trained proficiently in all four areas of the language and did not
have to rely heavily on supplements such as audio tapes or disks. Instructors used an adequate
mix of topic introduction, scaffolding, and group work that lowered the student’s “affective filter”
and made the learning process less stressful (Gan, Humphreys and Hamp-Lyons, 2004).
Extracurricular English Use. Successful students have also reported that they actively
used English outside of the classroom and sought additional activities to strengthen their newly
acquired skills (Gan, Humphreys and Hamp-Lyons, 2004). Successful students report that when
they have a positive experience in the classroom with strong instructors that are capable of
lowering their affective filter that they are more motivated to practice English outside of the
classroom (Gan, Humphreys and Hamp-Lyons, 2004). This applies to EFL and ESL
environments as most ELLs tend to self-segregate.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
18
Unsuccessful First Attempts and Possible Contributing Factors
The same research studies that examined first attempt success also examined multiple
factors associated with unsuccessful attempts: student GPA, forms of instruction, alignment of
instruction with the exams, compulsion, and location.
Student GPA. Going back to 1992, research studies have examined the correlation
between the exam scores and student GPA after entry into programs, and the findings found no
compelling evidence that the exams could predict student outcomes (Ayers & Quattlebaum, 1992;
Floyd, 2015). Students with low IELTS or TOEFL scores still completed their degree programs
with the same GPA as their counterparts with higher IELTS or TOEFL scores upon entry.
Alignment of instruction to exams. If instruction does not align with the exam, then it
could be the single most important factor identified with significant policy implications in every
country where English is taught as a foreign or second language. A sampling of class schedules
in comparison to the IELTS and TOEFL exam questions indicate varying degrees of alignment
between institutions (British Council, 2016; De Anza College, 2016; Educational Testing Service,
2016; Appendix G).
Compulsion. Another inhibiting factor is students being forced to learn English rather
than being given sufficient motivational reasons to want to succeed. Many countries require that
students learn English to a certain level of proficiency as a requirement for completing high
school and degree programs (Ayers & Quattlebaum, 1992). Students in De Anza ESL programs
tend to be in the program for a number of reasons: they want a degree from an English-speaking
institution; their workplace may require them to speak English; a family has been relocated to an
English-speaking country. If an EFL/ESL student is forced into any of these scenarios then their
degree of motivation towards success may be low.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
19
Location. Where the students are being taught, geographic location and place of
instruction, may have an impact on their ability to succeed with first attempt on exams. For
example, the instructors at their particular school may not be able to teach listening and speaking
skills due to their own lack of training, so they may rely on recorded material instead (Hadaway,
Vardell, & Young, 2009). Geographic location does not necessarily guarantee that the place of
instruction will be better than in another geographic location. For example, the California
Community College Chancellors database show a student success rate at San Jose City College
of only 18% but De Anza College, just 8 miles away, has a success rate of over 60%, which is
just below the average rate in Hong Kong. Although both colleges reside within the same County,
San Jose City College has a significantly higher low-income disadvantaged student population in
comparison to De Anza College, which may be a contributing factor.
Learning and Motivation Theory
This study uses a methodological framework that includes examining knowledge,
motivation, and organization, defined as follows:
Knowledge and Skills
There are multiple types of knowledge and influences that affect students when learning
and testing.
Knowledge types. Factual knowledge refers to the knowledge of facts, details, and
elements. Conceptual knowledge refers to knowledge of principles, theories, and models.
Procedural knowledge refers to familiarity with procedures, techniques, and methods.
Metacognitive knowledge emphasizes knowledge of cognition and awareness of one’s own
thoughts and actions (Anderson & Krathwohl, 2001); these are the four major types of
knowledge.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
20
Stakeholder knowledge influences. Several key factors can influence stakeholder
knowledge but ELL program structure and instructor behavior have been documented in studies
as having a significant impact on student success (Gan, Humphreys & Hamp-Lyons, 2004).
Specifically, EFL programs in many countries will emphasize reading and writing skill
predominantly while putting less emphasis on listening and speaking skills. In this same vein,
many EFL instructors may not be fluent themselves in listening and speaking and shy away from
those areas or leave the students with old recordings with which to practice on their own time.
Motivation
Persistence, which refers to an individual being motivated to start a goal, but lack of
persistence is being distracted by other goals and interests and thus the individual stops working
on the task at hand. Active choice refers to having the intent to act but failing to follow through
on the action and mental effort refers to individuals failing to utilize new knowledge to solve a
problem. These are the key aspects of concentration when discussing motivation (Clark & Estes,
2008).
Organization
An organization’s culture consists of many stakeholders and within these stakeholder
groups are organizational subcultures. According to Schein (2004), culture is created through a
group of people spending significant time together and adopting common beliefs and ideas. The
organizational leaders foster these beliefs that in turn act as models that create a social context
and model for which the work is performed.
Student Knowledge, Motivation, and Organization Influences
There are many subcategories with respect to student KMO that are examined in more
detail in this section.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
21
Student Knowledge and Skills
This study reviews literature that is pertinent to the understanding of the knowledge and
skills necessary for students to successfully pass the IELTS or TOEFL exams on their first
attempt. Influences examined in the relevant literature include procedural, factual, declarative,
and metacognitive knowledge and the associated skills necessary for the students to attain
successful achievement.
Guided by Anderson and Krathwohl (2001) the various knowledge causes and types are
explored: Students must have the basic knowledge to successfully pass the examination (factual
knowledge). Students must be aware of the various parts of the exams (conceptual knowledge).
Students must know how to take the exams (procedural knowledge). And students must be able
to reflect on knowledge through self-awareness (metacognitive knowledge).
Factual knowledge. Something that contains basic elements used for understanding and
organizing systematically is factual. Experts use it in communicating about their academic areas
of discipline (Anderson & Krathwohl, 2001). In a more generalized manner, factual knowledge
is necessary in almost all aspects of daily life when performing tasks because every task has
basic steps that are required to be successfully completed.
Students may not have the basic knowledge about the IELTS or TOEFL exams. This
basic knowledge comes in two forms. Prior to taking the exams, the student should understand
the separate elements contained within each area. Students must also have a minimum basic
understanding of the test material (Gan, Humphreys & Hamp-Lyons, 2004).
Conceptual knowledge. The basic elements taken together under a larger structure and
then seeing their interrelationships and how they are able to function together is concept
(Anderson & Krathwohl, 2001). Basic elements can function independently to a point but to
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
22
build more complex tasks, the elements must become interrelated such as classifications,
categories, principles, theories, models, and structures (Anderson & Krathwohl, 2001).
Students may be unaware of all the concepts addressed in the different exams (Gan,
Humphreys, & Hamp-Lyons, 2004). While students may understand the four separate elements
of the exams, they may not understand their interrelationships with respect to the taking of the
exams themselves. The speaking portion of the exams has an interrelationship with the listening
and reading sections, for example. Conceptual knowledge is crucial for students to successfully
complete this portion of the exam.
Procedural knowledge. How something is accomplished by way of understanding the
criteria and methods can be interpreted as procedural (Anderson & Krathwohl, 2001). Students
must know the specific classes to take to be prepared for the exams. Students may know little
about the specific areas of the exam to study. Students need to know how to speak and
understand approximately 8000 words to pass the IELTS/TOEFL exams (Esmail & Roberts,
2013).
Metacognitive knowledge. Cognition is the process of gaining knowledge and
understanding from thought, experience, and the senses. Metacognition is the knowledge of it in
general and the awareness of our own cognition (Anderson & Krathwohl, 2001). Understanding
one’s own knowledge is also crucial to pacing oneself with their goals. The temptation of
reaching the finish line early, without the proper foundation of factual, conceptual, and
procedural knowledge can inhibit success.
Students must have self-awareness as to their level of basic English language skills.
Students may understand individual words but they may not cognitively always understand a
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
23
sentence read or heard. Students may also rush their English learning and skip levels to try and
pass the exams more quickly, which is a cognitive choice (Floyd, 2015).
Student Motivation
This study further investigates literature into why students feel overwhelmed and make
assumptions about the exams rather than investigate and adequately prepare. As an example,
students may not have strong self-efficacy related to their content knowledge. The literature
explores the motivational factors that are affecting student behaviors (Greiner, 1996). Motivation
in this study, as it relates to students, is the belief in oneself in learning tasks, activities, and
factors related to both (Rueda, 2011). There are three motivational factors that will be examined
to determine the gaps in student performance on the IELTS and TOEFL exams as described by
Clark and Estes (2008).
Student active choice. It is not the intention of the student to pursue a goal but their
active pursuit of the goal that matters (Clark & Estes, 2008). Some students do not like to do
group work even though it is necessary for learning, while others know that they should practice
English each day but instead hang out with their friends who speak their first language. There are
also instructors that may not hold students accountable if they do not contribute to group
assignments (Brown, 1998), which may reinforce the choice not to participate. While on the
surface, the obvious answer is that they are taking the tests voluntarily and therefore active
choice is given, there are many factors under that umbrella of choice where actively choosing not
to do something may influence their outcome in a negative manner (the performance gap).
Student persistence. It is desirable for students to be able to prioritize their time with the
importance of their goals (Clark & Estes, 2008) and it is far too easy for students to become
distracted by less important goals. This study will examine how many goals students have and
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
24
how well they can prioritize their goals with their time. The study will then look at how well the
students are persisting with respect to their goals in the face of distractions. While the students
may have their goals prioritized correctly, distractions in different forms may cause the student
to change that priority. For examples, students may tend to give up or create a false belief that
they cannot learn if they do not pass the exams on their first attempt (Purwanto, 2012). One of
the goals of this study will be to identify these types of distractions and their correlation with the
identified gaps.
Student mental effort. The third area that must be evaluated is whether the student is
putting in adequate mental effort combined with the active choice and persistence (Clark & Estes,
2008). Some goals only require drawing from past experience, while others require greater
mental effort. Greater the mental effort required to succeed at a goal requires greater self-
confidence. Many students may not use English outside of the classroom because they do not
understand the amount of mental effort required to prepare for the exams (Gan, Humphreys &
Hamp-Lyons, 2004). If they are not clear on this then they may not do well on their first attempt,
which in turn can affect their motivational and active choices. This consideration must also be
considered when analyzing the performance gaps. Other areas of mental effort such as task value
and self-efficacy play a role in effort. It is imperative to understand the amount of value the
students are placing on ELL and how much confidence they carry with their efforts.
Organizational Factors
Processes, resources, and culture play the third part of gap analysis (Clark & Estes, 2008).
The students may need the resources, policies, and guidance necessary to align the curriculum
with their goals and objectives. This study will evaluate whether students are receiving adequate
support based upon the organizations current processes and resources. Additionally, the study
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
25
will evaluate if the organization is addressing culture in a way that maximizes support for the
student.
Processes and resources. On the surface, it may seem that the organization has enough
personnel and resources to accommodate a rapidly growing ESL population. This study
investigated how the resources are spread horizontally and vertically within the organization and
evaluated whether they are adequately supporting the area of student preparation for the IELTS
and TOEFL exams. There may not be enough counselors, teachers, and tutors trained to assist
students with preparation for the exams. Thus, students may need to seek additional counseling
outside of the organization, which in turn can add additional financial burden to their student
loan debt (Greiner, 1996).
Cultural model. With continued growth of English Language Learning demand, the
current model may not adequately support the needs and expectations of the students. The model
may not place the level of importance on that area that was not necessary in the past but is quite
necessary now. With significantly more international students demanding more ESL courses, the
organization’s cultural model must be evaluated to understand whether it is just expanding what
has previously existed or, evaluating the additional needs that the new student population also
brings. The exams therefore, may not be included as a part of the curriculum, counseling, or
support services (Purwanto, 2012).
Cultural setting. If the cultural model is just an expansion of a previous and smaller
model, then it may be true that teachers may not be required to learn about or make exam
preparation part of their lesson plans. This study will evaluated whether the ESL instructors were
aware of and prepared to address these exams with their current and future students. In previous
studies, it was found that many students who felt that their instructors were not fully aware of or
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
26
supportive of their efforts to prepare for these exams suffered from a lack of self-efficacy (Ayers
& Quattlebaum, 1992).
Teaching English to Speakers of Other Languages (TESOL)
This section introduces TESOL and what it means, why there is an increasing demand in
TESOL, the four English language learning areas, and the types of institutions teaching TESOL.
The Differences between ESL and EFL
TESOL encompasses teaching English in countries where English is the primary
language, in which case English as a Second Language (ESL) is taught. When TESOL is in a
country where English is not the first language then English as a Foreign Language (EFL) is
taught (Hadaway, Vardell, & Young, 2009).
When EFL is being taught, the curriculum is designed to consider that students do not
have regular access to English speakers within their own country. Lesson plans are designed
differently and may or may not include more intensive language labs to compensate for the
student’s limited ability to practice the language outside the classroom. ESL instruction assumes
that since the student is living in a place where English is the first language that students must
use the language outside of the classroom to “get by” in society. The reality is that ISs self-
segregate and rarely use English outside of the classroom (McKay, 2002).
Teacher training in both EFL and ESL is also a concern. EFL instructors may not be
proficient in listening and speaking skills, since these are typically given less weight than reading
and writing. Thus, students may be handed CDs and practice manuals and told to do it on their
own, possibly without being tested with some frequency to see if any learning is occurring. ESL
teacher training has another completely unique problem in that teachers are not required to have
any multicultural training (TESOL) if they are deemed to be competent English instructors.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
27
There is evidence to suggest that without this training that a more ethnocentric worldview will
exist within the instructor’s pedagogical scope. Therefore, this gap in cultural understanding can
lead to student learning not occurring effectively as it could and more students not passing
courses that they may pass with a TESOL trained instructor (Nunan, 2003).
An Increasing Demand for TESOL
The U.S. Department of State’s Bureau of Educational and Cultural Affairs (2015),
shows that the numbers of international students increased by nearly 100,000 in 2014-2015 to
974,926 with an average annual growth rate of ten percent. For better or worse, whether people
agree or disagree, English is becoming the lingua franca due to the increasing influence of global
business. Even countries with regimes considered hostile to the west require their college
students to master a certain level of English before they can receive their degrees (Gan,
Humphreys, & Hamp-Lyons, 2004).
The Four Areas of English Language Learning Tested in the TOEFL Exam
English language learning is typically divided into four areas of study, they are: listening,
reading, speaking, and writing; the TOEFL exam tests students in each of the four areas. Herein
lies the challenge for socioeconomically challenged students. Students with access to the
financial resources will take TOEFL preparation courses but they are costly (Templer, 2004).
The 39-year-old English Language Center located in Boston, Los Angeles, and Santa Barbara
has a minimum 4-7-week commitment, charging in 2017 an estimated $1850 starting fee. And
public U.S. institutions, such as De Anza College, do not offer any type of TOEFL preparation
courses (Appendix G).
Listening. Listening skills will mostly be neglected and relegated to the bottom of the
“importance” list. Thus, this is also typically the area of the entrance exams receiving the lowest
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
28
scores, which requires a knowledge of 8,000 – 14,000 words. Listening is a receptive skill that
requires a person to receive input and process it in real-time, comparing it to our own ideas and
experiences and then apply meaning (Helgesen, 2003). Listening is typically taught and grouped
with all four areas through the intensive intermediate level and then only with speaking through
low advanced level. These levels are relatively uniform across the spectrum of institutions in
terms of the level of student preparedness. In other words, a student at the low-advanced level is
close to or at being able to comprehend 6,000 words and strings of sentences. If students do not
have access to auxiliary workshops or support groups, they may take the exams completely
unaware of the level of understanding fully required to achieve the minimal score.
Reading. Most students fair well on the reading section in the entrance exams however,
that depends strongly on the EFL/ESL instructors they have in the more advanced levels. If the
instructor is not familiar with Asian culture, as an example, they will not know that emphasis is
put on memorization and not critical thinking with summaries. As a result, their students may not
be as well prepared as others. The multicultural training in this area is crucial. Reading is not
only a receptive skill but also a fluent process where readers combine information from text and
their own background to create meaning with the goal of comprehension (Anderson, 2003).
Reading is taught grouped with listening, speaking, and writing from a high beginning level
through an intensive intermediate level and then as an individual course from high intermediate
through advanced levels.
Speaking. Speaking is considered a real-time, productive oral skill producing utterances
systematically to create meaning (Bailey, 2003). Speaking is also a major issue in EFL
instruction because many instructors may, rightly or wrongly, believe that their own speaking
skills are not strong enough to teach this area and as a result, push the students to using
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
29
technology and manuals. In ESL, this area is also relegated to the lower level of importance with
formal classes ending at the upper-intermediate level and a few classes possibly offered as
electives without much promotion. Therefore, this area of the entrance exams may be scored low
by students on their first attempt.
Writing. Students who are prepared to take entrance exams like the IELTS and TOEFL
tend to be strongest in this area. One exception in this area may be grammar skills. A student’s
grammar skills can vary greatly with other students depending on the instructors that they have
had previously. Writing is both a physical and mental act for expressing and impressing and
requires processing and production skills (Sokolik, 2003). Writing is most probably considered
the most important area of EFL and ESL curriculum. Students receive the most intensive
curriculum and testing in this area.
International Schools, Colleges, Universities, and Instructor Credentials
There are generally three types of locations that an ELL will go to study English and
these institutions can vary greatly with instructor credential requirements and thus, capability.
They are international schools, colleges, and universities. Attending these institutions is not just
unique to countries where EFL is taught, it is also common to countries where ESL is taught as
well. It is important to understand the differences between these institutions and their instructors
when considering the success rates of students taking the IELTS or TOEFL exams for the first
time.
International schools. International schools tend to be the first path that people take
when learning English as a new language as these schools have curriculum designed for first
time ELLs. This is true for both ESL and EFL students (McKay, 2002). These institutions are
privately owned and cater to K-12 and adults. There are no government guidelines or regulations
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
30
with respect to ESL teacher qualifications. It is therefore up to the potential students to check on
the credentials of the teachers before signing contracts and entering into these programs, which
can also be costly.
Colleges. For students who gain a “high beginning” ability and can afford to attend, the
community colleges in the United States are the most common choice (Institute of International
Education, 2015). ESL instructors at community colleges are required to have a master’s degree
or equivalent to teach but they are not required to have any multicultural training, such as
TESOL. ESL programs, such as those at De Anza College, focus on reading and writing with
only a few quarters of listening and speaking emphasis (Appendix G).
Universities. Most undergraduate ESL students will transfer to a university from a
community college (Institute of International Education, 2015). In 2006, it is estimated that 14%
of all transfer students in the State of California were Community College ESL students
(Sengupta, R., & Jepsen, C., 2006). Many universities outside of the U.S. offer EFL programs to
their undergraduate students and in some countries, such as China, mastering a certain level of
English is mandatory in order to receive a Bachelor’s degree (Gan, Humphreys, & Hamp-Lyons,
2004).
Instructor credentials. Students need to be diligent in their background checks of
various institutions. Private institutions may hire instructors with bare minimum skills required
by their local, county, and state governments. In many countries, you simply need to be born in
America to teach EFL; blonde hair, blue eyes, and white skin being a plus (McKay, 2002). And
there is an underlying problem, even with the HE institutions that require instructors to have
advanced degrees; part-time instructors do not have the same access to training as their full-time
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
31
counterparts, and yet they account for more than 50% of instructors in HE within the State of
California in 2016 according to the California Community College Chancellor’s Office.
Summary
Many studies have been performed to analyze varying aspects of the TOEFL exam. The
literature selected for this study examines performance factors from several perspectives. The
purpose and significance of the IELTS and TOEFL exams, how they are administered, scored
and evaluated by institutions are explored. Understanding the first attempt passing rates of
students taking the exams and possible factors that are associated with successful and
unsuccessful scores. Learning and motivation theory are also examined as a part of performance
factors. The study examined what is in the students control (knowledge and motivation) and
what is out of their control (organization).
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
32
CHAPTER 3: METHODOLOGY
Purpose of the Project and Questions
The purpose of this study was to evaluate the degree to which international students
attending the De Anza College ESL program are prepared to take the TOEFL exam and succeed
on their first attempt. The definition of successful completion, for the study, was satisfactory
scores in all areas of the exam as required by an institution chosen by the candidate for purposes
of entry.
As such, the questions that guide this are the following:
1. To what extent is De Anza College achieving its goal for all international students
to be capable of passing the TOEFL exam on their first attempt?
2. What is the status of international students’ knowledge, motivation, and
organizational (KMO) influences needed to pass the exams on their first attempt?
3. What are the recommendations for organizational practice in the areas of KMO
resources?
In this chapter, best practices are discussed to answer these questions based on a
methodological, research-based manner by utilizing a specific framework, assumed performance
based influences from a stratified sampling of a primary targeted population. Assumed
performance-based influences, participating stakeholders, data collection, validation of the
performance, data analysis, limitations, and delamination were examined and taken into
consideration with respect to the methodology.
Methodological Framework
The conceptual framework was implemented from an adaption of the gap analysis
process model described by Clark and Estes’ (2008) an analytical implementation method that
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
33
helps to improve organizational performance by setting goals and investigating influences along
three dimensions of knowledge, motivation, and organization (Figure 1). The GAP Analysis
Process is a continuous seven-stage cycle that begins by clearly defining goals (Figure 1). Goals
are then compared to what is being achieved (Stage 2) with the difference(s) being the gaps
(Stage 3). The next step (Stage 4) identifies what the causes are for the identified gaps: the
knowledge of the stakeholders; their motivation factors; and possible organizational barriers that
may be preventing the stakeholders from achieving the stated Goals (KMO factors) and their root
causes identified. The last three remaining stages are to develop and implement solutions, and
evaluate whether those solutions are working – or to what degree they are working and then
repeat the cycle until the desired goal is achieved
Figure 1. The GAP Analysis Process Flow Chart.
The gap analysis model by Clark & Estes (2008) was adapted as an evaluation approach
for the study. Like an improvement model, the evaluation requires an analysis of the
organization’s goals and current achievements that will reveal gaps. The causes for the gaps were
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
34
identified along the three dimensions of the KMO and recommended solutions and
implementation are offered to the organization.
The methodological framework for the study is a mix of qualitative and quantitative
evaluations using descriptive statistics. Assumed KMO influences at De Anza College and its
ability to provide student achievement are generated based on personal knowledge and related
literature. The various influences are assessed by using surveys, interviews, literature review,
document analysis, and content analysis. The area of motivation reveals the level of choice,
persistence, and mental effort on the part of students preparing for their first attempt on taking
the exams, thereby contrasting what the organization can or should be doing to assist the students.
Recommendations are made on research-based solutions and evaluated in a comprehensive
manner.
Assumed Performance-Based Influences
The study evaluates performance based upon issues, needs, and assets using a gap
analysis process. According to Clark and Estes (2008) a gap analysis process is a problem-
solving approach to systematically improve individual performance and lift organizational goals.
This method avoids the presumptions that people tend to make that omit the real causes and
therefore, leads to inappropriate solutions and therefore, avoidance due to perceived complexities
that make the correction seem unattainable. The study includes three components to conduct a
thorough investigation into the performance gaps identified: informal scanning, knowledge,
motivation, and organizational theory; and a review of related literature concerning first attempt
performance on IELTS and TOEFL exams. Causes from the related literature have been
discussed in Chapter 2 and tables from this literature are included in this chapter. Following is an
examination of the knowledge, motivation, and organizational assumed causes for students
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
35
preparing to attempt the IELTS or TOEFL exams and the gap between successful and
unsuccessful students.
Preliminary Scanning Data
Informal conversations and observations conducted at the organization provided what is
currently known without having conducted a deeper study and are discussed below in three
categories and Table 2.
Knowledge and skills. At De Anza College, most international students are aware of the
IELTS and TOEFL exams however, most of the ESL instructors seem to only know of the
TOEFL exam. Those instructors that know of the IELTS exam may not know anything further
about the exam. Guidelines set forth for instructors to create lesson plans include a list of
approved textbooks, material, and expected learning outcomes but do not contain any reference
of alignment with the IELTS or TOEFL exams. Students who are looking for support services on
the campus cannot find them because they do not exist. Only one center is known of that has a
staff member and a volunteer instructor that can provide counseling and guidance to students for
exam preparation. As a result, students who cannot find these two individuals may give up and
go elsewhere to gain the knowledge necessary to prepare for the exams.
Motivation. Many ESL instructors are part-time adjunct faculty and are not paid for
anything other than their class time and “office hours” to meet with students each week. While
many different training opportunities are available, only full-time instructors are compensated for
their time attending additional training and the part-time faculty therefore have no incentive to
attend. Further, the small handful of full-time faculty spend forty-plus-hours each week on the
campus and have an opportunity to share knowledge and experience, whereas the part-time
adjunct faculty are only on campus to teach a class and then leave. Thus, much valuable
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
36
experience is only seeded with a few and not with the many and it is the students who will not
benefit in the end. When a student asks an instructor a question for information pertaining to the
IELTS or TOEFL exams and the instructor does not have an answer, they also have no real
incentive to follow-up with “colleagues” since they never can share information.
Organization. Several organizational issues could contribute to the problem being
evaluated, such as constrained budgets and increasing international student growth and demand.
International student enrolment is growing at a significant rate each year and the time it takes to
add or change processes can take many years due to the scale and layers of bureaucracy. It may
be that the gap issue with the exams has not been floated to a level high enough to affect the
necessary processes to begin to add to the curriculum. Further, due to the constrained budgets, it
is difficult to pay part-time faculty wages that are high enough to allow them to live in what is
considered the most expensive area of the United States. Many instructors are forced to move
away because they simply cannot afford to live in the area and teach. Teacher retention may also
be an organizational issue that is directly, or indirectly affecting the gap problem being evaluated.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
37
Table 2.
Summary of Sources about Assumed Causes of Knowledge, Motivation, and Organizational
Issues
Causes
Sources
Knowledge Motivation Organization
Scanning
interviews,
personal
knowledge
Procedural:
Students do not
know the specific
classes to take to be
prepared for the
exams.
Students know little
about the specific
areas of the exam to
study.
Students feel
overwhelmed and
make assumptions
about the exams
rather than
investigate and
adequately prepare.
The organization does
not provide counseling
or support services
that prepare students
for the exams.
Learning and
motivation
theory
Factual:
Students did not have
the basic knowledge
about the IELTS or
TOEFL exams.
Conceptual:
Students are unaware
of all the concepts
addressed in the
different exams.
Procedural:
Students need to
know how to speak
and understand up to
8000 words to pass
Active Choice:
Some students do not
like to do group
work even though it
is necessary for
learning.
Some instructors do
not hold students
accountable if they
do not contribute to
group assignments.
Persistence:
Students tend to give
up or create a false
belief that they
Goal setting:
The organization has
not given students a
clear path for
achieving the minimal
exam scores.
Cultural model:
The exams are not a
part of the curriculum,
counseling, or support
services.
Cultural settings:
Teachers are not
required to learn about
or make exam
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
38
Participating Stakeholders
The primary population for the area of study is international ESL students at De Anza
College. The college has the third largest population of international students at community
colleges in the United States, with nearly 3,000 enrolled each quarter. These students represented
an excellent opportunity for a stratified random sampling. The students were subdivided into
the IELTS/TOEFL
exams.
Metacognitive:
Students have the
basic understanding
of English.
Students can
understand the
individual words but
they may not always
understand a
sentence read or
heard.
Students may rush
their English
learning and skip
levels to try and pass
the exams more
quickly. They may
not know how to
pass the exams as a
result.
cannot learn if they
do not pass on their
first attempt.
Mental Effort:
Students do not use
English outside of
the classroom
limiting their ability
to practice.
preparation part of
their lesson plans.
Professional
Development:
Teachers are given
little to no training
about the exams.
Resources:
There are not enough
counselors, teachers,
and tutors trained to
assist students with
preparation for the
exams.
Related
research
literature
Students do not have
a good understanding
of the four areas of
the exams.
Students do not have
strong self-efficacy
related to their
content knowledge.
Teachers do not have
training on content or
pedagogical
knowledge to address
student needs in
preparing for the
exams.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
39
groups, with the first division being students who plan to take the IELTS or TOEFL exam again
to gain entry into a graduate school program. The groups were further divided into smaller
groups based upon country of origin, gender, age, GPA, and area of study. By stratifying the
students into many subgroups, the study had greater diversity in which to study the knowledge,
motivation, and organization of the students. For example, does their country of origin prepare
some students better for the exam than other countries (organization) and do certain groups
persist better or more frequently than other groups (motivation).
Since this study was an evaluation, what is learned about the students by stratifying them
into smaller subgroups created a more accurate baseline that was used to measure the success of
the institution where the study is being performed, De Anza College. While the primary purpose
of this study was to provide De Anza with an accurate evaluation of its performance around
preparing students for IELTS and TOEFL examination, an ancillary benefit could be information
that can be provided to governments about their performance in the same area as well. The goal
of this study is to improve English Language Learning and preparation for the IELTS and
TOEFL exams so that more students are prepared to pass with acceptable scores on their first
attempt.
Data Collection & Analysis
The Clark & Estes methodological framework was used to develop the instruments to
collect data for the study. Questions used within the survey and interviews concentrated on
understanding the knowledge, motivation, and organizational influence on the participating
stakeholders with respect to the problem being evaluated. Permission was obtained from the
University of Southern California’s Institutional Review Board (IRB) and the De Anza College
IRP. All participants provided informed consent and their identities will remain confidential with
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
40
use of pseudonyms. The validation of knowledge, motivation, and organizational assumed causes
was accomplished through online surveys, in-person interviews, and an in-depth document
analysis. Each of these items is discussed as follows:
Survey
There was a single survey that could be completed online at the student success center
website. Once IRB and IRP approval was received by the University of Southern California and
De Anza College then the survey was made available from October through December 2016. Of
the 2,400 students enrolled during this period, a subgroup of 419 student participants that made
2361 visits and spent 2014 hours at the listening and speaking center were selected. The survey
was distributed in English and contains 28 demographics, Likert scale, contrasting, and open-
ended questions. Knowledge, Motivation, and Organizational items were given equal weight in
the design of this project. Surveys have been stored on an encrypted external digital device and
the device is locked securely in a safe place. All demographic information collected is not
identifiable and is also encrypted. The survey instrument can be found in Appendix D.
Survey questions are Likert scale based and examined for frequencies. The survey was
available to all enrolled ESL students at the evaluation site (LSC) for a period of 10 weeks.
There were 27 questions that documented demographic information in addition to student
performance and their impression of curriculum and it effectiveness, or lack thereof. The final
question of the survey asked the student if they would agree to an interview on a set date and
time, should they be deemed qualified (Appendix D).
Interviews
Interviews were conducted in a quiet study room within the De Anza College Library.
Participants were chosen through the survey; each survey had a final question at the end asking
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
41
the survey-taker if they would be willing to participate in an additional live, one-on-one
interview (purposeful sampling). The interviews were scheduled to occur over a one-week period
with 10 students or until saturation occurred (no new information being received), whichever
came first. All interviews were conducted in English since the interviewees had a vocabulary of
more than 6,000 words necessary to communicate with a native speaker. Interviews were
confidential and pseudonyms used. Student participation was voluntary.
One week was scheduled for ten, 45-60 minute interviews and each interview was
recorded and transcribed. Primary analysis of surveys consisted of matching commonalities
observed to identify strengths and weaknesses that further provided an understanding of the
significant gap that exists.
Interviews consisted of 8-10 questions recorded over a 45 to 60-minute period and were
fully transcribed and coded into themes that align with the knowledge, motivation, and
organization primary categories, then coded further into issues, needs, and assets subcategories.
Interviews were conducted as a means of gathering additional information that cannot be gleaned
from a survey and provided additional questions and areas of examination. Students who were
considered “qualified” had taken the IELTS or TOEFL exam previously. The interviewees had
received mixed scores that for some were high enough to provide them with entry into an
institution and for the others not. They had taken these exams from different geographic
locations (Appendix E).
Document Analysis
For the purposes of the project, multiple documents were analyzed. The student success
center, listening and speaking center (LSC) quarterly student surveys dating back to 2011,
provided relevant voluntary student demographic data. The LSC quarterly usage reports dating
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
42
back to 2011 was matched with the student surveys to track what ESL support services were
used, when, and for what length of time per use, and overall use per quarter. Documents from the
California Community Colleges Student Success Scorecard that includes but is not limited to:
Momentum Points, Degree/Transfer Outcomes, and Career Technical Education Metrics
provided valuable inferential information such as when students expected to transfer and when
they could transfer, including but not limited to courses that needed to be retaken or further
additional courses taken before transfer. Finally, requests were made for general data from the
respective organizations that administer and maintain data for the IELTS and TOEFL exam.
All paper and electronic documents made available, including observation notes, have
been examined to triangulate the survey and interview data to further strengthen the analysis for
accuracy. Documents were retrieved for the past five years from the evaluation site and the
California Community College Chancellor’s database, in addition to observation notes
(Appendix F).
Observation Notes
Students were observed in a workshop environment at the LSC for knowledge,
motivation, and organizational skills. Workshops were conducted in a large, 12’ x 12’ cubical-
type room setting where the walls were not tall enough to block sound, thereby allowing the
investigator to observe and make notes without being obtrusive. Workshops were selected based
upon subject matter to better denote KMO factors; for example, a workshop on idioms
demonstrated student knowledge, and a workshop on academic skills would highlight and even
contrast student motivation and organization. The investigator taught some workshops and would
make personal notes immediately afterward so as not to forget salient moments that contributed
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
43
to the study. Observation notes (Appendix F) were used at the end of the study to triangulate the
other data collected.
Validation of the Performance
The study has attempted to validate its assumptions using surveys, interviews, and
document review. By triangulating the data, we expect to eliminate invalid assumptions and
concentrate on those that are supported by the analysis of the data collected. Summary tables
have been created for each of the KMO performance issues and questions pertaining to their
issues, needs, and assets are correlated with the instrument type that will be used (see the
example below):
Table 3.
Linking the Instruments to the Inquiry Questions
Research Question Survey Interviews Document Review
IQ 1 X X
IQ 2 X X
IQ 3 X X X
Trustworthiness of Data
The study uses triangulation of data between the instruments used to collect data,
which is surveys, interviews, and document analysis (Merriam, 2009). All survey instruments
have been originally generated and taken by current students within the program on a voluntary
basis. The instruments never asked for, or recorded in any way whatsoever, the students name or
any identifying number (such as social security or student identification numbers). Should any
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
44
respondents have been employees or students of mine, then a third-party administrator would
have been selected and agreed to administer the surveys, interviews, and observations in my
absence to avoid as much researcher bias as possible (Maxwell, 2013). Further, no document
reviews of my current employees or students have been used in the analysis. Finally, the matter
of interpretation, or the quality of conclusions within the study has been taken very seriously and
utilize as many overlapping tactics as possible to ensure that a rigorous accountability that
upholds merit, integrity, and ethics exists (Huberman, Miles, & Saldana, 2014).
Role of Investigator
I am a part-time ESL faculty member and volunteer workshop moderator at De Anza
College. As an ESL instructor, I teach one-to-two classes per quarter in the areas of listening,
speaking, reading, and writing. As a workshop moderator, I will teach multiple one-hour sessions
each week in listening and speaking. As a full-time researcher at an educational technology
(EdTech) company, I have access to a wealth of literature related to the areas of student ELL and
the IELTS and TOEFL exams. As the principal evaluator of this project, I will be conducting a
gap analysis as part of an evaluation of De Anza College’s ESL program to determine how they
can improve their curriculum to better support students who desire to take the IELTS and
TOEFL exams to enter graduate school programs in the United States or another country where
English is the first (native) language.
Within the scope of this project, students were aware that I am acting as an evaluator and
that all steps were taken to protect their anonymity and that the entire purpose of this evaluation
is to improve the program to help students improve their chances of passing the IELTS and
TOEFL exams on their first attempt. Students were reassured that participation in the survey and
interview was strictly voluntary and that their personal identity will be kept strictly confidential.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
45
Finally, students were made aware that I am conducting this project as a doctoral candidate and
that findings will be compiled and presented to administrators, who may have the authority to
decide what to do with the findings of this project.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
46
CHAPTER 4: RESULTS AND FINDINGS
Data collection consisted of voluntary surveys and interviews; observations; and
document analysis. Surveys were offered online to 419 students who used the De Anza College
Student Success Listening and Speaking Center on a regular basis from October through
December 2016. Of those 419 students, 46 students (11%) took the survey. The survey was
created using Qualtrics. Ten students were selected from the 46 students who took the survey for
interviews that were conducted over a one week period at the end of the data collection period.
The Student Success Listening and Speaking Center provided five years of general data on
student use of the center for purposes of further triangulation of the data collected. Following is a
breakdown of the results of the data collection.
The survey was voluntary and the country of origin of the survey taker was therefore,
completely random. Thus, the students that completed the survey were from China, Korea, Japan,
and Vietnam only, all Southeast Asian countries, which represents approximately 45% of the
ESL student population. Seventy-percent (70%) of the respondents identified themselves as
female and thirty-percent (30%) as male, whereas the overall female to male ESL population is
approximately equal. Most survey takers (78%) were between the ages of 18-24.
Table 4.
Survey Respondent Age Range
Age Range Percentage
18-24 78
25-30 17
Over 30 5
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
47
Ninety-five percent of the survey takers studied English in their home countries. Forty-
three percent (43%) studied English for 12-16 years in their home country. From this group,
twenty-seven percent (27%) only studied English in their home country; thirty-nine percent (39%)
studied English for one year outside their home country; twenty-two percent (22%) studied
English for two years outside their home country. Following is a breakdown of the KMO data
collection and analysis that was designed to understand the evaluation of the gap analysis of first
time passing rates for the TOEFL exam.
Results and Findings for Knowledge Causes
Knowledge causes in this evaluation focus on what English Language Learners think they
know in terms of the language itself and the four distinct parts and procedures of the TOEFL
exam. The survey results revealed an anomaly between student perceived knowledge and actual
test score results. Interviews would reveal that socioeconomic factors are an unexpected and
contending factor with student TOEFL test results.
Survey Results
Students were asked about their level of awareness of each part of the TOEFL exam. The
sections of the exam are listening, reading, speaking, and writing. There are four separate
sections of the exam, each worth 30 points for a total of 120. While some institutions only look
at the total score, others may look at the score for each section, requiring a minimum score from
each section. The purpose of the awareness question was to identify a student’s perceived
knowledge of the TOEFL exam. Eighty-percent (80%) of respondents either somewhat or
strongly agreed that they thought they were aware of all four sections of the exam. Twenty-
percent (20%) neither “agreed or disagreed”. The student’s high perceived awareness is
juxtaposed to the actual test results; lower test results are concentrated to areas of the exam that
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
48
are not rigorously taught in public education and require extracurricular investment and
preparation.
Participants were then asked if they felt they had been informed of all the classes that
they should take to be able to pass the TOEFL exam successfully on their first attempt. The
purpose of this question was to see if students who felt they were aware of the four parts of the
exam were equally as confident in their perception that they were adequately informed of all of
the classes that they should take in order to pass the exam on their next attempt. They numbers
were nearly identical with 82% of students strongly or somewhat agreeing that they believed that
they were fully informed. In addition to fully understanding each of the four sections of the exam,
it is equally important to understand the procedures involved with each section of the exam.
Respondents again believed similarly, with 83% strongly or somewhat agreeing that they
understood examination procedures.
Students were also asked if they were aware of the minimum number of words needed in
their lexicon to be able to score successfully. The purpose of this question was to probe deeper
into the student’s knowledge base. While it is important to understand the various parts and
procedures for taking the exam, in addition to the classes necessary, it is also essential to know if
students understand if they have reached a minimum level necessary to pass the exam with the
score that is required. Eighty-three percent of students responded that they agreed strongly or
somewhat.
On average, 82% of students strongly or somewhat agreed to all four questions, which is
significantly higher than actual first-attempt success rates revealed in other studies (Table 5).
Initial consideration was that there were misperceptions within a percentage of those students
whom answered. Triangulation of the data revealed that the same percentage of students were
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
49
from a higher socioeconomic background than the 18% whom indicated lower confidence levels
in the four questions.
Table 5.
Knowledge Surveys
Strongly
Agree
Somewhat
Agree
Neither Agree
or Disagree
Somewhat
Disagree
Strongly
Disagree
Understood
each part of
the exam
39 41 20 0 0
Informed of
all classes
51 32 10 2 5
Awareness of
examination
procedures
51 32 10 2 5
Awareness of
minimal
words needed
to pass*
43 40 8 5 5
* Total of 101 due to respondent hitting the back key and changing their answer resulting in a
double count.
Findings from Interviews
Overall, the students interviewed had studied English throughout most of their lives in
their home countries. The students that agreed to be interviewed all had a GPA of 3.0 or higher.
The purpose of the interviews was to understand student knowledge of the four parts of the exam
and to uncover information that may not have been revealed from the survey questions.
Socioeconomic Standing. The correlation between the students who participated in the
survey and the percentage of students who answered the survey as being in “strong” or
“somewhat” agreement matched their acknowledgement of coming from a higher socioeconomic
background and receiving additional training from private institutions (8 of 10). Two of the ten
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
50
however, revealed during the interview that they experienced financial hardship and had to find
other ways to prepare for the exam; Rebecca from Taiwan said she spent time searching multiple
libraries in order find the newest resources; Grace from China spent time finding free online
resources; both students taking help from economically stronger friends and working multiple
jobs.
Reading and writing. The most recurring pattern that was exhibited in all ten interviews
was that emphasis on reading and writing dominated their public school English studies. Andrea,
from South Korea, stated that she started learning English at age 10 and that now students begin
to learn in the first grade and that all classes focus solely on reading and writing. Connie from
Japan stated that she was required to take EFL grammar and reading classes to graduate. JC and
Grace from China also affirmed that EFL reading and writing skills were a mandatory part of
their high school curriculum. TJ from Vietnam also stated that EFL reading and writing are a
part of the public-school curriculum starting in middle school and continuing through high
school.
Listening and speaking. An unexpected opposite from reading and writing; public
schools in each country, according to the interviewees provided little instruction for listening and
speaking skills. Rebecca, from Taiwan, stated that the English classes that she took were more
“conversation wise” and akin to “how you would survive in an English-speaking country.”
Connie, from Japan, JC & Grace from China, Andrea from South Korea, and TJ from Vietnam
all stated that they concentrated little, if any, on listening and speaking skills. Students from
China, Japan, and Korea had private schooling alternatives available to them that supplemented
their listening and speaking skills for the exam. Students in Vietnam tend to rely on their
teachers as tutors outside of public school hours. In all cases, these extracurricular study
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
51
programs were dependent on the student’s family having the financial resources to cover the
costs.
Knowledge of the exam. Interviews revealed that focus on preparing to take the TOEFL
exam was viewed in relation to overcoming an obstacle rather than mastering English. It also
revealed that what students thought they knew about the exam was not always correct as was the
case with the survey question about knowing the minimum number of words necessary to
achieve a high score on the exam. All ten students interviewed stated that they had selected
“strongly agree” but when asked the actual number, every person interviewed got the correct
number range wrong (8000 – 14000).
Observations of students tested at the same level indicated that knowledge skills were
less relevant than motivational and organizational factors. Students who used the student success
center were practicing their English listening and speaking skills to a greater degree. Students
observed acknowledged to one another that they were only using English to communicate with
each other in the mixed cultural setting. The students would congregate with students who spoke
their home language when possible, so they did not use English unless it was necessary.
Documents that logged student visits were provided by the De Anza College Student
Success, Listening and Speaking Center. The documents showed that students who took the
survey visited the center an average of 18 times and spent an average 4.8 hours there each
quarter. Most of the students primarily attended workshops designed to help them improve their
English skills through a variety of topics. One-on-one tutoring sessions were the second most
used service followed by foreign language exchange meetups.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
52
Synthesis of Results and Findings for Knowledge Causes
Socioeconomic standing became a key factor in correlation to TOEFL exam passing rates.
English education in the student’s home countries is concentrated on reading and writing skills.
Listening and speaking tend to be glossed over and limited to use of old CDs because the
teachers are themselves, not fluent enough to teach the listening and speaking skills. Most
students rely on private business services to supplement the areas not covered in their public-
school education, primarily listening and speaking and preparation for the TOEFL exam. These
services are expensive and exclude international students with limited financial resources.
Results and Findings for Motivation Causes
The reason for examining student motivation is to determine the possible reasons why a
student may score high or low on the TOEFL and what level of influence their motivation may
have on their performance factors.
Survey Results
When asked if anybody else had an influence on the student’s choice to take the exam,
62.5% said that they had not been influenced by anyone else. Self-motivation is a critical factor
to persistence (Clarke & Estes, 2008) and is an important factor when preparing to take
standardized examinations. Remaining participants that felt someone had an influence on their
taking of the exam would feel some form of compulsion, which leads to lower motivational
efforts that can result in lower scores (Ayers & Quattlebaum, 1992). These numbers fall closer in
line with actual test result success rates.
Students who took the survey were asked if they actively choose to use English outside of
the classroom and three-quarters responded affirmatively. This is a key motivational indicator
when assessing possible student success on first attempt performance (Gan, Humphreys &
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
53
Hamp-Lyons, 2004). This number seems disproportionately high when compared to interviews
and observation notes however, the students selected for the survey were identified as already
highly motivated based upon their high usage of the LSC.
Respondents were evenly split when asked if they ranked the importance of preparing for
the exam higher than their other goals (Table 6). The division of importance of preparation on
the surface implies that there is a correlation between this area (47% do not consider preparation
important) and the low pass rate (as high as 62%). The interviews however, provided insight into
what students consider “preparation” when compared to a Western definition.
Table 6.
Motivation Surveys
Strongly
Agree
Somewhat
Agree
Neither Agree
or Disagree
Somewhat
Disagree
Strongly
Disagree
Influenced by
someone else
to take exam
35 27.5 25 10 2.5
Actively
choose to use
English
outside
classroom
46.15 30.77 17.95 0 5.13
Ranking
Importance of
Preparation
21.05 31.58 36.84 7.89 2.63
Findings from Interviews
The interview questions tried to reveal more information related to the survey questions;
specifically, who has influenced the students, how often do they practice outside of the
classroom, and where does their preparation rank in importance in comparison to their other
goals. Interviews also revealed that socioeconomic factors play a role.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
54
Socioeconomic factors. There is a perception that international students come from
wealthy families. This perception tends to be reinforced when these students are seen driving
into the campus parking lot in Ferraris and Maserati’s, as observed by this investigator. These
glaring observations can blind the shadow students who have scraped together what they can to
arrive at the college and pay their fees. While interviewing Grace, she revealed that she was
“contemplating moving into her car because she was struggling to pay her rent and eat”. Another
student who was not interviewed was spotted eating out of the campus trash cans regularly. He
turned out to be an international student in a similar situation. In both cases these students were
able to receive assistance once their situations were identified. Rebecca, from Taiwan, was also
identified through the interview as an economically challenged student although she had opted to
“work multiple jobs to maintain a reasonable lifestyle by significantly sacrificing sleep”.
Student influence. Most of students (62.5%) stated that somebody had an influence on
their taking of the TOEFL exam. All ten of the students interviewed cited their “culture” as the
primary cause of influence, with family contributing first and friends second. Andrea (South
Korea) explained that at home, the expectation for taking the exam and succeeding is not spoken
but implied, “it is just there”. Since this is the case with all of the interviewed students, they say
that they influence each other outside of the home.
Practice outside the classroom. Interviewees generally stated that they would study at
an outside institution for “approximately 20 hours per week for a period of six months” prior to
taking the exam and that they preferred to work in groups. Socioeconomic status also influenced
practice outside of the classroom. Grace (China), Connie (Japan), and Rebecca (Taiwan) all
explained that they came from lower socioeconomic backgrounds and had to find other ways to
practice outside the classroom, normally without “the group”. This included “utilizing audio-
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
55
video resources at libraries” (Connie & Rebecca), “hanging out at clubs where Americans would
gather” (Connie), or even “dating an American” (Grace). The other students who admitted being
from a higher socioeconomic status stated that they would study outside of school at private
institutions with their peers.
Ranking importance of preparation. Andrea gleaned some interesting information on
the way of thinking when it comes to preparation, “. . . they really concentrate on how they can
get the score, not for improving whole English skills.” For the students, getting “the score” is
more important than learning to command the language. Another aspect of preparation was again
money, and what kind of assistance could and could not be afforded by students in each region.
Tulip, from Vietnam, stated that “private training for the TOEFL exam (in her country) can be as
expensive as $1000USD for a preparation program”, which leaves many students looking for
alternate resources or just giving up altogether. Rebecca, from Taiwan, did not have the financial
resources to enroll in a program but instead of giving up she stated that she “studied on her own
during her free time to score well enough to get into a community college in the US”. All
students interviewed were highly motivated and they all cited wanting to study in the U.S. as
their number one reason. Socioeconomic factors had not been considered going into this problem
of practice and has now become a new performance factor that has been added as a result of
searching for motivation gaps.
Teaching ESL workshops as a volunteer provided this investigator a good place to
observe student motivation. One of the primary observations in this area was with students
practicing listening and speaking skills. Students who attended workshops at least one day each
week had higher listening and speaking skills than students that showed up only once or twice
during a quarter. Students who said that they were only there, “because they had to be” tended to
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
56
have a lower level of listening and speaking skills than those students who showed up on a
regular basis because they wanted the opportunity to improve.
The documents provided by the LSC showed that on average close to one quarter of the
international student population use the center at some point during the quarter. The student
success center is provided as an additional resource at no cost to students and the LSC is
dedicated primarily to international students. Documents revealed that many of the students that
use the center are in the extended opportunities program services (EOPS), which is also provided
to high-needs students here on F-1 student visas and lawful permanent residents. These extended
services seem to be a great benefit to students who cannot afford private tutoring programs.
Synthesis of Results and Findings for Motivation Causes
The majority of students that participated felt that they were influenced by other persons
or sources with their decision to take the exams. Their primary motivation was the ability to
receive an education in the United States. Nearly three quarters of these students choose to
actively use English outside of the classroom and about one half put their preparation for the
TOEFL exam as a higher priority than other goals. Students from all regions that participated
indicated that listening and speaking courses were virtually absent from their countries public-
school systems and it was up to them, and their families, to find outside programs to assist them
with their TOEFL exam preparation. Not all international students are wealthy and several of the
student participants had to find alternative ways to prepare for taking the TOEFL exam. This
finding can be a determining factor on a student’s motivation. Further, students are so focused on
passing the exams that they are becoming disconnected from focusing on learning the language
proficiently.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
57
Results and Findings for Organization Causes
The instruments were created to identify student perception on adequacy of internal
resources, external resources, and alignment of their courses and instructor knowledge with that
of the TOEFL exam and their perceived ability to pass the exam on their first attempt (Table 7).
Survey Results
The first question asked if respondents felt that they were given adequate access to
courses and counselors that would allow them to achieve their goal. One third of respondents did
not feel that they had adequate access to prepare them for passing the TOEFL exam on their first
attempt.
Following the question of adequate access to resources, students where then asked if they
were using additional resources outside of the primary organization where they are learning
English. As indicated in Table 13, just over half of respondents answered affirmatively.
Students were asked two similar questions that can overlap between knowledge and
organization. When asked if they believed that the ESL curriculum closely aligns with the exams
and then whether their courses fully prepare them for the exam, the results were almost identical.
The curriculum offered is the total of all available classes that, by completion, should prepare the
student to pass the TOEFL exam on their first attempt. The second question is asking their
opinion of the individual courses that they have taken to date. The curriculum and course
satisfaction should match with respect to alignment and the results did confirm what would be
expected in terms of student perception.
In addition to questions about curriculum and courses, students were asked about how
they perceived their instructor’s knowledge of the TOEFL and IELTS exam and addressed areas
specific to the exam while teaching. Students were evenly split on instructor familiarity and
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
58
slightly less confident that the material being taught addresses areas of the exam specifically.
Like the previous set of questions, the purpose is to see if students believed their instructors are
familiar with the exams and compare this to whether they perceive that instructors are teaching
what is on the TOEFL exam. The instructor familiarity and teaching perceptions should also
closely align, which they did, confirming what would be expected again in terms of student
perception.
Table 7.
Organizational Surveys
Strongly
Agree
Agree Somewhat
Agree
Neither
Agree or
Disagree
Somewhat
Disagree
Disagree Strongly
Disagree
Adequate
access to
courses
and
counselors
28.95 26.32 15.79 18.42 10.53 0 0
Using
outside
resources
23.68 0 34.21 26.32 10.53 0 5.26
Curriculum
aligns with
exam
15.38 0 28.21 46.15 7.69 0 2.56
Courses
align with
exam
12.82 0 28.21 41.03 10.26 0 7.69
Instructors
familiar
with exam
12.82 0 38.46 38.46 7.69 0 2.56
Instructors
addressing
specific
areas of
exam
15.79 0 26.32 42.11 10.53 0 5.26
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
59
Findings from Interviews
Interviews for organization causes revealed strategies that students use at De Anza with
respect to their access to resources, curriculum and course preparation for the TOEFL exam, and
instructor knowledge. The interviews identified organizational strengths contributing to a
student’s success and weaknesses that may be obstacles towards a student’s ability to succeed.
Access to resources. The organizations greatest strength would be the student success
center, which is unique and sets De Anza apart from other community colleges. This center
provides a wealth of resources such as tutoring and workshops, in addition to materials that
students would otherwise have to purchase or seek outside of the campus; all of this at no
additional costs to students. All students who attended workshops at least once a week also stated
that they were “listening to or watching English language based programming to help them with
their listening skills”. This is a critical strategy for improving listening and speaking skills.
Curriculum and course preparation. The curriculum is lite on listening and speaking
courses and the student learning outcomes (SLOs) for all courses do not specify any alignment
with the IELTS or TOEFL exams. Students have therefore, created strategies for coping with this
hindrance. For example, Anna from South Korea was “taught how to memorize patterns of
speech to improve her ability when taking the listening and speaking sections of the exam” since
the listening and speaking classes are limited. JC from China explained that the same problem
exists at public institutions in her country too, which is why “students learn strategies for passing
the exams without having to be fluent in the language”.
Instructor knowledge. Through no fault of their own, most instructors at De Anza are
perceived by students as having limited-to-no knowledge of the IELTS or TOEFL exams. This
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
60
lack of knowledge, combined with no inclusion of the exams into course SLOs makes it difficult
at best to design lesson plans that address preparation for these exams.
Synthesis of Results and Findings for Organization Causes
Students were evenly divided on perception of organizational causes. Half of students felt
that they had adequate access to resources and used programs outside of their primary institution.
Perception that the ESL curriculum, courses preparing them for the exams, instructor knowledge
of the exam, and teaching material would benefit the students were also divided. Student
strategies however, seem to be aligned based upon interviews no matter what the social-
economic status of the students. Highly motivated students from lower socioeconomic
backgrounds were capable of finding resources that substituted for the high-cost, private
preparation schools used by their counterparts.
Chapter Summary & Validated Causes
The data collection period lasted ten weeks, starting October 2017 and ending in
December 2017. The total international student population at De Anza College during the data
collection period was approximately 2,400 of which, 419 were eligible to participate and of those
46 took the survey and ten of them were interviewed. Knowledge gaps were identified with the
understanding of the areas of the TOEFL exams and the lack of listening and speaking skills
being taught in the student’s home countries. Motivational gaps did not exist due to the student’s
socioeconomic status and ability to pay for private programs (or not) outside of their public-
school education. The students however, were identified as highly motivated. Nearly half of all
students who participated were dissatisfied with their ESL curriculum, course, and instructor
knowledge and teaching skills as it pertained to their goal of passing the TOELF exam
(Appendix H).
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
61
CHAPTER 5: RECOMMENDATIONS
This chapter presents evidence-based recommendations to address the validated
knowledge, motivation, and organization causes now identified from the findings reported in
Chapter 4 that prevent students from achieving their desired score during their first attempt on
the TOEFL exam. This study has found that “first attempt” does not necessarily mean the first
time taking the exam, but rather, the first attempt at a score for a new institution. Many students
take the TOEFL in high school to see if they can be placed in a U.S. community college. Once in
the U.S., students may be required to take the exam to achieve entry into an undergraduate
institution and most certainly are required to take the exam for entry into a graduate school. Each
institution determines the minimum score for entrance: A total score of 60-70 (TOEFL) for
community colleges, 80-90 (TOEFL) for California State Universities, 90-100 (TOEFL) for
University of California colleges, and 100-120 (TOEFL) for private institutions.
This chapter is divided into five sections. Section-one presents the validated causes and
the selection and rationale for the key causes analyzed. The second section provides solutions
based upon the research theories that have been shown to effectively close identified gaps in
application. The next section describes and outlines the implementation plan that could be used
to bring into existence the solution to solve the identified gaps. The fourth section provides an
evaluation plan which can be used for two purposes: first to guide the implementation of the
proposed solution and second, to assess the progress after implementation to ensure that expected
outcomes are being achieved. The last and final section presents recommendations for future
research into the problem of practice.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
62
Validated Needs: Selection and Rationale
In this section, the validated KMO assumptions and newly identified causes are analyzed
based upon the surveys, interviews, observations, and document analysis presented in Chapter 4.
Ideally, each validated cause would have its own unique solution however, the causes associated
with this problem overlap to the point where individual solutions for each cause would be
illogical. Further, the problem is global and the solution needs to be practical enough to be
proven locally, and in a way that it can ultimately be repeated on a global scale.
Validated Knowledge Gaps and New Causes
As presented in Chapter 3, knowledge gaps were broken into procedural, factual,
conceptual, and metacognitive areas to examine. Collection of the data with respects to these
areas has been presented in Chapter 4 and new causes were identified by contrasting the survey
data with the interview data, which appears across all areas of the knowledge gap. Now that the
data has been examined and the validated gaps presented along with the rationale that has been
applied to determine their existence, this chapter proposes solutions to close these gaps.
Procedural knowledge. A majority of students who took the survey indicated that they
believed that they knew the processes involved with taking the TOEFL exam. Students
interviewed also confirmed that they had a procedural understanding of the TOEFL exam early
in their academic years. Procedural knowledge in this area seems to be a social norm, rather than
an institutional gap. This area is therefore, not seen as a validated cause.
Factual knowledge. Two areas of factual knowledge were observed: Basic knowledge of
the exams, and level of competence in the four areas of English. The surveys revealed that
students may have a relatively good basic knowledge of the exams themselves. Interviews
revealed that students may not be competent in all areas of English due to their public-school
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
63
systems in their home countries. Further, students rely on private institutions to supplement what
they do not receive in their public schooling. This revelation uncovered a new cause, which is
socioeconomic imbalance. Every student interviewed stated that listening and speaking skills
were barely taught, regardless of where the students were located. They cited their teachers own
lack of ability in these areas as the primary reason and thus, had to rely on outside, private
institutions, which can be very costly.
Table 8.
Validated Assumed Factual Knowledge and New Cause
Assumed Cause Recommendations New Cause
Students do not have a good
understanding of the four
areas of the exam.
Offer an elective course at De
Anza that teaches students the
four areas of the exam.
No
Socioeconomic imbalance
prevents students from
supplementing their listening
and speaking skills at outside,
private institutions.
Offer a “curved” curriculum at
De Anza that recognizes that a
student’s listening and
speaking skills may not be as
strong as their reading and
writing. Currently, students
are tested into a fixed level
with all four skill sets.
Yes
Conceptual knowledge. Students have a very acute conceptual knowledge of the exam
and what they should know when taking the exam. But as stated above in factual knowledge,
they are not necessarily being taught to the level that they may need to achieve their desired, or
needed score.
Metacognitive knowledge. An interesting phenomenon that was revealed in the
organization section was that students viewed learning English more as a strategy, rather than a
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
64
language to master. By triangulating all of the sources, it became apparent that students were
lacking in metacognitive knowledge of English and validated both assumed causes.
Table 9.
Validated Metacognitive Knowledge and New Cause
Assumed Cause Recommendations New Cause
Students have the basic
understanding of English.
Students can understand the
individual words but they may
not always understand a
sentence read or heard.
Consider review of the current
placement tests and verify if
they are doing an adequate job
of testing for this specific
cause.
No
Students may rush their
English learning and skip
levels to try and pass the
exams more quickly. They
may not know how to pass the
exams as a result.
Make available pamphlets in
the international student
program center that are
offered to students to increase
their awareness.
No
Students with limited financial
resources were unable to
supplement their training to
the degree necessary to
achieve their required scores.
Consider waiving or
supplementing international
student fees for the elective
course, if made available.
Yes
Validated Motivation Gaps and New Causes
Most students seem to investigate and do their due diligence to understand the exams and
adequately prepare. They feel overwhelmed because they are not sure if they have been
adequately trained for the moment, unless their socioeconomic situation allowed for that training.
Active choice. An interesting paradox developed with the participating students. They
unanimously agreed that within their cultural settings that they prefer to work in groups with
their friends. But when asked to work in groups with students’ other than their friends, they
resisted, which validated this cause. Not only did the students state that instructors did not hold
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
65
them accountable, they suggested that they went out of their way to avoid discussing group
participation to avoid potential conflict.
Table 10.
Validated Active Choice Causes
Assumed Cause
Recommendations New Cause
Some students do not like to
do group work even though it
is necessary for learning
Make pamphlets available to
students through the
international student program
to create awareness
No
Some instructors do not hold
students accountable if they do
not contribute to group
assignments.
Include this message as a part
of teacher training and during
teacher retreats.
No
Persistence. This study revealed that persistence could not be fully determined due to the
limited nature of the available population. The students who were able to participate were all
international students. Most came from wealthier families that can afford to pay not only the fees
for an education in the United States, but also the private institutions in their home country that
prepare them well for the exams. While the assumed causes for persistence could not be
validated, the new cause of socioeconomic imbalance again raised itself to the surface. Students
with a higher socioeconomic background tended to show higher self-efficacy than their
counterparts.
Table 11.
Invalidated Assumed Cause and New Cause
Assumed Cause Recommendations New Cause
Students tend to give up or
create a false belief that they
cannot learn if they do not
pass on their first attempt.
Invalidated No
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
66
Students from better
socioeconomic backgrounds
have higher self-efficacy.
Train counselors to be
sensitive to students from poor
socioeconomic backgrounds
so that they can be better
counseled and guided to the
student success center.
Yes
Mental effort. Triangulation of all studies indicate that students do not use English
outside of the classroom or campus environment. Students that had a better command of the
language claim to listen to native speakers by audio or video methods at least 30 minutes each
day. Those same students indicated that they try to “date” or “hang out” with native speakers,
rather than spend time with colleagues from their own culture.
Table 12.
Validated Assumed Cause for Mental Effort
Assumed Cause
Recommendations New Cause
Students do not use English
outside of the classroom,
limiting their ability to
practice
Create pamphlets for students
at the international student
program center, enhance
teacher and student awareness
overall
No
Validated Organization Gaps and New Causes
Organizational causes seem to be the most pervasive in this study. Again, triangulating
the data showed that while students surveyed indicated that they had a strategy for passing the
exams, the interviews revealed that those strategies were developed while the students attended
outside, private institutions in their home countries. None of the students interviewed said that
they had the ability to develop a strategy while in a public institution at home or at De Anza. The
interviews revealed that the student’s strategies were different than what we would interpret in
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
67
Western culture. Our interpretation is how best to master the language in order to achieve the
score, while their strategy is how best to get the score without mastering the language.
Goal setting. A distinction occurred that was not considered at the beginning of the study,
namely, that students attend both public and private institutions in their home country to achieve
their goals with respect to the exams. The assumed cause that De Anza has not given students a
clear path for achieving the minimal exam scores has been validated. A new cause was identified
during the study whereby students require additional private institutional help either in their
home country or in the U.S. to be able to successfully achieve the desired score on the TOEFL
on their first attempt.
Table 13.
Validated Assumed Cause for Goal Setting and New Cause
Assumed Cause Recommendations New Cause
The organization has not given
students a clear path for
achieving the minimal exam
scores.
Offer an elective course that
prepares students for the
TOEFL exam
No
Students require costly,
private institutions in their
home country or the U.S. to
create clear goals for exam
achievement.
Offer an elective course that
prepares students for the
TOEFL exam
Yes
This is not only a problem in the students’ homeland, it is also a problem at U.S.
institutions where the ESL curriculum does not fully take these exams into account either.
Cultural model. There is a pattern occurring here, which we also see in cultural settings
below as well. The EFL/ESL curriculum as described by students does not seem to support
preparation for the exams, nor do the counseling and support services. It seems to be completely
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
68
missing. The data clearly validates this point and introduces the new cause that students must
rely on a costly, private service in their home country or within the U.S.
Table 14.
Validated Cultural Model Cause and New Cause
Assumed Cause Recommendations New Cause
The exams are not a part of
the curriculum, counseling, or
support services.
Offer an elective course that
prepares students for the
TOEFL exam
No
Students must seek costly,
private services to obtain
support for the exams.
Offer an elective course that
prepares students for the
TOEFL exam
Yes
Cultural settings. This too was validated based upon teacher orientation towards the
exam and their classroom plans. One-hundred percent of students interviewed conveyed that
their public-school education was focused primarily on reading and writing skills. Teacher
ability to speak and pronounce correctly was minimal and relied primarily on audio resources
such as old cassettes or CDs.
Table 15.
Validated Cultural Settings and New Cause
Assumed Cause Recommendations New Cause
Teachers are not required to
learn about or make exam
preparation part of their lesson
plans.
Offer an elective course that
prepares students for the
TOEFL exam
No
Students must find and use
costly, private schools in their
home country or the U.S. to
supplement their exam
preparation.
Offer an elective course that
prepares students for the
TOEFL exam
Yes
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
69
Professional development. This assumed cause could not be validated by students. As
this investigator is an ESL instructor, it is known from first-hand knowledge that teachers are not
provided with any training, that this investigator is aware of, for preparing students in any way
for the exams.
Resources. Considering the wealth of importance given to education by political leaders
and the public in the United States, it is a complete paradox that there are never enough resources
to support student needs. After triangulating the data, this study seems to again conclude that
public school systems in the students’ home country fail to have the resources necessary for
students to be adequately prepared to take the exams. Costly, private institutions in the U.S. and
abroad are therefore becoming the norm for students who want to move forward but the quality
of education is dependent on a student’s socioeconomic status.
Table 16.
Validated Organizational Resources and New Cause
Assumed Cause Recommendations New Cause
There are not enough
counselors, teachers, and
tutors trained to assist students
with preparation for the exams
at De Anza college.
Offer an elective course that
prepares students for the
TOEFL exam
No
Students must seek costly,
private institutions to assist
them with preparation in the
U.S. and abroad. The quality
of resources is based upon a
student’s socioeconomic
standing.
Offer an elective course that
prepares students for the
TOEFL exam
Yes
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
70
Solutions
The solutions became more apparent over the course of reviewing the data for each KMO
section. Students are not receiving the full curriculum necessary for them to pass the TOEFL or
IELTS exam on their first attempt, as revealed by previously cited studies from Esmail &
Roberts, et. al., and the findings in Chapter 4. Students who are socioeconomically privileged
have access to the costlier, private institutions in the U.S. and abroad that are necessary to
supplement what the public-school system will not offer. Even students who manage to get to
U.S. colleges, while wealthier than their counterparts back home, are still struggling to keep up
with the expenses of education in this country, and then the ability to pass the exam to an
undergraduate or graduate school program. American community colleges do not have a full
curriculum that prepares these students for the exams. The following solutions are presented:
Elective TOEFL Preparation Courses
Elective courses are not required and only have a pass-fail option. This can be a single
course or multiple courses offered, depending on the institution, their budget and perceived
needs. This problem of practice is recommending a single course as a pilot to see how many
students register for the course, complete it successfully, and then go on to take and successfully
pass the TOEFL exam with the scores they needed to attend the institution of their choosing.
Implementation Plan
The solution will be proposed, developed, and integrated into the ESL program over a
three-year period. The objective of this solution is to provide an elective course to any student
that would like to receive supplemental training when preparing for the TEOFL exam as a part of
the ESL curriculum. This solution will address the findings in the research that suggest first
attempt performance on the TOEFL exam can be improved by offering elective courses that
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
71
better prepare students and can serve as a model to other community colleges and public schools
systems as an equitable solution to address the needs of all ELLs that want additional preparation
for the TOEFL exam.
The organization where this solution will be implemented is De Anza College, where the
research for the study was performed. Recognized as a national leader among U.S. community
colleges, the successful implementation of this solution will benefit De Anza as well as its future
international student ELL population.
Proposed Solution and Environment for Implementation
The proposed solution and implementation environment is related to an EFL/ESL area of
policy and its relation to the preparation of students to take the TOEFL or IELTS exam. Data
collection and analysis indicate that students must find and pay for additional training for these
exams outside of the public-school sector in the U.S. or their home country. These limits the
number of students to those whose families can afford to pay the high fees associated with these
outside programs. Some alternative solutions considered based on the findings from this research
included creating a set of standards that could be used globally or modifying existing public
programs to include preparation for the exams. These alternatives are too complex in terms of
worldwide public-school implementation and the time to implement. An alternate solution has
therefore, been created; elective courses that are specifically designed to help students prepare
for the TOEFL or IELTS exam in public-school systems. This can potentially be the most
expeditious alternative to implement with significantly higher chances of buyoff from the
primary stakeholders’ due to reduced financial risks and impact on other parts of the program
curriculum.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
72
While all organizations have their own unique cultures, there will still be some
commonalities within a EFL/ESL environment. There are the full-time instructors with seniority
that may want to change slowly, and the need for “norming” sessions with new and part-time
instructors to have a similar style of teaching throughout the organization. These instructors are
one of the primary stakeholder groups that must “sign on” to a policy solution (Kirkpatrick &
Kirkpatrick, 2011). By introducing a separate elective course for exam preparation, it will have
minimal impact on other courses offered within the existing curriculum, which in turn should
make it easier to gain support from this stakeholder group.
Administration is the other key stakeholder. Public HE institutions have layers of
bureaucracy and procedures for making any changes to policies and procedures. Rather than
trying to inject TOEFL exam awareness into existing curriculum, the introduction of an
independent, elective course may be the most expeditious and risk averse way of implementing
this alternative policy. Public institutions in America are frequently struggling to receive the
maximum funding available to provide the best services. As an added course, it is possible that
funding may not be available or it would be provided at a minimal level. Nonetheless, the
institution where this policy alternative would be proposed tends to be more open to advancing
changes that have a positive impact on student performance. It is fair to compare their model to
that of the Finnish system in many ways. Examples of this range from the competitiveness to
gain employment as an instructor, emphasis placed on student equity and welfare, such as equal
opportunities, and adequate food and shelter. The institution ranks highest in student retention,
degrees awarded, and transfer rates using a model like the Finnish system as described by
Sahlberg (2015).
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
73
This policy alternative also addresses ethical and risk concerns insomuch that as an
elective, only students who are interested would take the course, thereby not creating a burden
and wasting the time of students whom are not interested. With respect to risk, as an elective
course, if the number of students who enroll do not satisfy the requirements necessary for
continued funding, then the course can be cancelled with no impact on the EFL/ESL program
curriculum that is already in place. The next step is to consider the best way forward to try and
implement this policy successfully.
Implementation Action Steps
The proposed solution for De Anza College as outlined above requires implementation
action steps to move it forward. The TOEFL preparation elective course will be a non-
credit/non-transferable course. Not only does this provide advantages for the student, since they
do not need it to affect their GPA, nor do they need it to transfer, but it also avoids additional
steps and time towards implementation. To successfully implement this course at De Anza
College will therefore, require eight steps as explained below:
Course outline. A general course outline that begins with a course description will be
required, followed by course objective, student learning outcomes, and suggested texts and
materials. The person or persons championing the course should be the author(s) of the outline.
The outline will be developed in collaboration with other faculty members. The course outline
should be distributed to all Language Arts instructors with the goal of receiving more than 50%
of instructor feedback. The impact should be the creation of new awareness with instructors
about the importance of the filling the identified gap.
Departmental meeting. De Anza’s Language Arts Division has English, ESL, Reading,
and Speech departments. A department meeting will be called and invitations will be sent to all
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
74
faculty in all the division departments to attend. The reason for including all divisions in this
case is because many ESL students take courses within these other departments once they
complete the highest level of study within the ESL department. Because most of the ESL
students are international students, the members of the International Student Programs (ISP) will
also be invited to attend and extend their input. The purpose is to validate efforts and ensure that
the solution is moving forward in the correct direction. Many of the instructors will be
participating and becoming awakened to the importance of their awareness with this gap as it
relates to their course instruction.
Departmental vote. Upon completion of the presentation in the departmental meeting,
the department members will be asked to vote. A majority vote carries the new policy (course)
forward to the curriculum committee. A “yes” vote will be affirmation that efforts to educate
administrators and faculty were successful. A favorable vote can potential and positively alter the
lives of many student who will be taking the TOEFL exam.
Curriculum committee. A curriculum committee consisting of senior faculty and staff
will be organized and will advance the course outline and recommend possible texts and
materials. A more fleshed-out curriculum will consist of course objectives, student learning
outcomes, required texts and material and prerequisites. The committee will have undertaken the
task of understanding the essentials of student preparation for taking the TOEFL exam to build a
meaningful curriculum. The impact of this curriculum can be very specifically gauged on the
success of the student’s first attempt to pass the TOEFL exam after successful completion of the
course.
Departmental input and revision. Once the committee has completed their additions to
the course outline, it will be recirculated amongst the department again for final input and then
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
75
revision. During this step, it is imperative to ensure that all possible areas have been covered to
ensure that the identified gaps can be closed and therefore, input from all instructors has been
received and evaluated. The primary impact of this step is that by now the faculty and staff have
a deeper understanding about the TOEFL exam and its connection with their ESL courses.
Division meeting. A meeting will then be scheduled with the initiator(s), department
chair, and the dean of the Language Arts Division. The meeting will be a formal presentation of
the final course curriculum that will be presented to the department chair and the dean for their
formal approval. At this stage, it will be determined if the costs to add this elective course are in-
line with budget objectives, has a realistic assessment of the costs been realized to justify the
course, and have all direct and indirect costs been identified and agreed upon.
Curriculum defense. With the approval of the department chair and division dean, the
initiator(s) will return to the committee for a final defense and vote, which is more of a formality
since it will have already exhausted all required procedures. What is being done within this
defense is an analysis of understanding of the performance gap and has it been sufficiently
addressed to enable the gap to be successfully closed.
Introduction of course. Typically, step one will occur at the beginning of the Fall
quarter in late September since it is the beginning of the fiscal year. The final “yes” vote will
occur that following April and the course would be offered in the Fall of the following year. So,
from step one to offering will take approximately two years, and this is the shortest period since
it is a non-credit/non-transferable course. Once the course is introduced then the expectation will
be that the minimum number of students have registered and that all 100% pass the course and
receive their needed scores on their first attempt taking the TOEFL exam thereafter.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
76
Building Capacity
The solution to be implemented has been chosen because it aligns well within the current
organizational structures in place. There are currently six levels of required ESL courses offered
so that a student can complete their way through the equivalent of English 1B, which will satisfy
most HE institutions, such as California State Universities. These levels are labeled as high
beginning, low intermediate, intensive intermediate, high intermediate, low advanced, and
advanced. Two other elective courses are being offered within this structure. The table below
shows the mandatory classes on the left and the existing and proposed elective courses on the
right.
Table 17.
ESL Course Sequence (Level, Required Courses, Elective Courses)
STUDENT LEVEL REQUIRED COURSE ELECTIVE COURSE
High Beginning ESL-200
(10 units)
Listening/Speaking/Reading/Writing
Low Intermediate ESL-234
(10 units)
Listening/Speaking/Reading/Writing
Intensive Intermediate ESL-244
(10 Units)
Listening/Speaking/Reading/Writing
High Intermediate ESL-251 (3 units)
Listening/Speaking
252 (3 units) Reading
253 (4 units) Writing
Low Advanced ESL-261 (3 units)
Listening/Speaking
262 (3 units) Reading
263 (4 units) Writing
ESL-260 (4 units)
Pronunciation
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
77
High Advanced ESL-272 (5 units) Reading
273 (5 units) Writing
005 (5 units) Advanced
Reading/Composition
006 (5 units) Critical
Reading/Research for Writing
ESL-274 (4 units)
Grammar
ESL-27x (Proposed)
TOEFL Preparation
Timeline
The following is a list of action steps with the proposed dates of execution for each step
in the process. The process of adopting a new elective course typically takes an entire school
year (from Fall quarter through Spring quarter) due to the large number of faculty and staff that
need to be involved in the process and conflicts with time schedules. One full year will be spent
pre-selling the idea throughout the campus (Bolman & Deal, 2013) and then actively trying to
implement the plan will begin.
Table 18.
Implementation Plan Timeline
Process Proposed Start Date Proposed End Date
Course Outline August 15, 2018 September 15, 2018
Departmental Meeting October 15, 2018 November 15, 2018
Departmental Vote December 15, 2018 December 15, 2018
Curriculum Committee January 15, 2019 February 28, 2019
Department Input and
Revision
March 1, 2019 March 30, 2019
Division Meeting April 1, 2019 April 15, 2019
Curriculum Defense April 30, 2019 April 30, 2019
Introduction of Course September 2020 ~
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
78
Evaluation Plan
Evaluation is a crucial way to measure the impact our intended policy change is making
and what changes, if any, should be made to further our progress of closing the identified gaps
throughout the gap analysis cycle (Clark & Estes, 2008). By evaluating students’ progress
throughout the course, corrections and improvements to the course curriculum. Monitoring the
impact of post-course results and performance results on the TOEFL exam will allow for further
improvements and adjustments to curriculum over a greater period.
As part of the course’s student learning outcome, exams will be given that are very
closely based upon the TOEFL exam that the student would be taking. In fact, the final exam
should so closely resemble the TOEFL exam that if a student were to pass one, they would pass
the other. Students will be requested to participate in an optional tracking program so that the
institution where they studied would know their success rate each time that they take the exam,
since they can take it multiple times throughout their academic career.
The assessment of the impact of the introduction of the evaluation course will be done
through the four levels and 10-step processes model by Kirkpatrick & Kirkpatrick (2006). The
framework for the model identifies changes through four carefully followed levels: Reaction,
measuring how those involve react; Learning, the extent to which those involved change attitude,
increase knowledge, or improve skills; Behavior, the extent to which a change of behavior can be
observed; Results, the final results that occurred because of the introduction and execution of the
policy.
Expected Observable Results
There are a minimal set of results that are expected at the end of each level. This section
states what these expected results are followed by how each result will be measured.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
79
Level 1 reactions. Observable results should yield a majority sign-on by administrators,
faculty, and students for an elective course that they believe satisfies all objectives in terms of
quality, SLO, and KMO.
Level 2 learning. Teachers with a full understanding of the TOEFL exam and the testing
procedures will be able to teach a 12-week curriculum that prepares the students to successfully
pass the exam on their first attempt upon passing the elective course.
Level 3 behavior. Students should have higher, observable self-confidence by the end of
each course. The course should also be accepted with wide regard by faculty and staff after
several quarters whereby students have moved forward after the class and successfully passed
the TOEFL exam.
Level 4 results. After one full year, four consecutive quarters, the final results should be
that all students taking the elective course will meet or exceed their minimum needed test results
on the TOEFL exam.
Level 1 Reactions
There are four primary reaction levels that will be measured: Support for introduction of
the course; Successful student enrolment numbers; Students passing the course successfully;
Students passing the TOEFL exam on their first attempt after passing the course.
Reaction levels will be monitored throughout the entire policy implementation period
from inception through execution and post-course training and final TOEFL evaluation scoring.
Measuring will be done by anonymous Likert scale surveys, open group forums, and voting by
administrators and faculty. Students will be asked to complete a survey at the beginning and end
of the course that will measure their attitude, commitment, and confidence (Dye, 2011).
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
80
Table 19.
Level 1 Reactions
Implementation Evaluated Reaction
Support for and introduction of the new course. Written, oral, and voted feedback. Approval
and funding by administration. Course
curriculum written and approved by faculty.
Successful student enrollment numbers. The minimum number of students have
registered for the course for the course to be
offered. Below the minimum means the course
will be cut from that quarter’s schedule of
classes.
Students pass the course successfully. The expected passing rate for this course
should be very high; nearly 100%
Students pass the TOEFL exam on their first
attempt.
The expected passing rate for the TOEFL exam
should be 100%
Level 2 Learning
Learning will be measured through periodic exams throughout the course. The exams
will measure if students are understanding the testing processes and procedures and the four
areas of the TOEFL exam itself. The course will end with a mock TOEFL exam that students
will be able to review and correct prior to the course ending. The examinations will therefore
require that the instructors be able to give oral, as well as written exams.
Level 3 Behavior
In this level, behavior will be monitored in several ways, starting with motivational
efforts (Clark & Estes, 2008). Motivational factors can include monitoring class attendance,
diligence with homework assignments, group work, and exam results. Student attitude and self-
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
81
efficacy (Dye, 2011) can be monitored directly by the course instructor indirectly during class
time and directly through verbal office hours’ interviews.
Level 4 Results
There are many factors to be analyzed when looking for results and their impact.
Throughout the process of bringing the course into the program, was there a significant enough
impact on the administration and faculty involved to create enough motivation for instructors to
actively consider and add material to their lesson plans that consider student needs with respect
to the TOEFL exam? Have students taking the course demonstrated changed behavior that shows
increased knowledge, skills, and attitudes by the end? Did 100% of the students who took the
course pass the TOEFL exam on their first attempt with the scores that they needed? Have these
results demonstrated a level of success that other HE institutions recognize and adapt the same or
similar elective course or course? These results and the impact that they give spans many
stakeholders over a long period of time.
Future Research
The exams are real and they are gatekeepers, which in turn affects societies that need
specially trained people to serve our societal needs without delay. Questions should be asked as
to whether the exams are truly in alignment with global education from administrative, faculty,
and student perspectives.
Further research into the level of quality being assigned to listening and speaking skills
should also be explored. The identified solution in this problem of practice may not be the most
optimal. But it may serve as a starting point that does help students and can provide impetus for
other HE institutions to explore solutions too. It behooves western societies that use the IELTS
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
82
and TOEFL exams as gatekeepers to continue to do research into their impact and integration
into global educational systems.
Further research into the link between IELTS & TOEFL scores and student GPA and
graduation rates should also be continued more vigorously. Weighing the importance that these
exams play in moving students further through higher education in English speaking countries
and the desire to keep their talents after graduation behooves us to create the most equitable
system possible.
CONCLUSION
Testing Of English as a Foreign Language (TOEFL) is a GATEKEEPER to entrance into
an English language institution. Depending on institution requirement, a score that is too low can
delay entrance for otherwise qualified students. Studies have shown that over 60% of students do
not achieve the scores that they require on their first attempt (Esmail, 2013). Repeated studies
have demonstrated that a student’s GPA and motivation are not reflective of their TOEFL scores
(Ayers & Quattlebaum, 1992; Floyd, 2014). The study evaluated the reasons why a large
percentage of students do not achieve their desired scores when taking the TOEFL exam for
entry into a U.S. institution of higher education on their first attempt. The purpose of the study
was to evaluate the degree to which international students attending a Community College ESL
program were prepared to take the TOEFL exam and succeed on their first attempt. The
definition of successful completion, for the study, were satisfactory scores in all areas of the
exam as required by an institution chosen by the candidate for purposes of entry. The study is
significant for several reasons: Limited resources such as classrooms, seating, teachers, and more;
students who must return for additional instruction displace incoming students as a result. The
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
83
same students returning for additional instruction must also take on additional educational costs.
Eventually, students may stop seeking higher education in English speaking countries as a result.
The conceptual framework utilized for the study was a gap analysis process by Clark and
Estes (2008) that uses a continuous cycle that identifies gaps in knowledge, motivation, and
theory thereby reducing error in evaluation of the problem of practice. This framework drove the
research questions to what extent the college is achieving its goal for all international students to
pass the TOEFL exam on their first attempt; international student KMO influences needed to
pass the exam on their first attempt; and what recommendations can be given for organizational
practice in the areas of KMO resources. The methodology was a mix of qualitative and
quantitative evaluations using descriptive statistics. Assumed KMO influences were assessed by
using surveys, interviews, literature review, document analysis, and content analysis.
Findings validated some assumed KMO causes, invalidated others, and revealed causes
not considered before or during the study, which yielded valuable consideration for
recommendations. Listening and speaking skills seem to receive minor training in comparison to
reading and writing in both EFL and ESL programs based upon qualitative inference. The same
qualitative inference also seems to correlate with low TOEFL scores and low socioeconomic
status because those students do not have access to the private institutions that can make up for
the absence of listening and speaking studies in the public schools. Implications for practice and
policy range from a reconsideration by graduate school programs of their use of the TOEFL
exam and their scoring levels – weigh scores against a student’s GPA and their socioeconomic
background. Addressing the issue of listening and speaking courses in public schools must
happen but is recognized as a large and in many ways, a daunting task but should be addressed
nonetheless.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
84
Considering the scope and implications of the practice and policy changes necessary, the
creation of an elective course, or courses, that specifically prepare students for the TOEFL exam
may be the most expeditious and favorable solution. If successful, the solution may create greater
awareness, initiate policy changes, and most certainly assists the students who need it now.
Further research into the full scope and impact of both the IELTS and TOEFL exams on students
seeking higher education in English language countries should be performed; how closely linked,
or not, are the exams with relation to a student’s GPA and motivation. And how does a student’s
socioeconomic status and country of origin relate to exam scores with relation to their final
grades and program completion. It is this author’s hope and desire to see this problem of practice
furthered to the ultimately empirical benefit of all students worldwide who want to achieve a
higher level of education in English language academic institutions and contribute to societies
everywhere.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
85
References
Anderson, L. W., Krathwohl, D. R., & Bloom, B. S. (2001). A taxonomy for learning, teaching,
and assessing: A revision of Bloom's taxonomy of educational objectives. Allyn & Bacon.
Anderson, N. (2003). Reading. In Nunan, D. (Ed.), Practical English Language Teaching (pp.
67-86). New York, NY: McGraw-Hill.
Ayers, J. B., & Quattlebaum, R. F. (1992). TOEFL performance and success in a masters
program in engineering. Educational and Psychological Measurement, 52(4), 973-975.
Bailey, K.M. (2003). Speaking. In Nunan, D. (Ed.), Practical English Language Teaching (pp.
47-66). New York, NY: McGraw-Hill.
Bolman, L. & Deal, T. (2013). Reframing organizations: Artistry and leadership. San Francisco,
CA: Jossey Bass.
Brown, J.D.H. (1998) Does IELTS Preparation Work? An Application of the Context-Adaptive
Model of Language Program Evaluation. The University of Melbourne. p.p. 20-37
Clark, R.E., Estes, F. (2008). Turning research into results: A guide to selecting the right
performance solutions. Charlotte, NC: Information Age Publishing.
De Anza College (2016). Spring 2016 schedule of classes.
http://deanza.edu/schedule/classes/schsearch.html Accessed: March 6, 2016 @ 18:30
Dye, T. (2011). Understanding public policy. (13
th
edition). Upper Saddle River, NJ: Pearson
Education. Chapter 15 (Policy evaluation), pp. 323-340.
Educational Testing Service (2016). About the test.
https://www.ets.org/toefl/ibt/about?WT.ac=toeflhome_ibtabout2_121127 Accessed:
March 6, 2016 @ 16:07
Esmail, A., & Roberts, C. (2013). Academic performance of ethnic minority candidates and
discrimination in the MRCGP examinations between 2010 and 2012: Analysis of data.
BMJ, 347, f5662.
Floyd, C. B. (2015). Closing the gap: International student pathways, academic performance and
academic acculturation. Journal of Academic Language and Learning, 9(2), A1-A18.
Gan, Z., Humphreys, G., & Hamp ‐Lyons, L. (2004). Understanding successful and unsuccessful
EFL students in Chinese universities. The Modern Language Journal, 88(2), 229-244.
Greiner, K. (1996). How much student loan debt is too much? Journal of Student Financial Aid,
26(1), 7-16.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
86
Hadaway, N.L., Vardell, S.M., & Young, T.A. (2009) What every teacher should know about
English language learners. Boston, MA: Pearson Education, Inc.
Haynie, D. (2014). Number of international college students continues to climb. U.S. News &
World Report. http://www.usnews.com/education/best-
colleges/articles/2014/11/17/number-of-international-college-students-continues-to-climb
Accessed: December 27, 2015 @ 13:30
Helgesen, M. (2003). Listening. In Nunan, D. (Ed.), Practical English Language Teaching (pp.
23-46). New York, NY: McGraw-Hill.
Huberman, A.M., Miles, M.B., Saldana, J. (2014). Qualitative data analysis: A methods
sourcebook. (3
rd
Ed.). Los Angeles, United States: SAGE Publications, Inc.
Institute of International Education. (2015). Open doors data. http://www.iie.org/Research-and-
Publications/Open-Doors/Data/International-Students Accessed: February 27, 2016 @
17:40.
Kirkpatrick, D. & Kirkpatrick, J. (2006). Evaluating Training Programs: The Four Levels (3
rd
Edition). San Francisco, CA: Brett Koehler Publishers, Inc.
Maxwell, J.A. (2013). Qualitative research design: An interactive approach, (3
rd
Ed.) Los
Angeles, United States: SAGE Publications, Inc.
McKay, S.L. (2002). Teaching English as an international language. Oxford, UK: Oxford
University Press.
Nunan, D. (2003). Practical English language Teaching. New York: McGraw Hill.
Purwanto, S. (2012) IELTS as a Literacy-Based Proficiency Test. Excellence in Higher
Education 3. p.p. 46-51.
Rueda, R. (2011). The 3 dimensions of improving student performance: Finding the right
solutions to the right problems. New York, NY: Teachers College, Columbia University.
Sahlberg, Pasi (2015). Finnish Lesson 2.0: What can the world learn from educational change in
Finland? New York, NY: Teachers College, Columbia University.
Schein, E. H. (2004). Organizational culture and leadership. San Francisco: Jossey-Bass.
Schlesinger, J. (November 28, 2012). Student loan debt nears $1 trillion: Is it the new subprime?
CBS Money Watch.
Sengupta, R., & Jepsen, C. (2006). California's community college students. San Francisco, CA:
Public Policy Institute of California.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
87
Sokolik, M. (2003). Writing. In Nunan, D. (Ed.), Practical English Language Teaching (pp. 87-
108). New York, NY: McGraw-Hill.
Soria, K. M., Weiner, B., & Lu, E. C. (2014). Financial decisions among undergraduate students
from low-income and working-class social class backgrounds. Journal of Student
Financial Aid, 44(1), 2.
Templer, B. (2004). High-Stakes Testing at High Fees: Notes and Queries on the International
English Proficiency Assessment Market. Journal for Critical Education Policy Studies,
2(1).
United States Central Intelligence Agency. (2015). The world factbook,
https://www.cia.gov/library/publications/resources/the-world-factbook/ Accessed:
January 29, 2016 @ 14:28.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
88
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
89
Appendix A
Conceptual Framework for Addressing the Inquiry Questions: Assumed Knowledge
Four areas of knowledge areas are addressed to validate our assumptions: (F)actual,
(P)rocedural, (C)onceptual, and (M)etacognitive.
Conceptual Framework for
Addressing the Assumed Knowledge Question
Research Question Survey Interviews Document Review
I am aware of the of
the separate elements
of the exams (F)
X X
I believe that I have
been informed of all
the courses I must
complete in order to
successfully pass the
exam. (F)
X X
Please describe the
separate elements and
explain how you
learned about each
one. (F)
X X
Do you believe that
you can pass the
exam(s) if you have
not completed all of
the recommended
courses? (F)
X X
I am aware of the
examination
procedures with
respect to each
X X
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
90
section of the exam.
(C)
I have a strategy for
practicing the
interrelationships
between the listening
and writing sections?
(C)
X X
Please describe the
examination
procedures for each
section and how you
learned about each
one. (C)
X X
There is a reason for
the interrelationship
between the listening
and writing sections.
Please explain your
understanding of why
this is done. (C)
X X
I aware of the
minimum word
criteria for taking the
exam. (P)
X X
I have been guided in
the best methods for
learning the
minimum criteria by
way of courses
recommended. (P)
X X
The minimum
recommended word
count is 8000 words.
X X
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
91
Please explain what
you know about this?
(P)
Are the courses you
are taking (or have
taken) balanced with
respect to each
section in the exams
in your opinion. Why
or why not? (P)
X X
Do you plan to take
the exam(s) before or
after you complete
your courses? (M)
X X
Considering the
courses you have
already completed,
what is your level of
confidence that you
would pass the exams
if you took them
today? (M)
X X
Going back to the
question on whether
you will take the
exam(s) before or
after completing your
courses, explain why
you will make that
decision. (M)
X X
Going back to the
question about having
to take the exam(s)
today, what affects
your decision of
success or not based
X X
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
92
upon your current
knowledge? (M)
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
93
Appendix B
Conceptual Framework for Addressing the Inquiry Questions: Assumed Motivation
Three areas of motivation will be explored to validate assumptions: Active Choice,
Persistence, and Mental Effort.
Conceptual Framework for
Addressing the Assumed Motivation Question
Research Question Survey Interviews Document Review
How much choice did
you have with your
decision to take the
exams?
X X
I actively choose to
use English outside
of the classroom.
X X
Please describe the
reasons why you are
going to take these
exams.
X X
Please explain your
feeling about group
group work. Do you
think it helps with
your English
learning?
X X
I rank the importance
of preparation for the
exams equally
compared to my other
goals.
X X
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
94
If I do not succeed on
my first attempt when
trying something
new, I:
X X
Compared to your
other goals, please
explain your strategy
for maintaining your
priority.
X X
Referring to how you
react if you do not
succeed on your first
attempt, try to explain
in depth the reason
for your answer.
X X
How much effort do
you put into reading
and writing English
outside of the
required classwork?
X X
How much effort do
you put into listening
and speaking outside
of the required
classwork?
X X
Please explain your
reasoning for the
level of effort you put
into English skills
beyond what is
required in your
classes.
X X
To what degree will
X X
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
95
you contribute your
efforts beyond the
classroom towards
your of success on
the exam. Explain.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
96
Appendix C
Conceptual Framework for Addressing the Inquiry Questions: Assumed Organization
Multiple aspects of the organization are viewed in order to validate assumptions
including cultural and resource structures.
Conceptual Framework for
Addressing the Assumed Organization Question
Research Question Survey Interviews Document Review
Do you believe that
you have adequate
access to courses and
counselors to achieve
your goal?
X X
I am currently using
additional resources
outside of the
primary organization
where I am learning
English.
X X
Please describe what
additional resources
your organization
could provide such as
additional classes or
counselors. Be
specific.
X X
Please explain what
additional resources,
if any, you are using
and if you are not
using additional
resources then
explain why your
current resources are
X X
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
97
adequate. Be specific.
How adequately do
you believe your
current courses cover
all areas of the exam
for your preparation?
X X
How well aligned do
you believe the ESL
curriculum matches
the exams?
X X
Please describe what
areas of your
curriculum are
strongest towards the
exams and which
areas are the weakest.
Be specific.
X X
Do you believe that
some of your ESL
classes may be
unnecessary or others
missing completely?
X X
All of my ESL
instructors fully
understand the IELTS
and TOEFL exams.
X X
My ESL instructors
address ideas
specifically related to
the exams while
teaching.
X X
Please describe any
X X
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
98
experiences where
you felt that your
instructor did not
understand the IELTS
or TOEFL exams.
Please describe
experiences and
provide examples on
how you believe that
instructors should
provide additional
support in their
lesson plans to assist
you in your
preparation for the
exams.
X X
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
99
Appendix D
Survey Instrument
The IELTS & TOEFL Exams: First Attempt Evaluation Survey
Q1 HOW DO YOU IDENTIFY IN TERMS OF SEX/GENDER?
Male Female Other
a. 1. 2. 3.
Q2 WHAT IS YOUR AGE?
1. 18 - 24
2. 25 - 30
3. Over 30
Q3 WHAT IS YOUR COUNTRY OF ORIGIN?
Q4 DID YOU STUDY ENGLISH IN YOUR HOME COUNTRY?
• Yes
• No
Q5 HOW MANY YEARS IN TOTAL HAVE YOU STUDIED ENGLISH?
• 0
• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20+
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
100
Q6 HOW MANY YEARS HAVE YOU STUDIED ENGLISH IN YOUR HOME COUNTRY?
• 0
• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20+
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
101
Q7 HOW MANY YEARS HAVE YOU STUDIED ENGLISH IN ANOTHER COUNTRY?
• 0
• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20+
Q8 I AM AWARE OF THE SEPARATE SECTIONS OF THE EXAM.
Strongly agree Somewhat
agree
Neither agree
nor disagree
Somewhat
disagree
Strongly
disagree
A. 4. 5. 6. 7. 8.
Q9 I BELIEVE THAT I HAVE BEEN INFORMED OF ALL THE COURSES THAT I MUST
COMPLETE IN ORDER TO SUCCESSFULLY PASS THE EXAM.
Strongly agree Somewhat
agree
Neither agree
nor disagree
Somewhat
disagree
Strongly
disagree
B. • • • • •
Q10 I AM AWARE OF THE EXAMINATION PROCEDURES WITH RESPECT TO EACH
SECTION OF THE EXAM.
Strongly agree Somewhat
agree
Neither agree
nor disagree
Somewhat
disagree
Strongly
disagree
C. • • • • •
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
102
Q11 I HAVE A STRATEGY FOR PRACTICING THE INTERRELATIONSHIPS BETWEEN
THE LISTENING AND WRITING SECTIONS.
Strongly agree Somewhat
agree
Neither agree
nor disagree
Somewhat
disagree
Strongly
disagree
D. • • • • •
Q12 I AM AWARE OF THE MINIMUM WORD KNOWLEDGE FOR TAKING THE EXAM.
Strongly agree Somewhat
agree
Neither agree
nor disagree
Somewhat
disagree
Strongly
disagree
E. • • • • •
Q13 IF I TAKE THE EXAM TODAY I AM CONFIDENT THAT I WOULD ACHIEVE MY
DESIRED SCORES.
Strongly agree Somewhat
agree
Neither agree
nor disagree
Somewhat
disagree
Strongly
disagree
F. • • • • •
Q14 NOBODY ELSE HAD AN INFLUENCE ON MY CHOICE TO TAKE THE EXAM.
Strongly agree Somewhat
agree
Neither agree
nor disagree
Somewhat
disagree
Strongly
disagree
G. • • • • •
Q15 I ACTIVELY CHOOSE TO USE ENGLISH OUTSIDE OF THE CLASSROOM.
Strongly agree Somewhat
agree
Neither agree
nor disagree
Somewhat
disagree
Strongly
disagree
H. • • • • •
Q16 I RANK THE IMPORTANCE OF PREPARATION FOR THE EXAM HIGHER THAN
MY OTHER GOALS.
Strongly agree Somewhat
agree
Neither agree
nor disagree
Somewhat
disagree
Strongly
disagree
I. • • • • •
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
103
Q17 IF I DO NOT SUCCEED ON MY FIRST ATTEMPT WHEN TAKING EXAMS, I TRY
HARDER.
Strongly agree Somewhat
agree
Neither agree
nor disagree
Somewhat
disagree
Strongly
disagree
J. • • • • •
Q18 I PUT MAXIMUM EFFORT INTO READING AND WRITING ENGLISH IN
ADDITION TO MY REGULAR COURSEWORK.
Strongly agree Somewhat
agree
Neither agree
nor disagree
Somewhat
disagree
Strongly
disagree
K. • • • • •
Q19 I PUT MAXIMUM EFFORT INTO LISTENING AND SPEAKING ENGLISH IN
ADDITION TO MY REFULAR COURSEWORK.
Strongly agree Somewhat
agree
Neither agree
nor disagree
Somewhat
disagree
Strongly
disagree
L. • • • • •
Q20 I HAVE ADEQUATE ACCESS TO COURSES AND COUNSELORS TO ACHIEVE MY
GOAL.
Strongly
agree
Agree Somewhat
agree
Neither
agree nor
disagree
Somewhat
disagree
Disagree Strongly
disagree
M. • • • • • • •
Q21 I AM CURRENTLY USING ADDITIONAL RESOURCES OUTSIDE OF THE
PRIMARY ORGANIZATION WHERE I AM LEARNING ENGLISH.
Strongly agree Somewhat
agree
Neither agree
nor disagree
Somewhat
disagree
Strongly
disagree
N. • • • • •
Q22 MY COURSES FULLY PREPARE ME FOR ALL AREAS OF THE EXAM.
Strongly agree Somewhat
agree
Neither agree
nor disagree
Somewhat
disagree
Strongly
disagree
O. • • • • •
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
104
Q23 THE ESL CURRICULUM CLOSELY ALIGNS WITH THE EXAM.
Strongly agree Somewhat
agree
Neither agree
nor disagree
Somewhat
disagree
Strongly
disagree
P. • • • • •
Q24 MY ESL INSTRUCTORS ARE FAMILIAR WITH THE REQUIREMENTS FOR THE
IELTS AND TOEFL EXAMS.
Strongly agree Somewhat
agree
Neither agree
nor disagree
Somewhat
disagree
Strongly
disagree
Q. • • • • •
Q25 MY ESL INSTRUCTORS ADDRESS AREAS SPECIFICALLY RELATED TO THE
EXAMS WHEN TEACHING.
Strongly agree Somewhat
agree
Neither agree
nor disagree
Somewhat
disagree
Strongly
disagree
R. • • • • •
Q26 IF YOU HAVE PREVIOUSLY TAKEN THE IELTS OR TOEFL EXAM, DID YOU
ACHIEVE THE MINIMUM SCORE(S) YOU NEEDED ON YOUR FIRST ATTEMPT?
• Yes
• No
• HAVE NOT TAKEN THE EXAM
Q27 WOULD YOU BE WILLING TO PARTICIPATE IN A FOLLOW-UP INTERVIEW TO
GIVE A BETTER SENSE OF YOUR EXPERIENCE AND SHARE YOUR VIEWS ON
WHAT YOU HAVE BEEN ASKED ABOVE?
• Yes
• No
Q28 PLEASE PROVIDE YOUR CONTACT EMAIL ADDRESS FOR THE INTERVIEW:
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
105
Appendix E
Interview Instrument
LIVE INTERVIEW QUESTIONAIRE – 2016
PSUEDONYM __________ IELTS___ TOEFL___ DATE ___________
1. HOW MANY OF THE REQUIRED ESL COURSES HAVE YOU COMPLETED?
2. PLEASE DESCRIBE THE SEPARATE ELEMENTS OF THE EXAM AND EXPLAIN
HOW YOU LEARNED ABOUT EACH ONE.
3. HOW DO THE COURSES YOU ARE (OR HAVE TAKEN) CONNECT TO EACH
SECTION IN THE EXAM IN YOUR OPINION?
4. WHAT IS YOUR UNDERSTANDING OF THE RELATIONSHIP BETWEEN THE
LISTENING AND WRITING SECTIONS?
5. PLEASE DESCRIBE THE EXAMINATION PROCEDURES FOR EACH SECTION.
6. PLEASE DESCRIBE THE EXAMINATION ENVIRONMENT.
7. PLEASE DESCRIBE YOUR EFFORTS BEYOND THE CLASSROOM TOWARDS
YOUR SUCCESS ON RETAKING THE EXAM?
8. HOW DO YOU FEEL NOW ABOUT YOUR ABILITY TO PASS THE EXAM?
9. IF YOU WERE NOT TO PASS THE EXAM ON YOUR NEXT ATTEMPT, HOW DO
YOU THINK YOU WOULD FEEL?
10. IS THERE ANY OTHER INFORMATION THAT YOU WOULD LIKE TO SHARE?
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
106
Appendix F
TOEFL EVALUATION OBSERVATION NOTES
Notes were taken from a series of volunteer ESL workshops designed to improve listening and
speaking skills from September through December 2016.
• Every student asked, stated that they spoke their native language exclusively off-campus,
whether their English skills were proficient or not.
• Less proficient students stated that they were only in the workshop to fulfill a mandatory
requirement for an ESL class.
• Students who said that they listen to English language broadcasts on Radio or TV for at
least 30-minutes each day demonstrated a greater proficiency in their listening and
speaking skills.
• When offering “open subject” workshops, explaining the U.S. higher education system
and grade point averages has been requested multiple times.
• Noticing three types of students: those who attend a few workshops “because it is
required”, those who attend several workshops randomly out of interest, and those who
attend every workshop regularly (about one-third).
• Students who only achieved a high enough score to enter a community college (60-70
points) have adapted the “two-year” strategy to get into a University to obtain their
Bachelor’s degree, rather than retake the exam. (Randomly asked question).
• Only ~half of students planning to attend graduate school in the U.S. say they are aware
that they will need to retake the TOEFL exam as part of the application process.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
107
• Every student asked, unanimously states that emphasis is put on reading and writing in
their home country and therefore, their listening and speaking skills are their weakest area,
with no strategy on how to improve.
• Students with stronger listening skills stated that they watch American TV shows
(“Friends” is the most cited) back home and in the U.S., daily.
• When attending workshops about consonants or vowel sounds, none of the students were
aware that there are 24 and 15 sounds respectively. They are also unaware that they
cannot hear many of the sounds and need to spend time listening to native speakers.
• The word “strategy” seems to be used frequently when discussing the TOEFL exam. The
exam seems to be disassociated with learning English and more as an obstacle that needs
to be tackled to proceed further with a goal.
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
108
Appendix G
De Anza College ESL Courses and Sequences
REVISED 04/2012
Lower levels need to be completed, either by coursework or placement before students can move to next level
ENGLISH AS SECOND LANGUAGE COURSE SEQUENCE
Students who need beginning level ESL are advised to enroll in an Adult Education program.
Students must complete ESL 261, 262 and 263 to be eligible for ESL 273
ESL 234
(10 units)
Listening/Speaking/Reading/Writing
ESL 244
(10 units)
Listening/Speaking/Reading/Writing
ESL 251
(3 units)
Listening/Speaking
ESL 252
(3 units)
Reading
ESL 253
(4 units)
Writing
unit) unit)
ESL 261
(3 units)
Listening/Speaking
ESL 262
(3 units)
Reading
ESL 263
(4 units)
Writing
ESL 272
(5 units)
Reading
ESL 273
(5 units)
Writing
ESL 005
(5 units)
Advanced Reading/ Composition
ESL 006
(5 units)
Critical Reading/Research for Writing
High
Beginning
Low
Intermediate
Intensive
Intermediate
High
Intermediate
Low
Advanced
Advanced
ESL 200
( 10 units)
Listening/Speaking/Reading/Writing
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
109
Appendix H
Validated Assumed Causes for Knowledge
Assumed Cause
Students do not have a good understanding of
the four areas of the exam
Validated
Students have the basic understanding of
English. Students can understand the
individual words but they may not always
understand a sentence read or heard
Validated
Students may rush their English learning and
skip levels to try and pass the exams more
quickly. They may not know how to pass the
exams as a result
Validated
Validated Assumed Causes for Motivation
Assumed Cause
Some students do not like to do group work
even though it is necessary for learning
Validated
Some instructors do not hold students
accountable if they do not contribute to group
assignments
Validated
Students do not use English outside of the
classroom, limiting their ability to practice
Validated
Validated Assumed Causes for Organization
Assumed Cause
The organization has not given students a clear
path for achieving the minimal exam scores
Validated
The exams are not a part of the curriculum,
counseling, or support services
Validated
THE TOEFL EXAM GATEWAY: UNDERSTANDING PERFORMANCE FACTORS
AMONG COMMUNITY COLLEGE STUDENTS
110
Teachers are not required to learn about or
make exam preparation part of their lesson
plans
Validated
There are not enough counselor, teachers, and
tutors trained to assist students with
preparation for the exams
Validated
Abstract (if available)
Abstract
Research has shown that first attempt performance on the TOEFL exam can have an extremely low rate of success (Esmail, 2013). The TOEFL and IELTS exams have become gatekeepers into English language institutions and society. This case study utilizes a gap analysis framework to try and understand why students have such low performance rates on such an instrumental exam. The primary goal was to identify knowledge, motivation, and organizational barriers that are preventing students from successfully achieving their needed passing rates. Data were collected by use of surveys, interviews, data collection, and observations in a community college environment with a significant international student population. Findings showed that students rely on additional, costly, external private institutions to prepare them adequately for the TOEFL exam. A key factor of success then, has to do with a student’s economic ability to utilize these services. Offering an elective TOEFL preparation course at community colleges may be a more equitable way to assist ESL students with TOEFL exam preparation.
Linked assets
University of Southern California Dissertations and Theses
Conceptually similar
PDF
Factors contributing to the civic engagement of rural community college students
PDF
Relationships between academic and psychosocial skills of community college students
PDF
Perspectives of Native American community college students
PDF
An exploration of student experiences in a preparation program for online classes in the California community college system
PDF
Latino male community college students' persistence to transfer
PDF
My college coach: An online college admission counseling program for high school students
PDF
Persistence among low income community college students
PDF
Developmental math in California community colleges and the delay to academic success
PDF
Factors affecting the success of older community college students
PDF
Oppression of remedial reading community college students and their academic success rates: student perspectives of the unquantified challenges faced
PDF
Engineering my community cultural wealth: testimonios of male Latino community college engineering students
PDF
"God has my back": the role of faith-based institutions in preparing African-American students for college success
PDF
Reflective journeys: African American community college STEM students' perceptions on equity and access
PDF
The barriers and facilitators of academic success for Black male students at a community college: a gap analysis
PDF
Educational experiences and psychosocial well-being of newcomer Afghan refugee students in California community colleges
PDF
Recruitment of women faculty to staff a gender appropriate university in Afghanistan: an innovation study
PDF
Assessing persistence for low-income students at community colleges: the impact of student-parent relationships
PDF
Transfer supports equity: a quantitative investigation into the academic performance of transfer students
PDF
Service-learning and character development: an analysis of Up with People resulting in a model of global citizens for servant leadership
PDF
Comparing the effectiveness of online and face-to-face classes among California community college students
Asset Metadata
Creator
Bankston, Christopher Wendel
(author)
Core Title
The TOEFL exam gateway: understanding performance factors among community college students
School
Rossier School of Education
Degree
Doctor of Education
Degree Program
Global Executive
Publication Date
07/20/2017
Defense Date
05/10/2017
Publisher
University of Southern California
(original),
University of Southern California. Libraries
(digital)
Tag
Community Colleges,EFL,ELP,ESL,HE,IELTS,international students,KMO,OAI-PMH Harvest,tesol,TOEFL
Language
English
Contributor
Electronically uploaded by the author
(provenance)
Advisor
Ahmadi, Shafiqa (
committee chair
), Chung, Ruth (
committee member
), Filback, Robert (
committee member
)
Creator Email
bankston.christopher@gmail.com,cwbankst@usc.edu
Permanent Link (DOI)
https://doi.org/10.25549/usctheses-c40-406522
Unique identifier
UC11263197
Identifier
etd-BankstonCh-5569.pdf (filename),usctheses-c40-406522 (legacy record id)
Legacy Identifier
etd-BankstonCh-5569.pdf
Dmrecord
406522
Document Type
Dissertation
Rights
Bankston, Christopher Wendel
Type
texts
Source
University of Southern California
(contributing entity),
University of Southern California Dissertations and Theses
(collection)
Access Conditions
The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...
Repository Name
University of Southern California Digital Library
Repository Location
USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA
Tags
EFL
ELP
ESL
HE
IELTS
international students
KMO
TOEFL