University of Southern California Dissertations and Theses

Reducing unproductive learning activities in serious games for second language acquisition

(USC Thesis Other)

Reducing unproductive learning activities in serious games for second language acquisition

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content

REDUCING UNPRODUCTIVE LEARNING ACTIVITIES IN SERIOUS GAMES
FOR SECOND LANGUAGE ACQUISITION

by

Shumin Wu

A Dissertation Presented to the
FACULTY OF THE GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
(COMPUTER SCIENCE)

December 2008

Copyright 2008 Shumin Wu
ii
Dedication

To my baby Sandra
iii
Acknowledgments
This thesis arose in a great measurement from research of my PhD studies that has
been done since I came to the Center for Advanced Research in Technology for
Education at the Information Sciences Institute/University of Southern California
(ISI/USC). I was fortunate enough, having been working with many talented and
supportive colleagues during my PhD studies. Their work and thoughts have inspired
this work and encouraged me to continue the research.
I would like to convey my heartfelt gratefulness to my adviser Dr. Lewis
Johnson for his mental and financial supports over years. His guidance is
indispensable for the completion of this thesis, and nourishes my intellectual maturity
of pursuing research work. His constructive feedback tremendously helps me think
and grow. I learned from him not only concrete research methods but also earnst
attitudes of handing research work that I will benefit from in the long run.
My thanks go to my colleagues at ISI/USC, who offered ameliorative advice
for this work through constant meetings and discussions: Catherine Labore, Dimitra
Papadopoulos, Hannes Vilhjálmsson, Stacy Marsella, Shrikanth Narayanan, Abinav
Sethy, Ulf Hermjakob, Nicholas Mote, Ning Wang, Mei Si, and Prasan Samanti. I am
indebted to Catherine Labore for her meticulous work on constructing the skill
ontology.
I would extend my thanks to my supervisor and colleagues at the Alelo
Incorporation, where I did a research internship in the fifth year of my PhD program:
iv
Andre Valente, Brian Russell, Apoorav Gupta, Ben Moore, Robert Heuts, Greg
Hernandez, and Jorem Meron. They have provided me with valuable assistance on
verifying designs and improving code quality. Special thanks to James Reily, who
helped me collect data from the field; many thanks to Ellen O'Connor, Rebecca Row,
Gerardo Ritchey, Rafi Benjamin, Kerrin Barrett, Kami Ghahremani, Joel Harris,
Weiying Li, who enriched my work with diversified insights and practical feedback.
During my preparation for the PhD qualify exam, my committee members
have helped me improve this work with constructive comments and recommendation
of relevant literatures. I would like to express my sincere thanks to Dr. Chad Lane at
the Institute of Interactive Technologies, Ming-Deh Huang at the Department of
Computer Science of USC, Dr. Richard Clark at the Education Department of
University of Southern California, and Dr. Mark Riedel now with the Georgia
Institute of Technology.
I should thank the sponsors of this work over years. The TLCTS project was
first sponsored by DARPA, the US Marine Corps, Program Manager for Training
Systems (PM TRASYS), and US Special Operations Command, and US Navy.
Finally, I would like to thank my husband, my parents, and my in-laws for
their endless love and support throughout these years. They helped me take care of
my baby so that I could concentrate on my research. I could never make it without
their aids.

v
Table of Contents
Dedication .....................................................................................................................ii
Acknowledgments........................................................................................................iii
List of Tables..............................................................................................................viii
List of Figures ..............................................................................................................ix
Abstract ........................................................................................................................xi
Chapter 1: Introduction .................................................................................................1
1.1 Background and Terminology.......................................................................4
1.1.1 Games....................................................................................................4
1.1.2 Computer Games...................................................................................5
1.1.3 Serious Games.......................................................................................6
1.1.4 Intelligent Tutoring Systems .................................................................7
1.1.5 SLA and Cognitive Principles...............................................................9
1.1.6 The Motivation Theories.....................................................................11
1.1.7 Learning Activity and Productivity.....................................................14
1.2 Research Questions .....................................................................................15
1.3 Assumptions................................................................................................16
1.4 Limitations of This Study............................................................................16
1.5 Methodology Overview...............................................................................17
1.6 Major Contributions ....................................................................................20
1.7 Thesis Outline .............................................................................................22
Chapter 2: Related Work.............................................................................................23
2.1 Contemporary Serious Game Systems........................................................23
2.1.1 Serious Games for Job Training..........................................................23
2.1.2 Serious Games for Language Learning...............................................25
2.2 Problems and Challenges in Serious Games ...............................................30
2.3 Evaluation Models.......................................................................................35
2.4 Summary .....................................................................................................37
Chapter 3: Detecting Unproductive Learning Activities.............................................39
3.1 A Case Study on the TLCTS System ..........................................................39
3.1.1 Typical Interactions with the Mission Skill Builder ...........................41
3.1.2 Typical Interactions with the Mission Practice Environment .............44
3.1.3 Typical Interactions with the Arcade Game........................................47
3.2 Study Procedure ..........................................................................................49
vi
3.2.1 Experiment Purpose ............................................................................49
3.2.2 Method ................................................................................................49
3.2.3 Experiment Settings ............................................................................50
3.2.4 Result and Key Findings .....................................................................52
3.2.5 Behavior Patterns of the Unproductive Learning Activities ...............56
3.2.6 Behavior Patterns of the Productive Learning Activities....................64
3.2.7 More Statistics about Unproductive Activities ...................................66
3.2.8 Discussion on Behavior Patterns.........................................................67
3.3 Conclusion...................................................................................................71
Chapter 4: Learner Characteristics and Unproductive Learning.................................72
4.1 System Enhancement ..................................................................................72
4.2 Study Procedure ..........................................................................................78
4.2.1 Experiment Purpose ............................................................................78
4.2.2 Method ................................................................................................78
4.2.3 Experiment Settings ............................................................................78
4.2.4 Observations........................................................................................80
4.2.5 Results and Key Findings....................................................................81
4.3 Discussion ...................................................................................................86
4.4 Conclusion...................................................................................................88
Chapter 5: A Pedagogical Framework for Serious Games .........................................89
5.1 Architecture Overview ................................................................................89
5.2 Learner Service Layer .................................................................................91
5.3 Pedagogy Layer...........................................................................................93
5.3.1 Learner Model .....................................................................................94
5.3.2 Tutor Advice Model ............................................................................95
5.3.3 Tutor Service Layer...........................................................................102
5.3.4 Summary ...........................................................................................103
Chapter 6: Implementation and Evaluation...............................................................104
6.1 Training Standard......................................................................................104
6.2 Implementation Architecture Overview....................................................105
6.3 Implementing the Learner Model..............................................................107
6.3.1 Training Objective Profile.................................................................107
6.3.2 Survey Profile....................................................................................110
6.3.3 Skill Profile .......................................................................................113
6.3.4 Session Profile...................................................................................117
6.3.5 Other Profiles ....................................................................................121
6.4 Log System................................................................................................121
6.5 Tutor Advice Model ..................................................................................123
6.5.1 Examples of Tutor Advice ................................................................123
6.6 Evaluation..................................................................................................129
vii
6.7 Summary ...................................................................................................131
Chapter 7: Discussions, Future Work and Conclusion .............................................132
7.1 Discussions on Research Questions ..........................................................132
7.2 Discussions on the Framework .................................................................134
7.3 Future Work ..............................................................................................135
7.4 Conclusion.................................................................................................137
References .................................................................................................................139
Appendices ................................................................................................................146
Appendix A. Source File List................................................................................146
Appendix B. Evaluation Survey Questionnaire ....................................................149

viii
List of Tables
Table 2.1 Summary of literature reviews on problems about serious games..............35
Table 3.1 Experiment Data Overview.........................................................................52
Table 3.2 Interception of Training Log Data ..............................................................57
Table 3.3 Log Analysis of the Retrospect Pattern.......................................................58
Table 3.4 Log Analysis of the Onrush Pattern ............................................................60
Table 3.5 Log Analysis of the HelpRefusal Pattern....................................................62
Table 3.6 The Productive Learning Activities ............................................................64
Table 3.7 The More Productive Learning Activities...................................................65
Table 6.1 An Example of Computing the Skill Score...............................................117

ix
List of Figures
Figure 1.1 Learner’s ZPD Relative to the Curriculum Difficulty...............................11
Figure 1.2 The Two-Loop Model of Learning............................................................18
Figure 3.1 A Language Instruction and Practice Page the Mission Skill Builder.......42
Figure 3.2 An Utterance Formation Page in the Mission Skill Builder ......................43
Figure 3.3 A Scenario of a Café Place in the Mission Practice Environment.............45
Figure 3.4 Arcade Game Menu that Lists Four Vocabulary Categories.....................47
Figure 3.5 Arcade Game Speaking Mode ...................................................................48
Figure 3.6 Learning Statistics of the Mission Skill Builder........................................53
Figure 3.7 Learning Statistics of the Mission Practice Environment..........................54
Figure 3.8 Learning Statistics of the Arcade Game ....................................................55
Figure 3.9 The Sequence Diagram of Retrospect Pattern ..........................................60
Figure 3.10 The Sequence Diagram of Onrush Pattern..............................................61
Figure 3.11 The HelpRefusal Pattern.........................................................................63
Figure 3.12 Percentage of Unproductive Activities Against the Total Activities.......66
Figure 4.1 The Objectives Panel for Active Dialogs .................................................74
Figure 4.2 The Recording Quality and Pronunciation Feedback ...............................74
Figure 4.3 The Score Debriefing Dialog for Active Dialogs .....................................75
Figure 4.4 A Scenario When the Learner Offenses the Virtual Characters ...............76
Figure 4.5 A Progress Report (Skill Mode) ...............................................................77
Figure 5.1 Pedagogical Agents Architecture...............................................................90
x
Figure 5.2 Learner Service Layer................................................................................91
Figure 5.3 Pedagogy Layer .........................................................................................95
Figure 5.4 The Shift of Learner’s Position to ZPD in the Retrospect Pattern.............96
Figure 5.5 The Shift of Learner’s Position to ZPD in the Onrush Pattern..................97
Figure 5.6 The Shift of Learner’s Position to ZPD in the HelpRefusal Pattern..........98
Figure 5.7 The Composition of a Skill Space ...........................................................100
Figure 6.1 System Architecture of Tactical Iraqi
TM
Version 5.0 ..............................106
Figure 6.2 Utterance-Skill Space Mapping ..............................................................113
Figure 6.3 A part of the Bayes Tracing Net ..............................................................116
Figure 6.4 Interaction of Log System with Other Models ........................................122
Figure 6.5 Pre-quiz Advice .......................................................................................123
Figure 6.6 Post-quiz Advice......................................................................................125
Figure 6.7 Pre-scene Advice .....................................................................................126
Figure 6.8 Post-scene Advice....................................................................................128

xi
Abstract
Recent years have seen a large number of game-based training systems or serious
games developed for diversified learning domains. Despite the hypothesis that
computer games are motivator to promote learning engagement, however, researchers
reported various problems existing in these systems. One of the intractable problems,
for example, is that games incentives may direct learners to unproductive learning
activities, diverging from the original intention of educational software designers.
Skeptics began to question the worth of employing game techniques in training
systems, as constructing a serious game faces relatively longer development cycle and
consumes more expensive resources.
Do unproductive learning activities in serious games falsify the hypothesis
that games can promote learning? What kind of learners can benefit from the
didactics of learning by playing? What design issues should be taken into
consideration in order to reduce these unproductive activities? To what extent can
pedagogy harness the power of computer games and still preserve their fun elements?
This thesis attempts to answer these questions through in-depth research on
improving the learning productivity of serious games. Iterative exploratory studies
were carried for evaluating the training results and validating the design of a
particular serious game, called the Tactical Language & Culture Training System
(TLCTS), which coaches adult learners to rapidly acquire spoken communication
skills. Evidence of the previous study was used to establish the hypothesis for the
xii
next controlled study, which further serves as the basis of qualitative and quantitative
analysis on the generic issues of low learning productivity in serious games. Based on
the analysis, this thesis presents a pedagogical framework tailored for serious game
applications, which incorporate customized curriculum contents, implicit and explicit
feedback, game scoring, scaffolding and fading, online student assessment,
performance summarization as well as tutor advice facilities. This thesis also provides
implementation details of this framework as a case study. Preliminary results on
adoption of this framework indicate that it helps reduce unproductive learning
activities. We conclude that serious games should be engineered as coherent whole
that harmonizes the cognitive and game design principles rather than a mere container
of the curriculum contents.

1
Chapter 1

Introduction

The motivation of this research to a great degree stems from interesting phenomena
which I have experienced and observed over years of video game playing and studies,
and which are repeatedly reported in academic studies: playing games can help
learning.
I myself benefited from playing English-subtitled games when I was learning
English as my second language in China. When I was a freshman in college, my old
strategy of drill-and-practice started losing its effectiveness, partly because the
curriculum of English courses at college involve more advanced vocabulary and
complicated sentence structures than that of the middle school, and partly because I
had less motivation to concentrate on the long and tedious texts any more. Then it
happened that a friend of mine lent me an English-subtitled game called the Final
Fantasy
TM
VII, a role-playing game (RPG) produced by the Square Enix. I was soon
fascinated to this game, playing hours after hours relying on a dictionary to
understand the conversation between game characters, in order to resolve the riddles
and to advance the plot. After I completed the game, I found that I could memorize
many new words and phrases, and recognize their meanings in different contexts with
2
little effort. This positive experience encouraged me to try more games in English
subtitles and later Japanese subtitles when I was learning Japanese.
My experience was not alone, as many research studies have showed the
success of using videogames to promote learning. Soon after the first commercial,
off-the-shelf (COTS) computer games was made available, games’ positive effects on
learner motivation came into notice of researchers in the psychology field. Motivation
is regarded as a critical element to learning as it can increase learning efficiency in
educational systems (Lepper & Cordova, 1992). Empirical studies (Malone, 1981)
proved that games have potentials in improving teaching and learning, asserting that
games used as teaching devices can create intrinsically motivating instructional
environments, where deeper learning may take place. The findings suggest new
learning settings for certain domains (e.g. language training) where learning results
are considered not easily achievable in traditional teacher-centered classroom
environments (Jonnavithula & Kinshuk, 2005). Therefore, pioneers who studied the
game effects claimed that computer games can create “high-performance domains”
(DeSmedt, 1995), where computer games in particular can bring great interactivity to
learning materials, and therefore can engender a high level of motivation and
engagement (David, 1997).
However, COTS games like Final Fantasy
TM
VII are not designed for teaching
languages. Game scripts may be written for stories that happen in a fantacy world
rather than real life, and therefore the subtitles of these games are not suitable for
teaching day-to-day language usage. In addition, interaction with such systems is
3
silent, offering zero chance for players to practice spoken communication skills.
Besides, during most time of playing the game, the character which I was playing was
engaged in combats with evil witches and monsters, which involved little language
learning.
Wouldn’t it be lovely if a game were tweaked so as to allow users to engage in
effective learning activities as much as possible?
As a matter of fact, researchers already worked on innovations such as serious
game systems designed solely for educational purpose (Diller & Roberts, 2005;
Careless, 2005; McGrath & Hill, 2004; Korris, 2004). However, designing an
effective serious game is a non-trivial problem, because there is no guarantee that
effective learning would take place in serious games simply by installing the learning
materials in the game container (Cone et al, 2007).
This thesis explores the topic of how to build effective serious games to
greatly enhance the learning productivity. Generic issues in serious game
development were examined in the context of a case study on a serious game
designed for second language acquisition. Exploratory studies were launched to test
and validate the design of system.
In Section 1.1, we will take a look at the background of this study and
definitions of concepts used in this thesis. Section 1.2 introduces the current research
questions this study focuses on. Section 1.3 sets forth the assumptions. Section 1.4
covers the limitations of this study. Section 1.5 previews the methodologies
4
conducted for this study. Section 1.6 looks at the major contribution of this study. At
the end of this chapter, the outline of this thesis is presented in Section 1.7.
1.1 Background and Terminology
1.1.1 Games

Games are so much diversified activities that a simple definition often leads to either
insufficiency or over-generalization. The Marriam-Webster On-Line dictionary
defines game as “a physical or mental competition conducted according to rules with
the participants in direct opposition to each other”. However, as Clark Abt (1987)
pointed out, “not all games are contests among adversaries – in some games the
players cooperate to achieve a common goal against an obstructing force or natural
situation that is itself not really a player since it does not have objectives” (pp. 7).
Instead, he defined a game as “an activity among two or more independent decision-
makers seeking to achieve their objectives in some limiting context” (pp. 8).
Although the denotation of this definition can cover most games, it mentions little
about rules generally regarded as the source of fun that attract people to play.
Therefore, David Michael and Sande Chen (2005) advocate applying the definition of
“play” to “games”. Utilizing Johan Huizinga (1950)’s definition of play, Michael and
Chen (2005) redefined games as “a voluntary activity, obviously separate from real
life, creating an imaginary world that absorbs the player’s full attention” (pp.19).
This is a plausible definition for games, yet another feature of games which is not
addressed is that even though the game world is separate from reality, a game is
5
constituted of “patterns” that are abstracted from reality (pp. 40, Koster, 2004). This
addition discloses that games are based on models of the real world was explained in
details in the work of (Klein, 1985). In summary, a game is a voluntary rule-governed
activity undertaken by independent participants who attempt to achieve their
objectives in a fantasy world distinct form reality but present patterns abstracted from
reality.
1.1.2 Computer Games

Chris Crawford (1984) defined a computer game as a computer-controlled universe
where players are imposed by a set of rules. This definition still applies to today’s
computer games, however, technological innovations have propelled the growth of
multimedia platforms on top of which these games can run. Players have a wide range
of choices on where to play computer games, such as Internet, hand-held devices, and
game consoles, besides the traditional coin-op arcade machines, personal, and large
mainframe computers. Modality of computer games is also expanded. Not only had
classical mental games been transplanted to multimedia virtual environments, but also
games that involve physical exertion were created on various game consoles such as
the Nintendo Wii
TM
.
Compared with twenty years ago people are more ready to accept computer
games as culture equal to traditional entertainments. The number of computer game
players continues increasing. A 2007 report of the Entertainment Software
Association shows 67 percent of heads of households play computer games, and the
6
average game player is 33 years old having been playing for 13 years (ESA, 2007).
These so-called “gamers” play for their own sake without external rewards or goals.
In an online survey conducted by the sales tracking group NPD (www.npd.com) from
February to April 2008, 72 percent of 20,240 U.S. residents surveyed reports that they
play computer games, up from 64% in 2006. Among them, 42 percent engages in
online gaming. Regardless of the numbers, the most amazing fact, perhaps, is that
these gamers are all voluntary – they play games for their own sake without external
rewards.
1.1.3 Serious Games

Interest grows for constructing game-based learning environments as the computer
technology advances. The concept “edutainment” emerged in 1990s, dedicated to the
method of embracing the entertainment technology in traditional education forms.
However, this term was mostly used for preschooler learners and new readers
(Michael, 2005). The term “serious game” has been in existence since 1980s, but is
not made prevailing until around 2002, when the Serious Game Initiative
(www.seriousgames.org) was founded. Distinct from “edutainment”, serious games
apply for learners of all ages. Clark Abt (1987) claimed that serious games should
have “an explicit and carefully thought-out educational purpose” and “are not
intended to be played primarily for amusement”. This definition is unambiguous and
widely cited in serious games literatures. It identifies the main difference between a
serious game and entertainment games are that the purpose what the game is designed
7
for. Other than that, serious games are just like any game that is a structured,
voluntary activity constituted of goals and rules. Certain literatures construe the
serious game as any form of game, but in many e-learning or serious game literatures
this term overtly refers to computer games designed for educational purpose.
As Richard Clark (1983) argued that computers as a medium do not cause
learning, and it is the pedagogy employed by the medium that has impact on learning.
Furthermore, there is no guarantee that pedagogy will cause learning. Similarly,
serious games, which are delivered by computers or other media, may or may not
influence student learning outcomes. But there are strong belief held by some
researchers that serious games have positive impact on learning. James Paul Gee
(2003) argues that serious games are a semiotic domain with matrices of
environmental attributes such as well-ordered problems, pleasantly frustration, cycle
of expertise, as well as strong identities. In fact, he thinks games can provide learning
conditions than de-contextualized, skill-and-skill classroom activities.
1.1.4 Intelligent Tutoring Systems

An intelligent tutoring system (ITS) aims at providing automated feedback and
instructions, which are often personalized. ITS systems are more than games that are
installed with curriculum contents. ITS systems pursue learning by doing, whereas
serious games on learning by playing. According to the Maslow's hierarchy of needs
(Maslow, 1943), ITS systems mainly focus on learner’s cognitive needs (acquiring
new knowledge and skills), whereas serious games also address on their self-
8
actualization needs (winning the game), aesthetic needs (high-quality graphics and
music) besides cognitive needs. Furthermore, ITS systems are equipped with
advanced tutoring devices that have the knowledge of “when to give feedback,
scaffolding, and explanations, when to hold back error corrections and allow the
learners to infer that an error has been made” (Chi et al., 2001). These features are not
necessarily included in serious games.
A typical intelligent tutoring system consists of five components: an expert
model, a learner model, a tutor model, a user interface and an authoring system. The
expert model maintains the representation of expertise knowledge about the domain,
serving as an information source to the whole ITS system (Ong & Ramachandran,
2000). The learner model, or student model, maintains a knowledge base about “what
the user does and doesn't know, and what he or she does and doesn't have” (Ong &
Ramachandran, 2000). Nevertheless, more advanced learner models can be built for
different purpose, such as recognizing solution plans, tracking errors, or performing
student assessment (Zhou & Evens, 1999). A tutor model simulates the human tutor
knowledge and behaviors, such as providing feedback and instructions. A mature ITS
system is equipped with an authoring system which can be used to automate the
curriculum development, and therefore can facilitate the collaboration of multiple
tutors working remotely (Murray, 1999). In addition, an ITS system usually has a
graphical user interface through which the user can interact with the system.
ITS systems are considered as ideal platforms where various artificial
intelligence techniques can be applied such as knowledge representation, probabilistic
9
decision-making, and rule-based inference (Urban-Lurain, 2002), and are often seen
as supplemental tools to the classroom learning instead of a replacement of the human
tutors (Epstein & Hillegeist, 1990).
1.1.5 SLA and Cognitive Principles

Second Language Acquisition (SLA) is a subject of linguistics that studies the
phenomena of adult speakers learning to speak a language (L2) other than their native
language. Despite the wide scope of SLA research, there are basically two diametrical
stances of SLA researchers. One is Chomsky’s Universal Grammar (UG) framework.
This theory postulates that all languages share the same universal grammatical
structure innate to language learners. The structure consists of a set of unconscious
constraints on which the learner depends to decide whether a sentence given is
grammatically acceptable or unacceptable (Chomsky, 1965). Another theory called
constructivism stemming from the theories of developmental psychology and genetic
epistemology (whose representatives are Vygotsky and Piaget) does not acknowledge
the pre-existence of real world. The constructivism theory believes that language
learning is acquired solely through interaction with the environment (Geoff, 2004),
and therefore emphasizes “the instrumental and practical function of theory
construction and knowing” (pp.7, Denkin & Lincoln, 1998). Advocators of the former
are often referred to as “mentalists”. Advocators of the later are often referred to as
“behaviorists”. The two parties have different emphasis on many of the linguistics
terms.
10
One of the most influential constructivist theories that largely affect classroom
language teaching is Krashen’s Scaffolding Theory. This theory postulates that the
amount of comprehensible input that a L2 learner (acquirer) receives determines the
language acquisition (Krashen, 1982). The comprehensible input is any form of
information about the target language understandable at the learner’s current level of
linguistic competence (i), or just slightly beyond that level (i+1).
A parallel theory to Krashen’s i+1 concept is Vygotsky’s zone of proximal
development (ZPD) theory. ZPD is a zone that sits between “a learner’s current actual
level of development as determined by independent problem-solving and learner’s
emerging or potential level of development as determined through problem solving
under adult guidance or in collaboration with more capable peers” (Vygotsky, 1978).
Although originally the ZPD theory addressed the children’s mental maturation and
formation processes, it also applies to the adult learning. It gives a learner’s
developmental status in the immediate future with the aid from a tutor or advanced
collaborator, when this learner will master the skills that the learner does not master
at present.
Figure 1.1 describes the learner’s position relative to the ZPD zone. The
artificial curve is a ratio that represents the curriculum difficulty against the learner’s
skill development level. If this ratio falls into the ZPD zone, then the learnrer’s
likelihood of high learning efficiency is maximized. Otherwise, the learner may
remain in the zone of confusion or zone of boredom. Generally, when the learner falls
outside his/her ZPD, he/she is either not developing skills proficient enough or not
11
challenged enough. While a learner’s skills are poorly developed, the learner feels
confused; while a learner is not challenged enough, the learner feels bored. This
confusion and boredom may lead to low learning efficiency.

Figure 1.1 Learner’s ZPD Relative to the Curriculum Difficulty

Both the Scaffolding and ZPD theory suggest that the learner should be
challenged at a level that is just above the current level of development, a level that
will neither overwhelm the learner nor appear too easy to the learner. They imply that
there are good learning opportunities when the learner is facing this kind of challenge.
1.1.6 The Motivation Theories

Motivation drives people to do things. It is inevitable to take consideration of
trainee’s motivation if we intend to adjust the trainee’s activities. Therefore, design of
ZPD
Curriculum
Difficulty
Learner Skill Level
Zone of confusion
Zone of boredom
Difficulty-Skill Ratio
12
any training system shall be conform to the principles identified in motivation
theories.
One of the earliest and most well-known theories on motivation is Maslow’s
hierarchy of needs theory, which describes a pyramid of five-level needs:
physiological needs (e.g., breathing and food), safety (e.g., body and job security),
love/belonging (e.g., friendship and family), esteem (self-esteem and confidence), and
self-actualization (e.g., morality and creativity) (Maslow, 1943). This theory also
claims that people focus on higher-level needs only after the lower-level needs are
met. The first four levels of needs are often called D-needs (D stands for deficiency),
and are prioritized over the self-actualization needs, because one would feel anxious
and tense if these needs are not met. Critiques on this theory include the neglect of
simultaneously occurring needs at different levels, and individual differences that
related to their earlier experience (Neher, 1991).
McClelland’s three-need theory claims that a person’s specific needs are
acquired over time and shaped by their early experience. These needs are classified
into three categories: achievement, affiliation, or power (McClelland, 1975). People
who need achievements look for frequent positive feedback from others on their
performance. They tend to avoid low-risk tasks because these do not bring much gain
of satisfaction, and high-risk tasks because these might lead to failure. Affiliation-
motivated persons seek harmony and approval among their social circles. They tend
to avoid standing out in the public eye. Power seekers have great motivation to
13
control others (McClelland & Burnham, 1976). They strive to achieve their goals by
casting out other’s agreement or compliance.
Another perspective of classifying motivation based on sources rather than
individual’s needs is the theory on extrinsic and intrinsic motivation. Extrinsic
motivation is driven by external rewards or pressures (Deci, 1971). However, removal
of extrinsic motivators could stop people who were exposed to it from doing things.
In contrast, intrinsic motivation drives people to do things for the sake of fun or for
maintaining their self-concepts (e.g., believing it is a good thing to do) (Malone &
Lepper, 1987). Studies found out that replacing intrinsic motivators with external
drivers could raise an Overjustification Effect (Greene et al, 1976), which again stops
people from doing thing.
Some theorists attempted to link the motivation to task selection. The
Cognitive Evaluation Theory claims that people tend to select tasks that they think
they will be able to complete given their current level of competence (Deci & Ryan,
1991). This is because a person needs safety to feel the task is in control. The theory
further claims that if a person thinks a particular task is manageable he/she will be
intrinsically motivated to perform this task. Hence, selecting a task within one’s
competence can be a motivator to drive people to do it.
A relatively new theory on motivation is the Affect Perseverance theory
(Sherman & Kim, 2002), which emphasizes the emotion effect on people’s
motivation. Sherman and Kim’s studies showed that people’s affective preference
persevered even after the facts and evidence that gave rise to their original emotion
14
were invalidated. This suggests that emotion is a channel of motivation independent
of cognition based on rational purposes. It is possible to motivate people to undertake
a task by offering a rational purpose at first and then gradually engaging them
emotionally in this task. Their affect perseverance occurs after the rational purpose of
that task is removed.
1.1.7 Learning Activity and Productivity

There are controversies on defining the terms “learning activity”, “learning task”, and
“learning event” in the ITS and SLA fields (VanLehn, 1999; Ellis, 2003). These terms
in certain contexts are used interchangeably, and in other contexts are used to define
each other in a way that the mapping between a pair of them is a 1:n relationship. In
this thesis, a learning activity refers to a procedure that includes a step or a set of
steps to stimulate learning (with mental involvement) during interaction with the
computer system. A step is as small as a mouse click or button press. And a task is
defined as an activity with a specified objective, carried out to accomplish an
assignment or problem. For the definition of task in the SLA domain, more
specifically, we adopted the one widely used by the SLA community, which was
originally proposed by Rod Ellis (2003): a task of language learning is goal-oriented,
involving a primary focus on meaning and choice of linguistic resources, with a
clearly defined outcome. Therefore, a learning event is the process of construction or
application of skills to complete a task.
15
Productivity defines the ratio of benefits against costs. Learning productivity
in this context refers to the ratio of learning outcomes against the learning inputs.
Learning inputs include the training time, personnel (human tutors), and costs used to
create the learning environment. Learning outcomes are knowledge, skills or abilities
acquired by the learner through learning activities. An unproductive learning activity
is defined as a learning activity that yields few learning outcomes.
1.2 Research Questions

Educational software developers are now able to create more realistic simulations
with the innovative graphics techniques. Increasing needs of serious games have
been shown in a wide range of domains. Unsettled are the research issues as follows:
1. Do serious games provide productive learning, especially in training of
realistically complex problem-solving skills?
2. Do serious games enable productive learning for everybody?
3. How can we modify the existing serious game system to reduce learners’
unproductive activities and therefore to promote their learning productivity?
4. Should serious games be used as a replacement or supplement to traditional
classroom learning?
All of these concerns suggest design an effective serious games is a non-trivial
problem, and there is no guarantee that immersive learning takes place in serious
games simply by installing the learning materials in the game container.
16
1.3 Assumptions

The following assumptions underline the basis of further analysis and discussion in
this thesis.
1. Computer games are expressive contexts to convey language and culture
knowledge.
2. Language skills are developed through socially contextualized interactions.
3. Learning opportunities occur in learner’s action as well as reflection on their
mistakes and problems.
4. Human memory has a limited working memory and an unlimited long-term
memory. The amount of comprehensive input and the quality of such input to
the working memory determines the amount and quality of the learning
outcomes.
5. Learners prefer few tutoring interrupts when playing serious games.
6. Motivation is a desirable factor in learning.
1.4 Limitations of This Study

The limitations of this study include the following aspects:

1. This work focuses on a case study on different versions of a particular system.
Discussions and conclusions made in this thesis may only apply to the full-
fledged serious games with identical built-in pedagogy.
17
2. Subjects available for this study are moderately or highly motivated learners.
Subjects' motivation levels were obtained through interviews, which may or
may not reflect their real motivations during game-playing.
3. The author of this thesis is not an independent evaluator. Studies were
conducted as a part of verification process to validate the design intents.
4. This work is an exploratory study based on practical experience rather than
established theoretic foundations.
1.5 Methodology Overview

This study attempts to engineer a framework to reduce the unproductive learning
activities for serious games through continuous assessment and system enhancement.
It takes an instance of a serious game system at a particular development stage as the
starting point, revising its curriculum management and pedagogical assistance
capabilities and validating the design through data analysis. The work built on the
following two theories, which address the different aspects of training system design
and both advocates research in vivo:
1. The two-loop learning model was supported by empirical researches (Argyris
& Schön, 1978). The model involves three components: governing variables,
action strategy, and consequences. Governing variables are the input within
the learner’s acceptable range. Action strategy is the plan formulated by the
learner to maintain the acceptable range of governing variables. Consequences
are both the intended and unintended results of an action (Anderson, 1997).
18

Figure 1.2 The Two-Loop Model of Learning

Traditional single loop learning reflects the relationship between
Action Strategy and Consequences in a way that the learner will modify the
action plan according to the errors detected in results. In contrast, the double
learning loop involves modification of the learning systems’ underlying goals
and strategies (Argyris, 1982). Therefore, iterative enhancement on the
learning system is an integral part of the evolving learning process.
The two-loop learning model also covers the organizational learning,
based on the assumption that the training productivity at the organizational
level is aggregated individual training productivity. It considers the extra
cycle of error detection and correction through modification on the governing
variables of the learning systems.
2. The adoption-based design (ABD). ABD is a relatively new notion but has
been practiced for many years. Chan (2007) summarizes the current research
models in serious games and technology enhanced learning (TEL) and
classifies them into three types: dream-based, adoption-based, and humanity-
based. Adoption-based research supports evolutionary design and “intends to
prove the feasibility of spreading TEL in the real world practice” (Chan,
19
2007). Research of this type focuses on the knowledge transfer from
authenticate learning settings to regular field use. The original ABD defines
dual procedures that provide impetus to iterative revision on the current
curriculum design. In our study the first procedure is referred to as the process
that delineates the prototype of serious games, while the second procedure is
referred to as the process that adjusts the features of which the requirements
are identified in regular field use. Hence, the study conduces to and relies on
the successful adoption of effective educational software. On these grounds
student assessment performed in this study aimed at learning outcomes within
the digital learning environment and its effectiveness in real practice. The
assessment is thus used to verify and advance not only the curriculum design
of the serious games but also the pedagogical service that is intended to
operate the inputs to the learner’s working memory. The drawback of this
method is that it is laborious to track how learner exerts the acquired skills in
the field. Although this work has spared no efforts in understanding how
educational software performs in practice through surveys, the volatile
military situation often interrupted the pre-established data collection plan.
The following work examines the learner’s interaction patterns and
characteristics which are conductive to learning success and therefore should be used
as the anchor of formulating different learning plans. Performance scores and
interaction logs were analyzed to identify unproductive learning activities that
occurred in the logs of unsuccessful learners. Based on the analysis, we then revised
20
the design approaches and conducted new experiments with a group of fresh learners
with similar backgrounds to the former users, in hope of the new adjustments would
mitigate the ungracious factors that caused the unproductive learning.
The choice of a version of the training system itself an assessment tool is
unusual, but affords a number of advantages. It tests a wide range of cognitive
abilities relevant learning language, wider than what is typical of language training
system tests (classic evaluation models are discussed in Section 2.3). It gives us an
opportunity to examine the internalized learning results such as self-assessment of
learners’ own language performance, learning plan formulation to achieve mastery of
skills, and motivation invoked at different learning stages with different tasks. It
provides us the view of how learners are selecting tasks that they think language and
cultural skills that are potentially valuable can be learned. The aggregated facts allow
us aware of the motivation at the organization level as well as the individual level.
The data collected from the training sessions also gave us the opportunity to further
investigate the research questions that are of concern to this study.
1.6 Major Contributions

This thesis contributes to the serious game field in a number of ways as follows:
1. A pedagogical framework designed for mitigating the unproductive
activities in serious games. Traditional ITS models emphasize too much on
the tutoring service rather than coaching the learner as to allow them to
yield automatic productive learning activities through action and reflection.
21
They are not suitable for serious games, which advocate learning by playing
and prohibit frequent tutoring interrupts. This framework emphasizes the
coaching service that guides learners as to where they should focus their
efforts. It conforms to the principles of game design, SLA theory, and ITS
model. This framework employs a set of game-tailored pedagogy
techniques, such as customized curriculum contents, implicit and explicit
feedback, task debriefings and recommendations, online skill assessment,
performance summarization facilities and coaching service, in hopes of
reducing the unproductive learning activities that hamper the effectiveness
of utilizing the game stimulus to improve learning.
2. A qualitative approach analyzing and classifying the unproductive learning
activities in serious games. Many literatures have raised questions on the
effectiveness of serious games (Shaffer et al, 2004). Little has been done to
quantitatively analyze the real learning data, or further implement
compensations to improve the existing systems based upon the analysis.
This study examines the real training data collected from the field. Pattern
recognition techniques were used to extract the unproductive learning
activities from these data. Surveys were conducted among selective learners,
and their learning performances at different system evolvement stages were
compared to verify the effectiveness of the newly added features that are
intended to reduce the unproductive learning activities.
22
1.7 Thesis Outline

The rest of the thesis is organized as follows.
• Chapter 2 reviews the contemporary serious game systems, the problems and
challenges of these systems discussed in various literatures, as well as the
evaluation models used by researchers for evaluating the effectiveness of
these systems.
• Chapter 3 describes the approach of detecting and classifying the
unproductive learning activities in a case study.
• Chapter 4 examines the characteristics of learners who did both productive
and unproductive learning activities with the system having been described in
the case study.
• Chapter 5 presents a generic pedagogical framework for serious games.
• Chapter 6 describes the implementation and evaluation on this pedagogical
framework.
• Chapter 7 looks at the future work of this study and concludes this thesis.

23
Chapter 2

Related Work
Today a growing number of serious games have been deployed in miscellaneous
fields, gradually impacting our human beings’ learning culture. In this chapter we will
first review the existing serious games developed for various domains and purposes
up to date. The use of these systems not only carries new opportunities for learning,
but also elicits novel research questions on the actual productivity as well as design
issues. Followed by that, a survey of education and game design literatures will be
presented. Our focus is on problem analysis and proposed solutions identified in these
literatures.
2.1 Contemporary Serious Game Systems
2.1.1 Serious Games for Job Training

One of the earliest game-based training systems is the WEST system (Burton &
Brown, 1982), focusing on teaching basic mathematics (arithmetic and operator
precedence) and strategic thinking skills. The highlight of this system is that it
employs a virtual coach that provides feedback to help learners learn from their
mistakes, and to encourage them to use new strategies against the computer opponent.
Specifically, the coach is responsible for monitoring the game-playing, recognizing
and explaining weaknesses, as well as intervening minimally to offer suggestions.
24
Burton and Brown (1982) claimed that the coach’s function in game-based
educational systems should be able to transform non-constructive errors into
constructive ones. Here, non-constructive errors refer to the errors that contain not
enough information to improve the students learning behaviors, whereas the
constructive errors refer to the errors based on which learners can determine what
caused errors and how to correct them. Therefore, the coach should identify the issues
the learner may have been overlooked, e.g., a possible intervention could be in a
sequence of moves where the learner’s strategy differs from the expert’s strategy
significantly and reflects a substantial amount of moves that they should not have
been used (Burton & Brown, 1982).
The applications of serious games in military training are extensive and varied.
Games such as the DARWARS Ambush! (Diller & Roberts, 2005), HotZone (Careless,
2005), UnrealTriage (McGrath & Hill, 2004), and Full Spectrum Warriror (Korris,
2004) evolved mainly from the original first-person shooting and strategic games,
which, to a large extent, emphasize on situational authenticity, imposing high
requirements on the quality of simulation graphics. CyberCIEGE, a serious game for
cyber security training, is another kind of training system that accentuates scenario-
based interactive storytelling (Cone et al, 2007). The authors claimed that learners
acquired knowledge about what can be done and what cannot be done in a security-
critical system by playing scenarios that simulate real-life situations, e.g., receiving a
phishing email.
25
Certain commercialized serious games are designed for civilian uses such as
healthcare and corporate training. Companies that produce this kind of applications
include the BreakAway Ltd. (www.breakawaygames.com) and Forio Business
Simulations (forio.com). The former is one of the leading COTS software developers,
which has developed a basket of serious games for job training. The later builds
online games for business decision training and crisis management.
2.1.2 Serious Games for Language Learning

Predecessors of serious games for language acquisition can be traced back to as early
as 1990s. These so-called Computer-Assisted Language Learning (CALL) systems
were not explicitly named after “games” or “game-based systems”. Instead, they were
given all kinds of names such as microworlds, scenes or conversations, which reflect
certain game features such as fantasy, interaction, and role play. CALL systems teach
communication skills of target languages (listening, speaking, reading, and writing) at
different levels (beginner vs. advanced).
Inspired by the theory of problem-solving, some systems enable learners
acquire second language skills through solving real-life communication problems.
One of these systems is LingWorlds (Douglas, 1995), which is built upon multiple
graphical simulations that consist of a variety of movable objects. In the LingWorlds,
the tutor assigns a task through speech commands, each of which contains a digitized
phrasal lexicon. The learner then responds by moving objects in the microworld to
complete the task. The command will be repeated if the learner does not take any
26
action in a certain time interval. If the learner does, the system will analyze the
learner’s reaction to diagnose how much the learner has understood what the tutor has
said. The relevant part of speech command will be replayed if the system detects that
the learner fails to follow the tutor’s instructions. Therefore, the learner can focus on
the erroneous lexicons and improve their listening skills through immediate feedback.
Another microworld system is the MILT (Kaplan & Holland, 1995; Holland et
al., 1999), which demonstrates the approach of enabling a single platform to teach
multiple target languages. MILT builds on the Army Research Institute (ARI)’s
BRIDGE system (Sams, 1995), where a focus-on-form parser diagnoses the learners’
input sentences with a limit at the syntactic level. MILT has the built-in German and
Arabic natural language processing (NLP) parsers that perform semantic parsing, as
well as a graphical representation of microworld, where learners can type in
commands to manipulate the objects inside the microworld. Immediate feedback is
provided to correct their grammatical errors. MILT has a variety of task types and
tasks at different difficulty levels. Besides the microworld, MILT provides
supplemental fill-in-blank exercises to allow learners to practice production of words
and phrases, and simulated dialogs to practice production and understanding of the
target language. One research question MILT explores but does not give a conclusive
answer is who should take the initiative of task selection: the learner, the tutor or the
system. Another question unanswered is whether tasks should be chosen based on
performance or randomly.
27
Conversation-based systems are common for training speaking skills.
FLUENT (Hamburger, 1995), a conversation-based system, is a practice environment
where learners have conversations in the target language with intelligent tutors and
use speech as acts. For example, in a kitchen scene, learners can issue speech
commands to complete the task of having a cup of coffee. As the vocabulary of the
learner-tutor conversations is limited and expected, it reduces the complexity of the
automated speech recognizer (ASR). The Subarashii system (Bernstein et al., 1999) is
a similar but more complex system, as it teaches beginner level learners solve simple
communication problems through open-ended spoken dialogues, or “encounters”.
This system sets all the computer-animated virtual characters as monolingual
Japanese native speakers. Subarashii not only trains the learners in language
proficiency but also teaches them cultural knowledge, e.g. to refuse an invitation
politely. Because of the use of open dialogues, Subarashii attempts to capture
learners’ anticipated responses during one encounter in a dialogue flow chart, but
there are still many responses that are either invalid or cannot be recognized by the
ASR module. To compensate this deficiency, Subarashii uses help facilities such as
reminders, hints, and directions in English to assist learners to proceed with the
conversation.
Another conversation-based system is known as Virtual Conversation (Harless
et at., 1999), which teaches American military personnel Arabic language skills.
Techniques used by Virtual Conversation are akin to those in Subarashii, but more
emphasize on intermediate level learners. The system enables learners the interaction
28
to perform “face-to-face” conversations with computer-controlled virtual characters.
Studies on this system showed positive effects in learners’ confidence and motivation
to study Arabic, as well as proficiency in speaking, listening, and reading (Harless et
at., 1999).
Conversation-based systems that embed intelligent agents are often used to
train high-level communication skills such as negation and interrogation. Herr
Kommissar (DeSmet, 1995), a German-language intelligent computer-assisted
language learning (ICALL) environment, embodies a role-playing detective game. In
this detective game, learners play the role of a police investigator to interrogate
German-speaking suspects via text inputs. The interrogation is a goal-oriented task
with the purpose of uncovering the true murder. The interesting settings are intended
for triggering the learners’ curiosity and therefore to extend practice times. Because
of the uncertainty of learner inputs, predication-driven parsing was used to avoid the
defects of nested rule-based systems, instead of the traditional production rules. A
learner performance profile records cumulative evidence generated by a fined-grained
syntactic analysis that inspects all the inputs from the learner. DeSmet (1995) argued
that the reason why this approach was adopted is because a generative learner model
lacks of semantic understanding on learners’ erroneous performance.
Not negligible are hundreds of online language practice games, which, limited
by the bandwidth, are mostly mini-games with simple graphical interfaces. One of the
most widely played word games is called Hangman, where an animation showing a
hanged man approaching to death is displayed whenever the learner types a
29
mismatched character into the corresponding blank position of the target word.
Hangman, though small, yet includes all features as a game, including fantasy,
challenge, interaction, and feedback (the hanged man’s distance from death).
However, Hangman is far from a complete language tutoring system. Neither is it a
good test bed to perform serious research on improving learning efficiency, as the
interaction patterns of learners are very limited due to the transient nature of game
sessions. In addition, Hangman is in form of extrinsic fantasy – a form which has
weak relationship with the skills being trained – which is generally regarded as less
motivational compared with games that employ intrinsic fantasy in which the
learner’s success of completing the game severely relies upon one’s mastery of skills
(Malone, 1981).
In contrast with the serious games, some COTS language teaching software,
such as the Rosetta Stone language learning suite software and Learning Company’s
Learn to Speak series, still follow the traditional memorization-by-rote method, while
the so-called mini games in their software are no more than flashcards that project the
drill-and-practice model (Squire & Jenkins, 2003).
Another trend of designing language learning software is the distance learning
platform. With the thriving of social networks these platforms are replenished with
new modalities. The new generation of Web and Voice-Over-IP technology now
enables more interactivity of user operations within the Web browser than through
online learning communities such as newsgroups and forums. Examples of these
systems include the SOFTS Tele-training System (Donnelly, 2007) and Speak2Me

30
(LaPointe et al, 2004). The former is a multi-user language learning platform,
sponsored by the US Special Operations Command, which allows a group of learners
to perform distance learning with human tutors. The latter is online English learning
system that allows the learner to carry on interactive conversations with a virtual
character to intermediate-level Chinese speakers and supports discussion groups. The
difference of distance learning platforms and serious games are that the former
usually involve more teaching resources especially human tutors.
2.2 Problems and Challenges in Serious Games

Contradictory findings were reported in literatures regarding the effectiveness of
serious games, and little agreement has been reached (Randel et al, 1992).
Some researchers criticized that certain serious games failed to conclusively
demonstrate their effectiveness (Conati & Klawe, 2000; O’Neil et al., 2005).
Although experiments and surveys have shown some game-based environments can
achieve advanced learning efficiency (Ricci et al., 1996; Jonnavithula L. & Kinshuk,
2005), others have found the games do not have positive influence on increasing
learning efficiency. Empirical evaluations (Klawe et al., 1995) demonstrated that a
commercial game-embedded system that helps learners solve algebra word problems,
Counting on Frank, was less effective to improve learners’ performance than
traditional spreadsheet-based exercises, despite the fact that learners spend most of
their time on mathematical learning activities in the game. Within the group which
played Counting on Frank, no significant learner attitude changes on math were
31
detected, either. Parchman et al (2000) also reported both the form of computer-based
drill and practice and the form of computer-based instruction outperformed a
computer adventure game learning environment on all measures, and under the game
condition, learners performed no better than under the classroom condition.
Some researchers argued that games do promote learning, but the conditions
when games can be effective should be under scrutiny. Specific guidance and
teaching methods are reported important to the learner performance in serious games
(Szczurek, M., 1982; VanSickle, R. L, 1986; Randel, 1992; Kirkpatrick, 1994).
Empirical evaluation (Klawe, 1998) on two mathematical puzzle games, Super
Tangrams and Phoenix Quest, shows that games can enhance learning when coupled
with active teachers or supporting classroom activities. Another evaluation
(Henderson, 2000) on a game that teaches science to Grade Two students also claims
that games should be paired with proper working environments where classroom
curriculums and practices are available, in order to teach higher order cognitive skills.
These studies imply that games incorporating classroom-like environments can result
in a better learning performance. However, this study cannot manifest that games can
support learning as the game playing activities did not separate the classroom
activities.
Some researchers propound to find out how different game features influence
learning (Lepper & Malone, 1987; Voderer & Hartmann, 2003), and to develop
guidelines for serious game design (Rieber, 2001). Malone (1981) proposes a
theoretic framework that consists of a set of heuristics to guide the design of game-
32
based learning environments. Important principles proposed in this work includes
designing personally meaningful goals and uncertain outcomes to make an
environment challenging, using intrinsic fantasies to make an environment more
instructional, as well as providing a moderate complexity of new, incomplete or
inconsistent information to make an environment curiosity-provoking. Garris and
others (2002) view the game design as an information flow cycle that processes both
game and instructional interactions. In an input-process-output model, a game
development cycle is defined as a continuous process where each element of user
judgments, user behavior, and system feedback can the next recursively. The model
emphasizes the key elements that happen in learner’s cognitive aspects of learning in
relation with the game features and the instructional contents. Although in
implementation it is never easy to reconcile game-playing with effective learning,
these theoretical frameworks provide a well-grounded baseline for serious game
design.
Certain researches were conducted on finding out what elements in games
may give rise to unproductive learning activities, and therefore making compensation
for these “harmful” elements. A major factor to be blamed is the distracting elements
in games (Conati & Klawe, 2000), because “the game is not integrated with external
activities that help ground the game experience into the learning one”. It is observed
that learners engaged in activities that are entertaining but do not contribute to
learning, when given the freedom to explore the game world and to take action as
they like. In addition, many games adopt reward systems that are indented for
33
immediate reinforcement. However, it is likely reward systems can be a non-trivial
distracting factor, especially for learners who fail to build up long-term intrinsic
learning motivation through game play. As a result, learners tend to be “obsessed
with progress, scores and other non-learning components in the game, to the
detriment of the content” (Prensky, 2001). The existence of these distracting elements
suggests that learning objectives in serious games should be congruent with the game
objectives, so that distraction which drags the learners away from real learning can be
diminished.
Another view examines the effects of learners’ cognitive properties on the
learning outcomes within serious games. Some learners may become discouraged
because they face challenges beyond their current level of competency of completing
demanded tasks. This type of learners is considered having low tolerance in
frustration, or low self-efficacy, and thus fails to react constructively to the game
stimuli (Quinn, 2005). Another type of learners that perform unproductive learning is
those who lack self-regulation. The work in (Baker et al., 2004 & 2006) describes a
type of learner who attempts to find the “shortcuts” to succeed in the education
systems by exploring the system properties instead of the learning materials. Their
experiments also showed that under control condition where intervention from a
computer simulated tutor to prevent the learners “gaming” the system was available,
the percentage of learners “gaming” the system dropped 3% (not much difference),
while learners achieved a seemingly larger pre-post gain compared with the group
under experimental condition where no tutor intervention was provided. These
34
studies suggest the tutor interventions are necessary for learners of some cognitive
traits, e.g., low self-efficacy and less self-regulated learners.
Psychological educationalists argued that it was the teaching component
within interactive media that primary contributed to learning outcomes (Clark, 1983).
Although earlier studies have provided positive evidence showing that computer-
based instruction (CBI) can improve learning, Salomon (1984) disclosed the
fundamental reasons of inverse relationships between interest and achievements
based on compelling evidence. He brought forward a hypothesis that students are
interested in newer media because they expect it reduces the load of learning. While
this expectation may not be true, as a matter of fact, it was observed that learner
tended to spend less “mental effort” in hope of the instruction of new media makes
their learning easier and fun. Mental effort is defined as “the number of non-
automatic elaborations invested in learning” (Salomon, 1984). Salomon’s hypothesis
was validated in recent meta-analytic studies (Bernard et al, 2004). Bernard and
others’ studies also showed that as achievements increased in multimedia learning
student interest decreased. This theory applies to learners in game-based
environments as well. Learners who initially hold high expectation on games to
deliver easier courses are likely to find the real experience later on is against their
intuition. Hence, their motivation slopes. Pintrich and Schunk (2002) proposed a
solution to address this problem. They suggest various measures should be used for
continuous assessment on learners’ motivation such as measures on self-efficacy,
affect, goals, and mental effort.
35
We have summarized the aforementioned problems and challenges identified
in the current education and serious game literatures into Table 1. Together listed as a
separate column is the possible solutions proposed, targeting at these problems.
Standpoing Experimental Results Related Research Possible Solutions
Control the learning
conditions when
serious games are
applied
Learning can be promoted
when coupled with
classroom activities
Szczurek, M., 1982;
VanSickle, R. L, 1986;
Randel, 1992;
Kirkpatrick, 1994;
Klawe, 1998;
Henderson, 2000
Encompass
classroom learning
activities in games
Design games
following certain
guidelines, principles
or frameworks
Game features such as
challenge and fantasy can
influence learning outcomes
Lepper & Malone,
1987;
Voderer & Hartmann,
2003;
Rieber, 2001;
Malone, 1981;
Garris et al., 2002
Consider the
relationships of
game features and
learning in the
design stage
Find out unfavorable
elements in games
and make
compensations
Some games have
distracting factors and
incongruent objectives that
jeopardize learning
Conati & Klawe, 2000;
Prensky, 2001
Establish game
goals/rules
congruent to the
learning objectives
Track learners
cognitive properties
and enable tutor
intervention
Low self-efficacy and less
self-regulated learners do
not engage in learning
activities. These behaviors
can be reduced if tutor
intervention is provided
Quinn, 2005;
Baker et al., 2004 &
2006
Provide
suggestions/advice
during the learners’
unproductive
learning activities

Continuously monitor
the leaner’s motivation
Students with high interests
spend less mental effort and
obtain lower-level
achievements, and when
their achievements become
higher, they interest and
satisifaction decrease.
Salomon, 1984;
Bernard et al, 2004;
Pintrich and Schunk,
2002
Use various
measures to assess
learners’
motivation in
multiple
dimensions

Table 2.1 Summary of literature reviews on problems about serious games

2.3 Evaluation Models

36
Evaluation models of training having been developed over years in both academia
and industry have many kinds. One of the commonly used is the classic Kirkpatrick
model
1
. It outlines four levels of training effectiveness (1-4): reaction, behavior,
learning, and results (Kirkpatrick, 1994). Reaction refers to as how participants react
to the training system. Learning refers to as to what extent trainees acquire knowledge
and skills as a result of training. Behavior refers to as what extent trainees would
change their behavior in the real field as a result of the training. Results refer to as the
organizational results benefits from the training. Each level is linked to its preceding
or succeeding levels.
A number of critics made on the Kirkpatrick framework, one of which this
study concerns is that it “fails to take account of the various intervening variables
affecting learning and transfer” (Tamkin et al, 2002), when we wish to know the
factors that cause the low productivity of learning.
The CRESST model
2
of learning (Baker & Mayer, 1999) is another
commonly used evaluation method. It is composed of five families of cognitive
demands, which are content understanding, collaboration or teamwork, problem
solving, communication and self-regulation. Each family defines a framework for
design of instructions and testing, and can be measured either quantitatively or

1
This model was originally proposed in 1959.
2
The full name of CRESST model is National Center for Research on Evaluation, Standards, and
Student Testing (CRESST).

37
qualitatively. Because the CRESST model conforms to cognitive learning theories
and addresses features of modern training systems, some researchers advocate using
this model to evaluate the learning outcomes of educational software or serious games.
However, the CRESST model addresses little on learners’ affect status influenced by
game stimulus as a motivator to increase the learning engagement over time.
Therefore, some researchers recommend to use an augmented version with affective
evaluation included or an augmented version with affective or motivational views
included (O’Neil et al, 2005).
In the language training domain, the Defense Language Aptitude Battery
(DLAB) is designed to determine what kind of learners may pursue training as a
military linguist (Ellis, 1994). DLAB is often used for aptitude test. However, the
DLAB does not engage subjects in speaking the language, or using it for face-to-face
communication in culturally appropriate ways. Therefore, it is not suitable for
evaluating training systems intended for spoken communication tutoring.
2.4 Summary

Many serious games in the past decade have been developed for various educational
purposes such as job training and language learning in a wide range of domains, such
as military defense, business, healthcare, and organizational training. The work
enables extensive experiments and in-depth studies on serious games. Although the
positive effect of serious games seems promising, studies conducted by different
parties reported the inadequacy of using game techniques to achieve an effective
38
learning due to the distracting factors and unproductive learning behaviors. Therefore,
some researchers advocate treating serious games merely as a supplemental tool
paired with the classroom learning activities. Others suggest introducing pedagogical
service to help the learners exploring the game contents. Yet the proposed solutions
are needed to be elaborated and carefully crafted to attain the desired improvement.
Meanwhile, traditional evaluation models are less sufficient for evaluating the
productivity of serious games, especially for those systems which teach face-to-face
spoken communication languages.
39
Chapter 3

Detecting Unproductive Learning Activities

Productivity is an important measure of training efficaciousness, which assesses the
positive learning outcomes against inputs. Productive training systems are ideal for
modern organizations that suffer tight schedule and scarce resources when training
employees for urgent deployment or even life-critical tasks. Many people believe that
unproductive learning activities are inevitable in training because traditional training
systems fail to engage the learners, consuming excessive training time and resources
but yielding fruitless learning outcomes. However, we argue that these activities may
occur even in motivating systems such as serious games which successfully engage
learners. We will identify these unproductive activities through a case study in this
chapter.
3.1 A Case Study on the TLCTS System

We focus on a case study about the Tactical Language & Culture Training System
(TLCTS). This system is an excellent test bed for investigating the aforementioned
research questions. First of all, TLCTS possesses a full-fledged game environment
that implements multimodal interactions, exuberant curriculums, and rich graphical
interface. It is far more than a research prototype or demo application. Secondly,
TLCTS has a large group of users both from the military and civilian sides. It is a real
40
system used by real training, and is popular among military personnel deploying
overseas. Over ten thousand copies of TLCTS courses have been distributed nation
wide, and tens of thousands of learners have used it to date. The number is still
increasing. Hence, quantitative data analysis is feasible to evaluate, classify, and
summarize the learning outcomes. Thirdly, the TLCTS incorporates two games of
different types. It offers opportunities to find out what coaching strategies are specific
to the domain and what are for the game. Comparison between unproductive learning
activities occurring in different types of games expands breadth of this research. Last
but not least, this study has the full access to the source codes
3
, so revised design
approaches can be implemented and validated under control.
TLCTS is a second language acquisition system that is embodied in a serious
game form (Johnson et al, 2004). It helps learners quickly acquire knowledge of
foreign language and culture. The system has a number of instances that teach
different target languages. Tactical Iraqi
TM
teaches colloquial Iraqi Arabic, and is the
most widely used. Rez World
TM
teaches Cherokee, which is a Native American
language spoken in the southern United States; Tactical Pashto
TM
teaches Pashto,
Tactical Dari
TM
teaches Dari spoken in Afghanistan, ISpeakChinese
TM
teaches
Chinese Mandarin, and Tactical French
TM
teaches French as spoken in sub-Saharan
Africa. Different versions have been created for military forces in the United States,
German, Australia, and other countries. In addition, TLCTS also have instances that

3
I participated the software development in the Alelo TLT LLC.
41
include only civilian courses. All of these instances built upon a unified, extensive
game-based system architecture, which incorporates three learning environments: (1)
a set of interactive lesson modules that help develop basic cultural and language skills,
called the Mission Skill Builder (MSB); (2) a 3D role-playing game where learners
can explore freely and converse with game characters in simulated social settings,
called the Mission Practice Environment (MPE); (3) a 3D mini game, where learners
can practice their listening and speaking skills in first-person interaction mode, called
the Arcade Game (AG). Normally, learners first develop spoken communication
skills through a combination of interactive lessons that focus on particular skills, and
then practice the newly acquired skills in the interactive games.
In the following subsections, we will take a look at the typical interactions
with the three learning environments. The instance of the TLCTS system used as the
baseline is Tactical Iraqi
TM
Version 3.0.
3.1.1 Typical Interactions with the Mission Skill Builder

The Mission Skill Builder (MSB) environment contains interactive lesson pages,
review quizzes, passive and active dialogues. Lesson pages have the following types:
• Cultural instruction pages, which introduces cultural knowledge (social
customs, gestures, taboos, etc.)
• Language instruction and practice pages, which teaches the utterance
pronunciation, syntax, and semantics and allows the learner to practice the
utterance pronunciation.
42
Figure 3.1 shows a language instruction and practice page, where learners
perform interactive exercises, speaking into the computer and getting feedback on
their choice of responses and pronunciation. A typical user interaction in the context
of a learner learning to say “hello” in the target language could be like this. Learner
first hears the pedagogical agent pronounce the phrase, and then attempts to repeat it.
Learner’s speech is recorded and will be replayed.

Figure 3.1 A Language Instruction and Practice Page the Mission Skill Builder

Pronunciation errors are detected and motivational, corrective feedback is
provided in this case. Possible responses would be: "It seems that you said ‘salaamu’.
Please try again."
43
• Memory pages, which mainly test if the learner can speak the newly taught
utterances given the corresponding English words or phrases.
• Exercise pages, where learners can use their newly acquired language skills to
solve simple problems. Exercise pages include utterance formation, multiple
choices, and match exercise pages.

Figure 3.2 An Utterance Formation Page in the Mission Skill Builder

On an utterance formation page (Figure 3.2), a learner is expected to first
recognize the utterance spoken by a virtual character, and then to construct or produce
an utterance that is an appropriate response. Feedback is given when the user
confirms inputs.
44
For example, if the virtual character says “as-salam alaykum” (which means
“hello” in English), and the learner responds with “wa 9aleykum as-salaam” (which
means “hello back” in English), then the feedback is displayed as “Sounds like you
said ‘wa 9aleykum as-salaam’. That is correct.” (Figure 3.2)
Two types of dialogs were created inside MSB, as follows:
• Passive dialogs, where learners can watch conversations between two game
characters going on without risk of making mistakes. The passive dialogues
employ a game specific tutoring approach called “fish tank” (Gee, 2003;
Johnson et al, 2005), and usually are demonstrated in the beginning of lessons.
• Active dialogs, where learners can play a virtual character, having
conversations using the utterances taught with the current lesson. The active
dialogs adopt the “sand box” technique (Gee, 2003; Johnson et al, 2005),
serving as warm-up exercises before the learner plays complex game scenes.
In addition, MSB in Tactical Iraqi
TM
Version 3.0 has more than 40 lessons.
Each lesson is coupled with a review quiz. One quiz spans several pages. Quiz pages
are just like normal exercise pages, except that in quiz mode checking previous
instruction pagesis prohibited. Quiz scores are the percentage of correct answers
against the total number of questions. These scores are recorded in logs as indicators
of learner performance.
3.1.2 Typical Interactions with the Mission Practice Environment

45
The Mission Practice Environment (MPE) is the role-playing video game portion of
the TLCTS, including multiple mission scenes, where the learner controls a character
that can walk around in a free-exploring environment and converse with non-player
characters (NPCs). Learners can also select gestures by rolling the mouse wheel to go
along with their speeches. A typical interaction of the learner with the NPCs can be
described as follows: the learner listens to a speech said by one of the NPCs, and then
selects from a palette of gestures and toggles to the speaking mode by right-clicking
the mouse button. Then he/she can speak into a microphone to respond, and end the
speech with a mouse right click.

Figure 3.3 A Scenario of a Café Place in the Mission Practice Environment

46
Figure 3.3 displays a scene taking place in a café, where the learner’s
character is talking to an NPC. A simulated tutor is standing behind the learner’s
character playing a role of aide, who can suggest a specific foreign phrase to say, or
give a hint in English and let the learner decide how to say it. The learner’s task is to
find the address of the local leader. To complete this task, the learner needs to
convince the local people that his mission is peaceful and to build rapport with them.
After that, he can ask where to find the person in charge of the town. One of the
possible bad endings are the learner gets questioned and confronted by the local
people inside the café, if the learner speaks or acts rudely, or if he fails to gain trust
from others.
The learner can choose the difficulty level of a scene by logging in as a
beginner or experienced user. At the beginner level, the learner is allowed to
experience the scene while he can request hints in English from the aide. Also, the
system will delegate the learner’s speech to the aide in order to proceed the
conversation if the system detects the learner gets stuck at some point. At the
experienced level, the aide only provides hints of what to say next in Arabic. The
personalities and goals of the virtual characters will be altered, e.g. the arousal level
of the young man in the café will be increased so that a few errors in the learner’s
speech and manner would trigger him to stand up and point at the learner.
Each mission scene has a set of tactical objectives which decompose the task
into different subtasks. The learner can press a hotkey to view the progress made so
far toward objectives, and adjust their learning plans on a regular basis to complete
47
these objectives. For example, the first mission scene has the following objectives in
order: (1) Build trust with a local; (2) Get directions to man in charge; (3) Follow the
directions. Completion of later objectives relies upon completion of the preceding
ones. Therefore, unless a learner can achieve the previous objectives (by speaking the
correct sentences), he/she is not allowed to play the next plots, and therefore is not
granted the chance to skip more than one objective.
3.1.3 Typical Interactions with the Arcade Game

The Arcade Game environment trains the skills of giving and receiving directions,
landmarks, numbers, military ranks, and colors (Figure 3.4). These words and phrases
are all covered in the core lessons of MSB.

Figure 3.4 Arcade Game Menu that Lists Four Vocabulary Categories

48
The Arcade Game incorporates two learning modes: listening mode and
speaking mode. In listening mode, the learner listens to a voice command, and moves
the character which they play to the desired position on the map. The learner can pick
up a reward (Figure 3.5) if the character moves to the right place and get positive
points. A request of hint will get the learner’s total points deducted. In speaking mode,
the learner is expected to use speech to control the character to pick up rewards or
avoid enemy attack. Similar to the listening mode, he gains points if he speaks the
right command that controls the character to successfully pick up rewards, or looses
points if he uses hints or gets killed by a flying enemy.

Figure 3.5 Arcade Game Speaking Mode

49
3.2 Study Procedure
3.2.1 Experiment Purpose

The purpose of this experiment is to explore whether the example system has positive
effect on learning for massive learners working with it over a certain period of time,
and to explore the characteristics of unproductive activities, if there are any, captured
in training session logs.
3.2.2 Method

We used the Tactical Iraqi
TM
Version 3.0 itself to assess the learning success in
interacting with the system. Comparison study was used for this experiment. The
focal group studied the TLCTS system without intervention from supervisors. The
comparison group was accompanied with a supervisor who offered suggestion like
which part of the system the learner should focus on given their learning progress.
The study procedure is organized as follows:
1. Laying out the experiment settings, and installing Tactical Iraqi
TM
Version 3.0
on all lab computers.
2. Giving orientations to the subjects, and launching the experiment.
3. After the training, collecting data including the learner profiles and system
logs from the server.
4. Summarizing the session data to compare the time distribution of each group
within each environment.
50
5. Classifying the productive and less productive learning activities, extracting
the learner’s behavior patterns in the system logs, and Exploring key findings by
comparing these data.
3.2.3 Experiment Settings

A comparison study was conducted, which involved a total of 801 personnel from
two battalions (the 2/7 Marines and 3/7 Marines) who participated the training at the
Sim Center of 29 Palms. 291 learners of 2/7 (Group #1) and 384 learners of 3/7
(Group #2) experienced a training session that lasted for a two week period. Each
learner worked with TLCTS for approximately 10 hours per week. Group #1 learners
performed self-paced learning without a supervisor; Group #2 was monitored by a
supervisor in lab. Due to limitation of hardware conditions - the Sim Center has two
computer labs of about forty computers - the two groups would be trained in turn.
Group #1 was trained between June and September 2006, and Group #2 between
September and November 2006.
The curriculum selected for this assessment would introduce the candidates to
aspects of the phonology, morphology, syntax, and pragmatics of Iraqi Arabic. The
materials also would present non-verbal gestures and other cultural information
relevant to face-to-face encounters with people in Iraq.
The MSB lessons, which the subjects were expected to complete, are Getting
Started (a tutorial), Meeting Strangers (vocabulary, phrases and etiquette relating to
meeting strangers, possessive morphological endings), Introducing Your Team
51
(Arabic terms for military ranks, phrases and etiquette relating to making
introductions, definite articles, demonstratives, grammatical gender and agreement),
Explaining Your Mission (declaration of purpose, Arabic terms and phrases relating to
describe occupations and military units), and Getting Directions (articles, Arabic
terms for orientations and directions, and question formulation),
The MPE scenes, which the subjects were expected to play and complete, are
Tutorial Scene (Introduction to manipulating user interface), Find Your Way to the
Person in Charge (Build trust with the local, get directions to man in charge, and
follow the directions), Visit the House of the Person in Charge (Introduce yourself,
ask directions, and locate the person in charge), and Introduce Yourself to the Person
in Charge (Meet Jasim, and build rapport with Jasim).
The designated Arcade Game levels are Level 1 - Basic directions, Level2 -
Cardinal directions, and Level 3 - Using directions with colored landmarks, including
both the listening and speaking mode.
The subjects received orientation and were directed to complete the tutorial
lesson and scene before they started the formal curriculum. They were told that at the
end of the learning session, they were supposed to complete the MSB lessons to build
up their basic knowledge components as well as to complete all the required MPE
scenes and AG levels. Learning profiles were synchronized and uploaded to the
server where the supervisor can monitor the learners’ progress through a server-side
application of TLCTS called Dashboard.
52
3.2.4 Result and Key Findings

Table 3.1 summarizes the learning data of the two comparison groups. In this study, a
session is defined as the time interval between when a learner signs in the system and
when he/she signs off.)
Group #1 Group #2
With Supervisor? No Yes
# of subjects participated 382 419
# of sessions 817 450

Table 3.1 Experiment Data Overview

All the subjects of both groups have practiced the MSB lessons (N
1
=382;
N
2
=419). Averagely subjects of Group #2 spent more time (the numbers below are
represented in hours) on MSB than subjects of Group #1 did (M
1
=3.32, SD
1
=2.65;
M
2
=8.21, SD
2
=3.07, as depicted in the top right diagram in Figure 3.6). Also, subjects
of Group #2 started more lessons than subjects of Group #1 did (M
1
=3.32, SD
1
=2.65;
M
2
=8.21, SD
2
=3.07, as shown in the bottom left diagram in Figure 3.6), and
completed more lessons than subjects of Group #1 did (M
1
=1.92, SD
1
=1.14; M
2
=4.33,
SD
2
=2.89, as shown in the bottom right diagram in Figure 3.6. Left column is Group
#1, and right column is Group #2).
53

Figure 3.6 Learning Statistics of the Mission Skill Builder

Nevertheless, Group #2 has more subjects practiced on MPE in proportion
(N
1
=374, P
1
=97.9%; N
2
=227, P
2
=60.7%), spent more time on MPE averagely, and
started more scenes averagely, compared with Group #1 (Figure 3.7). An interesting
finding here is that despite of more scenes having been tried by Group #1, subjects of
Group #2 actually completed more scenes (M
1
=0.66, SD
1
=0.58; M
2
=2.23, SD
2
=1.97
as shown the bottom right diagram). Comparison showed that Group #2 users were
more productive with the same amount of time.

Unsupervised Supervised
54

Figure 3.7 Learning Statistics of the Mission Practice Environment

Similar findings were detected in the training log data associated with the AG
levels (Figure 3.8). A level is regarded as completed when a learner scores 1000 or
above in the listening mode, and scores 800 or above in the speaking mode. Group #2
subjects appeared to have performed more productive learning given the amount of
training time and curriculum units which have been started.

Unsupervised Supervised
55

Figure 3.8 Learning Statistics of the Arcade Game

These findings conform to the key findings presented in a final report to the
Special Operations (Surface et al, 2007). Moreover, recommendations were made by
that report as to use the TLCTS system as a supplemental tool for language learning
paired with regular classroom teaching.
Distinct productivity of these two groups suggests that learners of the
controlled and treatment groups developed different learning behaviors. It is obvious
that Group #2 subjects showed potentially more productive learning with supervisor’s
Unsupervised Supervised
56
intervention. We then took one step further by examining the characteristics of these
behaviors in the training log data.
3.2.5 Behavior Patterns of the Unproductive Learning Activities

Further analysis on the training logs of Group #1 revealed that the subjects of this set
had performed various kinds of activities that contribute less to learning. Some of the
activity sequences repeatedly occur in the training log, suggesting possible behavior
patterns existing in interacting with the TLCTS system.
Three independent unproductive learning activities were identified by
examining the learner’s behavior patterns. The following abbreviations are used for
convenience of notation:
• P: productive learning behavior
• AU: actual unproductive learning behavior
• PU: potential unproductive learning behavior
3.2.5.1 The Retrospect Pattern

Table 3.2 presents an example of interaction sequence intercepted from system log
about a learner’s learning events. It mainly records a situation when the learner
acknowledges his/her lack of prerequisite skills, and switch back to search for MSB
lessons that can improve his/her skills.
We can see from the sequence that this learner attempted to complete the first
mission scene (Task #8). Toward that, he spent some efforts to learn the prerequisite
57
language skills in Lesson 1 (Task #1). Particularly at Task #3, #4 and #5, he restarted
the mission scene twice and used many hints to help him get through the scene, but
failed consecutively three times. (The first objective in Scene 1 is easy to complete, as
long as a learner can say “hello” in Iraqi. Hence, the completion of the first objective
hardly indicates progress.) Then he began to recognize that his current skill level was
not enough to conquer the scene, so he chose to go back to study more lessons (Task
#6 and #7). When he tried the mission scene at Task #8, he succeeded finally.
However, he wasted time on going back and forth between the scenes and lessons.
This learner did not stop when he finished the first scene, moving on to the second
scene (as shown in Task #5). Instead of horning his skills for conversing with the
NPCs, he chose to “peep at” the speeches that can lead to the successful ending.

Task
#
Curriculum Unit Duration
(min)
Correct
Speech
Attempts
Total
Speech
Attempts
Exercise
Score

Quiz
Score
1 Lesson1 25.600 22 26 100% 0
Task
#
Curriculum Unit Duration
(min)
Speech
Attempts
Hints
Requested
Objectives
Completed
Mission
Completed
2 ActiveDialog1 1.700 3 3 1/1 Yes
3 Scene1 9.400 7 7 1/3 No
4 Scene1 13.600 20 20 1/3 No
5 Scene1 5.600 8 8 1/3 No
Task
#
Curriculum Unit Duration
(min)
Correct
Speech
Attempts
Total
Speech
Attempts
Exercise
Score
Quiz
Score
6 Lesson1 5.633 2 6 0 83.3%
7 Lesson2 27.367 27 42 25% 0
Task
#
Curriculum Unit Duration
(min)
Speech
Attempts
Hints
Requested
Objectives
Completed
Mission
Completed
8 Scene1 10.133 11 8 3/3 Yes
9 Scene2 19.450 11 22 0/2 No

Table 3.2 Interception of Training Log Data

58
The next step of analysis is to classify the productivity of the 9 learning
activities in Table 3.3. We can see from the table that Task #1 through Task #8, 3 out
of 8 learning activities are unproductive. When scanning Task #9, we have found that
4 out of 9 unproductive learning activities, and this learner had wasted a little more
than 25% of the total time spent on tasks towards completing the first mission scene.
Task # CU Description Productivity
1 Lesson1 Complete all exercises and skip the quiz P
2 Active
Dialog1
Complete without hints P
3 Scene1 Fail to complete the scene with a few attempts PU AU
4 Scene1 Fail to complete the scene using a lot of hints AU
5 Scene1 Fail to complete the scene with a few attempts AU
6 Lesson1 Complete the quiz left from Task #1 P
7 Lesson2 Complete a part of the exercise pages and skip the quiz P
8 Scene1 Complete the scene with a moderate use of hints P
9 Scene2 Fail to complete the scene using a lot of hints PU

Table 3.3 Log Analysis of the Retrospect Pattern

Here, CU stands for a curriculum unit. It can be either a MSB lesson, or a
MPE scene, or an AG level. Task #3 initially exhibited as a potential unproductive
learning behavior (PU), and was tagged as actual unproductive learning behavior (AU)
when Task #5 was detected.
The following is an example of the algorithm which we used to tag
productivity to an activity. We examine a smaller window that consists of 2
consecutive tasks. After Task #3 and before Task #4, we should have enough
information to determine that the learner was doing unproductive learning activities,
because the log indicates that this learner failed at the same scene twice. Especially at
the second attempt the learner heavily used hints to get through the scene, which
indicates that his skill level was not proficient enough to complete this task. Therefore,
59
we can safely convert the previous tagged PU task to AU task. We argue that if a
tutor intervention is available, it should be offered at this particular time. After Task
#9 is done and before the next task begins, evidence is scare so we can hardly
determine if it is an actually unproductive learning task, so the label remains PU for
the current time.
Despite the 25% time in vain, this learner is still a self-regulated learner,
because he finally constructed a learning sequence on his own to complete the
mission scene.
We name this pattern as Retrospect Pattern, which describes a sequence of
activities where learners search for curriculmns which can contribute to their learning
objectives.
On the fifth training day, this learner was able to complete the first three
mission scenes. Then he tried the more advanced scenes by abusing the hint facilities
(21 hints requested vs. 21 speech attempts in Task #7).
Situation of Task #8 and #9 is quite different. The learner tried a more
advanced scene immediately after he finished the previous scene. This is actually a
small example of the next pattern to be discussed.
The generic case of this pattern is summarized in a sequence diagram (Figure
3.9). We conducted this step because it is useful to design algorithms that automate
the pattern detection.
60

Figure 3.9 The Sequence Diagram of Retrospect Pattern
3.2.5.2 The Onrush Pattern

Table 3.4 describes a case where the learner devoted substantial time in playing the
mission scenes. This is a case when the learner lingered in the MPE environment
even if he could not make further progress, rushing to complete advanced tasks
beyond his current skill level.

Table 3.4 Log Analysis of the Onrush Pattern

Task
#
CU Duration
(min)
Speech
Attempts
Hints Objective
Completed
Mission
Completed
Productivity
1 Scene1 10.900 24 8 2/3 No PUP
2 Scene1 21.967 35 9 3/3 Yes P
3 Scene2 9.350 24 6 1/2 No PUP
4 Scene2 13.817 18 5 2/2 Yes P
5 Scene3 5.150 10 5 2/2 Yes P
6 Scene4 10.733 21 21 0 No AU
7 Scene5 5.183 11 5 0 No AU
8 Scene5 1.267 3 2 0 No AU
9 Scene5 6.75 4 1 0 No AU
Not making enough efforts in MSB,
usually skipping exercise pages and
quizzes or scoring low in quizzes
Playing a mission scene in
MPE and failing to progress the
conversation unassistedly
Switch to
MPE
MSB
begins
Switch
to MSB
Repeating
meaningless speech-
acts but not using
any help facility
Abusing the help
facility and
relying on hints to
continue the
conversation
Not completing
objectives after a long
interval or many
speech attempts
61
Scene 3 requires approximately the same skill set as Scene 1, and the learner
passed quickly. However, Scene 4 and Scene 5 require advanced language skills in
negotiation, and the learner’s continuous failure on these scenes indicates that he had
not mastered these skills. Unlike the previous example, this learner never resorted to
MSB lessons or formulated a learning plan at the end of that training day. As Task #2
and Task #4 turn out to productive, we update the previously labeled “PU” tasks to
productive ones as well. As a result, in this session 44.44% of his learning activities
are unproductive, accounting for 26.51% of his total time.

Figure 3.10 The Sequence Diagram of Onrush Pattern
We name this pattern as “Onrush”, because learners behaving like this pattern
rush to complete all content materials before they are ready for them. They never had
Completing simple
scenes
Playing more advanced scenes but
failing to progress the conversations
Continue
playing
MPE
MPE
begins
Continue
playing
MPE
Repeating
meaningless
speech-acts but
not using any help
facility
Abusing the help
facility and
relying on hints to
continue the
conversation
Not completing
objectives after a
long interval or
many speech
attempts

62
a chance to recognize their inadequacy in skill development. Usually this kind of
behaviors pairs with heavy use of the hint facilities.
Figure 3.10 describes this pattern in a sequence diagram. Based on the
diagram the computer program was written to capture the pattern detection when
scanning the training logs.
3.2.5.3 The HelpRefusal Pattern

Table 3.5 shows another behavior pattern found in the system logs. In this case, the
learner makes a number of speech attempts but cannot achieve an objective and
refuses to use hints.

Table 3.5 Log Analysis of the HelpRefusal Pattern
After 7 minutes passed since start of Task #1, this learner failed to achieve a
single tactical objective. Neither did he use any hints to help himself. In Task #1, this
learner used the hints once, which implies that he knows the availability of the hint
facility. But he had ceased using hints after that. It is likely that he was formulating a
approach to complete the game by blind guess. This behavior has been described as a
“help refusal” type in (VanLehn, 2006): a learner who refuses to use the hint even
when he needs it.
We utilize VanLehn’s work and name this behavior pattern as HelpRefusal
Pattern. The characteristic of this pattern is that in a long time interval the learner
Task # CU Duration
(min)
Speech
Attempts
Hints Objective
Completed
Mission
Completed
Productivity
1 Scene1 7.000 3 1 0/3 No AU
2 Scene1 7.683 7 0 0/3 No AU
63
remains in low performance and turns away from help facilities or auxiliary learning
materials.

Figure 3.11 The HelpRefusal Pattern
This behavior does not “straddle” across tasks, because the behavior does not
generate productivity to the current or next task. Again we use the pattern language to
describe it (Figure 3.11).
In Group #1’s data, there are 274 AG levels where learners scored lower than
3000 points. The median score for Arcade Game levels is about 8000 points. Scoring
below 3000 points is considered a failure. In 232 out of 274 cases, learners requested
zero hints. In Group #2’s data, hint refusal patterns account for 36 in a total of 41
cases. This implies in most of the low performance situations learners did not use the
hint facilities.
We did a comparison among learners who behave in this pattern in both game
environments. The results showed the two sets have only a few overlapped. The
learner who refuses using hints in the MPE games could use a moderate number of
Playing mission scenes
but failing to progress
the conversations
MPE
begins
Continue
playing
MPE
Not complete any
objectives after a long
time interval or many
speech attempts
Refusing the
help facility
64
hints in the Arcade Game, and vice versa. In other words, learners may exhibit
different behavior patterns in different types of games.
3.2.6 Behavior Patterns of the Productive Learning Activities

Besides the pattern extracted from the unproductive learning activities, we also
present what is the most effective method used when a learner fails to recall the
utterance in Arcade Games. This is the behavior patterns which we wish the learner’s
behavior can be transferred to.
Table 3.6 describes the way how those types of learners succeed. The learner
of this example acquired skills from hints in the listening mode, and then applied
these skills in the speaking mode.
Task
#
Duration
(min)
Level Mode
Total
Speech
Attempts
Total
Voice
Command
Score

Hints Productivity
1 5.267 2 Listening 0 33 11050 12 PU P
2 6.067 2 Speaking 24 0 13300 0 P

Table 3.6 The Productive Learning Activities

Level 2 is practice on cardinal directions. This learner used 12 hints in the
listening mode against the total 33 times of system voice commands received,
indicating he had not master the skill to understand cardinal directions at this point.
However, after enough practice in the listening mode, he had acquired this skill, as in
the following task which is the speaking mode of level 2 he could utter the correct
speech without using a single hint. The result manifests a remarkable progress. We
65
found that 2 cases in Group #1 belonged to this category, against a total 51 cases of
high hint requests (requested hints > 10). In Group #2, it is 8 out of 14 cases.
We regard the example in Table 3.6 as productive, as it yields positive
learning outcomes in the next task, although it may not be efficient. Because this
learner also spent time in the MSB lessons that teach the cardinal directions in the
target language.
In contrast, Table 3.7 presents a case where the learner practiced Arcade
Game level 1 listening mode without using any hints. Log data show that this learner
had mastered the relevant utterances he learns with the MSB, and succeeded in arcade
games of different modes. He continually achieved a high score in the speaking mode.
This is an example of efficient learning in an ideal way: a learner is capable of
planning the learning tasks and is able to obtain a high level of knowledge transfer
and retention.

Table 3.7 The More Productive Learning Activities

Because these three patterns are not overlapping, a detection program was
created to computer the frequency of these patterns. The program scans the training
logs (XML files), and counts the activity sequence that matches the sequence
diagrams of the three behavior patterns.

Task #
Durat
ion
(min)
Level Mode
Total
Speech
Attempts
Total
Voice
Command
Score

Hints Productivity
1 6.283 1 Listening 0 46 13650 0 P
2 4.783 1 Speaking 69 0 16250 0 P
66
3.2.7 More Statistics about Unproductive Activities

After the first scan over the log data was done when patterns were extracted, we then
performed a second scan for statistical analysis using the algorithms described in the
sequence diagrams. As a result, wee found the Retrospect pattern occurred in 50%
and 22% of the training days for Group #1 and #2, respectively. At the same time, the
Onrush pattern occurred in 18% and 9% of the training days for Group #1 and #2,
respectively. For the HelpRefusal Pattern, the statistics is 14% and 13% (Figure 3.12).
The less frequency of the three patterns may be caused by the supervision condition
of Group #2.

Figure 3.12 Percentage of Unproductive Activities Against the Total Activities

We also did a rough assessment about the learners’ allocation of learning time.
The results are striking. Within Group #1, 69.99% of the learners’ total learning time
was spent on unproductive activities, making no progress at all during MPE games.
Within Group #2, the percentage is 64.68%, which doesn’t show much difference.
Group #1

Group #2

67
This has reinforced our belief of the necessity of offering specific tutoring assistance
during gameplay.
As for the AG levels, we only found less than 20 and 5 cases of the Retrospect
pattern for each group, respectively. For the Onrush pattern, as the later success in the
game may convert the previously tagged PU activities to productive one, there are
even less cases having been found by the scan.
Naturally, the following questions emerged. Why were there less Retrospect
and Onrush patterns found in the AG log? Does the statistical result imply that
learners were doing better in AG-like environments than in MPE-like games? Or is it
because the AG levels are less challenging than the MPE games?
We argue that this is because even these unproductive activity patterns will
yield different degrees of productivity, and therefore learners of different
characteristics are inclined to exhibit more of one pattern than the other. Onrush
Pattern is generally less productive than Retrospect Pattern. Therefore, a self-
regulated learner is more likely to behave as Retrospect Pattern; whereas a less self-
regulated learner tend to get over the mission task by abusing the hint facilities and
thus exhibit the Onrush pattern.
3.2.8 Discussion on Behavior Patterns

Serious games allow learners to fully control their learning pace. The problem with
this approach is that most learners do not have the knowledge about the curriculum
materials as the tutors do, and therefore are likely to select the task inappropriate to
68
one’s current development level. Learners also tend to overestimate their skills and
play games forward prematurely. In the worse case, learners completely wasted time
on exploring the game environments rather than progressing step by step.
The appearance and frequency of the unproductive behaviors in the
experiment data suggest several things. First the unproductive behaviors are not a
neglectable part of the learning activities. With more than 50% of the learning time
spent on the unproductive behaviors within the MPE games, and more than 20% low-
scoring games of the total games played within the Arcade Game, we cannot bypass
these cases if the purpose of using games is to promote learning effectiveness. We
have built a fun and enjoyable learning environment that we hope to evoke the
learners’ motivation, but we cannot posit the falsehood that learners are always
engaged in both playing and learning. As a matter of fact, a large part of their
activities did not yield positive learning outcomes, as the in-game indicators (such as
scores) have demonstrated. Unproductive behaviors signal a demand of tutoring
intervention, whether learners consciously acknowledge it (as in the Retrospect
pattern), or do not consciously recognize it (as in the Onrush pattern), or even
intentionally evade it (as in the HelpRefusal pattern).
The second issue that the data inform us is that learners have distinct
behaviors in games of different properties. From a gamer’s perspective, the role-
playing MPE games are more complicated and more social compared with the Arcade
Game. It incorporates a story emerging from the interaction between the learner and
the NPCs. Learners need to plan their speeches and gestures in order to achieve the
69
tactical objectives. However, this kind of game does not explicitly tell the learner if
he/she is a winner or a loser. Neither does it impose pressure on response in a timely
fashion. Therefore, the game playing of MPE is less tense than that of AG. Arcade
Game does not have stories, and the character controlled by the learner walk around
in the game world alone. In the MPE, however, an aide agent is incarnated as the
learner’s company, through whom the learner can obtain translation or suggested
speeches to carry on the conversation. Both the learner and the aide agent can take
initiatives to present hints, whereas in the AG it is the learner who decides to use the
hints or not. Therefore, MPE may seem more intriguing and motivating to many
learners.
From the language training perspective, the speeches involved in the
conversations of the MPE games contain more words and phrases. To advance the
conversation, the learner is required to master a higher level of skills in terms of
pronunciation, comprehension, production and recall than the level of skills required
in the AG. Furthermore, learners can go through the AG games relying solely on
memorization by rote, as the voice commands repeat during the games, and the
speeches the learner is suppose to say are limited.
The difference of design of these two games also explains the distinction of
learner behavior patterns in interaction. In the MPE, even though the aide agent can
offer hints on the expected speech in English and Arabic, the learner would not be
able to memorize it if he/she did not build up enough skill level from the MSB
lessons due to the complexity of the speech. In the AG, however, speeches are simply
70
and associated to external graphics objects. Learners can master these speeches more
efficiently (especially in the listening mode) through using hints than studying MSB
lessons. Hence, Retrospect Pattern was rarely detected in the AG data.
Moreover, learners were more likely to get distracted and did unproductive
learning in the MPE, because walking around the game world, watching the NPCs’
activities or even summoning the aide agent for hints could be interesting experience,
although this experience contributes little to productive learning. The high frequency
of Onrush pattern displays that the learner enjoys playing games regardless of their
consistent failure to achieve the tactical objective, while it is less manifested in the
AG logs.
Lastly, HelpRefusal Pattern is shown in logs of both games. Do they arouse
for the same reason? We argue the game properties again play an important role in
making the distinction. In the MPE, it could take long time for the learner to figure
out what the NPC expects him/her to say, which decreases the speech attempts and
hint uses over a training session. In the AG, because every hint requested costs a
minus point in the Arcade Game. If a learner uses too many hints, the score could be
low enough to frustrate him.
Consequently, the fact suggests that we should use different coaching services
to help learner improve the learning productivity in different game environments. In
the MPE, the coaching service should target on the Retrospect and Onrush pattern,
whereas in the AG, the service should emphasize the HelpRefusal pattern.
71
The timing to provide tutoring advice also varies. Advice should not interrupt
the gameplay and diminish the fun of gameplay. In the MPE, coaching should be
harmonized to the fantasy context so that the learner will not perceive the abrupt
intervention of seemingly uncompromising instructions. In the AG, coaching could be
more intrusive as the learners’ action is episodic and has no story involved. However,
it should never be disturbing and annoyingly frequent.
Another message sent by this experiment is the call for digging out more
underlying reasons that cause unproductive activities. What factors can influence the
learning productivity? What are the relationships between these factors? Can we
control them to convert the unproductive behaviors to productive ones? If yes, what
approaches can we take? We attempt to answer these questions in the following
section.
3.3 Conclusion

Learner engagement is regarded conductive to learning productivity. However, our
findings showed that intensive learner engagement does not guarantee high
productivity. Unproductive activities still exist in motivational learning environments,
and may account for a substantial amount of the total learning activities. Further
examination on learners’ behavior patterns discloses that unproductive activities
occur when learners exhibited hasty action which overdrafts fun from game-play. We
conclude from this that pedagogical coaching service is needed to cut out these
unproductive learning activities, which targets at the problems identified in this study.
72

Chapter 4
Learner Characteristics and Unproductivie Learning

While Chapter 3 targets the question of whether the serious games provide productive
learning, this chapter focuses on the second research question raised in Section 1.2 of
Chapter 1: what kind of learners do comparatively more productive learning with
serious games? This is a typical “who” question in evaluating any kind of training
systems. We wish to test and validate the improved Tactical Iraqi
TM
system, and to
explore more opportunities for further improvements that take into consideration
various learner characteristics. Toward this goal an experiment was designed to find
the correlations between learner characteristics and learning productivity. The
experiment would be conducted in a similar setting to that of the case study in
Chapter 3.
4.1 System Enhancement

73
The second experiment in this thesis was conducted in December 2007, as a part of
the SEPTR study
4
, with the purpose to test and validate the effectiveness of Tactical
Iraqi
TM
4.0 and to identify opportunities for improvement of this system.
Compared with the Tactical Iraqi
TM
3.0 system, this version of system
integrates new features that are designed for facilitating learners to perform self-
diagnosis on their mistakes, to encourage learners to improve their performance, and
to discourage them from “gaming” the system by continuously using hints. The
improvements include the following features:
1. A side panel that informs the objectives of Active Dialogs. When leaner
enters the Active Dialog page within the MSB, a side panel will pop up,
announcing what the learner needs to accomplish in this conversation (shown in
the bottom left corner of Figure 4.1).
2. Feedback that reports the quality of recordings and pronunciation (Figure
4.2). (This part of work was mainly done by Yoram Meron, who was the primary
contributor of the Automated Speech Recognizer (ASR) for TLCTS.) In the past,
learners were confused and frustrated when ASR did not recognize their speech
for a number of times. This new feature allows them to diagnose what is going
wrong with their speech input. Colors of the traffic light icon (the bottom left of
Figure 4.2) denote the quality of recordings (green means good; red means bad).

4
The Simulation Enhanced Pre-deployment Training and Rehearsal (SEPTR) study was launched by
the US Marine Corps Training and Education Command (TECOM), which is designed to evaluate the
use of simulation-based training in preparing units for deployment overseas.
74

Figure 4.1 The Objectives Panel for Active Dialogs

Figure 4.2 The Recording Quality and Pronunciation Feedback
75
3. A score bar that displays the learner’s performance when interacting with
the virtual character (shown in the top right corner of Figure 4.1). The score is
assigned depending on how the learner conducts a speech: the learner scores less
if requesting hints.
4. A debriefing dialog that explains why scores are assigned. In this example
(Figure 4.2), the learner requests hints in the first speech, and says the second
speech without using hints. The debriefing page tells the learner: “Good job! You
properly exchanged greetings with Answer. Your score for this dialog is 11. If
you had avoided using hints you would have achieved a score of 20.” It also
further explains why the score is assigned in each turn.

Figure 4.3 The Score Debriefing Dialog for Active Dialogs

76
5. Enhanced game graphics and music that aims at influencing the learner’s
affect. Figure 4.3 shows the reaction of the virtual characters to the learner’s
omission of self-introduction, which is considered rude in the Arabic culture. The
red minus sign hovering over the virtual characters’ heads indicate that the
character’s opinion on the learner goes down.

Figure 4.4 A Scenario When the Learner Offenses the Virtual Characters

6. Colored feedback that indicates the recording quality. The red bar on the
top right corner of Figure 4.3 here shows that the quality of the speech recording
was poor perhaps because the learner was speaking too loudly.
77
7. A list of Course of Actions instead of a single hint (the top left corner of
Figure 4.3). When requesting hints the learner have to experience some cognitive
process in order to select the right hint to advance the current conversation.

Figure 4.5 A Progress Report (Skill Mode)

8. A Progress Report that displays what one has accomplished in previous
training (Figure 4.4). The report has two modes between which the learner can
toggle to view the estimate on the learner’s current skill level and the scores of
completed quizzes. The skill report is generated by a Learner Model that tracks
every action the learner does (details are presented in Chapter 6).
78
4.2 Study Procedure
4.2.1 Experiment Purpose

The purpose of the experiment is to find out the correlations of learner characteristics
and learning productivity.
4.2.2 Method

We used a sample of material from Tactical Iraqi
TM
Version 4.0 itself to assess the
learning productivity. We would compare subjects with different characteristics, e.g.,
with/without experience with earlier versions of TLCTS system. The study procedure
is organized as follows:
1. Setting up the experiment environment.
2. Collecting background information on each subject as well as self-
assessments of their interest and motivation to learn Arabic through a
questionnaire survey.
3. Launching the training session according to the experiment settings
4. After the session is over, collecting training logs and learner profiles.
5. Classifying the learning productivity into a range of categories.
6. Analyzing the correlations of learning characteristics with learning
productivity and identifying the key findings.
4.2.3 Experiment Settings

79
49 subjects from the 2/7 Marines were selected to take part in this experiment.
Subjects were organized into two groups of approximately 25, and would work with
the Tactical Iraqi
TM
Version 4.0 for about eight hours.
Firstly, they received an initial 20 minute orientation from a session proctor,
including the software demonstration and task explanation. The proctor then told the
subjects to strive to master the material, reviewing and repeating the learning
materials, exercises, and quizzes as necessary.
Due to the limitation of training time, the curriculum selected for this
experiment is at the beginner level. Subjects were directed to focus on four lessons:
Getting Started (a tutorial), Meeting Strangers (vocabulary, phrases and etiquette
relating to meeting strangers, possessive morphological endings), Introducing Your
Team (Arabic terms for military ranks, phrases and etiquette relating to making
introductions, definite articles, demonstratives, grammatical gender and agreement),
and Pronunciation Lesson 1 (easy Arabic consonants, long vs. short vowels, single vs.
double consonants).
The MPE scenes the subjects were expected to focus on are Tutorial Scene
(Introduction to manipulating user interface), Find Your Way to the Person in Charge
(Build trust with the local, get directions to man in charge, and follow the directions),
and Visit the House of the Person in Charge (Introduce yourself, ask directions, and
locate the person in charge).
80
The Arcade Game levels there learners were expected to experience are Level
1 – Basic Directions and Level2 - Cardinal Directions, including both the listening
and speaking mode.
The subjects then trained for approximately 45 minutes in the Tactical Iraqi
TM

Skill Builder, and spent ten minutes completing a short questionnaire. The
questionnaire asked whether the subjects had been deployed to Iraq before, if so how
many times, and how motivated they were to learn Arabic.
The proctor only provided the subjects with occasional technical assistance,
but otherwise left them to train on their own. After a 10 minute break, the subjects
were then directed to resume training in the Skill Builder for another 45 minutes.
They were then directed to spend twenty minutes in the Mission Game, and then take
another ten-minute break. Finally, the candidates completed another 30 minutes of
Skill Builder training.
4.2.4 Observations

Certain observations were recorded during the proctoring sessions. First, although the
subjects were instructed to focus on the Skill Builder lessons, some subjects still
remained in the two game environments that interested them until the proctor
specifically directed them back to the lessons. Secondly, some subjects left early for
various reasons. Thirdly, some subjects who had used TLCTS before tended to skip
Skill Builder lessons and devoted more of their training time to the game
environments.
81
4.2.5 Results and Key Findings

One of the 49 subjects was excluded from the analysis presented here because he did
not complete the survey questionnaire. As observed, a few subjects left the training
session earlier. Therefore, actual training time (as determined from the log data) had a
relatively high variance (time in Skill Builder: M = 1.08 hrs, SD = 0.72 hrs; time in
Mission Game: M = 0.92 hrs, SD = 0.55 hrs; time in Arcade Game: M = 0.36 hrs, SD
= 0.36 hrs). And this in turn resulted in high variance in performance scores in each
environment (Skill Builder score: M = 2.92, SD = 1.46; Mission Game score: M =
2.92, SD = 1.49; Arcade Game score: M = 2.48, SD = 1.38).
A weighted scoring system that takes the actual training time into
consideration is created to compute and represent the productivity. Each subject was
assigned a score between 1 (low) and 5 (high) for his performance in each of the three
learning environments: Skill Builder, Arcade Game, and Mission Game. The Skill
Builder scores were assigned according to the number lessons attempted, the number
of lessons completed with a high quiz score (80% or better), and the number of
individual language and cultural skills that the learner model indicated were fully
mastered
5
. The Arcade Game scores were assigned according to the number of levels
played, completed, and the number of hints requested by the learner to complete the
level. Similarly, the Mission Game scores were assigned according to the number of

5
The assessment on the learner’s language and culture skills are computed by the Learner Model, the
details of which will be presented in Chapter 5.
82
scenes played, the number of scenes completed, and the number of hints the learner
used to complete the scene.
Productivity scores were computed based on the environment performance
scores and time spent within each learning environment, using the following formula:
∑ ∑
× =
env
env
env
env env
T Score T oductivity / ) ( Pr …………………..(*)
,where env represents the three learning environments,
env
T is the time spent in
a particular environment, and
env
Score is the assigned score for this environment.
Scores are continuous values computed out of ordinal environment performance
scores.
The average productivity score for this population (N=48) is close to the
medium category (M=2.91, SD=1.13, %95CI = [2.585, 3.241]). We found 10 subjects
who achieved high performance scores (>4.0). 1 out of 10 scored 5 in all the three
environments; 3 out of 10 scored 5 in two environments, and the rest 6 scored 5 in
one environment. The best learners spent on average 2.5 hours pure training time with
the system (SD = 0.43 hrs).
The 11 characteristics we examined are categorized into 4 groups. The
personal trait category includes age, education, self-reported motivation to learn
Arabic language and culture, and experience of training with TLCTS before; the
military experience category includes rank, time in service, experience of deployment
to Iraq; the linguistic category includes language spoken other than English and
83
language formally studied; the music ability category includes self-rated musical
talent, ability to sing or play instrument, and experience of formal music training.
T-tests show that 32 trainees who identified their motivation greater or equal
to 4 outperformed the 14 trainees having motivation below 4 (t(44) = 2.038, p =
0.048). Older trainees (>=20 year old) scored lower than younger ones (<20), but the
difference is not statistically significant (t(46) = -1.491, p = 0.14). No significant
difference was found for education, either. The 21 trainees who received some
college education had performance close to the 27 trainees who received high school
degrees (t(45.75) = -0.383, p = 0.715). Interestingly, former TLCTS trainers did not
have superior performance than fresher users do. Rather, they scored a little lower
than those who have never trained with TLCTS before (t(46) = -0.123, p = 0.902) as
they would be expected to. The proctor observed that some former trainees devoted
little effort to the Skill Builder lessons and played a lot in the game environment, but
they were not able to complete the entire game, probably because their language skills
had decayed. Additionally, it also could be that some of the former trainees did not
learn much in the previous experience, or only spent a little time on the system.
Finally, among the former trainees there was a cluster of trainees who had both very
low motivation and performance.
In the military experience category, rank does not effect the training results, as
the average scores for three groups of different ranks are approximately the same
(Rank > E-3 Score: M = 2.88, SD=1.46; Rank = E-3 Score: M = 2.91, SD = 1.17;
Rank = E-2 Score: M = 2.95, SD = 1.04). However, the group with less than one year
84
of time in service and the group with more then one year had statistically different
performance (t(45) = 1.961, p = 0.056). As for experience of deployment to Iraq,
there is no significant finding between the group with the experience and the group
without (t(44) = -.822, p = 0.416).
Those who had studied another foreign language performed at a level that was
close to those who did not (t(46) = 0.115, p = 0.909). In the language experience
category, only 4 trainees speak a language other than English, so it is impossible to
draw conclusions about the role of foreign language fluency.
In the music ability category, no significant effect is found. Trainees who
rated their music talents higher seemed to score slightly lower than those who
identified themselves as “I have no talent in music” (t(46) = -0.551, p = 0.584).
Similarly, trainees who reported practicing singing or playing instrument were
outperformed by their non-practicing counterparts (t(45) = -1.091, p = 0.281).
However, those having taken formal music training scored a little higher (t(45) =
0.430, p = 0.669). But those results are not statistically significant to verify
hypotheses.
In summary, characteristics such as motivation and time in service seem
promising to be conductive to success. The correlation of productivity with other
characteristics is not statistically significant. The findings are then reinforced when
we compare the two groups of subjects. For convenience of notation, 10 subjects are
separated from the rest into the productive learning group (PLG), while the other 39
formulating the less productive learning group (LPLG)). We found out among the
85
subjects of PLG, 90% reported high motivation, and 70% served in military more
than 1 year. In addition, t-tests on PLG and LPLG show that motivation has
significant effects on the productivity (t(44) = 2.381, p = 0.021), while the effect of
time in service seems not statistically significant (t(9.07) = 1.036, p = 0.372).
There are some findings relating to the behavior patterns as well. The
unproductive activities are greatly reduced because of the well-organized orientation,
controlled learning sequence of this experiment, and limited curriculum selection.
Training logs show that frequencies of the Retrospect and HelpRefusal patterns are
minimal among the PLG subjects (0.6 and 0.0 times per subject for each pattern
respectively), and moderately low in LPLG subjects (1.3 and 0.2 times per subject for
each pattern respectively). In contrast, the OnRush pattern occurs a little more
frequently in both groups (0.8 and 2.4 times per subject for each group respectively),
with the population of 65% and 80% subjects of each of the groups have exhibited
this behavior pattern, respectively. There is another noticeable pattern associated with
the OnRush pattern: overall 60% of the subjects have used this strategy: when playing
the mission scenes, they first heavily used the hint facility to go through them, and
then replayed the scenes and finally completed them. Subjects of the PLG group
requested an average of 59.10 times of mission game hint facility compared with the
LPLG group which used only 20.97 times, averagely (t(9.87) = 2.382, p = 0.039).
As we can see the successful learners used different strategies in the Mission
Game and Arcade Game. The difference between these two games explains the
distinction of their behaviors. In the Mission Game even though the aide agent can
86
offer hints on the expected speech in English and Arabic, the learner would not be
able to memorize it if he/she did not build up enough skill level from the MSB
lessons due to the complexity of the speech. Therefore, they need to request hints
often. In the Arcade Game, especially the beginner levels, expected utterance are
relatively short and simple, and therefore medium-leveled skills can be directly
applied.
Lastly, when examining the activity patterns of the two groups, we found that
PLG subjects did particularly well in Skill Builder lessons, compared with the rest of
the trainees (quizzes completed: t(9.65) = 2.654, p = 0.025; skill mastered: t(46) =
2.691, p = 0.100). We believe that this provided them with good foundations to be
able to apply the language and culture skills they learned from the lessons to the other
game environments. This also verifies the findings of the experiment in Chapter 3. In
the Arcade Game, 60% of them never requested a single hint to complete a level, and
therefore were never penalized by minus points because of hint requests.
4.3 Discussion

Evidence is found in the data analysis that a sample of candidates could achieve
above medium productive learning results in one-day training with only one session
proctors (N=48, M=2.91, SD=1.13, %95CI = [2.585, 3.241]). Evidence is also shown
in an interview on December 13
th
, 2007 with several officers and soldiers from the
3/7 Marines who have been using the Tactical Iraqi
TM
system intermittently.
Nevertheless, the distinction of learning productivity given the same conditions
87
implies that not all learners can equally benefit from TLCTS. This conclusion is not
surprising as there is virtually no training system that is effective for everybody. The
contribution of this study is that it has identified a set of learner characteristics which
are potentially conductive to productive learning. The findings are coherent with
those of earlier experiments, which also show that deployment experience and
motivation of learning the target language are conductive to learning success
(Johnson & Beal, 2005). The findings suggest that learners with previous real-life
experience similar to the current simulated situation are likely to perform productive
learning. The experience might boost their motivation of learning the target language,
as they understand the importance of effective communication and are aware of what
language and cultural knowledge are most needed in certain situations. One of the
attractions of game-based learning is that games promote motivation.
Time in service is presumed as an indicator of learner’s maturity and self-
regulation. Although ranks do not seem to have significant effect on the learning
productivity, in reality it’s associated with time in service (according to the US Army
Enlisted Promotion Statistics). Moreover, user requirements illustrate the need of
customized curriculum materials to different trainees such as officers and soldiers.
6

The findings of this experiment suggest further enhancement should address
the following issues. Firstly, the experiment on Tactical Iraqi showed that promoting

6
The civilian version of TLCTS also has customized learning plans that target to learners with long-
term language studying or short-term travel purposes.
88
and maintaining motivation was critical to learning. Therefore, new features that will
be introduced to the game system should not cause detriment of motivation. In
addition, in this experiment the presence of a session proctors successfully reduced
the unproductive activities. The difference implies as a possibility of designing
coaching service which emulates the session proctor’s behavior. Meanwhile,
reviewing the game design principle, we learned that game stimuli work best when
the system leaves learners to engage in learning of their choice, rather than follow a
designated program of instruction. And that in turn requires coach service to provide
instructional planning capability that adapts to the learner's choices, and a learner
modeling and a tutor advising capability that works robustly regardless of the
learner's choices.
4.4 Conclusion

A critical lesson we learned from this experiment is that the design of serious games
need to consider how learner characteristics influence the learning productivity. Our
results indicate that motivation is overall a key predictor of learning success.
Motivation of learning the target languages could stem from diversified factors, such
as the previous deployment of service (or experience of living in the foreign country),
time in military service, or interest on the culture. We conclude that we need to
provide learners with that freedom of choice, yet we should also provide learners
advice of what to work on next, to make sure that they are being productive at all
times.
89

Chapter 5
A Pedagogical Framework for Serious Games

This chapter presents a pedagogical framework for serious games based on the work
in Chapter 3 and 4. It is designed as a generic solution to all serious game, evolving
through continuous assessment, modification and validation. The goal of the
framework is to provide game-tailored coaching service to reduce the learner’s
unproductive learning activities with minimal interruption during the game-play. It
also intends to maintain the learner’s own learning pace as well as their affect status.
5.1 Architecture Overview

Figure 5.1 depicts the architecture of the pedagogical framework. The architecture of
this framework consists of three layers of training service. The Learner Service Layer
is a highly interactive learning environment, composed of game applications as well
as the underlying game engines. This layer also includes components such as the
interview lesson (survey questionnaires), setting manager, and progress report
(interface only) which interact with the other two layers for data retrieving or storing.

90

Figure 5.1 Pedagogical Agents Architecture

The Pedagogy Layer harbors all the pedagogical intelligence, including a
Learner Model, a Tutor Advice Model, and a Log System. The Learner Model is
designed for tracking the training objectives and learning progress as well as
summarizing the progress and generating the progress report. The Tutor Advice
Model retrieves the information about learning progress from the Learner Model and
provides tutor advice based on this information. The Log System manages storage of
the data of learner interaction and internal component messages for off line
assessment use by independent researchers.
The Tutor Service Layer assists the human tutors or lab supervisors to monitor
onsite learning. The components in this layer communicate with the Pedagogy Layer,
91
retrieving the learner’s progress data and generating a detailed progress report. Tools
are also created to manage the group structure and personnel.
The following subsections will describe the design of these service layers in
details.
5.2 Learner Service Layer

The Learner Service Layer consists of four sub-layers: Curriculum Contents,
Customized Contents, Interactive Learning, and Pedagogy Service (Figure 5.2).

Figure 5.2 Learner Service Layer

Curriculum Contents contain the corpora of knowledge and skills developed
by human instructors or tutors.
92
Customization tailors contents and learning plans to the learner’s preferences.
The information regarding to the learner’s preferences are collected through the
following two components:
• Interview collects the learner’s background information and their self-
identified motivation of learning the curriculum materials. The learner
information will be used for research analysis as well.
• Setting allows the learner themselves to adjust the learning contents and
system features.
The Interaction sub-layer allows the learner to choose the form which they
prefer to acquire knowledge and horn skills.
• Instruction conveys the objectives and major points of the curriculum unit the
learner will work on. It also introduces the knowledge which the current
curriculum unit attempts to teach.
• Exercise helps the learner memorize the knowledge just taught. Exercises can
be light-weighted games which prepare the learner for taking complex game
scenes.
• Practice/Game Play supplies the game-based learning environment where the
learner can actively advance the development of their knowledge and skills.
• Test enables the learner to assess their knowledge and skill levels. Tests can
be review quizzes, exercises that the leaner made mistakes on, and exam game
scenes.
93
Pedagogy Service addresses the learner’s tutoring and affect’s needs. It
coaches the learner as to where they should focus efforts on. It includes the following
elements:
• Scaffolding controls the scaffolding and fading levels throughout the TLCTS
system. Most beginner-level learners are intimidated when facing a complex
learning system. Scaffolding can maintain their motivation at the initial stage.
• Feedback provides the immediate feedback on the recording and
pronunciation qualities, and helps the learner with self-correction.
• Progress Reports presents the history record and/or learner performance
evaluation to facilitate the learner’s self-assessment. It displays the output of
the Pedagogy Layer.
• Advice suggests to the learner what to work next, and what needs to be
improved. It displays the output of the Pedagogy Layer as well.
• Scoring provides quantitative evaluation of the learner’s performance relating
to a particular curriculum unit. Game scores are rewarding/penalizing stimulus
that promotes learners’ motivation. Record-breaking scores are expected to
encourage learners to challenge high difficulty levels and hence to make more
efforts in learning.
5.3 Pedagogy Layer

The Pedagogy layer consists of two models: Learner Model and Tutor Advice Model
(Figure 5.3). The Learner Model is a stand-alone component which can work
94
independently, while the Tutor Advice Model relies on what has been stored in the
Learner Model.
5.3.1 Learner Model

Learner Model is further decomposed to a Learner Profile (data repository), a
Training Objectives Tracker, and a Progress Report Generator.
• Learner Profile traces three kinds of information: the learner's background
information (including the self-identified motivation score), the learner's
interests (e.g., particular curriculum contents), and the learner's progress
(including estimate skill levels and quiz scores). A serious game can teach
many skills, and the learner’s mastery of these skills is assessed in the
Progress Tracker. Each skill level is represented as a quantitative score.
• Training Objectives Tracker tracks the learner’s short-term and long-term
training goals and objectives. This information is filled in a Setting component
in the Learner Service Layer, and can be updated whenever the learner
establishes new goals and objectives.
• Learner Progress Tracker tracks the learner’s interaction with the system.
Every action is assessed by this tracker, which updates the information that
represents the learner’s mastery of skills in the Learner Profile.
• Progress Report Generator summarizes the learning history and progress up
to date, feeding the data to the Progress Report in the Learner Service Layer.

95

Figure 5.3 Pedagogy Layer

5.3.2 Tutor Advice Model

Tutor Advice Model is composed of Learner ZPD Evaluator and Tutor Advice
Generator, as shown in Figure 5.3. This model mainly aims at providing advice to
reduce the unproductive learning activities. A deeper insight on these activities is thus
needed to resolve what kind of advice should be provided given the current learning
progress.
5.3.2.1 Interpret the Unproductive Patterns

The Learner ZPD Evaluator model is constructed based on the ZPD theory, which
was firstly introduced in Section 1.1.5 in this thesis. The model depicted in Figure 1.1
96
helps us understand under what circumstances the unproductive activities are likely to
happen for a regular learner.
In the retrospect pattern, the learner tries to complete a task with inadequately
developed skills. As a result, this learner ends up in the zone of confusion (the grey
circle area in Figure 5.4) and cannot perform efficient learning.

Figure 5.4 The Shift of Learner’s Position to ZPD in the Retrospect Pattern

After a number of times of failure, the learner diagnoses his/her problem with
the current mission task is because of the relatively low-level skills. The strategy of
this self-regulated learner uses is to increase his/her skill level. After he/she has
improved the skill level to a certain degree and selects to play the mission task again,
he/she is likely to shift the position into the ZPD (the while circle area in Figure 5.3).
Of course, they cannot always come back to the ZPD zone. Chances are that after
practicing the irrelevant pages or exercises with learning materials their skill level is
Learner’s
current skill
level
ZPD
Game challenge level
Learner Skill Level
Zone of confusion
Zone of boredom
Challenge
level of
advanced
scenes
Learner’s
enhanced skill
level
97
still below the required level to win the challenge. And this shows the necessity of
pedagogical support for this pattern of behavior.

Figure 5.5 The Shift of Learner’s Position to ZPD in the Onrush Pattern

Learners who behavior in the Onrush pattern have the ambition to complete
all the mission tasks as soon as possible. It is possible it is because they gain
confidence in the simple scenes with less challenge levels compared with their skill
level.
But once the mission scenes raise the challenge level to a degree higher than
what he can get over with the current skill level, this learner will be shifted to the
zone of confusion (the white circle area in Figure 5.5). He can of course lower the
challenge level of the mission task by abusing the hint facilities. And if he succeeds,
he can bring himself back the ZPD. We have some examples showing that learners
ZPD
Game challenge level
Learner Skill Level
Zone of confusion
Zone of boredom
Learner’s current skill level
Challenge level
of a little
advanced scene
Challenge
level of
advanced
scenes
Challenge
level of
simple scenes
hints
98
complete the mission takes under the assistance of hints. The strategy is acceptable as
the learner can learn from hints to advance their skill level. However, in the situation
that they fail to lower the challenge level to shift their position into the ZPD zone, the
potential unproductive behavior will be turned into an actual unproductive one.

Figure 5.6 The Shift of Learner’s Position to ZPD in the HelpRefusal Pattern
Compared with the previous two patterns, the HelpRefusal pattern least
benefits the skill development of learner. Because the learners’ inaction to acquire
required skills and or to access the learning materials or help facilities, the position of
this learner, once in the zone of confusion, is difficult to change (Figure 5.6).
Without tutor intervention or self-diagnosis, this learner will remain in the
zone of confusion, which is claimed to be the area where low efficient learning occurs.
5.3.2.2 Learner ZPD Evaluator

Learner’s
current skill
level
ZPD
Game challenge level
Learner Skill Level
Zone of confusion
Zone of boredom
Challenge
level of
advanced
scenes
99
A number of methods have been devised to detect, measure, and maintain the
learner’s ZPD (Luckin & du Boulay, 1999; Murray & Arroyo, 2002). However, those
so-called ZPD tutoring techniques are less suitable for highly interactive serious
games where in-game tutoring is not the optimal solution as it can hamper the learner
affect. On the other hand, due to the high interactive nature of games, learner actions
in serious games are less predictable than those in traditional tutoring systems, where
tutors already have difficulty in diagnosing learners’ correct and incorrect
conceptions (Chi et al, 2004).
A simple yet effective method is proposed in our framework to determine the
learner’s position to their ZPD. Given a properly defined skill hierarchy and skill
assessment algorithm, we use an analogue to represent the learner’s position to the
ZPD zone, which are a set of coverage ratios of learner acquired skills against the all
the skills required to complete a game.
We denote all the skills associated with a particular game (scene or episode or
level) as a skill space SP , which corresponds to the game challenge level in the ZPD
model. This SP is further decomposed to two subsets: essential skills (denoted
as ESK ) and optional skills (denoted as OSK ) (Figure 5.7).
Essential Skills are the primary skills intended for the learner to practice
independently (without using hints) within the current game. Therefore, learners must
have achieved relatively high scores (greater or equal than a threshold denoted as
esk
Th ) of these skills in related exercises or tests or other games before they start to
play the current game. Optional Skills are secondary important skills that learners are
100
expected to practice in the game. Learners must have exposed to these skills (scores
greater or equal than a threshold denoted as
osk
Th ) but not necessary to have achieve
a high score (
osk
Th >
esk
Th ). The skills in SP but not in ESK or OSK are
• simple skills that learners can pick up through action or using hint facilities;
• skills that have no impact on game endings;
• skills that cannot advance the game plot;

Figure 5.7 The Composition of a Skill Space

We now map different situations of skill mastery to learner’s position to ZPD.
• The learner has not acquired the essential skills
( } | {
esk sk
Th Score LAS sk s < ∧ ∈ ∃ , where LAS is the set of learner acquired
skills. In this case the learner’s position is determined to be far outside ZPD
and in the zone of confusion.
• The learner has mastered all the essential skills but only exposed to a few
number of optional skills (
optional
CR = / | |
optional
LAS | | OSK <
k
Th
cos
, where
optional
CR is the optional skill coverage ratio and
k
Th
cos
is the threshold of
optional skill coverage ratio). This case maps to the situation where the learner
Essential Skills
Optional Skills
Scene Skill Space
Other Skills
101
is close to ZPD but still in the zone of confusion.
• The learner has mastered all the essential skills and a large part of the optional
skill ( ≥
optional
CR
k
Th
cos
). In this case, the learner is considered as remaining in
ZPD, because with the aid of a couple of hints, learners will develop their
skills reaching the immediate future when they can complete the current
game.
• If the learner has mastered both the essential and optional skills, the learner is
probably in the zone of boredom. However, the game stimulus as well as
satisfaction of success might motivate the learner to engage in the learning.
5.3.2.3 Tutor Advice Generator

In this framework, two categories of tutor advice are created according to the timing
of when the advice is presented to the learner.
Pregame tutor advice is generated dependent on which of the above cases
occur when the learner tries to start a game
7
. The learner is either encouraged to
resume the game or suggested to undertake extra tasks in order to position themselves
in ZPD before starting the game.
Pregame advice is to arouse the learner’s awareness of potentially
unproductive learning activities, and help them diagnose the causes of these activities.
Meanwhile, learners may choose to ignore the advice, following their own learning

7
Concrete tutor advice and examples will be presented in Chapter 6.
102
plans. Therefore, they still have the freedom of choice and can keep their own
learning pace.
The other type of advice provided by the Tutor Advice Model is called
Postgame tutor advice. This kind of advice focuses on suggesting to learners to
improve their productivity through perfecting their less optimal performance if there
is any. The advice addresses a wide range of aspects, from critiques on the learner’s
the problems which have been revealed in their actions in the game which they just
came off such as excessive use of hint, omission of politeness terms, and inefficient
time management, to more constructive ones such as which part of the curriculum
they need to retake to horn the related skills.
5.3.3 Tutor Service Layer

In some lab sessions, human tutors are assigned to monitor the onsite learning. Tutors
need to know group and individual learning performance in order to provide
instructions. The data reporting tool in the Tutor Service Layer is designed to assist
tutors to more accurately evaluate the learner’s progress. It communicates with the
Learner Model, summarizing the aggregate learning results of the entire group as well
as the learning history of an individual learner, and presenting the report to tutors.
The information about a group of learners includes their total learning time with this
session and with each of the games, quiz and skill scores, and completion of
curriculum units. Reports also include the background information of each learner, so
that the tutor can make better judgment to help a specific learner.
103
5.3.4 Summary

Our previous work has identified that unproductive activities can occur even when the
training system successfully engages the learner. We argue that the reason why these
activities occur is because of the distance between the learner’s actual acquired skills
and the skills required by game play. An attempt was made to interpret the
fundamental causes of unproductive activities using the ZPD model. We then convert
the problem of finding out the learner’s relative position to ZPD to the problem of
evaluating the skill coverage ratio associated with one game. Based on that, we
crafted tutor advice that addresses the learner’s pedagogical and affect needs. These
features are interwoven into a generic pedagogical framework designed specially for
serious games.

104

Chapter 6
Implementation and Evaluation

A large part of the pedagogical framework has been implemented in the Tactical
Iraqi
TM
Version 5.0 system, which targets at the training standard of Interagency
Language Roundtable (ILR) Level 1 of spoken language proficiency (elementary
proficiency). The system enhancement was proposed by assessment of our
experiments as well as feedback from previous trainees, and evaluated by preliminary
surveys.
6.1 Training Standard

The Tactical Iraqi
TM
Version 5.0 system aims at training beginners to reach the ILR
Level 1. The ILR scale defines five levels of abilities of speaking, listening, reading,
and writing a language, which are the elementary, limited working, professional
working, full professional and native proficiency (http://www.govtilr.org/). Level 1 of
spoken language proficiency describes the following abilities:
• able to carry on face-to-face conversation on simple topics, such as asking for
directions, purchasing goods, ordering meals, and making appointments.
• able to carry on the conversation in a polite manner.
105
• able to make oneself understood through guides of native speakers, such as
slowed speech, repetition, and paraphrases (frequent misunderstanding is
allowed).
• able to form questions and answers on simple topics with limited proficiency.
This version also has interactive lesson plans for learners with different
backgrounds and interests, who are preparing for deployment overseas, and who have
a limited amount of training time. Trainees who are junior and senior enlisted are
presented with lessons and scenes that focus on basic missions such as manning a
checkpoint. Officer trainees are presented with advanced lessons and scenes, which
address house searching, information gathering, and dealing with civil affairs, besides
the basic ones.
6.2 Implementation Architecture Overview

Pedagogical features permeated through the Tactical Iraqi
TM
5.0 system. The
architecture (Figure 6.1) consists of three layers of training service. The learner
service layer consists of the Mission Skill Builder, the Mission Game and the Arcade
Game as well as the underlying game engines (an adapted Unreal Engine and Python-
based Mission Manager). It also includes components such as the interview lesson
(survey questionnaires), setting manager, and progress report which interact with the
other two layers for data retrieving or storing.
The Pedagogy layer harbors all the pedagogical intelligence, including
Learner Model, Tutor Advice Model, and Log System. The Learner Model consists of
106
a training objectives tracker, a learner progress tracker and a progress report
generator. The Tutor Advice Model provides service recommendation based on the
learner's current progress.

Figure 6.1 System Architecture of Tactical Iraqi
TM
Version 5.0

The Log System component manages storage of the data of learner interaction
and internal component messages for off line assessment use.
107
6.3 Implementing the Learner Model

The initial version of the TLCTS system such as Tactical Iraqi
TM
3.0 was equipped
with an inadequate learner model that stores limited learner data and suffered from
data missing constantly. The old learner model did not store the necessary
information such as how many speech attempts the learner has practiced during a
session. Neither does it save any of the session information. As a result, it is of little
use for data analysis.
Profiling the learner's information helps create a system to the learner's needs.
More detailed information has been added to profiles of Learner Model in additional
to the existing skill profile and quiz profile. The learner model now traces three kinds
of information: the learner's background, the learner's interests, and the learner's
progress.
We adjusted several features in the enhanced design. The new learner model
traces three kinds of information: the learner's background, the learner's interests, and
the learner's progress in terms of skill levels, scores, and time. The learner progress
tracker includes a learner profile, which further decomposed to several sub-profiles:
the Training Objective Profile, the Skill Profile, Quiz Profile, and Session Profile.
6.3.1 Training Objective Profile

The training Objective Profile stores the following information:
• Training Standard, which defines the training purpose.
108
o Basic Training Standard: this training standard aims at training for
immediate deployment or travel overseas.
o Advanced Training Standard: this training standard aims at training for
long-term proficiency and skill retention.
• Rank, which determines lesson plans which will be selected for the current
user.
o Junior enlisted (E1 – E4)
o Senior enlisted (E5 and above)
o Officer
o Civilian
• Task Skills, which were designed for determining lesson plans together with
other parameters such as ranks (in military versions) or occupations (in
civilian versions).
o General Language Skills: this option covers the minimal survival and
core languages but do not target for specific missions
o Basic Mission Skills: this option primarily involves basic command
language
o Advanced Mission Skills: this option covers missions that require high
proficiency, e.g., information gathering and military advisory missions
o General Language Skills + Basic Mission Skills
o General Language Skills + Advanced Mission Skills
109
• Subtitles, which determine the form of on-screen texts that mirror the
conversation carried on between the player and virtual characters.
o English + Target Language Phonetics
o No Subtitles
o English Subtitles
o Target Language Phonetics
• Display Text, which determines the form of display texts on the instruction
and practice pages.
o Arabic Transliteration
o Arabic Script and Arabic Transliteration
o Arabic Script
• Virtual Tutor, which determines if the tutor advice is active. When this value
is off, learners will not have the tutor advice service.
o Turn On
o Turn Off
Instructors and learners can specify their preference in a setting page.
Instructors can set the default settings for a group of learners through a server
application called the Dashboard. Therefore, for each of the parameters, there are two
attributes associated with it:
• provenance: who set the default values, the instructor or the learner.
110
• modifiability: determines if the setting items are modifiable. If the current
item is allowed to be changed by the learner, then it is enabled in the Settings
Page.
6.3.2 Survey Profile

Our previous studies showed certain correlations between learning productivity and
learner characteristics. We wish to find out more correlations and to validate the
findings through further data analysis.
An Interview Lesson was thus created and integrated to the system to
automate the data collection of learner background information and interests on
learning skills of different communication functions. The Interview Lesson includes
questions that address training standard, rank, gender, age, motivation, language level,
education background, formal language study experience, informal language study
experience, musical ability, application experience, game experience, and task
experience. The questions and options are defined as follows:
• Gender
o Male
o Female
• Age Range
o Below 20
o 20 – 24
o 25 – 29
111
o 30 – 34
o 35 – 39
o Above 39
• How is your motivation of learning this language? (5=Very high; 1=Very low)
o 5
o 4
o 3
o 2
o 1
• How many hours have you studied Iraqi Arabic?
o Zero
o Lesson than 20
o More than 20, but less than 40
o More than 40
• What is your education background?
o No college courses
o Some college studies
o College degree
o Graduate or professional school
• Have you ever studied any other foreign languages in school?
o Yes
o No
112
• Have you ever learned a foreign language while visiting or living in another
country?
o Yes
o No
• Rate your musical ability.
o 5 (professional musician level)
o 4 (able to play instruments or sing)
o 3 (have taken some music lessons)
o 2 (don’t play any instruments or sing)
o 1 (don’t have any sense of music)
• Have you used Tactical Iraqi before? For many hours?
o No, never.
o Yes, less than 20 hours.
o Yes, more than 20 hours.
• Do you often play computer games?
o No, I seldom play computer games.
o Yes, I do once or twice a month.
o Yes, I do several times a month
o Yes, I play almost every day.
• Have you ever been deployed or traveled in Iraq before?
o Yes
o No
113
6.3.3 Skill Profile

Skill profiles tracks the learner's skill levels, which are computed through a Skill
Model. The Skill Model consists of two parts: skill ontology and tracing model. The
skill ontology has a hierarchy of two levels: the upper level skills are task skills that
describe a specific task objective; the lower level skills are language or culture skills
that describe the ability of completing that task. For example, the upper level skill
“build rapport with acquaintances” includes languages skills “welcome guests”,
“greet a host”, “inquire politely about others' well-being”, etc. Each skill corresponds
to one or more utterances taught in the Skill Builder lessons (Figure 6.2). For example,
the “welcome guests” skill is associated with the utterance “ahlan wa sahlan”.
Currently the skills are tagged at the page, lesson and, mission scene level to simplify
the mapping between utterance and skills.

Figure 6.2 Utterance-Skill Space Mapping

114
A Bayesian approach (Bolstad, 2004) was adopted to compute the skill score.
There are a number of reasons why a Bayesian method was used. First of all,
although TLCTS is equipped with an advanced Automated Speech Recognizer,
accuracy penalty is inevitable due to accents, noises, or a combination of these two.
False negative recognition must be taken into consideration, at the risk of hurting the
learner’s motivation. This probability model avoids the case where the learner’s skill
score remains low because the speech recognizer continuously fails to recognize
his/her speech input. Secondly, chance of skill score increasing gives credit to
learner’s efforts especially when learner speech keeps rejected by the system. The
uncertainty of score increase encourages the learner to undertake more practice.
Lastly, estimate on the skill mastery also has uncertainty given a single learner action
being observed. However, accumulated evidence of a sequence of learner action can
increase the accuracy of estimate.
Bayesian network enables predictions about the occurrence of a particular
event based on encoded causal relationships among events. In other words, a
Bayesian network can utilize prior knowledge to infer new posterior knowledge given
evidence or observations. Specifically for Bayesian inference, the prior knowledge is
considered as estimate of the degree of belief in a hypothesis before evidence is
observed, and the posterior knowledge is the inferred degree of belief in hypothesis
after evidence shows up. The name Bayesian stems from Thomas Bayes’s theorem,
which states that:

115
) (
) ( ) | (
) | (
E P
H P H E P
E H P = ,
where H represents the hypothesis, P(H) is the prior probability of H without
considering E, P(E|H) is the conditional probability which represents the likelihood of
evidence E happening if the hypothesis H is true, P(E) is the marginal probability of
evidence E in all possible cases, and P(H|E) is called the posterior probability which
represents the likelihood of hypothesis H being true after evidence E is observed. P(E)
is often calculated as the sum of product of probabilities of any possible hypothesis
and conditional probabilities under the corresponding hypothesis, as follows:
∑
= ) ( ) | ( ) (
i i
H P H E P E P .
The Bayes Tracing Net (Figure 6.3) which we constructed is composed of two
sets of nodes: evidence nodes and decision nodes. The evidence nodes contain a
sequence of user behaviors observed during a time interval when he/she is interacting
with the system. These behaviors are represented as events, e.g., practices one
exercise on an utterance formation page is an event. Events are not necessary learning
events but in the tracing net we only care about learning-related events. Inside the
decision node (the skill update node in Figure 6.3) the model computes the likelihood
of which hypothesis is true, namely, whether or not to increase the corresponding
skill levels given the current observation and skill level. For example, if a learner
practices certain exercises on an utterance formation page, then the degree of belief
on the hypothesis that learners have acquired relevant skills to that page might be
increased because accumulated evidence shows the learner does these exercises
116
correctly. Probability values are approximated based on the past training data set. The
probability of skills being increased associated with “using hints in the Mission
Games” is lower than that associated with “doing speech-act without hints”.

Figure 6.3 A part of the Bayes Tracing Net

Formally, a Bayesian approach for inference using this tracing net includes the
following steps:
1. Matching: map the practiced utterance to the relevant skill
i
SK , and a
hypothesis
i
H :
i
SK is acquired.
2. Inference: estimate the degree of belief on
i
H given past learner actions
K A : 1 according to the conditional distribution of
i
H
3. Update: update the degree of belief on
i
H .
4. If there are a sequence of actions associated with
i
SK , then repeats step 1, 2,
and 3 for each action in time order.
117
5. If multiple skills are mapped to the current utterance, then for each skill repeat
step 1 through 4.
The skill score for
i
SK is defined as the degree of belief on
i
H . A simple
example of skill score computation is described as follows:
A learner has practiced on “marHaba” on a practice page for three times, with
first two speech attempts recognized as incorrect and the third speech attempt as
correct. The tracing net then maps “marHaba” to the skill represented as “ELO-0101-
02: use informal greetings”. “ELO-0101-02” is the skill’s unique id in the skill
ontology.
For Action 1: speech recognized as incorrect
Inference: P(SkillDecision = AcquireSkill | Observation = Incorrect) = 0.0
Update: P(SkillDecision = AcquireSkill) = 0.2
For Action 2: speech recognized as incorrect
Inference: P(SkillDecision= AcquireSkill | SpeechObservation = Incorrect) = 0.11
Update: P(SkillDecision" = AcquireSkill) = 0.29
For Action 3: speech recognized as correct
Inference: P(SkillDecision = AcquireSkill | SpeechObservation = Correct) = 0.31
Update: P(SkillDecision = AcquireSkill) = 0.45

Table 6.1 An Example of Computing the Skill Score

The computation process of this example is shown in Table 6.1. The skill
score for Skill ELO-0101-02 is 0.45 given the learner’s action history on this practice
page.
6.3.4 Session Profile

There are two types of session profiles: the current session profile stores more
detailed information about the learner's progress, and the past session profiles that
118
only store the aggregated information about the previous learning sessions. The
current session profile includes the following information:

Current Session Profile

This profile contains the detailed information about learner’s behavior and progress
of the current learning session, as follows:
1. The name of the learning environment the learner is focusing on.
2. Information for each Skill Builder lesson that the learner visited:
a. What is the name of the lesson?
b. How much time did they spend in the lesson?
c. Which active dialog did they try?
d. How many attempts on an active dialog?
e. For each active dialog attempt, what were the scaffolding settings (e.g.,
subtitles) in force at the time?
f. What dialog score did they obtain?
g. How many speech attempts did they make?
h. How many hints did they use?
i. Did they attempt the quiz? If so, for each quiz attempt
j. What score did they obtain?
k. How many speech attempts did they make?
l. For the instruction / exercise portion of the lesson, how much time did
they spend in that?
119
m. How many speech attempts did they make?
n. How many correct speech attempts did they make?
o. When did they start the lesson?
3. Information for each Mission Game scene that the learner visited:
a. What is the name of the scene?
b. When did they start the scene?
c. What were the scaffolding settings in force?
d. How much time did they spend in the scene?
e. How much of that time did they spend in Dialog Mode?
f. How much of that time did they spend in Explore Mode?
g. What score did they achieve in the scene?
h. How many speech attempts did they make?
i. How many hints did they use?
j. Information for each Arcade Game level that the learner visited:
4. Information for each Arcade Game level that the learner visited:
a. What is the name of the game level?
b. When did they start the game level?
c. How much time did they spend in the game level?
d. What score did they achieve?
e. How many speech attempts did they make?
f. How many hints did they use?

120
Aggregate Session Profile

This profile contains the summary information of previous learning sessions, as
follows:
1. What did the session start?
2. How long did the session last?
3. What is the session ID?
4. For each learning environment, how long did the learner spend time on it?
5. For the practice within the Mission Skill Builder lessons:
a. What lessons did the learner start?
b. What lessons did the learner complete?
c. How many speech attempts did the leaner make?
d. How many correct speech attempts did the learner make?
6. For the practice within the Mission Game scenes:
a. How many scenes did the learner start?
b. How many scenes did the learner complete?
c. How many speech-acts did the learner make?
d. How many hints did the learner request?
7. For the practice within the Arcade Game levels:
a. How many levels did the learner start?
b. How many levels did the learner complete?
c. How many voice commands did the learner follow?
d. How many speech-acts did the learner make?
121
6.3.5 Other Profiles

Quiz Profile

Quiz profiles record the learner's interaction with review quizzes. The information
included in this profile is:
• Total score of the quiz
• Learner achieved score
• Duration of taking the quiz
• Number of speech acts the learner has made

Pronunciation profile

Pronunciation profile encompasses the learner's scores in the pronunciation lesson.
The scores are outputs from an Automated Speech Recognizer, and will be set as the
initial scores when the learner visits the previously practiced pages.

Error Profile

Error profiles track the exercise pages which the learner failed to answer correctly.
These exercises will appear in the review quiz, allowing the learner to perform self-
correction.
6.4 Log System

The initial version of the TLCTS system such as Tactical Iraqi
TM
3.0 has a
preliminary device for log data collection, which heavily used the DOM tree structure
that slows down the system when the training session exceeds one hour. The original
122
design of this system expected a well-formed data structure that describes the
learners’ all activities within an environment. A side effect of this design is increasing
the time complexity of storing and retrieving the learner data. What is worse is that
when the system crashed due to some unexpected reasons, the whole log file would
be lost.

Figure 6.4 Interaction of Log System with Other Models

A new event-based streaming log system (Figure 6.4) was created for
convenience of collecting log data and learner profile data from the training sessions
in the field. This system prunes the redundant messages and adds levels of learner
events of granularity, such as practice with the lessons, pages, utterances, scenes,
conversations. It also tracks the learner’s time devotion in different learning
environments and in different modes. .
123
6.5 Tutor Advice Model

The Tutor Advice Model provides suggestions and service recommendation
contingent upon the learner’s current action and progress. There are four types of
tutor advice, which are classified based on where they appear: pre-quiz, post-quiz,
pre-scene, and post-scene.
6.5.1 Examples of Tutor Advice

Pre-quiz Advice

Figure 6.5 Pre-quiz Advice

124
Each lesson has a review quiz that allows the learner to test how well they have
mastered the skills learned from the pages and exercises of this lesson. The Tutor
Advice Model will assess the learner’s mastery of skill before one takes a review quiz.
If the learner has mastered enough skills, the tutor advice is given like this:
You have done well in the previous exercises. Start the quiz now!

If the learner has done poorly on the previous exercise pages, he will probably
get the following advice:
You haven't acquired enough skills in the previous exercises. Taking this quiz
now is rushing it a bit. How about reviewing the lesson first? To review the
lesson press the Restart Lesson button.

Post-quiz Advice

After the learner has taken the quiz, the Tutor Advice Model will evaluate the
quiz score and provides advice based on the score and the total score of this quiz.
If the learner has answered 80% or above questions correctly, the tutor advice
is shown as follows:

Good job! That's a flying score. Keep up the good work.

If the learner has answered above 60% but less than 80% questions correctly,
the tutor advice is shown as follows:
Well done! Come back to this lesson and study it some more. See if you can
bring your quiz score to 80% or better.

If the learner has answered less than 60% questions correctly, the tutor advice
is shown as follows:
Please study this lesson some more now, and see if you can improve your quiz
score.
125

Figure 6.6 Post-quiz Advice

Pre-scene Advice

The Tutor Advice Model then evaluates the learner’s mastery of skills and
provides advice before the learner starts a Mission Game scene (Figure 6.7).

126

Figure 6.7 Pre-scene Advice

If the learner has mastered both the essential and optional skills which are
involved with the current scene above a certain level, the tutor advice is shown as:

Your have enough skills to complete this scene. Try it!

If the learner has not mastered the essential skills, the tutor advice will try to
prevent the learner from playing this scene. For example, for the Civil Affair scene,
the tutor advice might be:

It seems you have not mastered the skill(s) to complete this mission: Ask and
answer questions about well-being of family, Respond to inquires about your
well-being, inquire politely about other's well-being.

127
I recommend that you review the following lesson(s): Discussing Family,
Advanced Family, Giving Feedback.

If the learner has mastered the essential skills but not the optional skills, the
tutor advice will suggest the learner to play the current scene, and remind the learner
that there is an option to review the relevant lessons first. For example, for the Civil
Affair scene, the tutor advice might be shown as:

You have learned most of the skills required for this scene, and you might find
the skill(s) useful as well: inquire politely about other's well-being.

You may either play the scene now or try to brush up these skills in the
lesson(s): Discussing Family, Advanced Family, Giving Feedback.

As the skills are tagged at both the lesson- and scene-level, the Tutor Advice
Model can infer which lessons the learner needs to review in order to improve the
skills needed in the current scene.

Post-scene Advice

The Tutor Advice Model observes how the learner has completed a particular
scene and provides advice after the learner enters the debriefing page (Figure 6.8).
After the learner finishes the scene of finding Anwar, tutor advice is displayed
in the debriefing page:
Your score for this dialog is 21. If you had avoided using hints you would
have achieved a score of 30.

Great work! You introduced yourself to someone. Sure, it’s a small step, but
others are watching and your positive interaction with Anwar will definitely
help your image.
128

If you want to further improve your skills, maybe you can review the following
lesson(s): Meeting Strangers.

Figure 6.8 Post-scene Advice

In the case when the learner did not perform well in the game, the tutor advice
would be:
It seems you have not mastered the skill(s) to complete this mission:
Introduce yourself by name.

You have learned most of the skills required for this scene, and you might find
the skill(s) useful as well: Use polite forms of address, Tell your/someone's
rank or title.

I recommend that you review the following lesson(s): Meeting Strangers

129
You may either play the scene now or try to brush up these skills in the
lesson(s).

If a bad game ending has happened, the tutor advice will suggest the learner to
revisit the lesson that covers the culture skills:
I suggest you go over the Lesson "Culture Reference" to learn more about
Arabic cultures.

If the learner requests too many hints in a scene, the tutor advice will suggest:
You've practiced 11 speechacts and requested hints for 8 times. It would be
nice if you could use less hints. If you feel you cannot make it without relying
upon hints, why don't you revisit the lessons in the Skill Builder and then play
this scene again?

If the learner spends too much time wandering around instead of conversing
with the virtual characters, the tutor advice will remind him:
You've spent 80% of the time on exploring the game world. Try to reduce the
exploring time next time.

If you want to further improve your skills, maybe you can review the following
lesson(s).

6.6 Evaluation

A preliminary study on Version 5.0 was conducted to test and validate the
effectiveness of the tutor advice, skill assessment, scoring, scaffolding (hint facility
and subtitles), and progress report (the complete survey questionnaire is in Appendix
B). Three subjects were selected for convenience. One of the three subjects had brief
experience with Version 4.0, and the other two are fresh learners. All of them
identified themselves as highly-motivated and self-regulated learners.
130
Subjects downloaded and installed the system on their computer, playing the
system for about two hours.
They were told to complete a list of tasks in their own preferred sequence:
• MSB lessons: Meeting Strangers, Introducing Your Team, Social Etiquette,
and Say it in Iraqi, including the exercise pages, active dialogs and review
quizzes.
• MPE scenes: Scene 1 (Start making contacts with locals and find a local
aid), Scene 2 (Find the person in charge)
• Play the speaking and listening mode of Arcade Game Level 1 (Basic
directions), Level 2 (Cardinal directions)
They were also notified to make a note when they ignore the tutor advice or
when they encounter inaccurate skill scores.
Subjects rated highly on usefulness of nearly all the pedagogical features.
They also achieved relatively high skills scores and quiz scores (average # of skill
mastered: 20.5; average skill score: 0.6 out of 1; average quiz score: 88.2 out of 100),
which shows positive evidence that the framework is potentially beneficial for
improving learning.
Subjects consider the tutor advice was effective on improving motivation to
play the games but not so much on improving motivation to learn the language itself.
They believe that they were not coerced to follow the tutor advice and chose the
games in their own pace. Written answers show that they think the reference feature
131
of tutor advice was particularly useful as it helped them locate the related lessons so
that they could quickly brush up their skills.
However, their answers also imply that although they think the skill scores are
accurate they do not check the progress report very often. Also, two subjects admitted
that they tended to depend on the hints to complete the games.
6.7 Summary

This chapter mainly describes a concrete example where we executed the pedagogical
framework into a specific system. The realized pedagogical framework includes five
major types of pedagogical service: scaffolding, scoring, progress report, feedback,
and tutor advice. All the pedagogical features are stitched within the system
architecture, interacting with different components of game applications.
Implementation details and peculiar design considerations are also presented for a
serious game with purpose made for language learning. Preliminary evaluation proves
the value of this framework.

132

Chapter 7
Discussions, Future Work and Conclusion
7.1 Discussions on Research Questions

After presenting the data analysis and solution, we now return to answer the four
research questions brought forward in the introduction part of this thesis.
• Do serious games provide productive learning, especially in training of
realistically complex problem-solving skills?
Our work showed that serious games such as TLCTS was effective for
learners with certain characteristics (e.g., motivation) but further evidence is needed
to conclude that serious games can provide productive learning as an independent tool.
However, serious games are promising for training in training of realistically complex
problem-solving skills.
L2 language learning is considered an arduous learning process for most
adults. The goal of rapid training even adds more difficulty. Yet post-surveys and
interviews of Tactical Iraqi
TM
4.0 and 5.0 both revealed that learners felt that the
revised systems with enhanced pedagogical features did help the learner achieve
language competency both in labs and in real field use. These results contribute show
potentials that serious games, if designed properly, can enable the learner to do
productive learning.
133
• Do serious games enable productive learning for everybody?
Findings of our experiments suggest beneficiaries of serious games are those
who have experienced similar situations simulated by game scenes, and therefore who
understand that the knowledge and skills which the system intends to teach were
critical to solve their problems. These learners also have high motivation of learning
the curriculum materials. Learners who have fairly low motivations appeared to
perform less productive learning.
• How can we modify the existing serious game system to reduce learners’
unproductive activities and therefore to promote their learning productivity?
Our practice is based on the iterative system improvement process that
involves continuous evaluation, problem identification and assessment, user
requirement collection and definition, implementation and testing. These
modifications on the existing systems include revision on underlying norms, inputs,
objectives, modeling processes, and codes.
Our investigation on fundamental reasons that causes the unproductive
activities reveals that when learners are outside their ZPD zone, they tend to perform
activities that contribute little to learning, such as “gaming” the game regardless of
their incompetence to practice their skills through game-play. Initial requirements for
system modification thus emerged from the learner’s requirement of adaptive
instructional planning and skill assessment capabilities. At the same time, freedom of
choice is given to learners as a means of motivating them. These requirements
suggest to us to use the guideline of constructing a series of pedagogy features to
134
provide coaching facilities without damaging the motivational game elements. These
features should be seamlessly integrated into a pedagogical framework.
• Should serious games be used as a replacement or supplement to traditional
classroom learning?
Our experiments have shown that the training system achieved best results
when supervised by session proctors or lab supervisors. We argue that although
proctors are recommended for group learning, they are not necessarily experienced
language tutors. But whether serious games can be used as an independent learning
tool without paired with classroom activities needs to be tested. Based on the data
results, we recommend learners with high motivation in studying the target language
to use it as an independent tool.
7.2 Discussions on the Framework

A few pedagogical frameworks have been developed for the intelligent tutoring
systems. However, our work is neither redundant nor trivial.
The work in (VanLehn, 2006) reviews the tutoring devices used in six classic
ITS systems and summarizes the two-structured tutoring framework. The outer loop
of this framework is mainly responsible for macro-level task selection, which can be
categorized into different forms such as learner-controlled task selection, fixed
sequence, mastery learning and macroadaptation based on the initiator and strategy.
In the inner loop of this model, the tutoring service mainly targets at assisting learners
to manage steps of completing a task. The assistance includes feedback of various
135
kinds, such as the minimal feedback (simply indicating correct/incorrect) on a step,
corrective feedback that specifies errors in the last step, next step’s hints, knowledge
assessment, and reviews on the learners’ submitted solutions.
We argue that existing pedagogical frameworks of many traditional intelligent
systems are not suitable for serious games. Firstly, the nature of serious games
prohibits even moderately frequent tutoring interrupts during game-play. For example,
tutoring feedback would hurt the smoothness of plot flow in game scenes. This in turn
hurts the player’s motivations. Moreover, the traditional ITS frameworks do not
emphasize learning opportunities that occur in learner action as much as on those that
occur in learner reflection. In contrast, our pedagogical framework adopts
instructional planning capability to encourage learners to learn from their action, to
review their problems and mistakes in problem-solving in executing their actions, to
find out the source of these problems and mistakes, as well as to adjust their action
based on progress report and skill assessment. Lastly, the past models address little
about maintaining learner’s affect status. The major superiority of serious games over
the traditional education software is its positive effects on learner motivation.
Therefore, tutoring techniques in this context are expected to become an integral part
of the game stimulus.
7.3 Future Work

136
The work this study tries to complete is vast yet the time is limited. Limitation of this
study is inevitable. Yet we hope the work presented in this thesis can light more
studies on serious game research. Suggestions for future work include:
• Detecting more unproductive behavior patterns.
Chapter 3 demonstrated our efforts on detecting and classifying the
unproductive activity patterns during the learner’s interaction with a language
learning serious game. We argue that that abstraction of these patterns is necessary
because it frees us from laborious inspection on seemingly random action. Instead, it
provides us with clearer views on the learner’s intention of formulating their learning
paces as well as problems and mistakes in this process. In this thesis, we have
demonstrated an approach of using computer programs which scan the training logs
and match sequences of actions to the description of pattern language. In our work
three patterns were identified from the training logs. We believe there are more
unproductive activity patterns as the system grows and as the trainees of TLCTS
increase. More inspection on these patterns is necessary for constructing pedagogy
targeting at specific problems existing in learning plan formulation.
• More automation on creating the skill ontology.
The pedagogical framework is built upon the skill assessment, which in turn is
based on the design of skill ontology. To ensure this framework works, well-
organized skill ontology is needed. However, this added overheads to the authors
when develop the curriculum contents, for they had to familiarize themselves with the
skill ontology, which are likely designed by other developers. And the overheads can
137
be very large if skill ontology is complex. In some learning domains, however, new
techniques can be used to facilitate this job. For example, TLCTS has some hundreds
of skill items interwoven in a two-level hierarchy. But as these skills are naturally
linked to linguistic parts (phonetics, morphology, syntax, etc.), NLP techniques can
be used to automate a part of mapping between the utterances and skills.
• More evaluations on the individual pedagogical features.
Our preliminary evaluation dispels the doubts that the tutor advice might hold
back the learner from trying the game and thus may decrease their motivation of
game play. This is considered particularly important as the pedagogical features are
expected to avoid counteracting the motivational factor of game elements. However,
this is only based on the experience of three subjects who are highly motivated and
self-regulated. The future experiments should bring in learners with more diversified
aptitude and experience.
7.4 Conclusion

Rich interactivity and simulation authenticity arms serious games in force. These
systems have enhanced the problem-solving environment of the classic intelligent
tutoring architecture (Poison & Richardson, 1988). However, the occurrence of
unproductive learning in these systems challenges the hypothesis that game elements
are conductive to learning. Exploratory studies were conducted to find out the
underlying reasons that cause the unproductive activities. Deep scrutiny on these
activities exposes learners’ incapability of formulating optimal learning plans due to
138
limited skill development. Findings also show learners with low motivation fail to
benefit from game stimulus. Therefore, additional instructional capability is created
as compensation to coach the learner to reduce the unproductive learning activities as
well as to boost their motivation.
A game-tailored pedagogical framework was proposed to integrate various
coaching services in a coherent entity. Initial evaluations showed the potential of our
work in improving the learning productivity and maintaining the learner’s motivation.
We conclude that serious games should be engineered as coherent whole that
harmonizes the cognitive and game design principles rather than a mere container of
the curriculum contents.
139

References
Baker, E. L. & Mayer, R. E. 1999. Computer-based assessment of problem solving.
Computers in Human Behavior, 15, 269–282.
Baker, R.S., Corbett, A.T., Koedinger, K.R. 2004. Detecting Learner Misuse of
Intelligent Tutoring Systems. Proceedings of the 7th International Conference on
Intelligent Tutoring Systems, 531-540.
Baker, R.S.., Corbett, A.T., Koedinger, K.R., Evenson, E., Roll, I., Wagner, A.Z.,
Naim, M., Raspat, J., Baker, D.J., Beck, J. 2006. Adapting to When Learners Game
an Intelligent Tutoring System. Proceedings of the 8th International Conference on
Intelligent Tutoring Systems, 392-401.
Bernard, R.M., Abrami, P.C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L.,
Wallet, P. A., Fiset, M., & Huang, B. 2004. How does distance education compare
to classroom instruction? A Meta-analysis of the empirical literature. Review of
Educational Research, 74(3), 379-439.
Bernstein, J., Najmi, A. and Ehsani, F. 1999. Subarashii: Encounters in Japanese
Spoken Language Education. CALICO Journal, 16 (3): 361-384.
Bolstad, W.M. 2004. Introduction to Bayesian Statistics. Publisher: John Wiley,
ISBN 0471270202.
Burton, R. & Brown, J. S. 1982. An investigation of computer coaching for informal
learning activities. In D. Sleeman & J.S. Brown (Eds.), Intelligent Tutoring
Systems, Orlando, FL: Academic Press.
Carless, S. 2005. Postcard From SGS 2005: Hazmat: Hotzone - First-Person First
Responder Gaming. [Available from:
http://www.gamasutra.com/features/20051102/carless_01b.shtml]
Chan, T.W. 2007. The four problems of technology-enhanced learning. Plenary
address to AIED 2007.
140
Chi, M.T.H., Siler, S., Jeong, H., Yamauchi, T., & Hausmann, R.G. 2001. Learning
from tutoring. Cognitive Science, 25:471-533.
Chi, M.T.H., Siler, S.A. & Jeong, H. 2004. Can tutors monitor students’
understanding accurately? Cognition and Instruction, 22(3): 363-387.
Clark, R.E. 1983. Reconsidering research on learning from media. Review of
Educational Research, 53(4), 445-459.
Conati, C. Klawe, M. 2000. Socially Intelligent Agents to Improve the Effectiveness
of Educational Games. Proceedings of AAAI Fall Symposium on Socially
Intelligent Agents - The human in the loop.
Cone, B., Irvine, C. E., Thompson, M. F., Nguyen, T.D. 2007. A video game for
cyber security training and awareness. Computers & Security 26(1): 63-72.
Crawford, C. 1984. The Art of Computer Game Design: Reflections of a Master
Game Designer. Mcgraw-Hill Osborne Media, ISBN 0078811171. [Available
from: http://www.vancouver.wsu.edu/fac/peabody/game-book/Coverpage.html]
Deci, E. 1971. Effects of externally mediated rewards on intrinsic motivation. Journal
of Personality and Social Psychology, 18, 105-115.
Deci, E.L. & Ryan, R.M. 1991. Intrinsic motivation and self-determination in human
behavior. In Steers, R.M. & Porter, L.W. (Eds.). Motivation and Work Behavior,
5th Edition. New York: McGraw-Hill, Inc., pp. 44-58.
DeSmedt, W. 1995. Herr Kommissar: An ICALL Conversation Simulator for
Intermediate German. New Jersey: Lawrence Erlbaum Associates, Inc., pp. 153-
174.
Diller, D., Roberts, B. 2005. DARWARS Ambush! A Case Study in the Adoption
and Evolution of a Game-based Convoy Trainer by the U.S. Army. Simulation
Interoperability Standards Organization, 18-23..
Donnelly, J. 2007. SOF Tele-training System. [Available at:
http://www.dtic.mil/doctrine/education/dlcc0407_socomsofts.ppt]
Ellis, R. 1999. Learning a Second Language through Interaction. Publisher: John
Benjamins.
141
ESA, 2007. Essential Facts about the Computer and Video Game Industry.
Entertainment Software Association (ESA). [Available from:
http://www.theesa.com/facts/pdfs/ESA_EF_2007.pdf]
Gee, J. P. 2003. What Video Games Have to Teach Us about Learning and Literacy.
New York: Palgrave Macmillan.
Geoff, J. 2004. Theory Construction in Second Language Acquisition. Publisher:
John Benjamins. ISBN: 9781588114815.
Greene, D. Sternberg, B. and Lepper, M. R. 1976. Overjustification in a token
economy, Journal of Personality and Social Psychology, 34, 1219-1234.
Hamburger, H. 1995. Tutorial tools for language learning by two-medium dialogue.
Intelligent language tutors: Theory shaping technology, V.M. Holland, J.D.
Kaplan, & M.R. Sams (Eds.), pages 183-199.
Harless, W.G, Zier, M.A. and Duncan, R.C. 1999. Virtual Dialogues with Native
Speakers: The Evaluation of an Interactive Multimedia Method. CALICO Journal
16 (3): 313-337.
Henderson, L., Klemes, J. & Eshet, Y. 2000. Just Playing a Game? Educational
Simulation Software and Cognitive Outcomes. Journal of Educational Computing
Research, Vol. 22, No. 1/2000, pp. 105-129.
Huizinga, Johan. (1955). Homo Ludens. The Beacon Press. ISBN: 9788420635392.
Johnson, W.L. and Beal, C. 2005. Iterative evaluation of a large-scale, intelligent
game for language learning. Artificial Intelligence in Education, IOS Press,
Amsterdam.
Johnson, W.L., Beal, C., Fowles-Winker, A., Lauper, U., Marsella, S., Narayanan, S.,
Papachristou, D., & Vilhjálmsson, H. 2004. Tactical Language Training System:
An Interim Report. Proceedings of the 7th Intelligent Tutoring Systems
Conference, Berlin: Springer.
Johnson, W.L., Vilhjalmsson, H. & Marsella, M. 2005. Serious Games for Language
Learning: How Much Game, How Much AI?, 12th International Conference on
Artificial Intelligence in Education , July 18-22, Amsterdam, The Netherlands.
142
Jonnavithula L. & Kinshuk. 2005. Exploring Multimedia Educational Games: An Aid
to Reinforce Classroom Teaching and Learning. In Uskov V. (Ed.), Proceedings of
the 4th IASTED International Conference on Web-Based Education (WBE 2005),
Anaheim, CA, USA: ACTA Press, 22-27.
Kallgren and Wood. 1986. Access to attitude-relevant information in memory as a
determinant of attitude-behavior consistency. Journal of Experimental Social
Psychology, 22, 328-338.
Kaplan, J.D., and Holland, V.M. 1995. Application of learning principles to the
design of a second language tutor. V.M.,Kaplan,J.D.,and Sams,M.R.(Eds.),
Intelligent Language Tutors: Theory Shaping Technology, 273-287. Lawrence
Erlbaum Associates,Mahwah, NJ.
Kirkpatrick, D. L. 1994. Evaluating training programs: the four levels, Publisher:
Berrett-Koehler. ISBN-13: 978-1576750421.
Klawe, M.M. 1998. When Does the Use of Computer Games and Other Interactive
Multimedia Software Help Learners Learn Mathematics? NCTM Standards 2000
Technology Conference, Arlington, VA.
Klawe, M.M., Super, D. & Westrom, M.L. 1995. Counting on Frank: Postmortem of
an Edutainment Product. [Available from:
http://www.cs.ubc.ca/nest/egems/home.html]
Klein, J.H., 1985. The Abstraction of Reality for Games and Simulations. The
Journal of the Operational Research Society, Vol. 36, No. 8, pp. 671-678.
Korris, J. 2004. Full Spectrum Warrior: How the Institute for Creative Technologies
Built a Cognitive Training Tool for the XBox. 24th Army Science Conference.
Koster, R. 2004. A Theory of Fun for Game Design. Publisher: Paraglyph. ISBN
1932111972.
LaPointe, D.K., Greysen, K.R.B., Barrett, K. 2004. Speak2Me: Using Synchronous
Audio for ESL Teaching in Taiwan. International Review of Research in Open and
Distance Learning, Vol 5, No 1, ISSN: 1492-3831.
Lepper, M.R. and Cordova, D.I. 1992. A desire to be taught: Instructional
consequences of intrinsic motivation. Motivation and Emotion, 16(3), p187-208.
143
Lepper, M. R., & Malone, T.W. 1987. Intrinsic motivation and instructional
effectiveness in computer-based education. In R. E. Snow & M. J. Farr (Eds.),
Aptitude, learning, and instruction: Conative and affective process analyses, Vol. 3.
pp. 255-286, Hillsdale, NJ: Lawrence Erlbaum.
Malone, T.W. 1981. Towards a theory of intrinsically motivating instruction.
Cognitive Science, 5, 333-370.
Malone, T. W., & Lepper, M. R. 1987. Making Learning Fun: A Taxonomy of
Intrinsic Motivations for Learning. R. E. Snow & M. J. Farr (Eds.), Aptitute,
Learning and Instruction: III. Conative and affective process analyses, pp. 223-
253, Hilsdale, NJ: Erlbaum.
Maslow, A.H. 1943. A theory of human motivation. Psychological Review, 1, 370-
396.
McClelland, D. C., & Burnham, D. H. 1976. Power is the great motivator. Harvard
Business Review, 54(2), 100-110.
McClelland, D. C. 1975. Power: The inner experience. Publisher: John Wiley & Sons.
ISBN-13: 978-0829001013.
McGrath, D., Hill, D. 2004. UnrealTriage: A Game-Based Simulation for Emergency
Response. The Huntsville Simulation Conference, Sponsored by the Society for
Modeling and Simulation International.
Michael, D. & Chen, S. 2005. Serious Games: Games That Educate, Train, and
Inform. Publisher: Thomson Course Technology. ISBN 1592006221.
Murray, T. 1999. Authoring Intelligent Tutoring Systems: An analysis of the state of
the art. International Journal of Artificial Intelligence in Education, 10: 98-129.
Neher, A 1991. Maslow’s Theory of Motivation: A Critique. Journal of Humanistic
Psychology, Vol. 31, No. 3, 89-112.
O'Neil, H., Wainess, R. & Baker, E. (2005). Classification of learning outcomes:
evidence from the computer games literature. The Curriculum Journal, Volume 16,
Number 4, Number 4/December 2005, pp. 455-474(20).
144
Ong, J. & Ramachandran S. (2000) Intelligent Tutoring Systems: The What and the
How. [Available at: http://www.learningcircuits.org/2000/feb2000/ong.htm]
Parchman, S. W., Ellis, J. A., Christinaz, D. & Vogel, M. 2000. An evaluation of
three computer-based instructional strategies in basic electricity and electronics
training, Military Psychology, 12, 73–87.
Pintrich, P.R., & Schunk, D.H. 2002. Motivation in education: theory, research, and
applications (2
nd
ed.). Publisher: Prentice Hall. ISBN-13: 9780135036532.
Prensky, M. 2001. Digital Game-Based Learning, Publisher: McGraw-Hill. ISBN
9780071454001.
Quinn, C. N. 2005. Engaging Learning: Designing e-Learning Simulation Games,
Publisher: Jossey Bass Wiley. ISBN: 978-0-7879-7522-7.
Randel, J. M., Morris, B. A., Wetzel, C. D. & Whitehill, B. V. 1992. The
Effectiveness of Games for Educational Purposes: A Review of Recent Research,
Simulation and Gaming, vol. 23 no. 3 : 261–76.
Rieber, L.P. 1996. Seriously considering play: Designing interactive learning
environments based on the blending of microworlds, simulations, and games.
Educational Technology Research & Development. 44(2), 43-58.
Rieber, L.P. 2001. Designing learning environments that excite serious play. In
Proceedings of the Annual Conference of the Australian Society for Computers in
Learning in Teritary Education (ASCILITE), Melbourne, Australia, Dec. 9-12,
2001.
Ricci, K., Salas, E., & Cannon-Bowers, J.A. 1996. Do computer-based games
facilitate knowledge acquisition and retention? Military Psychology, 8(4), 295-307.
Salomon, G. 1984. Television is “easy” and print is “tough”: the differential
investment of mental effort in learning as a function of perceptions and attributions.
Journal of Educational Psychology, 76(4), 647-658.
Sams, M. 1995. Advanced Technologies for Language Learning: The BRIDGE
Project Within the ARI Language Tutor Program. In Holland,V.M.,Kaplan,J.D.,and
Sams, M.R.(eds), Intelligent Language Tutors: Theory Shaping Technology ,273-
287. Lawrence Erlbaum Associates,Mahwah, NJ.
145
Shaffer, D., Squire, K.R., Halverson, & Gee, J.P. 2004. Video Game and the Furture
of Learning, Wisconsin Center for Education Research Working Papers.
Sherman, D.K. and Kim, H.S. 2002. Affective Perseverance: The Resistance of
Affect to Cognitive Invalidation, Personality and Social Psychology Bulletin, Vol.
28 No. 2, 224-237.
Squire, K.R. & Jenkins, H. 2003. Harnessing the power of games in education,
Insight, Vol. 3, No. 1, pp. 5-33.
Surface, E.A., Dierdorff, E.C., Watson, A.M. 2007. A.: Special Operations Language
Training Software Measurement of Effectiveness Study: Tactical Iraqi Study Final
Report. Special Operations Forces Language Office, Tampa, FL, USA.
Szczurek, M. 1982. Meta-Analysis of Simulation Games Effectiveness for Cognitive
Learning, Ph.D. thesis., Indiana University.
Tamkin, P., Yarnall, J., Kerrin, M. 2002. Kirkpatrick and Beyond: A review of
models of training evaluation. Report 392, Institute for Employment Studies.
Urban-Lurain, M. 2002. Intelligent Tutoring Systems: An Historic Review in the
Context of the Development of Artificial Intelligence and Educational Psychology.
[Available at: http://www.cse.msu.edu/rgroups/cse101/ITS/its.htm]
VanLehn, K. 2006. The behavior of tutoring systems. International Journal of
Artificial Intelligence in Education.
Vygotsky, L.S. 1978. Mind in Society: The development of higher psychology
processes. Publisher: Harvard University Process. ISBN: 978-0674576292.
Zhou, Y. & Evens, M. W. 1999. A Practical Student Model in an Intelligent Tutoring
System. Proceedings of the 11th IEEE International Conference on Tools with
Artificial Intelligence. Chicago, IL, pp. 13-18.
146
Appendices
Appendix A. Source File List
{Installation_Path}\Content\Iraqi

config_basis.xml: the configuration file that defines the parameters and all the
possible values for the parameters.

default_settings.xml: the file that stores the default values for the parameters
defined in the config_basis.xml

\PythonSources\Python

TLMessageProcessor.py: the superclass that defines the default message
handling behaviors.

TLScoreManager.py: the class that manages score schema and score updating.

TLScaffoldingManager.py: the class that was originally designed for
managing dyanmic scaffolding levels but was replaced by the
TLSettingPage.py

EventXMLLogger.py: the subclass to TLMessageProcessor, which serializes
the system event messages to XML log files. This class is wrapped in a
singleton class that regulates only one instance existing throughout the system
running.

\PythonSources\Python\pedagent

__init__.py: initialization file that defines the interface from which outside
components can access the pedagogical agent. __pa is a global singleton
instance. pa_notify() relays any message it receives from other components to
the internal __pa instance.

PedTalker.py: the class that defines the initialization behavior of the
pedagogical agent class as well as the message handling methods.

AFB.py: the class that handles progress report display, retrieving the quiz and
skill data from the Learner Model and sending to the TLMissionManager.

147
TAB.py (deprecated): the class that was originally designed for tutor advice
display but was replaced by the messaging system of Tutor Agent

\PythonSources\Python\pedagent\LM

__init__.py: package initialization file that stores the version, author
information

LearnerModel.py: the class that defines the Learner Model behaviors, such as
loading/saving learner profiles, updating the learner's progress, and
receiving/sending messages. It includes the training objectives tracker, a
learner progress tracker and a learner progress generator.

KonaData.py: the class that retrieves the curriculum data from a shelve object
that loads the *.sb file. It creates a cache for lesson titles and quiz profiles.

SkillModel.py: the class that manages the skill profile updating.

SkillOntology.py (deprecated): contents have been merged to the
SkillModel.py.

SkillTracingBayesNet.py: the class that creates and maintains a Bayes Net and
manages the skill level updating.

SkillMapObject.py (deprecated): was used by the old skill model for progress
report display.

SkillOntology.xml: the xml file that stores all the skill information, including
the skill name and skill hierarchy

SkillsBayesNet.dsl: the Bayes network file that creates a training net through
pySMILE.

SkillTraicingProbabilities.xml: the xml file that stores the probabilities within
the conditional probability tables between two nodes.

\PythonSources\Python\pedagent\TA

__init__.py: package initialization file that stores the version, author
information

TutorAgent.py: the class that defines the Tutor Agent behavior.
148

ZPD.py: the class that evaluates the learner's relative position to the ZPD zone.

\PythonSources\Python\pedagent\TA\ontology

__init__.py: package initialization file that stores the version, author
information.

QualNet(deprecated): was used for evaluating the qualification of the learner
to proceed to the next level, replaced by the ZPD.py file.

\PythonSources\Python\pedagent\TA\ontology\Iraqi

__init__.py: package initialization file that stores the version, author
information.

Curriculums.py: the class that stores the default lesson-skill, scene-skill
mappings.

Language.py: the class that stores the default language knowledge.

\PythonSources\Python\pedagent\TA\ontology\Iraqi\US_Marine_Corps

__init__.py: package initialization file that stores the version, author
information.

Curriculums.py: the class that stores the lesson-skill, scene-skill mappings for
the Iraqi US Marine Corps version (P1)

Language.py: the class that stores the Iraqi language knowledge for US
Marine Corps.

149

Appendix B. Evaluation Survey Questionnaire
This experiment is to evaluate the pedagogical features of the Tactial Iraqi
TM
Version
5.0 system. Please make a note when you ignore the tutor advice or when you think
any skill scores that incorrectly reflect your actual skill levels.
1. Login in the system using your user name same as your email account. Skip
the password field.
2. Complete the Interview Lesson
3. Adjust the Settings to your preference (on the Setting page)
4. Complete the following tasks (you may arrange the order of doing these tasks
in your own way):
a. Complete Lesson "Meeting Strangers", "Introducing Your Team",
"Social Etiquette", and "Say it in Iraqi". Try to finish the exercise
pages, active dialogs and review quizzes as much as possible.
b. Play Scene 1 (Episode 1, 2, 3) and Scene 2 (Episode 1, 2, 3). Try to
complete them.
c. Play Arcade Game Level 1&2 (speaking & listening mode). Try to
score high.
d. You may follow the tutor advice or ignore it. When you ignore it,
please make a note about the current situation and the reason why you
choose to ignore it.
150
5. Answer the following questions about the tutor advice, please.
a. Rate the usefulness of tutor advice in a range of 1-5 (1=not useful at all,
5=very useful)
b. How useful was the tutor advice before you took a quiz?
c. How useful was the tutor advice after you took a quiz?
d. How useful was the tutor advice before you started a mission scene?
e. How useful was the tutor advice after you finished a mission scene?
f. What's your expectation on tutor advice? Does the current tutor advice
service meet your expectation?
g. Do you agree the tutor advice prevents you from taking a quiz or a
mission scene? (1=don't agree; 3=somehow agree; 5=strongly agree)
h. Do you agree that you were coerced to follow the tutor advice?
(1=don't agree; 3=somehow agree; 5=strongly agree)
i. Do you agree that the tutor advice increases your motivation for
language learning? (1=don't agree; 3=somehow agree; 5=strongly
agree)
j. Do you agree that the tutor advice increases your motivation for
playing the games? (1=don't agree; 3=somehow agree; 5=strongly
agree)
6. Answer the following questions about the progress report, please.
a. How often did you check the progress report? (1=never checked it;
5=very often)
151
b. How accurate was the skill part of progress report? (1=not accurate; 5=
very accurate)
c. How useful was the progress report? (1=not useful at all; 5=very
useful)
d. Do you agree that the progress report increased your motivation for
language learning? (1=don't agree; 3=somehow agree; 5=strongly
agree)
e. Do you agree that the progress report increased your motivation for
playing the games? (1=don't agree; 3=somehow agree; 5=strongly
agree)
7. Answer the following questions about the game scoring, please.
a. How useful was the score debriefing in active dialogs? (1=not useful at
all; 5=very useful)
b. How useful was the scores in mission scene? (1=not useful at all;
5=very useful)
c. Do you agree that the scores increased your motivation for language
learning? (1=don't agree; 3=somehow agree; 5=strongly agree)
d. Do you agree that the scores increased your motivation for playing the
games? (1=don't agree; 3=somehow agree; 5=strongly agree)
8. Answer the following questions about the game subtitles, please.
a. What subtitles did you use, English, phonetics or mixed?
152
b. How useful was the subtitles in the mission game? (1=not useful at all,
5=very useful)
c. Do you agree that you relied upon the subtitles to complete the mission
games? (1=don't agree; 3=somehow agree; 5=strongly agree)
9. Answer the following questions about the hint facility, please.
a. How often did you request hints? (1=never requested a hint;5=often)
b. What kind of hints do you prefer, candidate acts or utterances?
c. How useful was the hint facility in the mission game? (1=not useful at
all, 5=very useful)
10. Tell me something about your Learning Attitude.
a. Are you are self-regulated learner? (1=not self-regulated; 3=somewhat
self-regulated; 5=very self-regulated)
b. Rate your self-efficacy, please? (self-efficacy refers to as people's
belief about themselves capable of in a certain manner to attain certain
goals)
(1=low; 3=medium; 5=high)

Abstract (if available)

Abstract Recent years have seen a large number of game-based training systems or serious games developed for diversified learning domains. Despite the hypothesis that computer games are motivator to promote learning engagement, however, researchers reported various problems existing in these systems. One of the intractable problems, for example, is that games incentives may direct learners to unproductive learning activities, diverging from the original intention of educational software designers. Skeptics began to question the worth of employing game techniques in training systems, as constructing a serious game faces relatively longer development cycle and consumes more expensive resources.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Learning social sequential decision making in online games

PDF

No-regret learning and last-iterate convergence in games

PDF

Machine learning in interacting multi-agent systems

PDF

Heterogeneous federated learning

PDF

Modeling the learner's attention and learning goals using Bayesian network

PDF

Hierarchical planning in security games: a game theoretic approach to strategic, tactical and operational decision making

PDF

Transfer learning for intelligent systems in the wild

PDF

Learning about the Internet through efficient sampling and aggregation

PDF

Learning controllable data generation for scalable model training

PDF

High-throughput methods for simulation and deep reinforcement learning

PDF

Towards understanding language in perception and embodiment

PDF

Improving decision-making in search algorithms for combinatorial optimization with machine learning

PDF

Single-image geometry estimation for various real-world domains

PDF

Behavior-based approaches for detecting cheating in online games

PDF

Sample-efficient and robust neurosymbolic learning from demonstrations

PDF

Leveraging prior experience for scalable transfer in robot learning

PDF

Reward shaping and social learning in self- organizing systems through multi-agent reinforcement learning

PDF

Identifying and mitigating safety risks in language models

PDF

Learning from planners to enable new robot capabilities

PDF

Theoretical foundations for dealing with data scarcity and distributed computing in modern machine learning

Asset Metadata

Creator Wu, Shumin (author)

Core Title Reducing unproductive learning activities in serious games for second language acquisition

School Viterbi School of Engineering

Degree Doctor of Philosophy

Degree Program Computer Science

Degree Conferral Date 2008-12

Publication Date 11/03/2008

Defense Date 09/03/2008

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag AIED,education,learning productivity,Motivation,OAI-PMH Harvest,serious games,SLA

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Johnson, William (committee chair), Clark, Richard (committee member), Huang, Ming-Deh (committee member)

Creator Email shumin.wu@gmail.com,wushumin@us.ibm.com

Permanent Link (DOI) https://doi.org/10.25549/usctheses-m1742

Unique identifier UC193305

Identifier etd-Wu-2353 (filename),usctheses-m40 (legacy collection record id),usctheses-c127-127486 (legacy record id),usctheses-m1742 (legacy record id)

Legacy Identifier etd-Wu-2353.pdf

Dmrecord 127486

Document Type Dissertation

Rights Wu, Shumin

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Repository Name Libraries, University of Southern California

Repository Location Los Angeles, California

Repository Email cisadmin@lib.usc.edu