University of Southern California Dissertations and Theses

Modeling the learner's attention and learning goals using Bayesian network

(USC Thesis Other)

Modeling the learner's attention and learning goals using Bayesian network

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content MODELING THE LEARNER’S ATTENTION AND LEARNING GOALS USING BAYESIAN NETWORK by Lei Qu A Dissertation Presented to the FACULTY OF THE GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (COMPUTER SCIENCE) August 2007 Copyright 2007 Lei Qu ii Dedication To my parents and family. iii Acknowledgements This dissertation could not have been completed without the support of many hearts and minds. First of all, I would like to thank my PhD committee members; in particular, my advisor Dr. W. Lewis Johnson and co-advisor Carole Beal. I am deeply indebted to them for their advice, mentoring, and support throughout my academic program. I also truly appreciate their kindness and generosity which made my study life more smooth and enjoyable. My sincere thanks are also extended to other committee members Dr. Barry Boehm, and Dr. Lynn Miller, for the invaluable efforts on reviewing the draft of my dissertation. I would also like to express my thanks and gratitude to several other people. Dr. David Pynadath provided graciously of his time in helping me building Bayesian models. My special thanks to some other colleagues, including Shumin Wu, Ning Wang and Mei Si, the discussions with whom have greatly benefited my study. Lastly, I would like to thank my family. Your support over pass years is greatly appreciated. iv Table of Contents Dedication ii Acknowledgements iii List of Tables vi List of Figures vii Abstract viii Chapter 1 Introduction 1 1.1 Motivation 1 1.2 History of ITSs 3 1.3 Nature of the problem 6 1.4 Research Approach and Propositions 9 1.5 Dissertation Outline 10 Chapter 2 A Survey of Related Work 12 2.1 Eye Gaze Tracking Systems 12 2.2 Attention Tracking Systems 13 2.3 Motivation and Emotion Detection Systems 15 2.4 Conclusion 19 Chapter 3 Background Study 20 3.1 Interaction between human Tutor and Students 20 3.1.1 Purpose of this Study 20 3.1.2 Study design 20 3.1.3 Data Collection 22 3.1.4 Data Analysis Results 23 3.2 Study for Math Teaching using ITS 24 3.2.1 Purpose of this Study 24 3.2.2 Study design 24 3.2.3 Data Collection 25 3.2.4 Data Analysis Results 27 Chapter 4 Attention Tracking Model 29 4.1 Tracking Learner Focus of Attention under Uncertainty 31 4.2 Dynamic Bayesian Network (DBN) 31 4.3 Event Capture System 32 4.4 Certain and Uncertainty Events 34 4.5 Modeling Learner Focus of Attention 35 4.6 Evaluation of Attention Model 39 4.7 Summary 41 v Chapter 5 Learner Attention Classifier 43 5.1 Approach and Data Sources 43 5.1.1 Tutoring System 43 5.1.2 Student participants 44 5.1.3 Motivation profiles 44 5.1.4 Motivation Profile 45 5.1.5 Teacher ratings 45 5.1.6 Student data records 45 5.2 Learner’s Attention Pattern 46 5.3 Results and Discussion 47 5.3.1 Learner Motivation 47 5.3.2 Teacher Ratings 48 5.3.3 Attention Patterns 48 5.4 Conclusions 56 Chapter 6 Inferring Learning Goals 58 6.1 WAYANG Test Bed 58 6.2 Attention Pattern Definition 60 6.3 Learner’s Learning Goals 61 6.4 Modeling Learning Goals 62 6.5 Model Initialization 65 6.6 Model Evaluation and Results 67 6.6.1 Scoring 68 6.7 Results 69 6.8 Discussion 75 Chapter 7 Contribution and Future Work 80 7.1 Contributions 80 7.2 Areas of Future Work 81 References 82 vi List of Tables Table 3.1 Learner Trait Assessment 27 Table 3.2 Learner Daily Mood Assessment 27 Table 4.1 Probabilities After T Seconds 35 Table 4.2 Average Accuracies for Different Windows 41 Table 4.3 Accuracies of Attention Tracking Model 41 Table 5.1 Mean Scores on Motivation Profile by Group 48 Table 5.2 Mean Proportion for Attention Patterns 52 Table 6.1 Descriptions of Attention Pattern 60 Table 6.2 Cross-tab of HS and IS Groups 73 vii List of Figures Figure 1.1 Five Components of ITSs 6 Figure 3.1 Virtual Factory Teaching System 21 Figure 3.2 Tutoring Window 22 Figure 3.3 Wayang System 25 Figure 4.1 Screen Shot of Eye Gaze Tracking Program 30 Figure 4.2 Screenshot of Eye Tracking Program 36 Figure 4.3 Attention Tracking Model 38 Figure 4.4 Dynamic Attention Tracking Model 39 Figure 4.5 Comparison of ETP, LA and BNM 40 Figure 5.1 Summed Proportion for Individual Student 49 Figure 5.2 Mosaic plot BY Motivation and Attention Group 53 Figure 6.1 The Interface of Wayang Outpost 59 Figure 6.2 Probabilistic Model to Infer Learning Goals 63 Figure 6.3 The Distribution of Students’ Pre-test Scores 69 Figure 6.4 The Distributions for Learning Goals 70 Figure 6.5 The Relation Between Pre-test and Independent Study Scores 71 Figure 6.6 Mean Scores of Pre-test for IS Groups 72 Figure 6.7 Mean Pre- and Post-test Scores for Learning Goals Groups 75 viii Abstract Intelligent Tutoring Systems (ITSs) have evolved dramatically from the simple prompt for remediation based on a wrong answer to the complex, adaptive systems of today that truly qualify as intelligent. Modern ITSs include a variety of applications that allow the emulation of a human teacher and the ITS acts as the student's private tutor, and interacts to effectively lend pedagogical assistance to the learner. Researchers have focused on modeling the learner’s cognitive processes while solving problems, i.e., the “model tracing” approach. However, there is growing recognition that learning involves more than cognition, and that students’ attention with an ITS also reflects “engagement” meaning the transient shifts in focus of attention and the emotions that are often associated with learning. Failure to track the learner’s engagement could cause the ITSs to interrupt the learner with advice when the learner does not really need it. The primary research questions addressed in this dissertation are (1) Can an ITS access learner’s focus of attention and how? (2) If so, can learner’s attention signals be used to infer learner’s engagement? The research work in this dissertation provides a probabilistic model for ITSs to track learner’s attention based on multiple sources of user input data. The learner’s attention sequences thus were machine classified into five finite-state machines indicating guessing strategies, appropriate help use, and independent problem solving; over 90% of problem events were categorized. Students were grouped via cluster analyses based on self reports of motivation. This indicated that students learned by ix themselves and learned from instructions, which are two primary learning goals when students were studying in ITSs. And a Dynamic Bayesian Network (DBN) was built to enable ITSs to have the ability to detect the attention shifts and learning states. With this ability, ITSs will be more sensitive to the cognitive and motivational states of the learner and then be able to promote the learner’s motivation through interaction with the learner. To validate this Dynamic Bayesian Network Model, experiments have been designed and performed on undergraduate students at University of Southern California and high school students at Los Angeles area. The evaluation results show that applying this model should enable ITSs to track learner’s attention and recognize learner’s learning goals. 1 Chapter 1 Introduction 1.1 Motivation Tell me and I forget. Show me and I remember. Involve me and I understand. - Chinese proverb The world of education has changed from an orderly world of disciplines and courses to an infosphere in which communication technologies have become increasingly important. In the 1960’s, researchers started looking for new educational paradigms to take advantage of breakthroughs in computer technology. Since then, there has been a consensus that the combination of artificial intelligence, cognitive science and advanced technologies could dramatically improve learning and problem solving. One such paradigm is known as intelligent tutoring systems (ITSs) or intelligent computer-aided instruction (ICAI), and has been increasingly pursued for more than three decades by researchers in education, psychology, and Artificial Intelligence (AI). ITSs have evolved dramatically from the simple prompt for remediation based on a wrong answer to the complex, adaptive systems of today that truly qualify as intelligent. Modern ITSs include a variety of applications that allow the emulation of a human teacher and the ITS acts as the student's private tutor, and interacts to 2 effectively provide pedagogical assistance to the learner. ITSs have been shown to be highly effective at increasing students' performance and motivation. For example, students using Smithtown, an ITS for economics, performed equally well as students taking a traditional economics course, but required half as much time covering the material [Shute 89]. Tutoring system researchers have focused on modeling the learner’s cognitive processes while solving problems, i.e., the “model tracing” approach. However, there is growing recognition that learning involves more than cognition, and that students’ attention with an ITS also reflect “engagement,” meaning the transient shifts in focus of attention and the emotions that are often associated with learning [Beck 05, Qu 05, Vicente 02, Johnson 05]. For example, a student may become bored or fatigued over the course of a session and deliberately enter incorrect answers in order to elicit the correct answer from the ITS [Baker 05]. Failure to track the learner’s engagement could cause the ITSs to interrupt the learner with advice when the learner does not really need it. This has presented many challenges to ITSs. Some of these challenges are: • The learner’s attention is characterized by a vast array of interaction data, • Uncertainty exists in tracking learner’s focus of attention, and • Difficulty in interpreting learner’s engagement based upon the learner’s attention and actions. 3 Without a well thought model, many ITSs will fail to motivate learners engaged in learning. The motivation behind this research is to design a framework, which can enable ITSs to track the learning attention, infer learners’ learning goals, and eventually interact with learners in more appropriate and effective ways. 1.2 History of ITSs ITSs originated in the AI movement of the late 1950's and early 1960's. Then, researchers such as Alan Turing, Marvin Minsky, John McCarthy and Allen Newell thought that computers that could "think" as humans do were just around the corner. Education in the United States during this period provided fertile ground for the development of ITS after World War II. Educational institutions were required to meet the challenges of preparing larger proportions of their students back from war to attend college. To do so, they sought ways to increase the efficiency of instruction. As Berliner notes: The War did not require theoretical elegance from its psychologists. It required solving practical problems, not laboratory problems, such as the problem of rapidly teaching masses of men to reach acceptable levels of competency in hundreds of specialty areas. With the help of psychologists, the task was accomplished [Berliner 92]. In this environment, people tried to use computer technology to meet the increasing demands of education. Researchers built a number of Computer Assisted 4 Instructional (CAI) systems [Uhr 69]. These systems generated problems which are designed to enhance student performance in skill-based domains, primarily arithmetic and vocabulary recall. Most of the system designers' efforts were devoted to the technical challenges of implementing these systems. These systems did not address the issues of how people learn. They assumed that, if systems presented information and knowledge of particular domains to the learner, the learner would understand it. In the 1970's, many researchers began to consider the student as a factor in the instructional system [Suppes 67]. Many systems have been developed to respond to users based on the history of the user’s responses. These systems were the first to model students, although they only modeled the students’ behavior and not on any attempt to model their other states, such as knowledge, affective and cognitive. Even though these systems were relatively simple, students who used these systems improved on measures of the relevant skills. About this time, psychology in education was questioning the assumptions of behaviorism. Piaget's theories of learning and constructivism began to take hold. Chomsky, along with Newell and others, introduced the ideas of symbolic information processing [Greeno 94], ideas that dovetailed with the AI community's interests in linguistics and natural language processing. In 1982, Sleeman and Brown first introduced the term Intelligent Tutoring Systems (ITSs) to describe these evolving systems and distinguish them from the previous CAI systems. ITSs can be classified [Sleeman 82] as 5 • Problem-solving monitors, which monitor results of students' problem-solving processes. • Coaches, which teach students knowledge in particular domains, • Laboratory instructors, which instruct students laboratory knowledge and tests, and • Consultants, which provide expert suggestion based on user's requirements. These ITSs focus on the representation of a domain expert's knowledge, an instructor's knowledge, and the particular student that is being taught. In 1990's, researchers identified the following models that constitute an ITS [Beck 96]: • The Student Model stores information that is specific to each learner. • The Pedagogical Model defines the teaching strategies that the ITS will employ. • The Domain Model contains the information about the subject being taught and as such usually constitutes the most area. • The Communications Model includes the screen layout and dialogues with the learner. • The Expert Model is a model of how someone who is an expert in that domain would represent this information. As shown in Figure 1.1, Expert Model, Pedagogical Module and Domain Knowledge will provide input data to Student Model, and then Student Model gives 6 students’ state data to Pedagogical Module. At last, the Pedagogical Module will interact with the students through Communication Model. In Figure 1.1, Student Model is one of the most important components in this framework, and it’s performance will affect the accuracy and efficiency for the interaction between ITSs and students. Figure 1.1 Five Components of ITSs 1.3 Nature of the problem There is a consensus that ITSs can motivate students and improve students' learning results by providing various interactive actions. ITSs have the ability to generate highly detailed feedback about problem solving [McArthur 93]. They can coach and model problem solving down to "atomic" levels of reasoning. The converse learning principle argues that learners need rich, variable granularity feedback [Anderson 87]. When learners accomplish a task they use their skills along with external tools to generate reasoning and visible performance. Although learning can happen with nominal feedback, generally richer feedback yields more accurate diagnosis of errors, thus faster learning. The need for rich feedback is 7 especially important when tasks are authentic and skills are embedded. Because so many skills may be used in the process of generating an answer or step, it may be difficult for students to locate their errors among many acceptable actions (often referred to as the "credit assignment" problem) and to draw a general inference from the errors, without detailed feedback. In most cases, the ITS selects the next task or problem, decides when the student needs support and feedback in problem solving, and determines the nature of the information the students receive. Students may tailor information; for example they may request more detailed explanations. But their latitude is usually highly circumscribed. The principle of high tutor control reflects an implicit belief that a competent tutor is usually in a better position to make decisions about what experiences and information students need to learn effectively than the students themselves. Of course, this assumes, at a minimum, that the tutor knows the content the students want to learn, and also knows the students' specific knowledge state -- what they know, and what knowledge they lack -- at any given time. The expert systems and student models of ITSs attempt to provide this expertise and to thus meet the demands of high tutor control of learning. A related feature of ITSs is that they are stimulated to action by student difficulties or impasses. In the tutor-controlled version of ITSs, for instance, feedback and help can be triggered by a user's error. By organizing learning around "atomic" tasks, and by choosing tasks that are demanding for students, ITSs attempt to maximize opportunities for impasses. But, the decision-making in response to 8 impasses for many ITSs is quite "thin". Interventions are immediate reactions to errors alone. For example, they are not conditioned by a plan to set up the learning experience beforehand or to help students define goals or plans. However, whether ITS reason extensively or not, virtually all reasoning attempts to recognize student impasses and overcome them. The learning principle that corresponds to teaching principle of impasse-driven coaching is to provide immediate feedback [Anderson 87]. Most ITSs, implicitly or explicitly, are built on the premise that a good learning system will provide detailed feedback as soon as an impasse is detected. Within the ITSs, the opportunity for enhancing the ability of ITS intervention is to treat learner's attention as a central construct and organizing principle. Learner’s attention is a critical factor in reasoning the intervention time and strategies by ITSs. We should consider use learner’s focus of attention as an important source of rich signals about learning goals, motivation and interest of the learner. Although there is a rich history of prior work on attention from cognitive psychology, we found there is still much we do not yet understand. Thus, beyond pooling results from previous psychological studies, we need to perform user studies that adopt or extend prior studies [Cutrell 02, Czerwinski 00, Scott 03]. Also there is growing recognition that learning involves more than cognition. This requires the need for new methods for ITSs. This method should be able to utilize learner attentional cues and trace not only cognition but also engagement of the learner. And then ITSs can be able to interact with learners in more appropriate and social ways. Therefore, the principal research questions being addressed in this study are: 9 In ITSs, how can we enable ITSs to have the ability to track learner's focus of attention, and how can we treat the learner’s attention as a construct and organizing principle to model affective states of the learner? 1.4 Research Approach and Propositions Although attention and motivation of the learner plays important roles during the interaction between ITSs and students, relatively little is known about modeling them while using an ITS. One approach is to use special sensors to capture attentional signals or physiological indices of emotional states such as interest, boredom, and frustration [D’Mello 05]. But the ability to integrate estimates of engagement into learner models has been constrained by the difficulty of using sensors in field-based learning environments, i.e., outside of the laboratory. For example, few school classrooms are equipped with gaze-tracking cameras or devices to sense student posture or pressure grip, at least not in numbers sufficient for classes that may include 35 or more students in the computer lab. Thus, our focus is on non-intrusive alternatives to building rich learner models from students’ interactions with the ITS interface: models that can be used to infer the learner’s learning goals and interpret his or her attention in order to deploy effective instruction and intervention. The focus of this thesis is on developing rich student model that can access student's focus of attention, and utilize it to model student's learning goals. The main research goals include: 10 • To examine the interaction between the tutor and the learner. It is important to understand how the human tutor monitors students in order to infer students’ focus of attention. What information does human tutor use? • To enable ITSs to access the multiple sources of information of learner interface actions like human tutors. These multiple sources information can include keyboard event, mouse event or learners’ eye gaze. • To model the reasoning of the learner’s learning goals. Even ITSs have the ability to access multiple sources of user input data and infer learner's attention, it is still unclear that how learner's attention can be related to their learning goals. For example, selecting the correct answer for a math problem can either represent the students try to solve the problem by themselves or guessing. Then it is necessary to investigate how learner's learning goals can be integrated by learner's attention and activities. • To enable ITSs to infer learner's learning goals using learner's focus of attention. And provide rich models of student for ITSs to help them to intervene the students at appropriate time with suitable strategies. 1.5 Dissertation Outline Chapter 2 surveys related-works in the areas of addressing attention tracking and cognition modeling in ITSs. 11 Chapter 3 presents the results from two experiment background studies and introduces two test beds for the thesis work of establishment of the attention tracking model and learning goal model. Chapter 4 describes the attention tracking model based on Dynamic Bayesian Network (DBN) to track learner’s focus of attention by utilizing multiple sources of attention cues. The evaluation results of this approach are also discussed. Chapter 5 presents the attention classifier for classifying learner’s attention when interacting with an ITS. And the evaluation results of this classifier are also discussed. Chapter 6 describes the learning goal model to infer learning goals of learner in real time. On top of learner’s attention patterns in an ITS, two types of learning goals are summarized. Then these two learning goals are modeled using DBN and evaluate results are discussed. Finally, Chapter 7 summarizes the contributions of this thesis study and proposes several areas for future research. 12 Chapter 2 A Survey of Related Work In this section, we overview some works related to provide learner instruction and intervention by tracking learner attention or assessing learner’s cognitive and affective state. 2.1 Eye Gaze Tracking Systems In the classroom, one of the important elements between teacher-student communications is known as the attention of the people. Teacher needs to attract students’ attention in order to effectively teach them. And students’ eye gaze information plays an important role in identifying a person’s focus of attention. The information can provide useful communication cues to teacher. For example, it can be used to identify where a person is looking, and what he/she is paying attention to. So this section will provide an overview of eye gaze tracking and attention tracking systems. Gaze determines the user's current line of sight or point of fixation. Initial inquiry into eye movement research began in the early 1900s [Rayner 98]. The fixation point is defined as the intersection of the line of sight with the surface of the object being viewed (such as the screen). Gaze may be used to interpret the user's intention for non-command interactions and to enable (fixation dependent) accommodation and dynamic depth of focus. We will concentrate on video-based gaze estimation here. The video-based gaze estimation approaches can be partitioned into three approaches: 13 • head-based approach • ocular-based approach • and combined head and eye-based approach. The head-based approach determines user's gaze based on the head orientation. In Rae & Ritter [Ritter 98], a set of Gabor filters is applied locally to the image region that includes the face. This results in a feature vector to train a neural network to predict the two neck angles, pan and tilt, providing the desired information about head orientation. The ocular-based approach estimates gaze by establishing the relationship between gaze and the geometric properties of the iris or pupil. The most common approach for ocular-based gaze estimation is based on the relative position between pupil and the glint (cornea reflection) on the cornea of the eye [Ebisawa 95, Ebisawa 98, Morimoto 99]. However, common eye tracking methods require special hardware attached to the head, restrict head movement, and/or require calibration [Duchowski 03] and eye movement data includes a great deal of noise due to individual and equipment variability. 2.2 Attention Tracking Systems Cognitive Psychology found that despite the overall impressive abilities of people to sense, remember, and reason about the world, our cognitive abilities are extremely limited in well-characterized ways [Horvitz 03]. The findings about our 14 limited attentional resources – and about how we rely on attentional signals in collaborating – have significant implications for how we design our model [Horvitz 03]. Attentional cue is an important source of rich signals about goals, intentions, and topics of interest [Horvitz 99, Horvitz 01]. Horvitz [Horvitz 99b, Horvitz 03b] presented methods for reasoning about the cost of interrupting users based on multiple streams of events including information generated by interactions with computing devices, visual and acoustical analysis, and data drawn from online calendars. They described methods for inferring attention from multiple streams of information, and for leveraging these inferences in decision making under uncertainty. Attention is one of the most mentioned problems of human computer interaction (HCI). The need for information and data is quite widespread. Especially in tutoring systems, the lack of knowledge and experience of users with the subject at hand makes ITSs harder to select the appropriate attentional focus and may easily cause a loss of focus. It is desirable in ITSs to assess, support and maintain users' attention. A large portion of research on human attention is based on the findings of cognitive psychology. For example, Raskin [Raskin 00] investigates how single focus of attention, and habit formation have important consequences on human ability to interact with computers. He proposes that habit creation is a mechanism that can be used to shift the focus of users from the interface to the specific target task. 15 The study above follows the classic "direct manipulation" school [Shneiderman 92, Shneiderman 97] which aims at supporting the attentional choices of the user by making the device "transparent" so that the user can focus on the task rather than on the interface. The wide range of systems designed with this aim is often referred to as transparent systems, a term also employed in ubiquitous computing [Abowd 99, Weiser 91]. Another area of research focuses on designing interfaces and systems capable of guiding the users in the choice of attentional focus. The system is seen as proactive, visible, and capable of supporting the users in their choices. This type of systems are often designed as artificial agents [Bradshaw 97, Huhns 97] acting as proactive helpers for the user [Maes 94, Negroponte 97] and they are frequently referred to as proactive/adaptive systems. Attentional mechanisms also provide a framework that reconciles the direct manipulation user interfaces approach and the interface agents approach as presented and exemplified by Horvitz [Horvitz 99]. 2.3 Motivation and Emotion Detection Systems There are a broad range of different research issues on learner affective state, such as emotion or motivation detection and adaptive instruction system. Such issues are of direct relevance to our work on pedagogical agents. Conati [Conati 04, Conati 04b] presented a probabilistic model of student affect based on Dynamic Bayesian Networks and designed to detect multiple emotions in 16 an educational game. Electronic educational games can be highly entertaining, but studies have shown that they do not always trigger learning. To enhance the effectiveness of educational games, they proposed intelligent pedagogical agents that can provide individualized instruction integrated with the entertaining nature of the games. The agent had access to the learner’s affective state and knowledge, and relied on a probabilistic student model, the agent generated tailored interventions aimed at helping students learn. However, it has been shown to be problematic in the system to accurately assess of student goals. For example, the model can only maintain a fairly accurate assessment of the learner current emotional state when the student’s goals can be correctly determined. Burleson and Picard [Burleson 04] used affective agents as peer learning companions to facilitate development of empathetic relationships with learners. They explored how characteristics of affective agents can influence perseverance in the face of failure. They took the approach of assisting users to modulate the effects of their own affective state. D’Mello et al [D'Mello 05] aimed to develop an agile learning environment that is sensitive to a learner’s affective states. They augmented an existing intelligent tutoring system (AutoTutor) [Graesser 99; Graesser 01, Graesser 04] that helps learners construct explanations by interacting with them in natural language and helping them use simulation environment. Their approach for affective state classification relied on an exclusive use of special sensing devices, such as IBM Blue Eyes to detect facial expressions and a Body Pressure Measure System 17 (BPMS) for posture information. But these have been mainly focused on assessing the learner’s emotion by his/her posture and special sensors (e.g. head tracker, pressure mouse and chair with a posture sensor). Angel de Vicente and Helen Pain [Vicente 98, Vicente 02] argued for the need of empirical studies that can help us analyse learner’s motivation. To this effect, they discussed a number of empirical studies they performed in order to inform the design of an ITS simulation that detects the motivational state of a student. The main aspects of the motivation diagnosis architecture in their method were a motivation self-report component and a motivation diagnosis component based on human teachers' motivation diagnosis knowledge, elicited via one of the mentioned empirical studies. This architecture was implemented as an ITS simulation in order to help agents evaluate these motivation diagnosis techniques. However the detection model was based on insufficient knowledge on learner’s task and focus attention. This insufficiency frequently results in inaccurate detecting. Hilary Tunley [Tunley 04] focused on user modeling issues such as adaptive educational environments, adaptive information retrieval, and support for collaboration. Their HomeWork project was examining the use of learner modelling strategies within both school and home environments for young children aged 5 – 7 years. The main content material being used by the project was based on the Number Crew, a popular mathematics televisions series developed by Open Mind Productions for Channel 4 Learning. Their model took into account the 18 informality and potentially contrasting learning styles experienced within the home and school, and improved the fit between the user and the resources. Rosemary Luckin [Luckin 04] described the initial design of the Coherence Compiler for the HomeWork project. The Coherence Compiler was responsible for maintaining narrative coherence across these materials and across devices so that the learner experiences a well ordered sequence that supports her learning effectively. Such support may be provided both through narrative guidance and tools to support the learner’s own personal narrative construction. Narrative guidance should be adaptive to the needs of the learner. It initially offered a strong ‘storyline’ explicitly linking new and old learning and then faded as the learner became more accomplished at making these links for herself. Such support may be provided both through narrative guidance and tools to support the learner’s own personal narrative construction. Azevedo [Azevedo 05, Azevedo 05b] examined the effectiveness of three scaffolding conditions (adaptive scaffolding (AS), fixed scaffolding (FS), or no scaffolding (NS)) on adolescents’ learning about the circulatory system with a hypermedia learning environment. Findings revealed that learners in both the AS and NS conditions gained significantly more declarative knowledge than did those in the FS condition. Also, the AS condition was associated with shift in learners’ mental models significantly more than the other conditions. Associated with these significant shifts in their mental models, learners in the AS condition regulated their learning by planning and activating prior knowledge, monitoring their 19 cognitive activities and their progress toward learning goals, using several effective strategies, and engaging in adaptive help-seeking. By contrast, those in the NS condition used fewer effective strategies, while those in the FS regulated their learning by using several regulatory processes which seemed to impede their learning. 2.4 Conclusion The research achievements discussed above are focusing on tracking user’s eye gaze and attention, or using different methods or sensors to assess learner’s cognition, emotion and motivational states. All of these research works provide help and insights in users' attention, motivation and emotion, but these methods are either requiring special instruments, which limit their wide application in classroom, or lack of integrated framework which connects the students' attention, motivation and emotion. This is the main objective of our study which will be discussed next. 20 Chapter 3 Background Study How might we access learner's focus of attention and engagement? In order to answer this question, two studies were designed to investigate the characteristics of two types of interactions: one was about the interactions between students and teachers; and the other is about that between students and tutoring systems. The aims of these studies were to extend prior results on attention and motivation from psychology to computing applications through experimental studies. 3.1 Interaction between human Tutor and Students 3.1.1 Purpose of this Study In this study, we tried to investigate how a human teacher teaches a student some basic concepts of factory management and helps the student to solve a factory management problem successfully. This was an important step towards understanding the interaction processes between teachers and students and establishing the student model. 3.1.2 Study design There were two applications running on student machines in this study. One was the Virtual Factory Teaching System (VFTS) [Johnson 00, Johnson 03, Dessouky 01], an on-line tutoring system for teaching industrial engineering concepts and skills, as shown in Figure 3.1. Another was the tutorial window, as shown in Figure 3.2, where students could read instructions and learn background knowledge on factory management. The student sat in front of the computer and 21 studied on it. And the teacher sat next to the student. In order to capture the interaction between students and teachers, three video cameras were installed to record the learning sessions. The first camera captured the faces of students. The second camera captured the screens of computers. And the last camera captured the interactions between students and teachers. Figure 3.1 Virtual Factory Teaching System 22 Figure 3.2 Tutoring Window 3.1.3 Data Collection An experienced teacher and three students from the University of Southern California were invited to participate in this study at 2002. This teacher taught the students about basic concepts on factory management and instructed students to learn how to use the VFTS. Each learning session lasted from 30 minutes to 45 minutes. In this experiment, data on the interaction between students and computer was collected, such as mouse events and keyboard inputs. The interaction data was recorded as a 3-tuple including a time stamp, event type (mouse or keyboard) and value. The video data for each session was also stored on the tape. 23 3.1.4 Data Analysis Results We reviewed the recorded video and also interviewed the teacher after this study finished and concluded that the human tutors used the following types of information to infer learner’s focus of attention, • Eye gaze, the focus of student's eye gaze. The tutor used this kind of information to infer the focused window or area of the student. Eye gaze helped the human tutor to follow the shift of the student's attention. But sometimes if the student shifted his/her attention too often or too quickly, the human tutor lost student's focus of attention temporarily, • keyboard inputs, which was a supplemental information for tutor to recognize the student's focus of attention. For example, if a student was typing in a parameter field, the human tutor was quite certain about the learner's attention area, and • mouse actions. There were two types of mouse actions. One was mouse click and the other was mouse movement. We found that the mouse click was very useful information for tutor to recognize student's attention. For example, if the student clicks a button, then the tutor can get the corresponding focus area of this student. But mouse movement is not as useful as mouse click because a person will move his/her mouse randomly on the screen. So the tutor usually will not reply on mouse movement to infer learner's focus of attention. We also investigated the intervention strategies used by human tutor in this study. For example, if the teacher find the student is stuck at some places, the 24 teacher will give this student very explicit instruction. But if the teacher thinks the student can solve the problem by himself/herself, usually the teacher will avoid intervening with the learner or just give the student some hints about how to solve this problem. 3.2 Study for Math Teaching using ITS 3.2.1 Purpose of this Study We already know a great deal about the kind of information that is used by a human tutor to infer learner focus of attention. But we still don't know or know very little about how student's attention or action reflects his/her engagement. More specifically, what will a student who is focused on solving problem generally do? What will a student who is “gaming” the tutoring system do? To answer these questions, we designed another study. 3.2.2 Study design In this study, students studied in an online math tutoring system Wayang, as shown in Figure 3.3. The students can solve math problems in this system and select correct answer from five answer options. There is a “Help” button at the bottom of the system. If a student has difficult to solve a problem, he/she can click this button to request instruction or hint from Wayang. The system will show students instructions or background knowledge using multimedia, such as animation and sound. 25 The students who were participating in this study needed to complete several questionnaires. Every student needs to finish two questionnaires: One is pre-study questionnaire and another is post-study questionnaire. Figure 3.3 Wayang System 3.2.3 Data Collection In order to investigate the relation among students’ attention, performance, motivation, and rating from teachers, data of the students' answers to the designed questionnaires was collected, as well as the ratings for each student given by the teacher. The ratings consisted of student's performance and motivation in the classroom study, which was evaluated by the teacher using a number scheme of 1-3 where 1 represented the lowest performance and motivation and 3 represented the highest). 26 Teacher ratings. The students’ mathematics teachers provided categorical ratings of individual students’ behaviors indicative of mathematics motivation and achievement in mathematics class. There are three categories of motivation: High self-regulation; grade-level (average) motivation; disengaged. Achievement ratings included three categories: Above grade-level; meeting grade-level expectations; below grade level, i.e., in danger of failing the class. All teachers were qualified mathematics teachers and each had more than 10 years of experience with high school math instruction. Student data records. Data records were extracted from the ITS database for each student. A single student’s data record consisted of a sequence of problem events, defined as the presentation of a problem, followed by the subsequent interface clicks (clicks on answers, requests for help) and latencies between clicks, terminated by the request for a new problem. Student questionnaires In the questionnaires, two types of data were collected about learner: • Learner trait data (as shown in table 3.1), and • Learner mood data (as shown in table 3.2 ). The measures were obtained for these variables through questionnaires. The questions on the questionnaires were draw from Monique Boekaerts’s On-line Motivation Questionnaire (OMQ) [Boekaert 02]. The OMQ is an experience- sampling method that can reliably capture students’ cognitions and feelings in 27 relation to specific learning tasks, without disturbing their task focus. The OMQ may gain information about the various reasons why students are not willing to invest effort and provide valuable information on the effect that different cognitions and feelings have on task engagement. Furthermore, this OMQ is a general method which can be used in different domains. Then Boekaerts’s OMQ is used in this study to generate the online questionnaires. Table 3.1 Learner Trait Assessment Trait Value Task attraction 1-5 (low to high) Self-efficacy 1-5 (low to high) Relevance 1-5 (low to high) Perceived difficult 1-5 (low to high) Expected success 1-5 (low to high) Table 3.2 Learner Daily Mood Assessment Mood Value Relax 1-5 (relaxed to tense) Confident 1-5 (confident to worried) Great 1-5 (great to bad) Ready to concentrate on learning 1-5 (yes to no) 3.2.4 Data Analysis Results In this study, we investigated student action data and performance in the ITS and extracted different types of attention patterns from the data. This study helped us to design and implement a classifier to predict students’ strategies while using the ITS, particularly their tendency to guess, to work independently, or to use the 28 multimedia help to learn. The results indicated that it should be possible to seed pedagogical models in ITSs in advance with learner profile data that is timely and inexpensive to elicit, and quite predictive of strategies that will be employed once students begin working with the ITS. I will discuss more detailed analysis for this study in Chapter 5. 29 Chapter 4 Attention Tracking Model In this chapter, I will introduce how the learner's focus of attention is modeled using probabilistic methods. The interface of this model includes three components, as shown in Figure 4.1: • WebTutor, which is an on-line tutorial used to teach learner instruction and concepts of industrial engineering, is the right bottom window. • VFTS on the left part of the screen is designed to help engineering and business students grasp complex factory dynamics that are difficult to teach in chalkboard lectures and impossible to experiment with in real factories. The system allows students, working alone or in teams, to build factories, forecast demand for products, plan production, establish release rules for new work into the factory, and set scheduling rules for workstations. They can run simulations, and an animated panel displays jobs progressing through their factory, with queue counts, finished goods counts, graphs, and reporting functions all available. • Agent Window is a text window used to communicate with the agent (or a human tutor in Wizard-of-Oz mode) and the right part is an animated character that is able to generate speech and gestures.which is the right top window, 30 Figure 4.1 Screen Shot of Eye Gaze Tracking Program Then the attention tracking model takes input from the WebTutor interface, the VFTS interface and Agent interface as well as eye gaze information, in order to infer learner focus of attention. These three components can provide ITSs with capabilities to gather information about learners’ states and their expected tasks. Therefore ITSs are able to track learner attention and detect learners’ motivation. Our model is similar to that of the Lumière Project [Horvitz 98], which also tracks user activities by monitoring learner actions and tracking learner tasks. The differences are that our system models at a more elaborated and sophisticated level about learner’s activity (e.g., eye gaze) and 31 learner task (from the WebTutor). It can therefore track learner activities with greater confidence. 4.1 Tracking Learner Focus of Attention under Uncertainty From our previous study, human tutor used rich sources of information to infer learner's focus of attention, such as eye gaze, keyboard and mouse. A number of these clues can be taken as direct signals about the attention of learner. For example, we can use an eye gaze tracking system to track learner eye gaze. Some special instruments, such as helmet, camera or sensors, are designed to track learner’s eye gaze. However, even with these expensive instruments, the position of learner’s eye gaze can not be decided by them with 100% accuracy. Thus, we turn to models that can be harnessed to reason about a user's attention under uncertainty. Such models and reasoning can unleash new functionalities and user experiences. 4.2 Dynamic Bayesian Network (DBN) A Bayesian network is a graphical model that encodes probabilistic relationships among domain variables [Nicholson 94]. The relationship between any set of state variables can be specified by a joint probability distribution. Bayesian Networks have been used in various applications which initially were static [Albrecht 97]. More recently researchers have used it in dynamic domains, where the world changes and the focus is reasoning over time [Dean 91]. The DBN framework provides a broad variety of modeling schemes that can be conceptualized in a single framework with an intuitively appealing graphical 32 notation [Pavlovi’c 99]. DBNs extend BNs to time series data modeling by considering the state transition between time slices. A Bayesian network B =< N, A, Θ> is a directed acyclic graph (DAG) <N, A> with a conditional probability distribution (CP table) for each node, represented by Θ. Each node n Є N represents a domain variable, and each arc in A between nodes represents a probabilistic dependency (see Pearl 1988). In general, a BN can be used to compute the conditional probability of one node, given values assigned to the other nodes; hence, a BN can be used as a classifier that gives the posterior probability distribution of the classification node given the values of other attributes. When learning Bayesian networks from datasets, we use nodes to represent dataset attributes. The two major tasks in learning a BN are: learning the graphical structure, and then learning the parameters (CP table entries) for that structure. As it is trivial to learn the parameters for a given structure that are optimal for a given corpus of complete data – simply use the empirical conditional frequencies from the data [Cooper 92]. 4.3 Event Capture System As we mentioned in the study of Chapter 3.1, a human tutor tracked learner’s multiple sources of input events, such as mouse, eye gaze and keyboard. Then the first step for our work is to capture these sources of event information in a tutoring system. In this section, we introduce our test bed and the event capture system. 33 An event capture system was designed and implemented. It will monitor the learner’s input from keyboard, mouse and a PC camera. For example, we capture such low-level states as the specific application currently in focus, whether the learner is typing or clicking the mouse. There are four types of information: • Learner eye gaze: Eye gaze is extremely useful for an agent to track learner attention. In our system we used a camera installed on learner machine to capture learner eye gaze. • Keyboard event: Through keyboard typing, learner will input values for factory parameters in VFTS or asks questions in Agent window. It can be used to indicate current focused window of learner. • Mouse event: Frequently, learner will submit results in VFTS or select certain keywords to ask questions in Webtutor by clicking mouse button. We evaluated the possibility of using mouse location on the screen as evidence for the current focus of attention. However, analysis of user behavior on the VFTS interface showed that mouse location is not a quite reliable indicator of focus. • Scrolling: Scrolling helps the learner to navigate through the whole tutorial in Webtutor. For example, learner can scroll tutorial to a new paragraph or go back to previous paragraphs. It can also be used to indicate current focused window of learner. The mouse event, keyboard event and scrolling were collected with three attributes: 34 • Timestamp, which contains the event occur time • Event Type, which describe the event types (including mouse or keyboard) • And Event Value, which collect the input value for this event. For example, if a learner clicks a button to “run” factory in VFTS, we will capture this button’s value. All these information will be saved into a database on the server through a connector to database in event capture system. The eye gaze of learner was captured by a PC camera installed on student’s monitor. The approach used in our study is to discover a mapping from a multiple dimension feature space, comprising features extracted from processed real-time video images of the student’s head, to the two dimensional space of the x-y pixel coordinates of the video monitor. For this purpose, it tracks the following landmarks: the pupil and two corners of each eye, the bridge of the nose, the tip of the nose, and the left and right nose corners. This approach is robust and it can be used easily in almost any environment where a camera is installed. It works without calibration, although accuracy suffers as a result, particularly in the vertical axis. Fortunately, we can combine the eye tracking information with other learner interaction data in order to improve accuracy. 4.4 Certain and Uncertainty Events The event capture system can monitor multiple sources of interaction data between student and test bed. We found that when certain events occurred, tutors 35 can infer learner focus of attention with near certainty. For example, if a learner clicks a button or scrolls a window, the focus of learner must be on that window, assuming that the hand and eye are coordinated. Therefore we studied the pattern for learner focus of attention after learner performs actions. Table 4.1 shows the probabilities of the learner’s focus remaining in the same window after learner performs actions at time n seconds. For example, if a learner performs a mouse click in the VFTS window at time n, then the probability of the learner’s focus remaining at the same window after 5 seconds is about 60%. Table 4.1 Probabilities After T Seconds T+1 T+2 T+3 T+4 T+5 T+6 >95% >85% >75% 65% 60% <50% 4.5 Modeling Learner Focus of Attention Given the results of our studies, we set out to build Bayesian model with the ability to infer learner focus of attention. In our Bayesian model, learner eye gaze information is reported by an eye gaze tracking program. Common eye tracking methods require special hardware attached to the head, restrict head movement, and/or require calibration [Duchowski 03]. This has limited their applicability in learning systems to laboratory experiments (e.g., [Gluck 00]). However, our model aims at an eye gaze tracking program that is unobtrusive and requires no special hardware and no calibration. We use a program developed by Larry Kite (as shown in Figure 4.2) in the Laboratory for Computational and Biological Vision at 36 University of Southern California to track learner eye gaze. It estimates the coordinates on a video display that correspond to the focus of gaze. Figure 4.2 Screenshot of Eye Tracking Program Building an effective model hinges on defining appropriate variables and states of variables. The following variables are employed in Bayesian model, as shown in Figure 4.3, FA (focus of attention) represents learner focus of attention, the states of which are VFTS window (VW), Agent window (AW), WebTutor window (WW) and other area (OA). ETP (Eye Tracking Program) represents where the focus of learner is based on eye tracking program, the states of which are VW, AW, WW and OA. ME (Mouse Event) represents where the mouse click event occurs, the states of which are VW, WW or no. SE (Scrolling Event) represents whether or not a scroll window event occurred in WebTutor, the states of which are WW and no. TE (Typing Event) indicates where the type events occur, the states of which are VW, 37 AW or no. The no state of a variable represents there is no such event or action for this variable. And the OA state for a variable represents learners are focusing on outside area of VW, WW and AW. FS (Focused Screen) is one mediating variable between FA and ETP describing which part of screen learner eye gaze focuses on. The states of FS are LS (left screen), RS (right screen) or OA. The reason for inserting FS between FA and ETP is that eye gaze program has low accuracy for vertical axis and without FS we have no clue to specify the probabilities for ETP. To construct the Bayesian network for a set of variables, we draw arcs from cause variables to their direct effects as shown in Figure 4.2 [Heckerman 95]. In the final step of constructing our model, we assess the probability distribution P(X|Parent 1 , Parent 2 , … Parent n ) for each variable X. We initialize the distribution of X for every configuration of Parent i based on our experiments. According to the model, agent will reason learner focus of attention based on interface events and learner gaze. For example, given ME = VFTS, TE = no, SE = no and ETP = VFTS, we have P(FA = VW) = 0.98, P(FA = WW) = 0.007, P(FA = CW) = 0.007 and P(FA = OA)=0.006. So the current focus window of learner should be VFTS. 38 FA ME SE TE FS ETP Figure 4.3 Attention Tracking Model As we mentioned above, history events can be used to infer learner focus of attention. We use Dynamic Bayesian Networks (DBNs) to handle the uncertainty over time in the learner focus of attention reasoning. DBNs are a framework for reasoning over time under uncertainty. They are suitable for modeling variables whose values change over time. Figure 4.3 shows the procedures that dynamic Bayesian model infers FA. Link between nodes FA (t-1) and FA (t) represents their temporal dependencies. The probability of node FA at time slice t can be given as P(FA (t) | FA (t-1) , ME , TE , SE , ETP ). It depends on probability of node FA at time slice t-1 and observed events. 39 FA ME SE TE FS ETP FA ME SE TE FS ETP t-1 t t+1 FA ME SE TE FS ETP FA ME SE TE FS ETP FA ME SE TE FS ETP FA ME SE TE FS ETP Figure 4.4 Dynamic Attention Tracking Model 4.6 Evaluation of Attention Model In order to evaluate the appropriateness and quality of our model, two groups of experiments were performed. The subjects in these experiments were graduate students at USC Information Sciences Institute. Each subject had one session of experiment lasting from 15 minutes to 25 minutes. Group A includes 6 experiments that are used to compare 3 different models to track learner focus of attention and group B includes 4 experiments that are used to evaluate overall model. In experiments of group A, three approaches have been performed to comparatively evaluate the effectiveness of inferring learner focus of attention: eye tracking program (ETP), learner’s actions (LA) in system and Bayesian network model (BNM). In ETP, results from eye gaze program represent the learner’s current focus of attention. LA method considers the last window that learner performs actions as current focus of attention. And BNM infers learner focus of attention by Bayesian network with uncertainty and certainty information. 40 In all of these experiments, human tutors inferred the learners’ focus during the experiments and recorded them using a program when they coached learners. Although these inferred data from observation of tutors does not match learners’ actual focus of attention completely, it still is fairly consistent with what learners are doing. So we compared results from the three approaches with these analysis data. 0 1 2 3 4 1 112131415161 718191 101 0 1 2 3 4 1 11213141 51 61 718191 101 0 1 2 3 4 1 1121 314151 6171 8191 101 0 1 2 3 4 1 11 21 314151 6171 81 91 101 Baseline 0 10 20 30 40 50 60 70 80 90 100 WW AW VFTS NON 0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100 WW AW VFTS NON WW AW VFTS NON WW AW VFTS NON ETP LA BNM Time Time Time 0 10 20 30 40 50 60 70 80 90 100 Time 0 1 2 3 4 1 112131415161 718191 101 0 1 2 3 4 1 11213141 51 61 718191 101 0 1 2 3 4 1 1121 314151 6171 8191 101 0 1 2 3 4 1 11 21 314151 6171 81 91 101 Baseline 0 10 20 30 40 50 60 70 80 90 100 WW AW VFTS NON 0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100 WW AW VFTS NON WW AW VFTS NON WW AW VFTS NON ETP LA BNM Time Time Time 0 10 20 30 40 50 60 70 80 90 100 Time Figure 4.5 Comparison of ETP, LA and BNM Figure 4.5 illustrates the inferred results from different approaches during the experiments. The figures shows BNM gets more accurate result compared to EPT and LA. The vertical axis indicates the four different windows. Table 4.2 compares the average accuracies of the three approaches. These results indicate that ITS 41 using BNM has more confidence for tracking learner attention than using ETP or LA approach. Table 4.2 Average Accuracies for Different Windows NON VFTS Agent Window WebTutor Window ETP 78% 71% 56% 61% LA 0% 22% 37% 26% BNM 78% 84% 81% 88% As shown in table 4.3, the ITS is able to detect periods of fixation comparable to what learner actually does, i.e., how long learner has already focused on the VFTS or WebTutor. When agent infers the learner’s focus of attention is on “other area” for some time, e.g., more than 5 seconds, agent can recognize the learner is not focused on system, e.g., the learner walks away or turns away from the screen. Table 4.3 Accuracies of Attention Tracking Model Time after learners change focused window Accuracy of Bayesian network’s results 1 second > 60% 2 seconds > 75% > 3 seconds > 85% 4.7 Summary This chapter presented a model for pedagogical agents to use learner’s attention to detect motivation factors of learner in interactive learning environments. This 42 model is based on observations from human tutors coaching students in on-line learning tasks. It takes into account learner’s focus of attention, current task, and expected time required to perform the task. A Bayesian model is used to combine evidence from learner eye gaze and interface actions to infer learner focus of attention. Although this work has focused specific programs and interfaces, we can generalize the approaches and techniques used in our work to different domain. When applied to a new application domain, we need to know the possible actions on each window and the learner’s learning characteristics in this domain. Then we can use such information to rebuild and initialize the Bayesian Network, and make it capable of tracking user’s attention in the new domain. 43 Chapter 5 Learner Attention Classifier Many ITSs can provide learners effective instructions. However such systems will not always successfully motivate learners' learning because learners do not always use them effectively. This chapter presents a study where high school students’ attention sequences with a mathematics ITS were machine classified into five finite-state machines indicating guessing strategies, appropriate help use, and independent problem solving. 5.1 Approach and Data Sources 5.1.1 Tutoring System In this study, the ITS, which is called Wayang Outpost, is a web-based application providing instruction in high school mathematics, as shown in Figure 3.3. Wayang Outpost is an online system with multimedia instructions such as animation and sound. Students viewed a series of word problems. Each problem included five answer options; students could choose an answer at any point and receive feedback (e.g., when an answer was selected, a red “X” or green checkmark indicated if the answer was right or wrong). Students could also request a multimedia explanation of the solution by clicking the “Help” icon. Explanations were constructed as an ordered sequence of individual hints leading to the correct answer. Individual hints included information presented in one modality (e.g., text, or animation, or audio) to avoid excessive cognitive load [Mayer 03] but the complete explanation for a problem included hints with a range of modalities. 44 Student actions were recorded in the server database, including clicks, sequence, and latencies between clicks. 5.1.2 Student participants The study included high school students from three schools in an urban area serving a highly diverse student population. Students worked with the mathematics tutoring system as part of their regular classroom mathematics instruction. Data records were available for 85 – 91 students. 5.1.3 Motivation profiles In the first session, students completed an on-line self-report instrument used to assess mathematics motivation that was integrated into the tutoring system application. The on-line instrument was derived from integrating on-line and paper- and-pencil questionnaires previously shown to have high reliability and validity [Boekaerts 02, Eccles 93]. Because academic motivation is believed to be domain- specific, items were specific mathematics. The 10 item instrument included two questions addressing each of five constructs: • math self-efficacy; • beliefs that math is important to learn; • liking of math; • expected success in math; and • difficulty of math. 45 5.1.4 Motivation Profile The Motivation Profile also included an item designed to assess the learner’s beliefs about math ability: “entity” beliefs reflect the view that math skill primarily reflects native ability, whereas “incremental beliefs” indicate the learner believes that skill can be enhanced through effort [Dweck 02]. Students clicked on a rating scale to indicate their answer. Answers were automatically recorded into the Wayang Outpost server database. 5.1.5 Teacher ratings The students’ mathematics teachers provided categorical ratings of individual students’ behaviors indicative of mathematics motivation and achievement in mathematics class. Motivation ratings included three categories: High self- regulation; grade-level (average) motivation; disengaged. Achievement ratings included three categories: Above grade-level; meeting grade-level expectations; below grade level, i.e., in danger of failing the class. All teachers were qualified mathematics teachers and each had more than 10 years of experience with high school math instruction. 5.1.6 Student data records Data records were extracted from the Wayang Outpost database for each student. A single student’s data record consisted of a sequence of problem events, defined as the presentation of a problem, followed by the subsequent interface clicks (clicks on answers, requests for help) and latencies between clicks, terminated by the request for a new problem. 46 5.2 Learner’s Attention Pattern Learner’s attention patterns represent how students might work with the ITS. Rules for each are described below. Students’ data records were machine scanned and each student’s problem events were classified. (In the rules presented below, the limit of 10 seconds was generated after viewing a sample of traces of high- achieving students, on the grounds that if these skilled students required at least 10 seconds to read a problem or a hint, it was not likely that other students could do so in less time. Analyses conducted with 5 and 15 second windows yielded similar effects and interpretations.) • Independent-a. Problem is available for at least 10 seconds, followed by the selection of the correct answer. We infer that the student read the problem and solved it correctly without ITS assistance. • Independent-b. Problem presented for at least 10 seconds, followed by an incorrect answer choice; another 10 or more seconds; followed by the correct answer. We infer that the student read the problem, computed an incorrect answer, and revised to the correct answer, without ITS assistance. • Guessing. Student selected one or more answers within 10 seconds of the problem presentation; no help was viewed. We infer that the student did not read the problem and clicked on answers until the correct answer is discovered. 47 • Help abuse. The student clicked on “help” with interclick intervals of less than 10 seconds. We infer that the student did not attend to the hint but was searching for the correct answer. • Learning. The problem was presented for at least 10 seconds; help was requested and presented for at least 10 seconds before an answer was selected or another hint was requested. We infer that the student read the problem and the help, i.e., was trying to learn. 5.3 Results and Discussion 5.3.1 Learner Motivation Students’ average scores for the five math motivation constructs (self efficacy, value, expected success, difficulty, and liking of math) were subjected to hierarchical cluster analysis yielding 3 groups. Mean scores for the three groups may be viewed in Table 5.1. Students in Group 1 (N = 50) appeared to have average motivation in math: they expected to pass, thought math might be important to their future, and liked it a bit less than other academic subjects. Group 2 students (N = 21) showed a distinctly different pattern: they did not like math and did not think they had much ability in math. Group 3 students (N = 12) had high beliefs in their ability, liked math much more than the other groups, and thought math was very important to learn. 48 Table 5.1 Mean Scores on Motivation Profile by Group Efficacy Liking Value Diff Exp Succ Group 1 3.19 2.47 3.36 2.75 3.37 Group 2 1.89 1.60 3.14 2.58 1.90 Group 3 3.91 3.86 4.79 3.83 4.16 5.3.2 Teacher Ratings Teacher ratings of motivation were highly correlated with students’ self-reports of math motivation. A chi square analysis indicated that teacher motivation and achievement were significantly associated, e.g., students rated as performing above grade-level expectations also tended to be the same students who were rated by teachers as high in motivation. However, about 35% of the students were rated by teachers as having average to high motivation but also as being low in achievement, i.e., they were in danger of failing their math class. 5.3.3 Attention Patterns Students completed an average of 31 math problems, with a range of 10 to 90. As may be seen in Figure 5.1, over 90% of students’ problem events (totaling about 2,635) multi-step problems) could be classified into of the five action patterns. In Figure 5.1, the x axis is the student id from 1 to 95. The y axis is the percentage of each student’s actions that have been classified as one of five action patterns. For example, in Figure 5.1, student 44 (90%) completed 30 math problem and 27 of 49 them can be classified as one of the five action patterns. The remaining problems included cases of skipping or partial work on a problem (e.g., the student quit out of the tutoring application before completing a problem). Overall, students’ behavior with the ITS could be described in terms of finite state machine representations. 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1 4 7 10131619 22252831 34 37404346 49525558 61 64677073 76798285 88 9194 Student Recognize Rate Figure 5.1 Summed Proportion for Individual Student The most common pattern was independent-a problem solving (32%); 21% were independent-b problems. Students were classified as attempting to learn on 22% of the problems. Guessing occurred on 19% of the problems, and help-abuse was observed on only 1% of the problems. More globally, students solved 53% of the problems on their own (Independent-a and –b combined); used help in an effort to learn on 22% of the problems; and “gamed” the system (Help-abuse and Guessing combined) on 20% of the problems. 50 Our work is thus consistent with that of others in that students’ spontaneous use of the multimedia help available in the ITS was relatively low [Aleven 00]. This is a concern because, first, students do not have the opportunity to benefit from the scaffolding if they do not ever access it and, second, from the knowledge engineering perspective, considerable resources go into the development of multimedia scaffolding, much of which students rarely see. Thus, our next analyses focused on the relation of student characteristics (motivational profile and achievement) with action patterns. One question was whether we could proactively identify which students were most likely to solve the ITS problems independently, i.e., without incorrect answer attempts or accessing the multimedia help. Teachers’ ratings of achievement were strongly predictive of students’ independent problem solving, specifically, Pattern A, in which the student read the problem and chose the correct answer without error and without ITS assistance. A analysis of variance on students’ Independent-a scores with teacher achievement (above-grade; grade-level; below grade-level expectations) as the grouping factor indicated that high achieving students solved significantly more problems on their own (47.8%) than average (39%) and low- achieving students (19%), F(2,82) = 13.436, p < .001. Although not particularly surprising, this result provides some indication that teachers’ perception of their students’ math knowledge was accurate, in that students who were independently rated as performing above grade expectations were more likely than other students to solve the ITS problems on their own. It also suggests that prior achievement 51 could be used proactively to select a steeper difficulty curve for some students, rather than reactively in response to student problem solving behavior with the ITS, as reactive systems can lead to an overemphasis on easy problems [Arroyo 03]. We are currently adding a feature into the ITS so that teachers can enter their ratings of student achievement before students begin to work with the system. Teachers’ assessments of their students’ skills also matched students’ own perceptions of how well they expected to do in math. An analysis of variance on students’ scores for Expected Success (part of the Motivation Profile) with teacher achievement ratings as the grouping factor revealed that high-achieving students thought they would do better in math than lower-achieving students, F(2,79 ) = 32.233, p < .001. Mean ratings (from a range of 1 to 5) were 4.08 for high- achieving students, 3.31 for students with average achievement, and 2.47 for students who were performing below grade expectations. Thus, the end-users of the ITS (students in math classes) had a reasonably accurate sense of how well they were doing in math, and this self-assessment predicted an important aspect of their behavior with the ITS: independent and accurate math problem solving. To investigate patterns in students’ use of the ITS, proportion scores for the different patterns were used in a hierarchical cluster analyses yielding three groups. (Guessing and Help abuse scores were combined to produce one score, due to the low rate of Help Abuse.) Results may be viewed in Table 5.2. 52 Table 5.2 Mean Proportion for Attention Patterns Guess Learn Independent-a Independent-b Group 1 0.33 0.14 0.16 0.32 Group 2 0.13 0.55 0.14 0.13 Group 3 0.09 0.17 0.55 0.13 Group 1 students (N = 33) were most likely to guess while working with the tutoring system. Interestingly, these students were also likely to solve problems by making an incorrect guess and then viewing at least one hint (Independent-b pattern). Group 2 students (N = 14) were the highest users of the multimedia help for learning (55% of the problems). Group 3 students (N = 36) were most likely to solve the problems accurately and without using ITS help. These results indicate that individual students have systematically different strategies while using the tutoring system, and that these strategies can be described in terms of finite state machine components. Our next question of interest was whether students’ initial self-reports of motivation might explain which students subsequently adopted particular strategies as they worked with the ITS. A chi-square analysis on a cross-tab of students classified by motivation group and ITS action pattern group showed a significant relation, χ2(4,79) = 23.26, p < .001. 53 Figure 5.2 Mosaic plot BY Motivation and Attention Group As may be viewed in Figure 5.2, students who had high math self-efficacy, liked math, and thought that math was very important to learn (Group 3) were most likely to solve the ITS math problems accurately and independently (right column). There were relatively few students who fit this description. In contrast, the largest group included students with average math motivation (left column) who seemed to be equally divided between those with a high proportion of guessing, and those who worked independently. More interesting was the group with low mathematics motivation (center column): About half had high guessing ratings, but the others had high learning attention pattern rates. In fact, proportionally speaking, students who had low self efficacy, low attraction to math, and low expectations for success were most likely to use the ITS in a way that suggested an effort to learn, i.e., reading the problem and viewing the hints. The relatively high rate of learning- oriented ITS use by low motivation students suggests that technology-based instruction has potential to reach students who are not doing well with regular 54 classroom instruction; such students are known to avoid seeking help from teachers and classmates [Karabenick 98, Newman 02; Turner 02]. The opportunity to learn from software may offer an appealing alternative because the student can seek help in private. Additional support for this interpretation was found in an analysis focusing on the average number of hints viewed per problem. An analysis of variance on mean number of hints per problem with Motivation Group as the grouping factor showed a significant effect, F(2,82) = 20.525, p < .001. Post hoc Tukey comparisons (a = .05) indicated that low motivation students viewed an average of 1.94 hints per problem, whereas average and high motivation students viewed fewer hints (0.37 and 0.34, respectively). This result suggests that the low motivation students were not only more likely to use the ITS resources, they drilled deeper into the resources once accessed by viewing multiple hints. Of course, the high motivation students were also more likely to successfully solve problems independently, indicating that they did not need to view the help to succeed. Again, however, the point is that the ITS may offer low motivation students an effective way to improve their problem solving skills, and they seem willing to accept this offer. This is especially striking given that the ITS allowed them to progress through the problems by guessing, yet it was the average motivation students who were more likely to follow this strategy. Having established that students’ motivation helped to predict patterns of attention with the ITS, a related question was whether we could evaluate the relative contribution of students’ motivation and their prior math achievement (based on 55 teacher ratings). A logistic regression was conducted on the cross-tab of student motivation group and teacher rating group (high, grade-level, or below-grade level achievement), with action pattern strategy group as the outcome factor. The results for the whole model indicated a significant lack of independence, χ2(8) = 30.945, p < .001. This means that the relative proportion of guessing, learning and independent problem solving exhibited by individual students as they worked with the ITS was not independent of their motivation or achievement in math (as rated by their teachers). More significantly, likelihood ratio effects tests indicated that motivation group membership contributed to the model, χ2(4) = 12.055, p < .01, but that teacher achievement rating did not, χ2(4) = 7.737, p = 0.12, N.S. This result lends support to our claim that student motivation – the constellation of beliefs about one’s ability and the value of learning the subject – must be considered in the design of tutoring systems, in addition to the more traditional focus on cognitive modeling. We attempted to learn more about the largest group of students: those with average mathematics motivation (N = 50) who were roughly split between those who tended to guess (44%) and those who tended to solve ITS problems independently (40%). One might expect that, within this subgroup, the independent problem solvers would be those with higher math skills, yet teacher achievement ratings did not predict ITS strategy for this group of students. However, the learning orientation item on the Mathematics Profile was predictive to some extent. Specifically, students with average mathematics motivation who had higher 56 guessing rates were more likely to report that they held an “entity” theory of intelligence, whereas peers who more often worked independently were more likely to hold an “incremental” view of intelligence, χ2(2) = 8.812, p < .05. Our interpretation of this result is limited because only one item was used to assess students’ beliefs about the role of native ability versus effort in learning mathematics. Still, this finding is consistent with our view that students’ self- reported beliefs about learning predict aspects of their behavior while using an ITS, independent of their achievement in the domain. 5.4 Conclusions We have shown that multiple data sources can be integrated and used to classify students in terms of the constellation of beliefs that they bring to the learning situation. These data were readily acquired from users, and were consistent with teachers’ knowledge of their students’ achievement and motivation. The classifications also predicted students’ strategies while using the ITS, particularly their tendency to guess, to work independently, or to use the multimedia help to learn. In addition, our empirical approach led to the identification of students who described themselves as disengaged and discouraged about their ability to learn math, but who were at least as likely (and in some cases more so) as other students to use the ITS in a manner suggesting they were attempting to learn. The results indicate that it should be possible to seed pedagogical models in advance with learner profile data that is timely and 57 inexpensive to elicit, and quite predictive of strategies that will be employed once students begin working with the ITS. 58 Chapter 6 Inferring Learning Goals This chapter describes the study on modeling student learning goals, including the goal of learning by using multimedia help, and the goal of learning through independent problem solving. This model is based on Dynamic Bayesian Network (DBN) and utilizes students’ interface actions while working with the ITS and inter-action interval latency: data that can be captured automatically as students work, without the need for additional sensors or equipment. It was trained with interface action and inter-action interval latency data from 115 students, and then tested with data from an independent sample of 135 students. Estimates of learning goals from the model predicted student performance on a post-test of math achievement, whereas pre-test performance did not. Students with relatively weak math skills were more likely to have strong hint-based learning goals, whereas students with stronger math skills were more likely to learn through independent problem solving. Among low-achieving students, use of hints to learn was associated with improvement from pre- to pos-test. 6.1 WAYANG Test Bed Before proceeding to describe the model extended from previous study, we describe the ITS, Wayang Outpost. It is the tutoring system used as testbed in our work. It is a web-based application providing instruction in mathematics to high school students. (A demonstration version may be viewed at http://k12.usc.edu.) 59 Figure 6.1 The Interface of Wayang Outpost Figure 6.1 shows a screen shot of Wayang Outpost. It presented the student with a series of math problems. As illustrated in the screen shot shown in Figure 1, each problem included five answer options. Students could choose an answer at any point and receive feedback (e.g., when an answer was given by student selecting an option, the feedback with a red “X” indicated the answer was wrong, whereas the feedback with a green checkmark indicated the answer was right). Students could also request a multimedia explanation of the solution by clicking the “Hints” icon. And the explanations were constructed as an ordered sequence of individual hints leading to the correct answer. For example, in the screen shot above, angles that are important for the solution are highlighted with an animation. Individual hints included information presented in one modality (e.g., text, or animation, or audio) to avoid excessive cognitive load (cf., [Mayer 03]) but the complete explanation for a problem included hints with a range of modalities. Students could choose an 60 answer when viewing explanations. Therefore students can decide the use of strategies to solve the problems. After students finish a problem, they could click “Next” icon to proceed and then Wayang Outpost selects a new mathematic problem for students. 6.2 Attention Pattern Definition In previous study, five attention patterns were identified based on students’ interface data and inter-action interval latency with respect to individual problem, as summarized in Table 6.1. Table 6.1 Descriptions of Attention Pattern Action Pattern Description Independent-a Problem is available for at least 10 seconds, followed by the sel of the correct answer. We infer that the student read the proble solved it correctly without ITS assistance. Independent-b Problem presented for at least 10 seconds, followed by an inc answer choice; another 10 or more seconds; followed by the c answer. We infer that the student read the problem, comput incorrect answer, and revised to the correct answer, withou assistance. Guessing Student selected one or more answers within 10 seconds o problem presentation; no help was viewed. We infer that the s did not read the problem and clicked on answers until the c answer is discovered. Help abuse The student clicked on “Hints” with interclick intervals of less th seconds. We infer that the student did not attend to the hint bu searching for the correct answer. 61 Learning The problem was presented for at least 10 seconds; help was req and presented for at least 10 seconds before an answer was selec another hint was requested. We infer that the student read the pr and the help, i.e., was trying to learn how to solve the problem. Note: In table 6.1 the thresh value of 10 seconds was generated after viewing a sample of traces of high-achieving students, on the grounds that if these skilled students required at least 10 seconds to read a problem or a hint, it was not likely that other students could do so in less time. Analyses conducted with 5 and 15 second windows yielded similar effects and interpretations. 6.3 Learner’s Learning Goals These action patterns capture certain solving strategies that students are used to solve individual problems. So they can’t be obtained in real time. A student must complete the problem before ITS can decide which action pattern it is. In order to enable ITS to monitor solving strategies of students in real time, two learner goals are modeled on top of the five action patterns as followings. • Hint-based Study: represents the learning goal where the student’s request for instructions when solving problem. Student may read through these instructions and solve the problem relying on the help provide in ITS. • Independent Study: represents the learning goal where students are reluctant to request help or believe they are capable to solve the problem by themselves. 62 Each goal can be derived from certain interface actions and inter-action interval latency. For instance, after clicking “Hints” button, a student views the instruction for a period of time more than thresh value T. This action indicates the student goal of Hint-based Study. If a student selects a correct answer for a problem without using the multimedia resources in the ITS, this action indicates the student goal of Independent Study. According the design of multimedia instructions in the ITS, the goal of Hint- based study will affect the goal of Independent Study, because a student that have viewed instruction carefully usually results in deeper understanding of the problem. For example, if a student clicks the “Hints” button and views the instruction, this student will have more chance to figure out the correct answer. 6.4 Modeling Learning Goals The understanding and prediction of student goals is hard for tutoring systems, even for human tutors. The difficulty is largely due to the uncertainty in the mappings between students’ goals and their interface actions. In order to handle the uncertainty in the modeling task, our model uses a Dynamic Bayesian Network (DBN) [Dean 89] to reason students’ goals of learning. Bayesian network is a graphical model that encodes probabilistic relationships among state variables and the relationship between any set of state variables can be specified by a joint probability distribution. More recently researchers have used it in dynamic domains, where the world changes and the focus are reasoning over time. For instances, researchers began to user DBN to infer user cognitive and affective states in real 63 time. However, to date most relevant work focused on using user interaction data to detect such states. For example, Conati et al’s work [Zhou 03, Manske 05] seeks to rely on Dynamic Bayesian Networks (DBNs) to assess student affective states during interaction with educational games and motivate student to engage in educational games. Our model not only relies on interface actions but also the inter- action interval latency which is also important to recognize student goals of learning. Hint-based Study T i Independent Study Reading time for help Solving time for correct answer Solving time for incorrect answer Ask for help Help based Study Independent Study Reading time for help Solving time For correct answer Solving time for Incorrect answer Ask for help T i+1 Figure 6.2 Probabilistic Model to Infer Learning Goals A DBN consists of time slices representing relevant temporal states in the process to be modeled. A time slice is created in this network after each student 64 action, to capture the evolution of student goals as the study proceeds. Figure 6.2 shows two time slices of DBN to assess student goals of learning. The Hint-based Study and Independent Study nodes in Figure 6.2 represent the levels of two types of goals that student may have when studying in tutoring systems. Hint-based Study has four states: High, Normal, Low and None. Independent Study also has four states: Very High, High, Normal and Low. Hint-based Study has None state because students may not have goals to learn from instructions. Student actions and inter-action interval latency are modeled as following nodes: • Ask for help: models whether the student requests for help or not • Reading time for help: denotes the comparison between the periods of time for students to study the instructions in the ITS and a thresh value of 10 seconds. • Solving time for correct answer: indicates the comparison of the period of time that student spends for finding correct answer and a threshold value of 10 seconds. • Solving time for incorrect answer: represents the comparison between the period of time that student spends for solving the problem with an incorrect answer and a thresh value of 10 seconds. As discussed earlier, the learning goal of Hint-based Study depends on whether student requests for help or not and the period of time that student spends on the 65 instructions, as indicated by the links from the Ask for help and Time for reading help nodes to Hint-based Study node. The learning goal of Independent Study depends on the period of time that students spend for solving problem without help, by selecting a correct or incorrect answer, replied on his/her own mathematic knowledge. The links from Solving time for correct answer and Solving time for incorrect answer nodes to Independent Study node reflect the relation. The links between Hint-based Study nodes and Independent Study nodes across time slices Ti and Ti+1 reflect the temporal dependencies for the learning goals. For example, a student that previously requests for instructions is more likely to request multimedia instructions again. As discussed earlier in our previous study, the learning goal of Independent Study will be affected by the learning goal of Hint- based Study. This is indicated by the link between Hint-based Study node at Ti and Independent Study node at Ti+1. Our model currently assumes the learning of Independent Study is Normal and the goal of Hint-based Study is None after student begins a new problem. 6.5 Model Initialization An initial detailed structure for the model in Figure 2 was defined by using the data from 115 students in two high schools who worked with the ITS. These students completed 2151 problems. The ITS logged 6468 actions into the database. The study gave us the following information: 66 Student actions with timestamp. When students worked with the ITS, we logged all the interface actions of students with a timestamp. The action data included keyboard and mouse events. There are following types of interface actions: • Begin: represents student begins a new math problem. • Hint: indicates student clicks hint button in order to request a hint. • Correct: denotes student selects the correct answer. • Attempt: represents student attempts to answer the problem by selecting an incorrect answer. For each action of student, one data record <Student, ProblemID, Actiontype, Timestamp> will be logged into database. Student is the student id in system. ProblemID is the problem number related to this action. Actiontype is one of the four types of actions: Begin, Hint, Correct and Attempt. And Timestamp is the time stamp for this action. Levels of Learning goals for students. From students’ action data, the period of time Time action that student spends for each action except Begin action are calculated. In order to identify levels of learning goals on action A t , the following situations of learning goals are identified by using the next action A t+1 and the comparison between Time action and 10 seconds: 1) Situation with certainty levels of learning goals. For instance, student studies the instruction for more than 10 seconds, and then selects correct answer. In this situation, the learning goal of Hint-based Study has certainty level as High. 67 2) And situation with uncertain levels of learning goals. For instance, student studies the instruction for less than 10 seconds and clicks “Hint” button again to request more instructions. In this situation, level of Hint-based Study can not be identified as High, Normal or Low because we don’t know whether student has already known the knowledge in the instructions or student just abuses it. Then we used probabilities for these possible levels of learning goals. Corresponding levels of learning goals for each action of students are identified for this situation. However, this identification mostly relied on the experimenter’s subjective observations. Using the data of interface actions with timestamp and levels of learning goals from experiment study, the conditional probability tables (CPTs) are defined in the model. 6.6 Model Evaluation and Results Data from an additional 135 high school students were used to evaluate the model’s performance. These students first completed a pre-test of math skills, and then worked with the ITS during their mathematics classes. Students completed 30 problems on average. Students’ actions were recorded and time-stamped as described above. After the activity, students completed a post-test. There were two versions of the test, counter-balanced across students. For example, Student A might receive Version 1 as a pre-test and Version 2 as the post-test, whereas Student B would receive Version 2 on the pre-test and then Version 1 as the post- test. Versions were established in prior work to be equivalent in difficulty. 68 6.6.1 Scoring Pre- and post-test. The tests were taken by students with their computers. Answers were scored automatically for accuracy, and scores were downloaded for analysis. Quantification of rating levels for learning goals. In the model, symbolic rating is used for learning goals representation. For instance, learning goal of Hint- based Study is identified as High, Normal, Low and None. And learning goal of Independent Study is identified as Very High, High, Normal and Low. To automatically evaluate our model, these symbolic ratings are converted into quantitative ratings for assessing learning goals. For Hint-based Study, a quantitative rating of 3 represents High, 2 represents Normal, 1 represents Low, and 0 for None. For Independent Study, a quantitative rating of 4 is identified to represent Very High, 3 for High, 2 for Normal, and 1 for Low. These numeric ratings for Hint-based Study and Independent Study are also called HS and IS values. Learning goal estimates. On each math problem presented to the student, the model updated its estimates of the student’s Hint-based Study and Independent Study after each action with the ITS interface. When the problem was completed (i.e., the student either clicked “Next” to skip the item or selected the correct answer) the HS and IS values were summed and divided by the time for the problem. The problem values were averaged for the number of problems completed by the student. Scores ranged from 0 to 3 for Hint-based Study. This 69 was because in practice, a student could have a score of 0 in the model if he or she never requested multimedia help from the ITS. In contrast, scores for Independent Study ranged from 1 to 4. As long as the problem was available to the student, it was possible that the student was attempting to solve it, so this scale was anchored at 1 as the low value in the model. Figure 6.3 The Distribution of Students’ Pre-test Scores 6.7 Results Students’ performance on the pre-test was not terribly strong; the mean proportion correct was 0.31, with a large standard deviation (0.21). As may be viewed in Figure 6.3, the distribution was quite skewed, with most students performing poorly but some doing fairly well. 70 Figure 6.4 The Distributions for Learning Goals Distributions for HS and IS scores are shown in Figure 6.4. The mean for HS scores was 2.24 (on a scale of 0 - 3), and the mean for IS scores was 2.49 (on a scale of 1- 4). Thus, the model estimates students were somewhat more likely to have the goal of learning through using multimedia help, relative to the goal of learning through independent problem solving. For each measure, quartile scores were used to classify students into four groups, e.g., students with High, Normal, Low or None of Hint-based Study, and students with Very High, High, Normal or Low of Independent Study (Note: Here the same rating scheme is used for both Hint-based Study and Independent Study. But they are different from the level representations used in the DBN model because here rating scheme is derived from quartile scores.). Relation of students’ pretest performance to IS scores. One component of the model estimates the probability that the student is attempting to solve the problem independently. Relevant actions include sufficient latency between HS IS 71 problem presentation and first student action to suggest that the student has read the problem. Other relevant action patterns include delays between answer attempts suggestive that the student is trying to debug his or her own errors. It seemed reasonable to predict that students with better initial math skills would be more likely to engage in Independent Study, because such students would have the skills to find the problem solutions on their own, without needing to view the multimedia help features. The relation between pre-test and Independent Study scores is shown in Figure 6.5. The bivariate fit was significant, F(1,33) = 92.099, p < .001, with r 2 = 0.409. Figure 6.5 The Relation Between Pre-test and Independent Study Scores A one-way Analysis of Variance was conducted to evaluate the relation more directly, with Independent Study Group as the between-subjects factor and Pre-test score as the dependent measure. The results indicated a significant effect of Independent Study group, F(3,131) = 41.549, p < .001. Tukey’s HSD tests with α levels set to 0.05 indicated that Very High of Independent Study (Group 3) 72 students had significantly higher scores on the math pre-test than other students. Recall that these students had the highest model estimates of trying to learn by solving the problems independently. Mean scores are shown in Figure 6.6. Figure 6.6 Mean Scores of Pre-test for IS Groups Relation of model estimates to learning outcomes. We were also interested in whether the model estimates of ITS learning goals were related to students’ performance on our outcome measure of learning: scores on the math post-test. For example, a student with low IS and low HS scores might be expected to learn little from the ITS activity. In contrast, it would be reasonable to predict that a student who had a high HS score would have learned enough to show improvement on the post-test. 73 To investigate the relation of actions with the ITS to learning outcomes, we assigned students to groups based on the cross-tabulation of HS and IS group membership. Table 6.2 shows the cross-tabulation for students as a function of their HS and IS groups. For example, 14 students who were in the Low HS and also the Low IS study groups were assigned to the Low-Both group 0. Students who were in the High HS but the Low IS groups were assigned to Group 1 (N = 53). Those with low HS but High IS were in Group 2 (N = 51). Students who were High for both measures were in Group 3 (N = 15). Table 6.2 Cross-tab of HS and IS Groups None HS Low HS Normal HS High HS Low IS 2 6 6 19 Normal 3 3 14 14 High IS 415 13 1 Very Hi 24 10 1 0 As may be viewed in the Table, most of the students fell into Groups 1 and 2. There were relatively few students whose behavior indicated consistently low engagement with the ITS. In turn, relatively few students were estimated by the model to be High for both HS and IS. Is there a positive relationship between students’ behavior with the tutoring system, as estimated by the model, and whether they learned as the result of the tutoring? Students’ post-test scores were used to classify them into Math 74 Achievement groups based on quantile cut-off values (as with the pre-test). A logistic regression was conducted to test the contributions of Pre-test Group (High, Average, Low or Very Low Math Achievement), Learning Goal Group (Low, HS>IS, IS>HS, High) and the interaction term to predictions of Post-test Group. The results indicated that the whole model was significant, χ 2 (7) = 56.756, p < .001. Pre-test Group did not contribute significantly to the model, χ 2 (1) = 0.367, N.S. This indicates that knowing how the student performed on the pre-test of math skill by itself is not sufficient to predict her or his post-test score reliably. In contrast, there was a significant contribution of Learning Goal Group to the model, χ 2 (3) = 16.928, p < .001. This indicates that students’ actions while working with the tutoring system accounted for significant variance in their scores on the Post-test. However, this effect was qualified by a significant interaction between the Pre-test Group and Learning Goal Group terms, χ 2 (3) = 14.866, p < .001. To interpret the interaction term in the logistic regression model, the mean test scores for the four Learning Goal Groups are shown in Figure 6.7. As may be observed in the Figure, students whose actions indicated both low use of Hints and also low effort to solve problems independently (Group 0) were likely to have low scores on the pre-test. These students also did not improve much as the result of tutoring. In contrast, students with similar pre-test scores but relatively high Hint Study scores (Group 1) showed significant improvement on the post-test. That is, 75 students in Groups 0 and 1 started out with similarly weak math skills, but those who used the ITS hints improved whereas the others did not. Figure 6.7 Mean Pre- and Post-test Scores for Learning Goals Groups The picture is different for students with high IS scores (Group 2). These students have better math skills to begin with, as indicated by their higher pre-test scores, and they also are more likely to work independently. Thus, the interaction effect in the model indicates that HS and IS both explain variance in pre- to post- test score change, but that these factors function differently for students with weak or strong math skills. 6.8 Discussion Our first goal was to create a model that would estimate the student’s different learning goals and continually update the estimates as the student worked on a problem. The DBN model constructed estimates of students’ hint-based study and independent study from interface action records from one sample, working with relatively limited data: specifically, which interface items the student selected, in 76 what order, and for how long. The model was also able to generate HS and IS estimates for a second sample and to relate these estimates to student learning. It should be noted that the ITS interface was highly structured; on any particular math problem, the student’s possible actions were quite limited (choose answers, view help, replay, skip, etc.) which contributed to the model’s success. However, our objective was to turn this structure to our advantage by linking the actions to estimates of the student’s goal, and the DBN model was able to generate and update these estimates. The second goal of the project was to refine the notion of learner “engagement” by identifying two distinct learning goals suggested by self-regulated learning theory: One goal is to use available resources – here, the hints available in the ITSI – to learn, whereas another pathway to learning is to apply one’s existing knowledge to a new problem, and to self-correct errors until the solution is achieved. Traditionally, ITS research has focused on encouraging students to use help resources effectively. The present results suggest that help-use may not be necessary or optimal for all learners, and the DBN model provides a first step towards being able to recognize learners with different goals from their interactions with the interface. The third goal was to relate the DBN estimates of learning goals to a learning outcome measure, here, scores on the math post-test. Students varied in their rates of HS and IS, and in the balance of the two learning strategies: Some students did not use the help features very much if at all, whereas others did so at a high rate. 77 Some students who did not use the help features appeared to work hard to figure out the solutions on their own, whereas others simply guessed their way through the problem to the correct answer. Self-regulation theory as well as prior research would predict that this should affect how much students learned from the activity. Consistent with this prediction, students with low learning goals did not show significant improvement in their test performance. However, those with high HS or high IS goals did improve. Thus, the DBN estimates were related to an independent measure of student learning. We also found that students’ learning goals, as indicated by their HS and IS estimates, were related to their initial math achievement. Specifically, students who started the ITS activity with weak skills, as indicated by their low pre-test scores, tended to have high HS scores relative to students with better math skills who were more likely to have high IS scores. Perhaps this is not surprising: students who already know at least some of the mathematics concepts and skills targeted in the ITS will have a better chance to solve the problems through effort. What is more interesting is that many of the better students chose to learn through their own efforts rather than by viewing the ITS help features. Although we did not have the opportunity to ask students why they preferred to work on their own, a few informal observations along with other work suggests that attributions may play a role. Finding the answer on your own, even if you make many errors, can be interpreted as support for your ability, whereas using multimedia help may undermine this interpretation. Interestingly, some students did view the multimedia 78 help after they had arrived at the correct answer, saying that they wanted to confirm that they had done the problem correctly. The results also indicated, for students with poor math skills, use of ITS help was associated with improvement from pre- to post-test. However, there were some students with poor math skills who did not use the hints and did not learn much, if anything, from the ITS activity. There were relatively few such students (10% of the sample). However, it will be important in future work to learn more about why these students were reluctant to engage in hint-based study, and how to increase their engagement with the ITS. The DBN model offers a way to recognize these cases from students’ actions, as a first step towards effective intervention. One question is why Group 3 students, whose actions suggested the highest level of engagement (both HS and IS scores), did not show significant improvement from pre- to post-test. If, as estimated by the DBN model, these students were highly engaged in learning from the ITS, why did they not perform better on the post-test? One possibility is that these students’ actions were not classified correctly by the DBN, which is sensitive to the latencies associated with students’ actions. For example, if the student takes some time to review the problem before attempting an answer, the model will assign higher values for IS than when the student attempts to answer very quickly. However, it is possible that a long latency before the first response might reflect student boredom. In addition, at least some correct answer selections that increase IS estimates in the model may have been due to lucky guesses; there are five answer options for each math 79 problem, so chance performance would be 20%. Thus, Group 3 students’ temporal data may have suggested high engagement when the students were actually bored or detached. Even so, it should be noted that there were only 15 students in this group, meaning that the DBN model generated estimates of HS and IS that were strongly related to the outcome measure (post-test scores) for 89% of the sample. Additional work will be required to identify why the model did not function well for the remaining students. 80 Chapter 7 Contribution and Future Work 7.1 Contributions The following list summarizes the main contributions of this thesis: 1. This thesis analyzed how human tutor interacted with students and which sources of learner’s information that will be used by human tutor to infer learner’s focus of attention 2. This thesis provided a probabilistic model for ITSs to track learner’s focus of attention under certainty and uncertainty sources of input information. Combined learner’s mouse, keyboard and eye gaze information, this tracking model can get more than 85% accuracy to recognize learner’s focus of attention and it improved the confidence of ITSs to make decision of intervention. 3. This thesis described a classifier to classify learner’s attention sequence. Five attention patterns were classified and over 90% of problem events were categorized. 4. This thesis proposed two learning goals of learner based on learner’s attention patterns, including the goal of learning by using multimedia help, and the goal of learning through independent problem solving a model to access learner’s learning goals in real time. 5. This thesis described a model based on Dynamic Bayesian Network (DBN) by utilizing students’ interface actions to infer learner’s learning goals in real time. 81 7.2 Areas of Future Work Our work has been focusing on helping ITSs to generate more appropriate intervention at right time. Several attractive suggested directions for future research include: • Extending the use of the DBN model of inferring learner’s focus of attention and learning goals to different ITSs in order to improve the ITSs’ ability to interact with the learners. • Validating the DBN model of inferring learner’s learning goals with real- time data about cognitive workload and attention. A measure of self- regulation will be considered in order to learn if students’ strategic behavior in classroom-based situations predicts their HS and IS estimates while working with an ITS. 82 References [Abowd 99] Abowd, G. D. Software Engineering Issues for Ubiquitous Computing. Proceedings 21st International Conference on Software Engineering (ICSE'99), Los Angeles, CA, USA. 75 – 84, May 1999. [Albrecht 97] Albrecht, D. W.; Zukerman, I.; Nicholson, A. E.; and Bud, A. Towards a Bayesian model for keyhole plan recognition in large domains. In Jameson, A.; Paris, C.; and Tasso, C., eds., Proceedings of the Sixth International Conference on User Modeling (UM '97), 365-376. SpringerWien New York, 1997. [Aleven 00] Aleven, V., & Koedinger, K. R. Limitations of student control: Do students know when they need help? In G. Gauthier, C. Frasson, & K. VanLehn (Eds.), Proceedings of the 5th International Conference on Intelligent Tutoring Systems, pp. 292-303. IOS Press. 2000. [Anderson 87] Anderson, J.R., Boyle, D.F., Farrell, R, and Reiser, B.J. Cognitive principles in the design of design of computer tutors. In P. Morris (Ed.), Modeling Cognition, Wiley, 1987. [Arroyo 03] Arroyo, I., Murray, T., Beck, J. E., Woolf, B. P., & Beal, C. R., A formative evaluation of AnimalWatch. Proceedings of the 11th International Conference on Artificial Intelligence in Education. IOS Press. 2003. [Azevedo 05] Azevedo, R., Cromley, J.G., Winters, F.I., Moos, D.C., & Greene, J.A. Adaptive human scaffolding facilitates adolescents' self-regulated learning with hypermedia. Instructional Science (Special Issue on Scaffolding Self- Regulated Learning and Metacognition: Implications for the Design of Computer- Based Scaffolds), 33, 381- 412. 2005. [Azevedo 05b] Azevedo, R., & Hadwin, A.F. Scaffolding self-regulated learning and metacognition: Implications for the design of computer-based scaffolds Instructional Science (Special Issue on Scaffolding Self-Regulated Learning and Metacognition: Implications for the Design of Computer-Based Scaffolds), 33, 367-379. 2005. [Baker 05] Baker, R. S., Corbett, A. T., Koedinger, K. R., & Roll, I. Detecting when students game the system, across tutor subjects and classroom cohorts. In L. Ardissono, P. Brna, & A. Mitrovic (Eds.), User Modeling, pp. 220-224. Berlin: Springer-Verlag, 2005. 83 [Beck 96] Beck, J., Stern, M., Haugsjaa, E. Applications of AI in Education, Special issue on artificial intelligence, pp. 11 – 15, ACM Press, New York, NY, USA, 1996. [Beck 05] Beck, J. E. Engagement tracing: Using response times to model student disengagement. In C-K. Looi et al. (Eds.), Artificial intelligence in education: Supporting learning through intelligent and socially informed technology, pp. 88- 95. Amsterdam: IOS Press, 2005. [Berliner 92] Berliner, D. C. The Science of Psychology and the Practice of Schooling: The One Hundred Year Journey of Educational Psychology from Interest, to Disdain, to respect for Practice. Paper presented at the American Psychological Association, Washington, D.C, 1992. [Bradshaw 97] Bradshaw, J. Software Agents: AAAI Press/The MIT Press, 1997. [Boekaerts 02] Boekaerts, M. The on-line motivation questionnaire: A self-report instrument to assess students' context sensitivity. In P. R. Pintrich & M. L. Maehr (Eds.), Advances in Motivation and Achievement, Vol. 12: New Directions in Measures and Methods (pp. 77-120). New York, JAI (Elsevier Science). 2002. [Burleson 04] Burleson, W. & R. W. Picard. Affective Agents: Sustaining Motivation to Learn Through Failure and a State of Stuck, Social and Emotional Intelligence in Learning Environments Workshop In conjunction with the 7th International Conference on Intelligent Tutoring Systems, Maceio - Alagoas, Brasil. 2004. [Conati 04] Conati, C. & Zhao, X. Building and Evaluating an Intelligent Pedagogical Agent to Improve the Effectiveness of an Educational Game. International Conference on Intelligent User Interfaces 2004. Madeira. Portugal. 2004. [Conati 04b] Conati, C. & Mclaren, H. Evaluating A Probabilistic Model of Student Affect. Proceedings of ITS 2004, 7th International Conference on Intelligent Tutoring Systems, Maceio, Brazil. p. 55-66. 2004. [Cooper 92] Cooper, G.F. and Herskovits, E. A Bayesian Method for the induction of probabilistic networks from data. Machine Learning, 9 (pp. 309-347). 1992. [Cutrell 02] Cutrell, E., Czerwinski, M., and Horvitz, E.. Notification, Disruption, and Memory: Effects of Messaging Interruptions on Memory and Performance, Proceedings of Interact 2001: IFIP Conference on Human-Computer Interaction, Tokyo, Japan, July 2001. 84 [Czerwinski 00] Czerwinski, M., Cutrell, E., and Horvitz, E.. Instant messaging: Effects of relevance and time, In S. Turner, P. Turner (Eds), People and Computers XIV: Proceedings of HCI 2000, Vol. 2, British Computer Society, p. 71-76, 2000. [Dean 89] Dean, T. and Kanazawa, K. A Model for Reasoning about Persistence and Causation. Computational Intelligence 5(3):142-150, 1989. [Dean 91] Dean, T. and Wellman, M. Planning and Control. San Mateo, California , Morgan Kaufmann, 1991. [Dessouky 01] Dessouky, M.M., Verma, S., Bailey, D., & Richel, J. A methodology for developing a Web-based factory simulator for manufacturing education. IEEE Transactions, 33, 167-180, 2001. [D’Mello 05] D'Mello, S. K., Craig, S. D., Gholson, B., Franklin, B., Picard, R. W., and Graesser, A. C., "Integrating Affect Sensors in an Intelligent Tutoring System," In Affective Interactions: The Computer in the Affective Loop Workshop at 2005 International conference on Intelligent User Interfaces (pp. 7-13) New York: AMC Press, 2005. [Duchowski 03] Duchowski, A. T. Eye Tracking Methodology: Theory and Practice. Springer-Verlag, London, UK, 2003. [Dweck 02] Dweck, C. Beliefs that make smart people dumb. In R. J. Sternberg (Ed.), Why smart people do stupid things. New Haven CT: Yale University Press. 2002. [Ebisawa 95] Ebisawa, Y. Unconstratined pupil detection technique using two light sources and the image difference method. Visualization and Intelligent Design in Engineering and Architecture, pages 79-89, 1995. [Ebisawa 98] Ebisawa, Y. Improved video-based eye-gaze detection method. IEEE Transactions on Instrumentation and Measurement, 47(2): 948-955, 1998. [Eccles 93] Eccles, J., Wigfield, A., Harold, R. D., & Blumenfeld, P. Age and gender differences in children’s self and task perceptions during elementary school. Child Development, 64, 830-847. 1993. [Gluck 00] Gluck, K.A., Anderson, J.R., Douglass, S., Broader Bandwidth in Student Modeling: What if ITS were "Eye"TS?. Intelligent Tutoring Systems 2000. 504-513, 2000. 85 [Graesser 99] Graesser, A. C., Wiemer-Hastings, K., Wiemer-Hastings, P., Kreuz, R., & TRG. AutoTutor: A simulation of a human tutor. Journal of Cognitive Systems Research, 1, 35-51. 1999. [Graesser 02] Graesser, A. C., VanLehn, K., Rose, C., Jordan, P., & Harter, D. Intelligent tutoring systems with conversational dialogue. AI Magazine, 22, 39-51. 2001. [Graesser 04] Graesser, A. C., Lu, S., Jackson, G. T., Mitchell, H., Ventura, M., Olney, A., & Louwerse, M. M. AutoTutor: A tutor with dialogue in natural language. Behavioral Research Methods, Instruments, and Computers, 36, 180-193. 2004. [Greeno 94] Greeno, J. G., Collins, A., Beranek, B., & Resnick, L. B. Cognition and Learning. In D. Berliner & R. Calfee (Eds.), Handbook of educational psychology (pp. 1-51). 7/31/94 draft: CEP 900 Course Pack, 1994. [Heckerman 95] Heckerman, D. A tutorial on learning with bayesian networks. Technical Report MSR-TR-95-06, Microsoft Research, Redmond, Washington, 1995. [Horvitz 98] Horvitz, E., Breese, J., Heckerman, D., Hovel, D., and Rommelse, K. The Lumière project: Bayesian user modeling for inferring the goals and needs of software users. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, pages 256--265, Madison, WI, 1998. [Horvitz 99] Horvitz, E., Jacobs, A., Hovel, D. Attention-Sensitive Alerting, Proceedings of UAI '99, Conference on Uncertainty and Artificial Intelligence, Morgan Kaufmann Publishers: San Francisco. pp. 305-313, July 1999. [Horvitz 99b] Horvitz, E., Jacobs, A., Hovel, D. Attention-Sensitive Alerting, Proceedings of UAI '99, Conference on Uncertainty and Artificial Intelligence, Morgan Kaufmann Publishers: San Francisco. pp. 305-313, 1999. [Horvitz 01] Horvitz, E. and Paek, T. Harnessing Models of Users’ Goals to Mediate Clarification Dialog in Spoken Language System. In Proceedings of the Eight International Conference on User Modeling, July 2001. [Horvitz 03] Horvitz, E., Kadie, C. M., Paek, T., Hovel, D. Models of Attention in Computing and Communications: From Principles to Applications, Communications of the ACM 46(3):52-59, March 2003. 86 [Horvitz 03b] Horvitz, E., and Apacible, J. Learning and Reasoning about Interruption. Fifth International Conference on Multimodal Interfaces. Vancouver, British Columbia, Canada, pp. 20-27, 2003. [Huhns 97] Huhns, M. N., & Singh, M. P. Readings in Agents: Morgan Kaufmann, 1997. [Johnson 00] Johnson, W.L., Rickel, J.W., and Lester, J.C. Animated pedagogical agents: Face-to-face interaction in interactive learning environments. International Journal of Artificial Intelligence in Education, 11, 47-78, 2000. [Johnson 03] Johnson, W. L. Interaction Tactics for Socially Intelligent Pedagogical Agents. In Proceedings of the Intelligent User Interfaces, 2003. [Johnson 05] Johnson, W.L., Rizzo, P., Lee, H., Wang, N., and Shaw, E. Modeling Motivational and Social Aspects of Tutorial Dialog. ITS Workshop on Human Tutorial Tactics and Strategies, 2005. [Karabenick 98] Karabenick, S., Strategic help seeking: Implications for learning and teaching. Mahwah NJ: Erlbaum. 1998. [Luckin 04] Luckin, R., Underwood, J., du Boulay, B., Holmberg, J., & Tunley, H. Coherence Compilation: Applying AIED Techniques to the Reuse of Educational TV Resources. Intelligent Tutoring Systems 2004: 98-107. 2004. [Maes 94] Maes, P. Agents that reduce work and information overload. Communications of the ACM, 37(7), 30 – 40, 1994. [Manske 05] Manske M. and Conati C. Modelling Learning in Educational Games. Proceedings of AIED 05, Proceedings of the 12th International Conference on AI in Education, Amsterdam, July 19-23, 2005. [Mayer 03] Mayer, R. E., Dow, G. T., & Mayer, S. Multimedia learning in an interactive self-explaining Environment: What works in the design of agent-based Microworlds? Journal of Educational Psychology, 95, 806-812, 2003. [McArthur 93] McArthur, M., Lewis, M., and Bishay, M. The Roles of Artificial Intelligence in Education: Current Progress and Future Prospects. David McArthur, Matthew Lewis, and Miriam Bishay. (1993) RAND DRU-472-NSF. A very good overview with lots of basic information about intelligent tutoring systems, 1993. [Morimoto 99] Morimoto, C.H.; Koons, N.; Amir, A. & Flickner. M. Frame-rate pupil detector and gaze tracker. IEEE ICCV’99 FRAME-RATE Workshop, 1999. 87 [Negroponte 97] Negroponte, N. Agents: from direct manipulation to delegation. In J. Bradshaw (Ed.), Software Agents (pp. 57-66): AAAI Press/The MIT Press, 1997. [Newman 02] Newman, R. S., How self-regulated learners cope with academic difficulty: The role of adaptive help seeking. In S. Pape, B. Zimmerman, B., & F. Pajares (Eds.), Theory into practice: Special issue: Becoming a self-regulated learner, pp. 132-138. Columbus, OH: The Ohio State University. 2002. [Nicholson 94] Nicholson, A. E. and Brady, J. M. Dynamic belief networks for discrete monitoring. IEEE Transactions on Systems, Man and Cybernetics, 24(11):1593-1610, 1994. [Pavlovi’c 99] Pavlovi´c, V., Rehg, J. M., and Kevin, J.P.C., A Dynamic Bayesian Network Approach to Figure Tracking Using Learned Dynamic Models Murphy SLDS-ICCV99, 1999. [Qu 05] Qu, L., and Johnson, W. L. Detecting the learner’s motivational states in an interactive learning environment. In C-K. Looi et al. (Eds.), Artificial intelligence in education: Supporting learning through intelligent and socially informed technology, pp. 547-554. Amsterdam: IOS Press, 2005. [Raskin 00] Raskin, J. The Humane Interface: New Directions for Designing Interactive Systems: Addison-Wesley, 2000. [Rayner 98] RAYNER, K. Eye Movements and Information Processing: 20 Years of Research. Psychological Bulletin 124, 3, 372-422, 1998. [Ritter 98] Ritter, H. & Rae, R. Recognition of human head orientation based on artificial neural networks. IEEE Transactions on Neural Networks, 9(2):257-265, 1998. [Shneiderman 92] Shneiderman, B.Designing the User Interface: Strategies for Effective Human-Computer Interaction: ACM Press, 1992. [Shneiderman 97] Shneiderman, B. Direct manipulation versus agents: path to predictable, controllable, and comprehensible interfaces. In J. Bradshaw (Ed.), Software Agents (pp. 97-106): AAAI Press / MIT Press, 1997. [Shute 89] Shute, V., R. Glaser, and K. Raghaven. 1989. Inference and Discovery in an Exploratory Laboratory. Learning and Individual Differences, Ackerman, P., R. Sterberg, and R. Glaser, eds., pp. 279-326. [Sleeman 82] Sleeman, D. and Brown, S. J. Intelligent Tutoring Systems. Computers and People Series. Academic Press, Inc., London, 1982. 88 [Scott 03] Scott, D., McCrickard, Catrambone, R., Chewar, C. M., and Stasko, T. J.. Establishing Tradeoffs that Leverage Attention for Utility: Empirically Evaluating Information Display in Notification Systems. International Journal of Human- Computer Studies, 2003. [Suppes 67] Suppes, P. Some theoretical models for mathematics learning. Journal of Research and Development in Education, 1, 5-22, 1967 [Tunley 04] Tunley, H., du Boulay, B., Luckin, R., Holmberg, J., & Underwood, J. Up and Down the Number-Line: Modelling Collaboration in Contrasting School and Home. 2004. [Turner 02] Turner, J. C., Midgley, C., Meyer, D. K., Dheen, K., Anderman, E. M., Kang, Y, & Patrick, H., The classroom environment and students’ reports of avoidance strategies in mathematics: A multimethod study. Journal of Educational Psychology, 94, 88-106. 2002. [Uhr 69] Uhr, L. Teaching machine programs that generate problems as a function of interaction with students. Proceedings of the 24th National Confernece, 125-134, 1969. [Vicente 98] Vicente, A. D. & Pain, H. (1998). Motivation Diagnosis in Intelligent Tutoring Systems. Intelligent Tutoring Systems 1998: 86-95. 1998. [Vicente 02] Vicente, A., & Pain, H. Informing the detection of the student’s motivational state: An empirical study. In 6th International Conference on Intelligent Tutoring Systems, pp. 933-943. Biarritz, France, 2002. [Weiser 91] Weiser, M. The computer of the 21st century. Scientific American, 265(3), 66-75, 1991. [Zhou 03] Zhou X. and Conati C. Inferring User Goals from Personality and Behavior in a Causal Model of User Affect. In Proceedings of IUI 2003, International Conference on Intelligent User Interfaces, Miami, FL, U.S.A. p. 211- 218. 2003.

Abstract (if available)

Abstract Intelligent Tutoring Systems (ITSs) have evolved dramatically from the simple prompt for remediation based on a wrong answer to the complex, adaptive systems of today that truly qualify as intelligent. Modern ITSs include a variety of applications that allow the emulation of a human teacher and the ITS acts as the student's private tutor, and interacts to effectively lend pedagogical assistance to the learner. Researchers have focused on modeling the learner's cognitive processes while solving problems, i.e., the "model tracing" approach. However, there is growing recognition that learning involves more than cognition, and that students' attention with an ITS also reflects "engagement" meaning the transient shifts in focus of attention and the emotions that are often associated with learning. Failure to track the learner's engagement could cause the ITSs to interrupt the learner with advice when the learner does not really need it.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Understanding the relationship between goals and attention

PDF

An intelligent tutoring system’s approach for negotiation training

PDF

Modeling social and cognitive aspects of user behavior in social media

PDF

Transfer learning for intelligent systems in the wild

PDF

Artificial Decision Intelligence: integrating deep learning and combinatorial optimization

PDF

Using social networking technology to improve collaborative requirements elicitation, negotiation, prioritization and evolution

PDF

Learning to detect and adapt to unpredicted changes

PDF

Building straggler-resilient and private machine learning systems in the cloud

PDF

Data scarcity in robotics: leveraging structural priors and representation learning

PDF

Scaling up temporal graph learning: powerful models, efficient algorithms, and optimized systems

PDF

Building and validating computational models of emotional expressivity in a natural social task

PDF

Modeling dyadic synchrony with heterogeneous data: validation in infant-mother and infant-robot interactions

PDF

Inter professional education and practice in the health care setting: an innovative model using human simulation learning

Asset Metadata

Creator Qu, Lei (author)

Core Title Modeling the learner's attention and learning goals using Bayesian network

School Viterbi School of Engineering

Degree Doctor of Philosophy

Degree Program Computer Science

Publication Date 07/12/2007

Defense Date 03/07/2007

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag artificial intelligence,attention,classification,dynamic Bayesian network,engagement,intelligent tutoring system,OAI-PMH Harvest

Language English

Advisor Johnson, W. Lewis (committee chair), Beal, Carole (committee member), Boehm, Barry W. (committee member), Miller, Lynn Carol (committee member)

Creator Email leiqu@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-m614

Unique identifier UC1432889

Identifier etd-Qu-20070712 (filename),usctheses-m40 (legacy collection record id),usctheses-c127-516556 (legacy record id),usctheses-m614 (legacy record id)

Legacy Identifier etd-Qu-20070712.pdf

Dmrecord 516556

Document Type Thesis

Rights Qu, Lei

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Repository Name Libraries, University of Southern California

Repository Location Los Angeles, California

Repository Email cisadmin@lib.usc.edu