00001.tif

PDF

Download

Open document

Flip pages

Copy asset link

Request this asset

Request accessible transcript

Transcript (if available)

Content IMPASSE-DRIVEN TUTORING FOR REACTIVE SKILL ACQUISITION by Randall William Hill, Jr. A Dissertation Presented to the FACULTY OF THE GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (Computer Science) August 1993 Copyright 1993 Randall William Hill, Jr. i UMI Number: DP22865 All rights reserved INFORMATION TO ALL U SERS The quality of this reproduction is d ep en d en t upon th e quality of the copy subm itted. In the unlikely ev en t that the author did not sen d a com plete m anuscript and th ere are missing pag es, th e se will be noted. Also, if material had to be rem oved, a note will indicate the deletion. UMI' Dissertation Publishing UMI D P22865 Published by P roQ uest LLC (2014). Copyright in th e D issertation held by the Author. Microform Edition © P roQ uest LLC. All rights reserved. This work is protected against unauthorized copying under Title 17, United S ta tes C ode ProQuest P roQ uest LLC. 789 E ast Eisenhow er Parkway P.O. Box 1346 Ann Arbor, Ml 4 8 1 0 6 - 1346 UNIVERSITY OF SOUTHERN CALIFORNIA THE GRADUATE SCHOOL UNIVERSITY PARK LOS ANGELES, CALIFORNIA 90007 This dissertation, written by Randall William Hill, Jr. under the direction of hXs Dissertation Committee, and approved by all its members, has been presented to and accepted by The Graduate School, in partial fulfillment of re quirements for the degree of ph. P. C p5 > ra * , H DOCTOR OF PHILOSOPHY Dean of G raduate Studies D a te . . . . . . . . . . . . A u g u s t ^ . 9 9 9 3 , DISSERTATION COMMITTEE Chairperson To Marianne, whose love and encouragement made this work possible A cknow ledgem ents As I look back on the efforts of the last few years I am amazed at the num ber of people who have had a part in the production of this J dissertation. I owe an enormous debt to the community of family, friends, colleagues, and teachers who have encouraged, supported, criticized, and refined my work on the research described herein. A t the risk of overlooking the contributions of some, I will attem pt to acknowledge the efforts of a few of these "co-workers." The first person I wish to acknowledge is my wife, M arianne Haver Hill. To say that I am indebted to Marianne would be an understatement: her love, patience, support, and encouragement saw me through many difficult times. Yet, I know that she does not consider my "debt" to her to be something I have to pay back, because she has never put conditions on her love for me. I cannot imagine going through life's journey with anyone else. I owe much to my parents, Randall and Jerry, for the love of learning that they instilled in me. Dad taught me the meaning of discipline, hard work, and integrity through his exam ple and his w ords. Mom's enthusiasm for reading and her sense of curiosity infected me with the spirit of research. The sense of love and stability that they provided me and my brothers in our home gave nurture to the various pursuits we have each taken in our lives. My debt to my wife’ s parents, Lynn and Joan Haver, is also enormous. Their love toward me as a "son" has made me richer, and I am blessed every day by their daughter, who reflects their compassion and ideals in j many ways. } My brothers, Kevin and Bryan, have kept me laughing whenever we J are together. Somehow I lose some of my seriousness when I am with j them and I am glad to have brothers who are also good friends. Astrid and Benjamin have brought joy to my life and I look forward to more family time together now that I am done with school. Among the many people I count as friends, there have been a few in recent years who have been particularly supportive. The Shoracks: Todd, Karie, Ted, and Wesley have been family to me and Marianne these past three years. I am thankful for their listening ears and loving hearts. Chuck and Terry Fields are special friends who have been available day and night. Thanks to the Purgasons: Lee for always being up for a bike ride to Green Valley and to Kitty for being a good friend to Marianne. I can always count on Bryan Sands for a midweek lunch and I am thankful for a shared "adventure" with Russ and Sue Tenpas. The Jet Propulsion Laboratory has been a great place to work these past nine years. I have received financial support from the institution, which has shown a real commitment to educating its employees. I have also received tremendous moral and professional support from my managers, past and present: A1 Ellman, Mike Rodrigues, Lloyd Keith, Ann Griesel, Ed Records, Erich Corduan, Barry Cooper, A1 Silliman, Mark Browne, and Tammy Rimmer. Mark has not only supervised me but he has been a good friend as well. Tammy has been one of my biggest supporters, and I hope to one day have as much know-how about running a project as she possesses. With respect to this dissertation I am especially indebted to Lynne Cooper for promoting and managing my research on intelligent tutoring systems. Many of the concepts in this dissertation are a direct result of Lynne’ s research efforts in the Deep Space Network. Members of Lynne's group also supported many aspects of my work: a special thanks to Lorrine Lee, Kristina Fayyad, Kathy Sturdevant, Juan Urista, Elmain Martinez, i Crista Smyth and Erin Kan. Kathy was especially helpful in implementing ! the user interface for the training system; without her expertise the effort j w ould have required a great deal more w ork than I had time to accomplish. Thanks to Robert L. Miller for not only believing in me but also for ! , helping me to acquire funding for my research. Thanks also to Les Deutsch | , and William Rafferty of the Telecommunications and Data Acquisition j Technology Development section for their support. I hope to make their investment in me yield some good dividends for the Deep Space Network. Among the fellow USC students who have travelled this road with me, a few have made the trip easier by their assistance and humor: Nicholas Rouquette, Jon Lieb, and Bill Betts. A special thanks goes to Ben Smith for critiquing my research at various phases. I w ish to thank my dissertation committee: Les Gasser, Paul Rosenbloom, George Widmeyer, and W. Lewis Johnson. Each member has provided me the intellectual mentorship needed to become a researcher. Les Gasser guided me through the early years in my Ph.D. program; he encouraged me to free myself from the shackles of the "dom inant paradigm." I am still learning what this means, but Les has made me see that creative thought requires going beyond the old ways of doing things. Paul Rosenbloom welcomed me to the Soar group at the University of Southern California’ s Information Sciences Institute, and the interaction with this group of people in itself contributed greatly to the direction of my research. Paul's insights and challenging questions forced me to clarify many issues throughout the time I was developing my tutoring system. Paul always seems to know the right question to ask to get at the heart of a matter. Finally, I wish to thank my dissertation advisor, W. Lewis Johnson. Lewis took me under his wing nearly three years ago and since that time he has invested countless hours in me in terms of meetings, phone calls, and reviews of my work. Lewis has always treated me with respect, even when my ideas have not been well formed or articulated; his patient questions and ways of getting me to think through the issues are the marks of a great teacher and advisor. Lewis' encouragement to build a cognitive model and to use Soar for its implementation had a big impact on the direction of my research, taking it to places that I had not anticipated. It has been a great experience and I am thankful that I have had the privilege of working with someone I respect so much. The research described in this dissertation was carried out by the Jet Propulsion Laboratory, California Institute of Technology, under contract j with the National Aeronautics and Space Administration. C ontents 1. Introduction 1 1.1 The general problem: the acquisition of reactive skills 2 1.2 The specific problem: tutorial interaction 5 1.3 Impasse-driven approach to tutoring in reactive task domains 6 1.4 An interactive task domain: Link Monitor and Control (LMC) system 9 1.5 The REACT intelligent tutoring system 12 1.6 An example of impasse-driven tutoring 13 1.6.1 The task to perform 13 1.6.2 Tutoring with REACT 14 1.7 REACT's approach to tutoring 18 1.8 Related work 18 1.8.1 Tutoring implementation methods 19 1.8.1.1 Model tracing approaches to intelligent tutoring 19 1.8.1.2 Plan recognition approaches to intelligent tutoring 20 1.8.1.3 Overlay-based models 21 1.8.2 Cognitive modeling and theories of tutoring 22 1.8.3 Real-time task domains 25 1.9 Key contributions of this research 26 1.10 A guide to the dissertation 28 2. REACT as an ITS Architecture 30 2.1 Introduction 30 2.2 LMC training system architecture 31 2.2.1 Graphical user interface 31 2.2.1.1 LMC directive menu 31 2.2.1.2 LMC event log 33 2.2.1.3 LMC system displays' 34 2.2.1.4 REACT tutor window 35 2.3 LMC simulator 36 2.3.1 Device models 36 2.3.2 Command processor 37 2.3.3 O utput functions 38 2.4 REACT tutor 38 2.5 Comparison to traditional ITS model 39 2.5.1 The expert model 40 2.5.2 The student model 41 2.5.3 The environment module 43 vi 2.5.4 The tutor model 44 3. Tutoring with REACT 46 3.1 Introduction 46 3.2 Procedure manuals as resources 47 3.3 How tasks would be performed in an ideal world 49 3.4 The effects of the real world on task performance 51 3.5 Other Operator behaviors in the real world 53 3.6 What Operators need to know 56 3.7 Tutoring with REACT 57 3.7.1 Explaining an action constraint violation 59 3.7.2 Plan dependency violations example 60 3.7.3 Goal failure example 61 3.8 Summary 62 4. A Cognitive Model of Reactive Skill Acquisition 63 4.1 Introduction , 63 4.2 Overview of the Soar architecture 64 4.2.1 Soar cognitive architecture 65 4.2.2 Useful modeling features of Soar 65 4.3 Models of problem solving behavior 66 4.3.1 Novice problem solving 67 4.3.2 Expert problem solving 68 4.4 Knowledge compilation 71 4.5 Failure recovery and learning 73 4.5.1 Missing knowledge 74 4.5.2 Incorrect knowledge 77 4.5.3 Learning while recovering from failure 78 4.6 Results and design desiderata 79 4.6.1 Learning by doing 80 4.6.2 Learning about preconditions 81 4.6.3 Rote learning is incomplete 81 4.6.4 Learning occurs in a goal context 82 4.6.5 Learning via impasse 83 4.6.6 Detecting hidden knowledge gaps 84 4.6.7 Recovery from incorrect knowledge . 84 4.7 Conclusions: Benefits and limitations of the cognitive model 85 5. Situated Plan Attribution 88 5.1 Assumptions about plans and action 89 5.2 Situated plan attribution 90 5.2.1 The representation of plans, goals and action 91 5.2.1.1 Plan state attribute 91 5.2.1.2 Plan execution status attribute 92 V ll 5.2.1.3 Plan goal status attribute 93 5.2.2 Order among plans: temporal dependency network , 94 5.2.3 Representing the situation 95 5.2.4 Problem space description of situated plan attribution 96 5.2.5 The processing cycle with no student impasses 100 5.2.5.1 Waiting for input 100 5.2.5.2 Device state changes 101 5.2.5.3 Processing a student action 101 5.3 Impasse recognition 101 5.3.1 Action constraint violations 103 5.3.2 Goal failure 104 5.3.3 Plan dependency violations 106 5.4 Comparison to other approaches 108 5.5 Scope and limitations of situated plan attribution 110 6. Impasse Explication 112 6.1 Problem space hierarchy for impasse explication 113 6.2 Action constraint impasses 115 6.2.1 Explication using the expert cognitive model 116 6.2.2 Detecting and explaining the impasse 118 6.2.3 Resolving the impasse , 121 6.3 Goal failure impasses 122 6.3.1 Explicating a goal failure impasse 123 6.3.2 An example of goal failure explication 125 6.3.3 The explanation of a goal failure 126 6.4 Plan dependency impasse 126 6.5 The tutor improves over time 128 6.5.1 Impasse recognition chunks 129 6.5.2 Impasse explication chunks 129 6.5.3 Improvement 130 6.6 Conclusions 130 6.7 Related work 131 7. Results 132 7.1 Introduction 132 7.2 Prototype summative evaluation 135 7.3 Method 135 7.3.1 Task assignment 136 7.3.2 Pre-test phase 137 7.3.3 Training phase 138 7.3.4 Post-test phase 138 7.3.5 Summary of potential impasses by mission 139 1 7.4 Analysis of summative evaluation results 141 7.4.1 Data collected on individuals 141 viii 7.4.2 Summary and analysis of results 143 7.5 Evaluation by questionnaire 146 7.5.1 Background information 146 7.5.1.1 Questions for gathering background information 147 7.5.1.2 Results 147 7.5.2 Formative evaluation of Hypothesis 1 147 7.5.2.1 Questions related to Hypothesis 1 148 7.5.2.2 Results ,148 7.5.3 Formative evaluation of Hypothesis 3 149 7.5.3.1 Questions related to Hypothesis 2 150 7.5.3.2 Results 151 7.5.3.3 Conclusions about questionnaire 153 7.6 Assessing the effectiveness of situated plan attribution 153 7.6.1 Evaluation m ethod 154 7.6.2 Evaluation of El 155 7.6.3 Evaluation of E2 156 7.6.4 Evaluation of E3 157 7.6.5 Evaluation of E4 157 7.6.6 Evaluation of E5 158 7.7 Summary 158 8. Conclusions 161 8.1 Summary of thesis 161 8.2 Contributions to the understanding of tutorial interaction 162 8.3 Generality of methods used for impasse-driven tutoring 165 8.4 Direction of future work 167 Bibliography 170 Appendix A: Summative Evaluation Data 179 Appendix B: Acronyms 182 ix List of Figures 1.1 The VLBI task with a representative set of plans and operators 10 1.2 Procedural summary sheet for the assigned task 14 1.3 An example of tutoring with REACT 15 2.1 LMC training system architecture 31 2.2 LMC directive menu: VLBI subsystem directives shown 32 2.3 LMC training system event log 33 2.4 A example system display: NCB Configuration Table (NCNF) 34 2.5 REACT tutor window 35 2.6 Simulator device models 36 2.7 Traditional model of an ITS 39 2.8 REACT’ s expert model 40 2.9 REACT's student model 42 2.10 REACT's environm ent m odule 43 2.11 REACT’ s tutor model 45 3.1 The VLBI task with a representative set of plans and operators 48 3.2 Plans for Configure-DSP and Coherence-Test sub-tasks 48 3.3 Executing Configure-DSP and Coherence-Test in an ideal world 50 3.4 Executing Configure-DSP and Coherence-Test procedures in the "real world" 52 3.5 Actual event log illustrating impasse-based Operator behaviors 54 3.6 A tutored session with REACT , 58 4.1 The VLBI task with a representative set of plans and operators 67 4.2 Verification and recovery problem spaces for Load-Predicts problem space 69 4.3 Preconditions for Load-Predicts operator 70 4.4 Postconditions for Load-Predicts operator 70 4.5 Pseudo-code novice chunk that applies the NLOAD command for the Load-Predicts Operator 72 4.6 Pseudo-code expert chunk that verifies a precondition for the Load-Predicts Operator 73 4.7 Pseudo-code expert chunk that applies the command for the Load-Predicts operator 73 4.8 NLOAD command rejected 75 4.9 NLOAD command accepted and successfully terminated 77 5.1 Plans and their attributes 91 5.2 Plans contain operators, which have attributes 92 5.3 Plans have goals 93 5.4 Partial Temporal Dependency Network for VLBI Task 94 5.5 Device model example 95 5.6 Problem space hierarchy for situated plan attribution 97 5.7 Example of an action constraint violation 103 5.8 Example of a goal failure impasse 105 5.9 Example of a plan dependency violation impasse 107 5.10 Example of situated plan attribution allowing a plan dependency violation 107 6.1 Problem space hierarchy for impasse explication 114 6.2 Resolving an action constraint violation impasse 117 6.3 Event log with action constraint violation impasse example 118 6.4 Student preconditions of load-predicts operator ,119 6.5 Actual device states 119 6.6 Partial explanation of why NLOAD failed 120 6.7 Explanation of why NLOAD failed/ continued 120 6.8 Impasse resolution for NLOAD, continued 121 6.9 Resolving a goal failure impasse 123 6.10 Example of a goal failure impasse 125 6.11 Resolving a plan dependency violation impasse 127 6.12 Tutor's performance on action constraint impasse before and after learning, measured in seconds 130 7.1 Summary of potential mission impasses 139 7.2 Example of data collected on each subject 141 7.3 Time (in seconds) required to resolve potential action constraint impasses 142 7.4 Total time required to perform tasks in seconds 143 7.5 Average time per command in seconds 143 7.6 Reaction time for potential action constraint impasses, with and w ithout the tutor. 144 7.7 Did the subject successfully achieve the task goals? 145 7.8 Background information 147 7.9 Results related to Hypothesis 1 148 xii 7.10 Results related to Hypothesis 3 151 7.11 Evaluation criteria for Hypothesis 2 154 7.12 Total number of commands for tutor to interpret 155 7.13 Impasses detected by tutor by category 155 Chapter 1 Introduction This thesis describes a new approach to intelligent tutoring in interactive task domains that require goal-oriented, reactive skills. The proposed ap proach, called "impasse-driven tutoring," implements a theory of tutorial interaction in an interactive task domain. The theory addresses the ques tion of when and how to interact with students as they perform tasks in an interactive domain; it predicts that the best time to interact is when the student is at an impasse point in problem solving, where an impasse is an inability to make progress due to a lack of knowledge. The theory also pre dicts w hat kind of knowledge should be communicated to the student dur ing the interaction with the tutor. The theory of tutorial interaction underlying impasse-driven tutoring is based on hypotheses about learning and tutoring that are m ade from a cognitive model of problem solving and skill acquisition in an interactive domain. The cognitive model was designed to improve its behavior over time by acquiring and compiling new knowledge; it is able to revise its problem solving procedures to respond to dynamically changing situations, recover from errors, and it recovers from incorrect knowledge. Using the model, which simulates the behavioral transition from a novice to an ex pert, a num ber of predictions are made about how tutoring can affect skill acquisition in interactive domains. The impasse-driven tutoring methodology is implemented in an intel ligent tutoring system called REACT. REACT uses a technique called "situated plan attribution" to detect student impasses and an expert cogni tive model to explicate them. The tutor is implemented in Soar, an inte grated problem solving and learning architecture (Laird et al., 1987; Newell, 1990; Rosenbloom&Newell, 1986), and it is integrated w ith a monitor and control system simulator, which provides the training application domain for the work described in this thesis. 1 A pilot study was conducted that suggests that the hypotheses about tu torial interaction implemented in REACT have validity. The study im plem ented a prototype summ ative evaluation w here students perform tasks on a training system that uses REACT. Performance data and feed back from a questionnaire were used to evaluate the effectiveness of the system. 1.1 The general problem: the acquisition of reactive skills Monitor and control systems are ubiquitous in our increasingly automated society: power plants, factories, environmental control systems and opera tions centers all use computers to control other machines and to monitor their status and health. Typically only a portion of the work to be done in the problem domain is automated, leaving tasks requiring judgm ent and specialized knowledge to a human Operator, who m ust use these monitor and control systems to perform complex tasks in a dynamic and sometimes unpredictable world. The skills required for performing interactive tasks have two basic characteristics: they are goal-oriented and reactive. The Operators of monitor and control systems perform tasks in an in teractive and dynamic environment: they m ust achieve task goals using limited resources, which include items such as time, data, and cognitive processing. Task goals are typically achieved by executing standard proce dures or plans, while at the same time reacting to the state of the world (e.g., Wilkinson, 1992; Cooper et al., 1991; Cooper, 1991; Lee&Hill, 1992; Hill&Lee, 1992; Dvorak, 1987). In contrast to domains such as mathemati cal problem solving, which typically involve little interaction with the world (other than manipulating symbols), interactive tasks usually are di rectly involved with changing the state of the world. Due to the dynamic and at times unpredictable nature of the world, however, it is not always possible for a problem solver to execute a plan flawlessly; actions have un intended effects, assumptions behind the actions may be wrong, and events beyond the control of the problem solver may intervene. This is the gen eral idea behind "situated action," which has been put forth by Suchman (1987): 2 The foundation of actions by this account is not plans, but local interactions with our environment, more and less informed by reference to abstract representations of situations and of actions, and more and less available to representation themselves. The function of abstract representations is not to serve as specifica tions for the local interactions, but rather to orient or position us in a way that will allow us, through local interactions, to exploit contingencies of our environment, and to avoid others. While plans can be elaborated indefinitely, they elaborate actions just to the level that elaboration is useful; they are vague with respect to the details of action precisely at the level at which it makes sense to forego abstract representation, and rely on the availability of a particular, embodied response. (Suchman, 1987, p. 188) Suchman's view of action changes the conventional role of the plan from being the determining factor in behavior to a more moderate role as a mere resource. . . . plans are resources for situated action, but do not in any strong sense determ ine its course. While plans presuppose the em bodied practices and changing circumstances of situated ac tion, the efficiency of plans as representations comes precisely from the fact that they do not represent those practices and cir cumstances in all of their concrete detail. (Suchman, 1987, p. 52) This ultimately means that though a problem solver's goals may re m ain the same, the means of accomplishing them may vary, depending on the state of the world. It also means that even the goals may change if the world's state warrants it. If we take the situated action view of plans in interactive domains, there are some significant ramifications for how O perators should be trained in the skills to perform interactive tasks. One of the first observa tions is that it is not sufficient to merely "teach by telling," which essen tially means describing how to do a task in terms of a plan or procedure. Since a plan will never be sufficiently detailed to convey all of the knowl edge needed to perform a task that is situated in the w orld, something more is needed. A second observation follows from the first. Since action is situated in the world, it makes sense that learning how to perform a task m ust also be 3 situated in the world. This lends credence to the concept of "learning by doing." In his essay entitled Interest and Effort in Education, John Dewey m ade the following observation about learning and doing: There is nothing new or striking in the conception of activity as an im portant educational principle .... To make the idea of ac tivity effective, we must take it broadly enough to cover all the doings that involve growth of power— especially of power to real ize the meaning of what is done. (Dewey, 1939, p. 607) One hypothesis of what is gained by "learning by doing" is that it leads to an understanding of how to achieve a goal in the context of the real world. In other words, learning by doing helps the doer to acquire knowl edge about the details of the situation that affect how an action is taken, thus bridging the gap between theory and practice. This learning is accom panied by the other benefit of practice, i.e., increased efficiency, which is ex plained by the power law of practice (Snoddy, 1926). Recent cognitive theories of learning and skill acquisition such as ACT* (Anderson, 1983) and Soar (Newell, 1990) address the role of doing in learn ing more formally. In these and other theories, knowledge falls into two categories: declarative and procedural, where the procedural knowledge is acquired through a compilation process that takes place as the problem is being solved. W ithout problem solving, the learning of a procedural skill does not occur. The third observation is that merely "learning by doing" may not be sufficient, or at least not the most efficient way of acquiring skills in an in teractive task domain. Since the real world is such a complicated place that it is nearly impossible to describe all of the details of how to perform a task, then it doesn't make sense to believe that a student can efficiently acquire an understanding of all of the nuances of action by merely performing the task. The solution to this dilemma is to supplem ent "learning by doing" with "learn from a tutor." By this I do not mean to abandon learning by doing, rather, this means learn from a tutor in the context of doing. Benjamin Bloom found that students who had been personally tutored received achievement test scores above 98% of the students who were in 4 conventional classroom settings (Bloom, 1984). He noted the difference in style between tutoring and conventional classroom teaching: It is very different in a one-to-one tutoring situation where there is constant feedback and corrective process between the tutor and the tutee. If the explanation is not understood by the tutee, the tutor soon becomes aware of it and explains it further. (Bloom, 1984, p. 11) Bloom’s study confirms the conventional w isdom behind apprentice ship training, coaching and other forms of tutoring. This has also been one of the driving motivations behind the development of intelligent tutoring systems, and it is the motivation for asking the next question: How should an intelligent tutor interact with a student in a task domain requiring reac tive, goal-oriented skills? This question will henceforth be called the "tutorial interaction problem." 1.2 The specific problem: tutorial interaction In an empirical study of hum an tutors, Deborah Galdes noted that the tuto rial interaction problem involves knowing w hat to discuss and when to discuss it. Furthermore, deciding what to discuss was usually "related to ei ther correcting a definite error or identifying if a potential error was truly a definite error" (Galdes, 1989, p. 225) Since tutors do not have access to the student’s mental states, it becomes necessary to determine when and how to interact by observing the student performing the task and recognizing key indicators (Galdes, 1989). The basic problem addressed by this thesis is to find a solution for the tutorial interaction problem in an interactive task dom ain, that is, when and how should the tutor interact with the student? The first step that was taken in this research toward answering the tuto rial interaction question was to develop a detailed cognitive model of reac tive skill acquisition for an interactive domain, namely, the Link Monitor and Control (LMC) System used in NASA's Deep Space Network (DSN). The overall purpose of the LMC cognitive model was to gain an under standing of skilled behavior and learning in this domain. The model ac 5 complished this in several ways: it accounts for problem solving behavior at novice and expert levels of skill; it models skill acquisition and its effects on behavior; and, it accounts for how problem solvers can recover from several types of impasses in the domain, which, as it turns out, is a key element of skill acquisition (Hill&Johnson, 1993a). The second step was to develop a method for tutorial interaction in an interactive task domain that took into account the lessons learned from the cognitive model. The basic approach focuses on detecting student impasses in the course of performing a task. The technique em ployed by the tutor for interpreting the student's behavior is called "situated plan attribution." This approach incorporates the notion of situated action in its way of view ing the student: its goals are to give the student the flexibility to react to the situation in the world and to detect impasses that can lead to key learning opportunities by the student. This general approach to answering the tuto rial interaction problem is called "impasse-driven tutoring." 1.3 Impasse-driven approach to tutoring in reactive task domains The LMC cognitive modeling work yielded some interesting insights into problem solving and skill acquisition in this domain. Moreover, these in sights led to an approach to solving the tutorial interaction problem in the LMC domain called impasse-driven tutoring. The basic idea behind impasse-driven tutoring is that the best time to interact with the student is while he or she has the goal to acquire some additional problem solving knowledge. The cognitive model hypothesizes that students naturally try to acquire new knowledge w hen they have reached a problem solving impasse; thus, during the interaction the tutor m ust provide the student with the knowledge needed to understand the nature of the impasse and how to resolve it. To understand w ant is meant by an impasse, let us first consider how the term has been used elsewhere. The concept of an impasse is not new: Brown and VanLehn (1980) and VanLehn (1982, 1983) defined impasses as points in problem solving where the student's knowledge was not sufficient to continue performing a pro cedure correctly: "An impasse occurs when the step they believe should be executed next cannot be performed. If they are in a test-taking situation, 6 where they may not receive help, they perform a specific, simple kind of problem solving, called 'repair'." (VanLehn, 1988, p.19) VanLehn devel oped his "repair theory" to explain the errors that students make when they manufacture ways of getting around an impasse (VanLehn, 1983). The Soar cognitive theory (Laird et al., 1987; Newell, 1990) uses the con cept of "impasse" also, though it can have a somewhat different meaning from the definition just given. Soar has a num ber of architecturally de fined impasse types; impasses spawn subgoals to search a hierarchy of prob lem spaces for a means of resolving the impasse. Soar's learning mecha nism , called chunking, is based on this im passe-driven subgoaling. Chunking accounts for learning at two different levels in Soar: the symbol level and the knowledge level (Rosenbloom et al., 1987; Dietterich, 1986). Likewise, Soar impasses can be viewed as being caused by gaps in the prob lem solver's knowledge at either of two levels: the symbol level or the knowledge level. Knowledge level impasses in Soar most closely resemble the type of impasse described by Brown and VanLehn; they can be viewed as resulting from a lack of know ledge at the global level (i.e., the knowledge does not reside in any problem space.) Symbol level impasses, on the other hand, occur because of a lack of local knowledge, which results in a subgoal being form ed to search other problem spaces for the knowledge. The definition of impasse used in my research has a similar meaning to Brown and VanLehn's definition: an impasse is an obstacle to problem solving that results from either a lack of know ledge or from incorrect knowledge. Whereas Brown and VanLehn appear to limit their definition to the execution of a procedure, that is, impasses occur at the primitive ac tion level, my definition extends this notion to plans and goals also. In other words, just as there can be knowledge gaps at the action level, thereby creating an obstacle to taking the next step in a procedure, there can also be knowledge gaps concerning which procedure to execute and which goals need to be achieved. Impasses caused by a plan or goal knowledge gap, like an action level impasse, can be expected to result in situations where the problem solver either attempts to acquire new knowledge to cope with the 7 uncertainty of w hat to do next or else take an errorful or nonoptim al ac tion. The LMC cognitive model was developed in Soar to take advantage of the relationship betw een im passes and learning that exists in that architecture. Impasses in the LMC domain, as I have just defined them, are all realized as Soar impasses. This is consistent with the Soar view of a knowledge level impasse: it results from a lack of knowledge at a global level. Soar’ s symbol level impasses, which arise because the necessary knowledge is in another problem space, do not count as "impasses" in the sense that I have defined them. Hence, all impasses in the LMC cognitive model are realized as Soar impasses, but the reverse is not true. When a know ledge level im passe occurs in the cognitive model, a subgoal is created that contains methods for acquiring the necessary knowledge to resolve the impasse. Hence, the cognitive model suggests that impasses provide a natural place to tutor problem solvers since this is where they will have the goal to acquire the knowledge needed to correctly perform a task. This is the motivation behind impasse-driven tutoring, and this dissertation describes the first system to explicitly implem ent an im passe-driven approach to tutoring. To accomplish this, two problems had to be addressed. The first problem was to recognize when the student has reached an impasse. The second problem was to decide w hat to say to the student once the impasse has been detected. The process of recognizing an im passe can be som ew hat tricky. Impasses at the action level are often recognizable because the student makes an error or attempts to do something that the world will not permit, thus errors are cues that the student is at an action level impasse. Plan and goal impasses are somewhat more difficult to detect, however, since the symptoms may indicate only a "potential" rather than an actual impasse. The challenge for the tutor in the case of a potential impasse is twofold: First, it m ust be able to recognize potential impasses and interact with the student before it becomes too difficult to explain the impasse. The longer an impasse goes unresolved, the more likely it is that related impasses will arise as a consequence. The second challenge of the tutor is to interact with 8 the student in such a way that it will not disrupt the "learning by doing" that is occurring. If the impasse is potential then it is likely that the student is unaware that there is a problem and therefore does not have a goal to ac quire additional information or knowledge. If the tutor interrupts under these conditions, then it is not clear how damaging the break will be since the student's train of thought would be shifted from the problem to the tu tor. The potential advantage of a non-hum an intelligent tutor is that it may be possible to build subtle hints into the user interface that can serve as a less disruptive means of tutor-student interaction in these cases. Additionally, the tutor can also m anipulate the state of the sim ulator to force an impasse to which the student would likely attend. Once the decision has been m ade to interact with the student, the sec ond problem for the tutor is to explicate the impasse. The tutor m ust be able to explain the nature of the impasse in terms that will both enable the student to understand the source of the impasse and the how to avoid it or work around it in the future. In addition, the tutor m ust help the student to resolve the current impasse, which in m any task domains may involve more than re-applying an action. 1.4 An interactive task domain: Link M onitor and Control (LMC) system The task domain in which the tutorial interaction problem is addressed by this thesis is the Link Monitor and Control (LMC) system used in NASA's Deep Space Network (DSN). The DSN is a worldwide system for navigat ing, tracking and communicating w ith all of NASA's unm anned inter planetary spacecraft. As the name suggests, the LMC system is used by op erations personnel to m onitor and control a DSN communications link during an interaction with a spacecraft. A communications link is formed by assigning a collection of equipment to an LMC system for a particular mission; the link typically includes a large antenna dish (34 or 70 meters in diam eter), its electromechanical controllers, a receiver/exciter, a DSCC spectrum processor (DSP) subsystem, a precision power monitor (PPM), a hydrogen maser, frequency and timing subsystem (FTS), phase calibration generators, digital tone extractor (DTE), and numerous other devices. 9 Mission: VLBI " i . V i Plans: Configure-DSP Coherence-Test see Acqulre-Data see Playback-Data , 1 V V . 1 _ Commands: Load-Predicts Set-SAT-Value e • • Select-Recorder Figure 1.1 The VLBI task with a representative set of plans and operators The tasks perform ed by LMC Operators involve configuring and cali brating the communications link, testing the configuration for coherency, acquiring data from the spacecraft and playing back the data to the network operations control center. Each mission consists of a num ber of other simi lar tasks. The way to perform each task is described by a procedure1 man ual, which provides the Operator with command sequences for each of the subsystems. Figure 1.1 shows a portion of the task hierarchy for a VLBI2 mission. Note that the figure shows three basic levels in the hierarchy: mission, plan, and command. What Figure 1.1 does not convey is that the order in which the plans and commands are executed is not completely specified; though some partial orderings exist am ong procedures and commands, there is a lot of room for variability from one mission to the next (or from the way one Operator performs the tasks to the next.) To accomplish a VLBI mission, the Operator performs the plans that are shown in Figure 1.1: Configure-DSP, Coherence-Test, and so on. The Configure-DSP plan has operators3 to load the mission-specific prediction data file (Load-Predicts), set the attenuation values on the Interm ediate Frequency Video Down Converter (Set-SAT-Value), select a recording de vice to capture the mission data (Select-Recorder), and so on. Applying an operator involves issuing a command (e.g. the Load-Predicts directive is NLOAD predicts-file). 1 These written procedures are often referred to as plans. 2 VLBI stands for Very Long Baseline Interferometry. 3 An operator with a small "o" denotes a function that is selected and applied to a state to get a result. This is in keeping with the Soar notion of an operator (Laird et al., 1987). 10 It w ould appear that the tasks in the LMC domain are straightforward and would require little or no training-just follow the plans given in the m ission procedure manuals. In developing the LMC cognitive m odel, however, we found that this is not the approach taken by dom ain experts. Through extensive interviews with expert operators and system engineers, we determ ined that the procedure manuals only provide a subset of the knowledge needed to successfully perform the tasks associated with a mis sion. W hat is generally lacking in the procedure manuals is a complete de scription of the required device state conditions before and after a com m and is issued. Expert Operators possess a knowledge of the preconditions and postconditions for each command and verify these conditions are satis fied before and after issuing the commands. O perators who lack this knowledge may find it difficult to complete even simple plans, since the commands may be rejected, or worse, they m ay put the device into an in correct state for the current plan or mission. For example, one of the pre conditions for the Load-Predicts operator is that the predicts-file being loaded must be present on the system. If the Load-Predicts command is is sued for the predicts-file named "JK" (i.e., the Operator issues NLOAD JK), it will be rejected if a file by that name is not present in the predicts file di rectory. To complicate matters, during a two hour mission an Operator may in teract w ith five major subsystems, comprising fifty different devices, for which more than 250 unique attributes m ust be monitored and 500 event notice messages processed. The sheer quantity of monitor data accentuates the difficulty of executing the control procedures. Unexpected device state changes, device failures, and the slow reaction time of certain devices can cause procedural impasse and Operator confusion. If a command is issued when one of its preconditions is not satisfied, then it is likely to be rejected, or worse, it will put the device into an undesirable state. When Operators observe that a precondition is not satisfied, they have to know how to react. Consequently, procedurally-defined command sequences are not sufficient to accomplish m ost task goals. The plan acts as a guideline, b u t the Operator m ust bring other knowledge to bear on the performance of a task. 11 1.5 The REACT intelligent tutoring system An impasse-driven tutoring method has been implemented in REACT, an intelligent tutoring system for training Link M onitor and Control (LMC) system Operators. It is designed to support performance-oriented training, thereby helping the student to acquire the procedural skills by actually do ing the task as opposed to just reading about it. The Operator performs mission-related tasks on an LMC simulator, w here the sim ulator devices respond to the Operator's commands in the same w ay as the actual system. Consequently, the Operator cannot rotely execute procedures w ithout at tending to the state of the system. Commands are rejected when the simu lator devices are not in the right state, and mission goals may not be achieved if the commands are not given the correct param eters, which usually involves searching through system displays and other mission re sources for the desired values. The Operator m ust therefore attend to the state of the simulator devices during the execution of a procedure in order to react to the actual situation rather than assuming that everything is in an ideal state. The tutor observes the student's activities on the simulator, which con sists primarily of issuing command directives, and it observes the simula tor's responses, both in terms of whether or not the simulator accepted the directive, and if it did, the changes in device state caused by it. The REACT tutor may choose to interact w hen it recognizes that the student has reached an impasse of some sort, whether it is actual or potential. For this reason we call REACT an impasse-driven tutoring system; the tutor fo cuses on recognizing and explicating student impasses during the perfor mance of a task. The REACT tutor is implemented in Soar, an integrated problem solv ing and learning architecture (Laird et al., 1987; N ew ell, 1990; Rosenbloom&Newell, 1986). Problem solving in Soar involves searching for an operator to apply to a state in order to achieve a goal. When a Soar problem solver is unable to make progress toward a goal (i.e., it reaches an im passe), then the architecture supports the creation of a subgoal to achieve to continue the search. Subgoals lead to searching in other prob lem spaces, which form a problem space hierarchy. When a desired result 12 is achieved in a subgoal, Soar's chunking mechanism saves the results in new productions; these learned productions, or "chunks," save not only the results that were achieved in the subgoal, but they also save the condi tions that led to the subgoal's creation in the first place. Hence, the next tim e a similar problem solving situation arises, the chunk can fire, and thereby save the need to search in a subgoal’ s problem space for a solution. 1.6 An example of impasse-driven tutoring To illustrate the interaction between the student and REACT, consider an example where the student has been assigned a task on the LMC simulator with the REACT tutor. The next subsection describes the assigned task and the resources given to the student for performing it. This is followed by an example of how REACT interprets the student’ s performance of the task. 1.6.1 The task to perform The task stated below requires the student to issue a sequence of directives to the LMC simulator. In this case the student is required to execute the procedures for configuring the DSCC Spectrum Processing (DSP) Subsystem and performing the Coherence Test on the total configuration. The DSP is one of several subsystems that m ust be configured during the precalibra tion phase of a VLBI mission and Operators configure it by issuing direc tives containing param eters that m ust be gathered from a num ber of dif ferent sources. The Coherence Test, on the other hand, is a procedure that evaluates the continuity of the communications pathw ay from the an tenna receiver through the subsystems in the link, and it verifies that each local oscillator is phase coherent with the hydrogen maser. T ask: Perform the pre-calibration procedures for a Very Long Baseline Interferom etry (VLBI) mission. Specifically, configure the DSCC Spectrum Processing (DSP) Subsystem and perform the coherence test on the configuration. C o n d itio n s: Perform the task on the LMC simulator, using its command m enu window to issue the directives, the event log to monitor directive responses and event notice messages, and the system displays to evaluate the state of the sim ulator devices. The REACT tutor will monitor your progress and provide advice 13 when you make an error. You will also be provided a procedural summ ary sheet and a set of support data for the mission. G oals: Complete the procedures w ithout any directive rejec tions. Correctly set the values for all of the Configuration Table entries. Com plete the initialization of the N arrow Channel Bandw idth (NCB) recording program (abbreviated NCB-Pgm) and determ ine whether the configuration is coherent. C o n flg u re-D S P C o h e re n c e -T e s t V NLOAD <x> V NPCG <x> V NRMED <x> V NRUN <x> V SAT oc> V N D TEoo V X A Too V N FFToo V NTOP <x1> <x2> V OFST <x> Figure 1.2 Procedural summary sheet for the assigned task The sum m ary sheet for the tw o procedures, Configure-DSP and Coherence-Test, is shown in Figure 1.2. It gives a sequence of commands that will, under ideal circumstances, achieve the goals of each procedure. Each directive requires the Operator to supply one or more param eters, which m ust be determ ined from the resources provided (i.e., from system displays, support data, etc.). The procedural summaries do not include the directives that activate the various system displays, of which there are m any. 1.6.2 Tutoring with REACT Figure 1.3 shows a student performing the assigned task and REACT's interpretation when it detects impasses. The items shown in the Event Log column appear the same as they would in the actual LMC Event Log. The Event Log column lists the directives and their param eters in the exact se quence they were sent by the Operator. In the row following each directive there is a response message that was issued by the subsystem. The response indicates whether the directive was accepted or rejected. If it is accepted, the system responds with a COMPLETED message, which does not mean that it has been completely executed, only that the directive will be pro cessed by the system. If the directive is not accepted a REJECTED message 14 # Type Event Log REACT's Explanation 1 OD > V NLOAD JK 2 DR > COMPLETED. 3 OD > V NRMED L D O 4 DR > REJECTED. LDO DISABLED The NRMED command failed because one of its preconditions was unsatisfied: LDO should be in the ONLINE mode instead of the OFFLINE mode. Issue the command: V LDO E then Issue the command: VNRMED LDO 5 OD > V LDO E 6 DR > COMPLETED. LD O : ONLN 7 OD > V SAT 55 8 DR > COMPLETED. 9 OD > VXAT13 10 DR > COMPLETED. 11 OD > V NTOP 20.0 30.0 12 DR > COMPLETED. 13 OD > V NPCG M AN 14 DR > COMPLETED. You started the Coherence-Test procedure be fore you finished the Configure-DSP procedure. Issue the Command: V OFST o 15 OD > VOFST2.7 16 DR > COMPLETED. 17 OD > V NRUN COLD 18 DR > COMPLETED. You failed to achieve one of the goals of the Configure-DSP procedure: SAT = 12. Issue the command: V NIDLE REC then Issue the command: V SAT 12 19 OD > V NDTE E 20 DR > COMPLETED. 21 OD > V NFFT E 22 DR > COMPLETED. Legend: OD = Operator Directive, DR = Directive Response Figure 1.3 An example of tutoring with REACT 15 and short explanation will follow. There is normally a time delay associ ated with executing a directive; once a directive has been executed an indi cation will be given by an event notice message or a system display. In the example, REACT m ade three interpretations of the student's per formance, at lines 4,14, and 18. In each case REACT detected a student im passe before the interaction. These three explanations happen to be in re sponse to impasses in each of categories recognizable by the REACT tutor. For instance, the impasse on line 4 was immediately recognizable as an ac- tion-constraint violation because the Operator's directive was rejected. The impasses on lines 14 and 18 were more subtle since the LMC sim ulator did not give any indication of a problem. The explanation on line 14 resulted from the tutor recognizing that the tw o procedures, Configure-DSP and Coherence-Test, w ere being inter leaved when it expected one procedure to follow the other. This is known as a plan dependency violation, and it teaches the Operator about the inter dependencies that are defined among procedures. The final explanation was triggered by the fact that REACT recognized that the O perator had completed the Configure-DSP procedure w ithout achieving one of its goals, namely, that the SAT (S-Band Attenuation) should be set to the proper value. This category is known as a goal failure impasse, and it represents a potential rather than an actual impasse situa tion since it is likely that the student is not even aware of a problem. In the instance on line 18, the student would have been able to complete the mis sion w ithout evidence that there was a problem, since it w ould not have appeared until much later when recorded data was being processed. For this reason, REACT explained the impasse as near to its source as possible, making available tutorial input that could help the student to recognize an error long before it might otherwise manifest itself. It is evident that REACT not only recognizes impasses but also expli cates them. Note that each of the interpretations in Figure 1.3 contains not only an explanation of the impasse and what caused it, but it also gives the student a way of resolving it. For example, at line 4 REACT tells the stu dent to first send the LDO E directive and then the NRMED LDO directive. This impasse solution was dynamically derived by REACT with respect to 16 the current state of the devices; this avoids having to store canned solu tions for each type of error that the student will potentially make. At line 16 REACT explicates the plan dependency violation by pointing out which plan had not been completed and then it lists the missing direc tive. In this case REACT does not param eterize the directive for the stu dent since this is not the part of the task that the student failed; the failure was in recognizing the interdependency among plans and the explanation is kept at this level. In later sections I will discuss w hat happens when an impasse can be recognized as more than one type and how these conflicts are resolved. For instance, there are cases where a student's directive may violate both a plan dependency and an action constraint. Finally, the goal failure impasse at line 18 was explicated by first point ing out which goal had not been achieved. As was the case with the action constraint impasse at line 4, REACT derived a solution for the impasse. As before, it is necessary to issue a directive that is not part of the regular pro cedure: NIDLE REC. This directive changes the state of the NCB-Program from the RUN m ode to the IDLE mode before issuing the OFST command. N ote that the impasse that is being tutored was not caused by the state of the NCB-Program. Indeed, the SAT directive was not rejected. So why does REACT decide to issue the NIDLE REC com mand prior to re-issuing the SAT command? The answer to this question has to do with the m an ner in which REACT interpreted the student's behavior in this case. Note that REACT did not explicate the impasse until line 18, even though it could have done so at line 8, where the error initially occurred. In goal failure impasse situations, REACT delays its explication until after it ap pears that the student has not noticed the error and has moved on to an other procedure. Hence, REACT recognized the goal failure impasse before line 18, but delayed the explication until after the student issued the V NRUN COLD command, which is part of a different procedure than the one whose goal was not satisfied. Because of this delay, REACT has to take into account the effects actions (e.g., NRUN COLD) that were taken before the interaction was initiated. The NRUN command happens to change the m ode of the NCB-Program to RUN, which violates one of the precondi 17 tions of the SAT command, hence, REACT derived a solution that took this into account. 1.7 REACTs approach to tutoring Im passe-driven tutoring is achieved in the following m anner. REACT recognizes im passes w ith a m ethod called situated plan attribution (Hill&Johnson, 1993b), which uses plan descriptions and a variety of other resources to recognize impasses. The term plan attribution is used rather than plan recognition because it does not assume that the student's actions are controlled by plans. Instead, this approach takes a situated action view of plans, which treats plans as resources that guide and orient action, but where action is ultimately contingent on the state of the w orld (Suchman, 1987). The REACT tutor attributes plans to the student both by evaluating the current state of the task and w hat plans it expects the student to be exe cuting, as well as by observing the actions taken by the student and compar ing it to its expectations. REACT explicates impasses using an expert cognitive model. Impasse explication consists of understanding and explaining the impasse as well as determ ining a way to resolve it. REACT generates an explanation of the impasse in the process of resolving it. The resources used by REACT's ex pert cognitive model includes the Event Log, which records the interaction between the Operator and the LMC devices by keeping track of the sequence of directives and responses; state information from the sim ulator devices; and the attributed plans and goals associated with the impasse. The expert cognitive model is an integrated part of REACT; it is im plem ented as a hierarchy of problem spaces that can be searched once an impasse has been identified. 1.8 Related w ork REACT is related to three bodies of research and the purpose of this section is to show how REACT contributes to each of these areas. I call the first body of research tutoring im plem entation m ethods, and I com pare REACT's tutoring approach to systems that are based on model tracing, plan recognition and overlay-based models. The second body of research is 18 cognitive m odeling and theories of skill acquisition, w here I relate the m odeling work that guided the design of REACT to other such efforts. Finally, the third body of work concerns other ITS developm ent that has been done for reactive task domains similar to the LMC domain addressed by REACT. 1.8.1 Tutoring implementation methods The com parison betw een REACT and system s using these other ap proaches is done on the basis of how the tutorial interaction problem is ad dressed in each. The tutorial interaction problem concerns how the tutor decides that interaction is necessary, and once the interaction is initiated, what it says. 1.8.1.1 Model tracing approaches to intelligent tutoring The m odel tracing approach to tutoring uses an executable performance model to search for a production rule path that produces the same behav ior as the student's observed behavior (Anderson et al., 1990). This ap proach has been used in num erous intelligent tutoring systems: e.g., the Lisp Tutor (Reiser, 1985; A nderson et al., 1990), the Geom etry Tutor (Anderson et al., 1990) and the Algebra Tutor (Anderson et al., 1990). A variation of model tracing is differential m odeling (Wilkins et al., 1988), where the student's decisions are com pared to those produced by an exe cutable expert model. In the model tracing approach, student actions are modeled in terms of the productions that produce them, consequently the performance model ends up being highly detailed. The problem w ith this approach is that the performance model is, by necessity, non-deterministic, m aking the inter pretation of student action very complex at times since there may be many interpretations for a single action (Anderson, 1988; Anderson et al., 1990). This becomes even more difficult when the tutor has to model a lot of in termediate mental states, such as in W ard's work (Ward, 1990), where the number of possible paths may become very large. The functional purpose of the performance model is to give an account of the student's behavior so that the tutor can decide when to interact. 19 REACT accomplishes this purpose while rejecting the use of a production- based performance model to do the interpretation. Instead of trying to ac count for the student's behavior via a cognitive simulation, REACT m od els only as much detail of the task as is necessary to give a reasonable inter pretation. It focuses on impasse detection via situated plan attribution, for which it uses knowledge about action, plans, goals and the current situa tion to determ ine whether the student is having difficulty. 1.8.1.2 Plan recognition approaches to intelligent tutoring Whereas model tracing uses an executable performance model to interpret student behavior, plan recognition approaches to tutoring attem pt to un derstand behavior by m atching the student's actions w ith a declarative plan structure or description of action. One form of plan recognition in volves matching the observed actions to a library of plans and selecting the best interpretation (Genesereth, 1982; Johnson, 1986,1990; Calistri, 1990; Kautz, 1986). A variation of this approach interprets student actions with a procedure net or action gram m ar (Burton, 1982; Rickel, 1988; W arriner et al., 1990; Chen, 1991). REACT uses a form of plan recognition in that it matches observed actions to plans, but it takes a more situated view of ac tion than do some of these other approaches, allowing for deviations from the standard plans in response to the situation. In the plan m atching approach, errors are detected with mal-plans or difference rules and tutoring is done after the problem solving session has been completed (Johnson, 1986, 1990). This contrasts with REACT, whose goal is to detect impasses as soon as they occur so that it can make available tutorial input as close as possible to the impasse point. On the other hand, the action grammar approach detects errors in one of two ways. One of the methods treats actions that cannot be parsed by the "grammar" as errors (Rickel, 1988; Warriner et al., 1990; Chen, 1991), while another m ethod uses a buggy model of a procedure net representation of the task to detect errors (Burton, 1982). Though these can be efficient ways of detecting errors, thereby allowing the tutor to interact close to an im passe point, this approach is limited by the necessity to declaratively repre sent all of the contextual variations that might affect a student’ s actions. In 20 other words, it does not appear to be a good fit for environments requiring a lot of situated action, where the student may have to deviate from a plan in order to react to the situation in the world. In contrast, REACT assumes that a student's behavior is going to be both plan-like and reactive in an attem pt to achieve goals in the world. The net effect of this approach is to allow more flexibility in interpreting student actions, allowing the student to react to situations in ways that do not fit a standard plan approach to performing the task w ithout penalizing them for doing so. It also means that some impasses are going to occur as a result of situations that dem and a deviation from the standard plan or pro cedure, but the student doesn't know w hat to do or doesn't recognize the problem. The plan recognition approaches described above can only cope with these scenarios when all of the possibilities have been pre-elaborated, whereas REACT handles them in a dynamic fashion. 1.8.1.3 Overlay-based models An overlay is a declarative data structure that is used for student modeling (VanLehn, 1988; Clancey, 1986). It typically contains a target set of skills and knowledge that the student should acquire during training, and they are m arked as such as the student is observed to have mastered each item. At any given time the model of the student is a proper subset of the overlay, which is used to show the gaps in the student's knowledge. (Anderson et al., 1990) refers to the process of updating and using the overlay knowledge tracing. The use of an overlay for m odeling the student is a nearly universal technique since it can be used by the tutor to make choices about the cur riculum or about which diagnostic hypothesis to choose w hen there is more than one interpretation of the student's performance. Moreover, it is possible to use the overlay to guide the tutoring w ithout having to use a technique like model tracing or plan recognition to interpret the student's actions. Instead, the student’ s actions may only be judged at an atomic level w ith respect to a very specific action associated with a skill. The rea son that I mention this approach is that a fair num ber of training systems avoid the difficulty of diagnosing complex procedures by decom posing 21 them into their part-tasks and testing them at a very low level. The as sum ption is that the sum of the part-task skills is equal to the composite skill. While there is evidence that this approach to training can be effective (e.g., see Frederiksen&White, 1989; W hite&Frederiksen, 1990), it does not negate the usefulness of whole task training and tutoring at some point in the student's development. The LMC cognitive model predicts that students will learn best by per forming the task in its goal and situation context, therefore at some point the student needs to perform the task as a whole. REACT can support part- task training— the detection and explication of impasses should be easier in part-task training than in whole task training— but, REACT is designed to be used when all of the task elements are present and being exercised together. The reason for the approach taken in REACT is because the cognitive model predicts that tutoring will be m ost effective when it is given in the context of an impasse, and it is expected that some impasses will occur where knowledge is being integrated at the plan and goal levels, which is where a part-task approach may not suffice. 1.8.2 Cognitive m odeling and theories of tutoring The situated plan attribution methodology used in REACT was developed from the lessons that w ere learned by developing a detailed cognitive model of the LMC domain in Soar (Hill&Johnson, 1993a). Soar is a theory of hum an cognition, and much of what we learned from building the cog nitive model arose from the Soar architecture. One of the most prominent examples of another intelligent tutoring m ethodology that is based on a theory of hum an cognition is model tracing, which was previously dis cussed. The model tracing paradigm has its roots in ACT* and its succes sor, PUPS (Anderson, 1982; 1983; 1989; Anderson et al., 1990). The relation ship between model tracing and ACT*/PUPS is analogous to the relation ship between the situated plan attribution and Soar. There are some high- level similarities between ACT*/PUPS and Soar: both use a production model of long-term memory, though the ACT*/PUPS model has a separate declarative memory; both theories include a concept of a working mem ory; and both have models of learning, though they account for it some 22 w hat differently. The main point of this discussion, however, is that both model tracing and situated plan attribution have been informed by explicit, im plem ented theories of hum an cognition. Cognitive modeling techniques have been used to build process models that sim ulate hum an behavior in the subtraction dom ain by a num ber of different researchers. Langley and Ohlsson developed a cognitive diagnosis technique for explaining subtraction errors (Langley& Ohlsson, 1984; Langley et al., 1990; Ohlsson&Langley, 1988). They expanded on Newell an d S im on's p ro b lem -sp ace ap p ro ach to co g n itiv e d iag n o sis (Newell&Simon, 1972) to search for solution paths that w ould produce the observed student errors, and then used machine learning to hypothesize the specific productions that would lead to this solution path. Their work has a principle in common w ith the cognitive m odeling behind REACT: both assume that errors arise from misapplying learned cognitive operators (Ohlsson&Langley, 1988). On the other hand, Langley and Ohlsson's work differs from REACT's cognitive model in that their approach focuses only on how to diagnose misconceptions; unlike the REACT cognitive model, their w ork does not explain how the students acquire skill that will im prove their behavior. Kurt VanLehn's SIERRA system implemented a theory of skill acquisi tion in the subtraction domain (VanLehn, 1982; VanLehn, 1983; VanLehn, 1987). The goals of this w ork were to explain student errors, for which Brown and VanLehn had already developed their rep air theory (Brown&VanLehn, 1980), and to determine the best conditions for acquir ing knowledge through inductive learning of subprocedures. Repair the ory basically says that student errors result from creatively mis-applying procedures or rules at impasse points, i.e., at places in the problem where the student doesn't know w hat to do next (VanLehn, 1983). This work also inspired VanLehn's theory of im passe-based learning (VanLehn, 1990), which has many of the same motivations as REACT's impasse-driven style of tutorial interaction. The notion of impasse-based learning is key to hier archical problem solving and learning in the Soar architecture (Rosenbloom&Newell, 1986). Furtherm ore, the cognitive m odel that we 23 built for the LMC domain could only acquire new skills after it noticed and attended to a problem solving impasse. The LMC cognitive m odel differs from VanLehn's w ork in that SIERRA acquires knowledge about procedures inductively from a teacher's examples, while learning in the LMC cognitive m odel depends on the Soar chunking mechanism and on the acquisition of knowledge about op erator preconditions. According to the LMC model, these preconditions are initially added to long-term memory as declarative statem ents provided via tutoring (not by example), though the process of transform ing this declarative data into the form of a production is sim ulated in the model. There are currently a number of other research efforts that are looking into the issue of concept learning in Soar (Rosenbloom et al., 1987; Rosenbloom et al., 1988). Another difference between the LMC model and VanLehn’ s work has to do with the nature of problem solving in the two domains. Others have built, or are building, detailed cognitive models similar to the LMC cognitive model. Blake W ard developed a cognitive m odel in Soar of a student solving electrostatics problems (Ward, 1991). It carefully models many details of the problem solving process, such as perception and focus of attention; however, it does not learn. Ralph Morelli4 is work ing to extend W ard's work, and his model does learn. However, it does not address questions of how misconceptions are acquired or recovered from, nor how tutorial interaction m ight influence the learning process. Conati and Lehman (Conati&Lehman, 1993) are modeling problem solving and learning in a microworld learning environm ent called "Electric Field Hockey." It can recall failed problem solving attem pts, in order to avoid previous mistakes. However, this dom ain is m ore lim ited than the do main that we are concerned with. It does not involve interaction w ith a complex environm ent during an extended problem solving episode. The learning issues that their work addresses are largely orthogonal to the is sues addressed by this work; the problem solver in that domain learns se quences of operations to perform to achieve a goal, whereas in our domain the focus is on learning the conditions under which operations are appro 4 Trinity College, Hartford, CT, work in progress 24 priate to be perform ed and the effects that they should be expected to achieve. 1.8.3 Real-time task dom ains Though I have been calling the LMC an interactive task domain, it bears a close resemblance to task domains that have been characterized as "high- performance" (Regian, 1991; Fink, 1991) and "real-time" (Ritter&Feurzeig, 1988). Beverly Woolf (Woolf, 1993) characterizes real-time domains as re quiring adaptive and highly dynamic behavior, and w here the tasks m ust be perform ed w ithin deadlines. Events in the dom ain are unpredictable: the effects of actions are often delayed and the state of the world can change unexpectedly. Hence, real-tim e task domains appear to have all of the characteristics of an interactive task domain; the m ain differences are in the degree to which time plays a factor in performing the task. Examples of high perform ance and real-time tasks include: Regian's INFL1TE training system, which is concerned with landing a jet using only the flight instrum ents (Regian, 1991); Ritter and Feurzeig's Trainer for Radar Intercept Officers (TRIO), which trains basic tactics of high-speed air intercepts to F-14 pilots and radar officers (Ritter&Feurzeig, 1988); Fink’s mission control console training for NASA's Johnson Space Center (Fink, 1991); and the P2T2 Intelligent Trainer used for training operators how to use the robot arm on NASA's space shuttle (W arriner et al., 1990; Chen&Barbee, 1990) There are varying degrees of reactivity needed for real tim e domains, depending on how tight the task deadlines are as well as how unpredictable the environm ent is. INFLITE provides drill and practice. Student diagnosis is done mostly using the overlay m ethod along with an evaluation of skill with a set of metrics. N o model tracing or plan recognition used here. This domain re quires a high degree of automaticity compared to the LMC domain, where plan summ aries are used and information m ust be sought after in the do main prior to taking an action. TRIO makes use of daem ons to monitor task-critical parameters in order to detect events that could cause a mission failure. These daemons are used to invoke helpful messages for the stu dent, which appear to serve m uch the same purpose as impasse detection 25 by REACT. The difference in this case is that the daemons are triggered by critical events, which is one form of im passe recognizable by REACT. According to (Ritter&Feurzeig, 1988), only a limited num ber of daemons can be active at any given time, due to the relative computing cost of the daemons on the com puting resources. It is not clear to w hat extent this limitation would be reduced by simply using a more powerful com puter or w hether the limitation is inherent to the im plem entation of the match al gorithm for the daemons themselves. In contrast, REACT recognizes im passes with Soar productions, which do not have the same kind of compu tational limitations; (Doorenbos, 1993) reports on an im plem entation of the Rete m atching algorithm used in Soar that w ould allow systems of over 100,000 rules w ithout a significant perform ance degradation. One other difference from REACT is that the prim ary tutoring in TRIO is done during a post-session debriefing. The intensity of the task makes in-session tutoring disruptive. The Mission Control Operations Console trainer uses a form of a proce dure net, or directed graph task description to evaluate the student’ s per formance. The P2T2 Intelligent Trainer also uses procedure nets to repre sent task skills and evaluate student performance (Warinner, 1990). 1.9 Key contributions of this research The contributions made by this research are summ arized below. • A theory of tutorial interaction in interactive domains. This research addresses the question of deciding when and how to interact with a student who is perform ing a task in an interactive domain. It hypothesizes that it is best to interact with the student at an impasse point, where an impasse is an inability to make progress because of a lack of knowledge. Three im passe categories are identified: action-constraint, plan dependency, and goal failure. The reasons for interacting with the student at impasse points are given by a computational cognitive model that was constructed to ex plain problem solving and skill acquisition in an interactive domain. The model predicts that students have the goal to acquire new knowledge at an impasse point; it also predicts that it is best to acquire new knowledge in 26 the context of the goals where it will be used. These two predictions can conflict when the tutor detects that the student has reached a potential im passe and is not aware of it. In these cases the tutor m ust assist the student in having the goal to acquire to new knowledge. Once an impasse has been detected by the tutor, the theory makes predictions about the knowledge that should be communicated to the student, which depends on the type and context of the impasse. Beyond m aking hypotheses about how to tutor students in interactive task domains, the theory was implemented in an actual tutor, REACT, and a pilot study that evaluated the hypotheses was conducted. The results of the pilot study, which was a prototype summative evaluation, indicate that the theory may be valid. • Situated Plan Attribution: a new approach to student modeling. To im plem ent the theory of tutorial interaction required designing a tutor that could recognize when the student was at an impasse. Situated plan attribu tion im plem ents an abstract form of m odel tracing; it resembles plan recognition in that uses declarative plan and goal structures to keep track of w hat the student has done and to make predictions about w hat the student should be doing, but it avoids the use of mal-plans to explain aberrant be havior. The purpose of situated plan attribution is to detect when the stu dent deviates from the behavior predicted by the model; a deviation from the model is categorized as one of the three impasse types and the task of explicating the impasse is given to the expert cognitive model. This ap proach combines features of both model tracing and plan recognition while avoiding some of the difficulties associated w ith each of these techniques, and its use is not limited to a particular task domain. • Cognitive modeling: a methodology for designing tutoring systems. The theory of tutorial interaction that was implemented in REACT is based on a cognitive model of problem solving and skill acquisition in an interac tive domain; the cognitive model accounts for the behaviors of novice and expert Operators and it also gives an explanation of how a novice acquires expertise. The cognitive model m ade a num ber of hypotheses about skill 27 acquisition that were used in guiding the design of a tutor for an interac tive dom ain. By sim ulating the student/O perator, the cognitive m odel suggested ways that learning could be enhanced, given processes involved in perform ing tasks in the task domain. The underlying methodology that is suggested by this research is that constructing an executable cognitive model can inform the design of an intelligent tutor. In this research the cognitive model suggested a num ber of hypotheses about skill acquisition that had a direct bearing on answering the tutorial interaction question. These hypotheses, in turn, influenced the design requirements of the tutor, and they were implemented in REACT. 1.10 A guide to the dissertation The dissertation is organized in the following manner. Chapter 2 gives the reader a composite picture of the LMC Training System and the REACT tu tor. This chapter begins by describing the training system's architecture and its functional components; the approach taken in REACT is com pared to the traditional m odel of an intelligent tutoring system. W ith this back ground, the rem ainder of the chapter gives more a description of how the student interacts with the training system. Chapter 3 serves two purposes: it characterizes the difficulties of prob lem solving and skill acquisition in the LMC domain and it dem onstrates how REACT helps to overcome these obstacles. These purposes are served through the use of some detailed problem solving examples: task perfor mance in an ideal w orld, task perform ance in the real w orld, and how REACT tutors students in the real world. Chapter 4 describes a detailed computational model of skilled behavior and learning in the LMC task domain. The purpose of the model was to enable us to make predictions about the effectiveness of various tutoring and curriculum design choices. The model covers the problem solving be haviors of both novices and experts, and it also demonstrates the skill ac quisition process. It is capable of improving its performance over time, ac quiring new knowledge and recovering from incorrect knowledge. Chapter 5 gives the details of how the REACT tutor recognizes impasses using a m ethod called situated plan attribution. 28 Chapter 6 describes how REACT explicates impasses. Chapter 7 describes the results of evaluating the tutor w ith a pilot study. Chapter 8 presents the conclusions and the directions that the future work will take. 29 Chapter 2 REACT as an ITS Architecture 2.1 Introduction This chapter gives a functional overview of the LMC Training System, which com prises a graphical user interface, an LMC sim ulator and the REACT tutor. The overview is organized in the following manner. First, I describe the functional components of the training system: w hat they do, how they are connected, what data flows among them and how control is exercised. As a part of the overview, the training system architecture is analyzed from tw o perspectives. First, in the course of describing the training environm ent I compare it to the actual LMC system. The purpose of this comparison is to establish which elements of the training system are m eant to resem ble the actual LMC, and w hich ones are unique to the training system. The second analytical perspective compares the LMC training system architecture to the traditional intelligent tutoring system (ITS) model that has frequently been described in the literature. Though the traditional ITS com ponents are identifiable in REACT, their im plem entation in Soar leads to some interesting architectural results once the tutor has begun to "chunk" its ability to recognize and resolve impasses. Whereas there is a growing em phasis on m odularizing the ITS components into separate ob jects th a t m u st com m unicate th e ir resu lts w ith one an o th er (W arren&Goodman, 1993), the REACT architecture tightly couples these functions by implementing them in a single Soar problem space hierarchy. Though these functions are im plem ented in separate problem spaces, thereby m aintaining their distinct identities, once chunks are built for these problem spaces, the tutor's knowledge is compiled into a less m odu lar but highly efficient form. This is an im portant feature for impasse- driven tutoring, since it is critical for the tutor to be able to intervene while the student is at the impasse point, but it runs counter to the conventional notion of how an intelligent tutoring system should be configured. 30 2.2 LMC training system architecture The LMC Training System has three functionally distinct components: a graphical user interface, an LMC sim ulator and the REACT tutor (see Figure 2.1). I describe each of these components in detail in the following subsections. User Interface LMC Directive Menu LMC Event Log LMC System Displays REACT Tutor Window Task Progress Display Directive Device Display Data Direct ive&Response Impasse Tutoring Task Progress Data LMC Simulator Command Processor Device State Models Output Functions Device State Data Directive&Response REACT Tutor Impasse Recognizer Expert Cognitive Model Figure 2.1 LMC training system architecture 2.2.1 Graphical user interface The student interacts with the LMC training system using a graphical user interface, which is im plem ented in Motif. The w indow s through which these interactions take place fall into five categories: (1) the LMC Directive M enu, (2) the LMC Event Log, (3) the LMC System Displays, (4) the REACT Tutoring window, and (5) a set of Task Progress Displays. Note that the first three items in this list directly correspond to existing LMC user interface components. The last two items exist so that the tutor can communicate w ith the student. 2.2.1.1 LMC directive m enu The LMC Directive Menu w indow is used by the student to take control actions on the LMC simulator (see Figure 2.2). This w indow functionally corresponds to the command input window in the actual LMC console, but there are some differences between the two. The actual system requires the 31 O perator to type in a param eterized directive using only a keyboard and a procedure manual. Subsystem: OptionLabel I VLBI — < D ire c tiv e s : V N DTE <v> V NFFT <v> V NIDLE R E C V N L M T X <vl> <v2> <v3> V N L M T S <vl> <v2> <v3> V N L O A D <v> V N P C G <v> V N P C G C <vl> <v2> '■! M RtlEIl i'v : V N R U N <v> V NSKIP <v> V N T O P <s><x> V OFST <v> V P M E D <v> V SAT <sat> V XAT <xat> S electio n V N R M E D L D C j[ Send J C lear J Help Figure 2.2 LMC directive menu: VLBI subsystem directives shown The training system, on the other hand, provides the Operator w ith the ability to select and issue directives from an alphabetically ordered menu. Once selected, the directive appears in command selection window, where the student types in a directive's param eters. W hen satisfied w ith the param eterized form of the directive, the student clicks on the "Send" button, and the command is issued to the simulator. Thus, the only typing that is required in the training system involves providing param eter values for the selected directive. These features eliminate som e of the typographical errors that occur in the actual system. 32 t m f 2.2.1.2 LMC event log The Event Log W indow (Figure 2.3) in the LMC Training System func tionally corresponds to the actual LMC Event Log. The Event Log is a vi sual record of the directives that have been issued from the LMC to the various subsystem s in the link. The directives are labeled in the first column with an "OD", which stands for "Operator Directive". The log also shows directive responses and event notice messages from each of the subsystem s. A directive response, which is labeled w ith a "DR" for "Directive Response", is sent by a subsystem to the Event Log to indicate that the directive was received and also to tell w hether it was accepted or rejected. If a command is accepted, a COMPLETED message is sent; and if it is not accepted a REJECTED message will appear. Event notice messages are sent by the subsystems to indicate significant device state changes and warnings. There are num erous categories of event notice message: emer gency alarm (E!), critical alarm (C!), w arning alarm (W!), deviation advi sory (DA), completion advisory (CA), progress advisory (PA) and others. An example of an event notice message is shown in Figure 2.3: "CA V SET JK LOADED OK". This is a completion advisory that indicates that the predicts set was successfully loaded. O D V L O A D J K D R V REJECTED. N O T A L L O U E D IN N R U N N O D E O D V NIDLE R E C D R V C O M P L E T E D . NIDLE R E C INITIATED. O D V L O A D JK D R V C O M P L E T E D . L O A D O F N C B PREDICTS S T A R T E D . . . S E E N E X T C A V S ET J K L O A D E D OK. O D V N R M E D L D O D R V C O M P L E T E D . N R M E D : L D O p: Figure 2.3 LMC training system event log 33 N C H N Dwl SYNTHF(MHz) LO(HHz) R F (MHz) 1 ? 0.4 305.420013 2000.0000 2300.00000 3 4 5 R ♦ ♦ ♦ ♦ ♦ ♦ • ♦ 1.8 340.420013 8100.0000 8400.54000 7 8 9 10 1 1 12 N L O A D : JK SUBSET: O N E O N E W A Y P O C A N O D E : pnode.text FR E Q : pfreq_text N P C G (MODE): N O N E DESIRED: SPur 1.00 Z (PER T O N E ) XPwr 1.00 Z (PER TO N E ) SAT/XAT (atten uatio n): -1 -1 S X C M B (combiner) N O D E : sxcnb NFFT: DISAB Spacecraft: sc_text Year: 1993 N D S S : D S S 14 N B I4 : 250.000 K H z OFFS: 0.0 U S FTS: fts N C Q M B ; 100.00000000 KHZ(5/0) N T E H P : S 0.0 X 0.0 DTEV: NO TE : N R M E D : PH E D : N E W DISAB L D O N O N E (P C G Tone Monitor) B E S T (o f PBD): dest UDT: DRIFT JTR H A G NLHTS: 0.035 1.00 0.007 N LM TX: 0.039 1.00 0.007 udt Figure 2.4 A example system display: NCB Configuration Table (NCNF) 2.2.1.3 LMC system displays To m onitor an actual link in the LMC system, the Operator accesses device data through a variety of different standard system displays that can be shown on the LMC console terminals. There are dozens of displays from which to choose, and there is a lim ited am ount of screen space, so the Operator often has to shuffle the displays in order to track im portant device states. The LMC Training System currently only represents a small subset of the displays available to the actual LMC. The training system 's displays closely follow the format and representation of the actual system's displays. For example, Figure 2.4 shows the NCNF1 display, w hich contains the NCB2 configuration table used by the DSP3 subsystem. This is one of the prim ary displays used by the O perator during the DSP configuration procedures, which are described later. Just as in the actual system, the table values in the display change from red to green as they are set by the Operator. N ote that the change in color indicates that a value has been 1 NCNF stands for NCB Configuration Table. 2 NCB stands for Narrow Channel Bandwidth. 3 DSP stands for Deep Space Communications Complex (DSCC) Spectrum Processing. 34 changed by the O perator, but it does not indicate that the value will achieve the goals of the mission. The O perator is responsible for accom plishing the mission goals and merely uses the display to evaluate the state of the devices. 1 f Figure 2.5 REACT tutor window 2.2.1.4 REACT tutor w indow All of the user interface features described thus far correspond to an exist ing feature in the actual LMC system. The REACT Tutor window (Figure 2.5) is unique to the training system; it is the prim ary means for the REACT tutor to communicate w ith the student. The REACT tutor sends messages to a scrollable text display area in the w indow whenever it de cides to intervene as a result of recognizing and explicating an impasse. The message shown in Figure 2.5, was generated immediately after the first directive was sent by the student, which is shown in the event log window in Figure 2.3. The REACT tutor is explaining to the student why the NLOAD directive was rejected by the LMC simulator. In this case the directive was rejected because the device nam ed NCB-Pgm has an attribute nam ed MODE that should have had a value of NIDLE, but instead has a 35 Q uit Displays Windows Help Level: OptionLabel I Beginner Task S election: OptionLabel | g LL D elta D O R — C O M M A N D N L O A D FAILED B E C A U S E O F A N UNSATISFIED PRECONDITION DEVICE: N C B -P G M ATTRIBUTE: M O D E C O R R E C T VALUE: NIDLE A C T U A L VALUE: N R U N C O R R E C T C O M M A N D : V NIDLE R E C C O R R E C T C O M M A N D : V N L O A D J K value of NRUN. The device, attribute, value inform ation used in the ex planation all corresponds to data items that can be found in system dis plays. After explaining the cause of the failure, REACT explains how to re solve the current impasse: first issue the V NIDLE REC com mand and then the V NLOAD JK command. The V NIDLE REC com m and puts the NCB-Pgm into the desired NIDLE MODE; and V NLOAD JK accomplishes the original goal of loading the predicts file. 2.3 LMC sim ulator The LMC simulator models the state of the devices in the communications link. It is designed to react to Operator directives in the same m anner as the actual system . There are three functional com ponents in the sim ulator: (1) the device models, which are a collection of objects that m aintain the state of the link devices, (2) a com m and processor, which parses Operator directives and causes device states changes to occur, and (3) a set of output functions, which generate the messages to the user about the state of the devices. The simulator is im plemented in 'C', and each of its components are described in more detail in the following subsections. D evice T ype D evice In stan ce A ttribute V alue NCB-PROGRAM NCB-PROGRAM MODE NIDLE R EC O R D E R LDO M ODE ENABLED VLBI-PREDICTS JK TYPE NCB Figure 2.6: Simulator device models 2.3.1 Device m odels Each device is m odeled as an object having a unique name, a type, and a collection of attribute-value pairs that are used to represent the device's state. For example, Figure 2.6 shows a portion of the state information for three devices: the NCB-Program4 the VLBI predicts set nam ed JK, and the recording device nam ed LDO. The NCB-Program is a unique device, so its nam e is the same as its type; it has an attribute nam ed MODE w ith a value 4 NCB-Pgm stands for Narrow Channel Bandwidth Program, which is the program used for running the DSP while conducting a VLBI mission. 36 of NIDLE.5 The device named LDO is one of several possible data recorders used during VLBI missions. It also has an attribute nam ed MODE, and its value is ENABLED. The device nam ed JK, like the data recorder, is one of m any possible VLBI predicts set names. It has an attribute nam ed TYPE w hose value is NCB, m eaning that the predicts are m eant for use in a narrow channel bandw idth m ission as opposed to a w ide channel bandw idth mission. The Operator has access to device model state data via messages sent to the Event Log (Figure 2.3) from the sim ulator and through subsystem dis plays such as the NCNF shown in Figure 2.4. The subsystem displays are driven by the sim ulator, and any change to a device state is imm ediately reflected in all of the relevant active subsystem displays. The REACT tutor, unlike the Operator, has access to all of the device m odel state data, which is reflected in Figure 2.1 by the arrow labeled "Device State Data" and "Directive&Response". Every change in a device state is perceived by REACT, though it is possible to hide device state inform ation from REACT, if desired. In addition, REACT perceives the communication between the student and the simulator, which takes place in the form of directives and responses. REACT's internal model of the devices and the student-sim ulator communication is updated imm ediately after a change occurs.6 2.3.2 Command processor All student directives are processed by the sim ulator's command processor before applying them to any devices. In Figure 2.1 the arrow labeled "Directives" denotes the flow of O perator com mands from the graphical user interface to the simulator's com mand processor. The com mand pro cessor reads the directive and its param eters and checks link devices to in sure that they are in a legal state for applying the directive. If a device state is incorrect for a directive, it is rejected and a message is generated and sent to the Event Log. In the case of a rejection no device state change occurs. If 5 NIDLE stands for IDLE in the Narrow Channel Bandwidth program. 6 The updates to the internal model occur at the beginning of each Soar elaboration cycle. For more information on the details of the input/output cycles in Soar, see the Soar user's manual (Laird et al., 1990). 37 all of a directive’ s conditions are satisfied, the command processor accepts the directive and changes the appropriate device states. N ote that the command processor is doing a lot of the constraint check ing w hen it comes to verifying whether an action is legal or not. This re flects the w ay that the link subsystems are implemented, so it is not a spe cial feature that was added just for the training system. As we will see later, this constraint checking is an im portant feature used for identifying impasses. 2.3.3 O utput functions The outp u t functions generate directive response messages and event notice messages to the Operator via the Event Log in the graphical user interface. In addition, these functions u pdate active system displays containing device attributes whose values have changed. This accounts for the arrow s in Figure 2.1 pointing from the LMC sim ulator to the User Interface labeled "Device Display Data" and "Directive&Response". The tutor also perceives the "Directive&Response" m essage and uses this in form ation w hen to detect w hen an action constraint violation has oc curred. 2.4 REACT tutor The REACT tutor watches the LMC sim ulator devices and all student- sim ulator interactions w ith the purpose of recognizing student impasses. It uses a technique called situated plan attribution to recognize the im passes. Once recognized, REACT explicates the impasse through the use of its expert problem solving knowledge. Situated plan attribution and im passe explication are the main topics of Chapters 4 and 5, so they will not be discussed here. Rather, let us focus for a m om ent on some architectural issues related to the fact that REACT is implemented in Soar. Soar operates on a decision cycle, which is composed of an elaboration phase and a decision phase (Laird et al., 1987). Each elaboration phase may have num erous elaboration cycles. At the beginning of each elaboration cycle Soar updates its internal model of the w orld by adding external in puts. So when I say that REACT watches the sim ulator and the student- sim ulator interactions, this means that every change in the sim ulator and 38 every interaction between the student and the sim ulator is perceived al m ost instantaneously. This creates a very tight coupling between the tutor, the sim ulator, and the user interface. Consequently, impasses are almost always recognized before the student can take another action, m aking it possible to give tutoring to the student w ithin seconds of the im passe’ s detection. A benefit of using Soar is the uniform representation of knowledge as search in a hierarchy of problem spaces (Laird et al., 1987; Newell, 1990). The im passe recognition and im passe explication functions are im plem ented as problem spaces in the same hierarchy. These tw o functions fit the traditional ITS roles of student m odeling, diagnosis and expert model, respectively. Though the traditional ITS model does not preclude these m odules from being tightly coupled, there is a tem ptation to treat them as modules that need to communicate w ith one another as separate, self-contained programs. This is not the case in REACT, especially as the know ledge from these problem spaces is chunked and it becom es unnecessary to search for the operators that will recognize the impasse or explicate it. Thus, Soar's learning m echanism (Rosenbloom&Newell, 1986) is another source of implementational efficiency that can be used to make the tutor a highly integrated and responsive entity. Student Model ------------------------ 2.5 Com parison to traditional ITS m odel The traditional model of an intelligent tutoring system (ITS) has the four com ponents shown in Figure 2.7: an expert model, a student model, an en v iro n m en t m odule, and a tu to r m odel (B urns& C apps, 1988; Environment User Interface Simulator Tutor Model Student Expert Model Figure 2.7 Traditional model of an ITS 39 Burns&Parlett, 1991; Goldstein, 1982; Macmillan et al, 1988; Clancey, 1986, 1987; W arren&Goodman, 1993; Wenger, 1987). Each of these elements is identifiable in the LMC Training System, but their im plem entation is som ewhat different from the m odular decom position that is inferred by the traditional model. In the LMC Training System, the expert m odel is identifiable as the expert cognitive model in the REACT tutor. The student model is updated by the im passe recognizer and the expert cognitive m odel. The environm ent m odule is im plem ented as the User Interface and the LMC Simulator, and the tutor is implicitly im plem ented in REACT in term s of the style of interaction that results from im passe-driven tutoring. In this section I discuss in more detail how REACT compares to the traditional ITS architecture. User Interface Directive Menu LMC Event Log System Displays REACT Tutor Window Task Progress Display Directive Device Display Data Directive&Response Impasse Tutoring Task Progress Data Link Simulator Command Processor Device State Models Output Functions Device State Data Directive&Response REACT Tutor Impasse Recognizer Expert Cognitive Model Expert Model Figure 2.8 REACT's expert model 2.5.1 The expert model The traditional expert m odel, w hich is som etim es called the dom ain model, is a representation of the knowledge that is supposed to be learned by the student. It can serve both as a driver, providing instructional con tent for a simulator or for interactive dialogue, and it can act as an evalua tor, determ ining w hether the student’s behavior is correct or incorrect. 40 (Anderson, 1988) identifies three types of knowledge that may be contained in the expert model: declarative, procedural and causal, which is consistent with his own theories of skill acquisition and hum an cognition (Anderson, 1983). In the LMC Training System, the REACT tutor contains w hat is tradi tionally known as the expert model; the expert cognitive model shown in Figure 2.8 corresponds to the traditional expert model. The expert cogni tive model is implemented as a Soar problem space and it is derived from the LMC cognitive model that was developed in order to understand the differences between novice and expert behavior and the skill acquisition process that enables a novice to become an expert (Hill&Johnson, 1993a). The model provided insight into the issues of recognizing impasses in the LMC dom ain, deciding when to intervene, and determ ining w hat should be said at the tim e of the intervention. Besides influencing the design of REACT, this modeling w ork had the added benefit of providing the expert m odel that could be incorporated, w ith some m inor changes, into the tu tor's problem space hierarchy. REACT's expert cognitive model is not used to perform model tracing, a la Anderson and W ard (Anderson, 1990; W ard, 1991). In tutoring systems that use model tracing, the expert model is used to generatively trace the student's m ental states that w ould account for the observed behavior. Likewise, REACT's expert model is not used only for generating alternate solutions. Rather, it explains the sources of impasses as well as resolving them. This capability in REACT is called "impasse explication" and it is di rectly derived from the cognitive m odeling work (Hill&Johnson, 1993a). Since that model was capable of recovering from impasses, this capability is very useful in the expert model of REACT. 2.5.2 The student model The student model is typically a data structure that represents the student's level of understanding of the dom ain being taught (VanLehn, 1988; Clancey, 1986). This data structure is m anipulated by a set of diagnostic routines that attem pt to infer know ledge gaps and misconceptions. The diagnostic procedure inevitably compares the student model to the expert 41 model, w hich m ay contain both declarative and procedural knowledge, and it attem pts to determ ine the differences. Two of the dom inant diag nostic paradigm s in use today are m odel tracing and plan recognition. Model tracing (Anderson, 1988; Anderson, 1990; VanLehn, 1988) attempts to interpret the student’s behavior step-by-step, thus creating the need to have detailed access to each of the student's mental states. The results of assessing the student's mental state are used to update the student model, which Anderson calls knowledge tracing (Anderson et al., 1990). User interface Directive Menu LMC Event Log System Displays REACT Tutor Window Task Progress Display Directive Device Display Data Directive&Response Impasse Tutoring Task Progress Data Link Simulator Command Processor Device State Models Output Functions Device State Data Directive&Response REACT Tutor Impasse Recognizer Stude it Model Expert Cognitive Model Figure 2.9 REACT's student model REACT models the student in terms of actions, plans and goals. The student assessment function is performed by REACT’ s impasse recognition problem spaces, w hich use a m ethod called situated plan attribution to understand the student’s behavior. REACT's student m odel, show n in Figure 2.9, is updated by the tutor as it monitors the student's progress on a task. Situated plan attribution uses know ledge about the task to create expectations of w hat plans and actions the student should be executing at any given time. If the student performs an action that is inconsistent with the tutor's expectations, then it is diagnosed as an im passe and the tutor generates an explanation that can be used in the interaction w ith the stu dent. Situated plan attribution im plem ents an abstract form of m odel tracing; it differs from model tracing as described by Anderson et al. (1990) 42 in that it does not involve running a cognitive sim ulation of the student in order to diagnose an impasse. Rather than trying to understand the student's behavior by running a model to generate the identical behavior, this approach generates expectations about behavior and only uses an executable cognitive model to explain behavior that deviates from w hat w as expected. In this sense, situated plan attribution resem bles a plan recognition approach to student m odeling (Genesereth, 1982; Johnson, 1990; Calistri, 1990) in that the student's actions are m atched to plans that describe how to perform the task. It differs from plan recognition, however, in that it does not make use of an explicit representation of buggy knowledge, commonly called a mal-plan. The plan matching perform ed in situated plan attribution informs REACT of the student's progress on the task as well as alerting it to potential impasses. Once a potential impasse is detected, the expert cognitive m odel determ ines w hether the im passe is actual and, if so, explicates it accordingly. 2.5.3 The environm ent m odule The environm ent m odule is the part of system w ith which the student in teracts (Burton, 1988). It defines w hat the student is able to do in the training environm ent. Environment Module User Interface Directive Directive Menu ~ ~ Link Simulator Command Processor Device State Models Output Functions LMC Event Log Device Display Data Directive&Response Device State Data Directive&Response System Displays REACT Tutor Window Impasse Tutoring Task Progress Date! REACT Tutor Impasse Recognizer Expert Cognitive Model Task Progress Display Figure 2.10 REACT's environm ent m odule 43 The components of the LMC Training System that constitute the envi ronm ent m odule are the LMC sim ulator and the User Interface windows, shown in Figure 2.10. The LMC sim ulator drives the system displays and the event log responses to the student’ s actions. The User Interface w in dows provide a realistic LMC environm ent w ith which the student can in teract. In addition to providing a sim ulation of the target system, REACT com m unicates w ith the student through the Tutor W indow , w hile the Task Progress windows give the student a high level view of how well they are performing the task, both in terms of the procedures being executed and the goals that m ust be achieved. The com m unications through these windows are driven by REACT: the explanations in the Tutor W indow are generated while REACT is explicating an impasse, and the Task Progress W indow's contents directly reflect REACT's m odel of the student’ s task perform ance. Hence, the User Interface is driven by tw o sources: the LMC simulator and REACT. The LMC simulator provides the state of the world that is re flected in the system displays, and REACT interprets and explains the stu dents actions w ith respect to the state of the w orld. W hile the User Interface is im plem ented as a separate com ponent, the source of the ex planations that are communicated to the student is REACT. 2.5.4 The tutor m odel The tutor model, which is also known as the instructional or pedagogical model is a model of the teacher. It incorporates theories of teaching and learning so that it can, on the basis of a diagnostic evaluation of the stu dent, alter the teaching strategy or content of the lesson in the process of com m unicating know ledge from the expert m odel to the student (Halff, 1988; Wenger, 1987; Woolf, 1991). REACT is an im passe-driven tutoring system. This approach is based on a cognitive m odel theory of how students learn (Hill&Johnson, 1993a) and as such the tutor model is implicit in REACT's design. The decision to interact w ith the student is built into the w ay that REACT interprets the students actions, via situated plan attribution, and in the w ay that it expli cates impasses with the expert cognitive model. Thus, REACT's architec- 44 ture, as indicated in Figure 2.11, is infused w ith a model of teaching, which is based on the notion of detecting and explaining impasses. User interface Directive Menu LMC Event Log System Displays REACT Tutor Window Directive Task Progress Display Device Display Data Directive&Response Link Simulator Command Processor Device State Models Output Functions Device State Data Directive&Response Impasse Tutoring Task Progress Dat REACT Tutor Impasse Recognizer Expert Cognitive Model Tutor Model Figure 2.11 REACT's tutor model In the current implem entation of REACT, tutorial advice is sent to the Tutor W indow for every detected impasse. This approach will be modified in future versions of the system by allowing the user to select the level of help desired. Regardless of the level of help the student chooses, however, REACT will continue to detect and explicate all im passes, w ith the help level determ ining whether the explication is communicated or not. 45 Chapter 3 Tutoring w ith REACT 3.1 Introduction This chapter serves two purposes. First, it provides a set of examples that describe the nature of the knowledge and skills needed to perform tasks in an interactive environm ent such as the LMC system. The idealized view of task perform ance in this dom ain comes from a procedure m anual description. Procedure manuals are good examples of w hat Suchman calls a resource (Suchman, 1987): they delineate the basic structure, content and goals of a mission's plans w ithout providing all of the details that are nec essary to execute the procedures in the real world. Thus, a procedure m anual can serve to orient the O perator, b u t there is a lot of m issing knowledge that m ust be acquired by other means and from other resources. The problem w ith the procedure m anual perspective of task execution is that in the real w orld the assum ptions of the procedure m anual, which are often unstated, m ay be situationally invalid, creating a need for the Operator to deviate somewhat from the ideal plan in order to accomplish a procedure's goals. There are aspects of the environm ent that require the O perator to be able to react in a reasoned and timely fashion, and until these skills are acquired there will be situations where the Operator reaches a problem solving impasse, w here either an incorrect action is taken or a goal is not achieved. In addition to these situationally-based im passes, however, I also observe that Operators sometimes get confused about the structure of the the procedures themselves, which also results in impasse- based behavior. The second purpose of this chapter builds on the first: it describes how REACT tutors students so that they can acquire the skills that are necessary for perform ing "real world" tasks. Several examples are used to give the reader a functional perspective on how REACT interacts w ith the student. A detailed discussion of how REACT decides to interact and w hat to say is reserved for subsequent chapters. 46 3.2 Procedure m anuals as resources To begin a discussion of the skills required for the LMC system, let us start w ith the idealized view of how to perform a task that is given by the proce dure m anuals and see how this w ould look during their execution in an ideal world, where all of the assumptions about the state of the link devices are valid. For each mission that an Operator may be assigned there is a set of pro cedure m anuals that describe how to perform the tasks associated w ith the mission. M ission tasks involve interacting w ith a com munications link, which is composed of m any different devices and subsystems. The tasks include configuring, calibrating, and testing the link, followed by a period where data is acquired by communicating w ith a spacecraft. The acquired data m ust subsequently be played back from the recording device to the N etwork Operations Control Center (NOCC). Figure 3.1 shows a small portion of a typical mission's procedural hier archy. In this case, the m ission is Very Long Baseline Interferom etry (VLBI) Delta Differential One-Way Ranging (Delta-DOR). The purpose of this type of VLBI mission is to obtain a very accurate m easurem ent of the position of a spacecraft. According to the description given in (DSN-OPS, 1991), this is accomplished by simultaneously recording the one-way dow n link from a spacecraft at two widely spaced antenna sites and using this data to measure the difference in the time-of-arrival at the two sites. These m easurem ents are used to establish an angular position of the spacecraft w ithin the plane form ed by the antenna sites and the spacecraft. Similar m easurem ents are taken by a pair of antenna sites that are in a plane orthogonal to the original one. In each case, measurem ents of an angularly nearby extragalactic radio source are also taken. The recorded data is shipped to a processing center where it is correlated and the actual scientific m easurem ents are determ ined. Because of the length of tim e it takes to record and process the data and the physical distances that separate the recording sites from the processing center, operational errors affecting the m easurem ents m ay not become apparent until a long tim e after the mis sion is completed. Consequently, it is often the case that the Operators never receiving feedback about errors (Resch, 1991). 47 Each mission has a set of procedures associated with i t For instance, the VLBI m ission has procedures (i.e., plans) for configuring the DSCC Spectrum Processing (DSP) Subsystem, perform ing the coherence test on the communications link to determ ine its continuity and phase coherence, acquire data from the spacecraft, and play back the data to the NOCC, and so on. There are similar procedures for each of the subsystems used in the link. Mission: Plans: Commands: V LBI Acquire-Data Configure-DSP Load-Predicts Coherence-Test Seled-Recorder Set-SAT-Value Figure 3.1 The VLBI task with a representative set of plans and operators Each of a mission’ s procedures involves taking num erous actions. For instance, the Configure-DSP procedure shown in Figure 3.1 requires the O perator to take actions to load the predicts set (Load-Predicts), set the S- band attenuation value (Set-SAT-Value) and select a recording device (Select-Recorder). Configure-DSP Operator Command Load-Predicts V NLOAD <x> Set-SAT-Value V SAT <x> Set-XAT-Value V XAT <x> System-Temperature VNTOP <x1><x2> Select-Recorder V NRMED <x> Set-Offset V OFST <x> Legend: V - Subsystem ID, o-param eter variable Coherence-Test Operator Command Phase-Calibration- Gen V NPCG <x> Run-NCB-Program VNRUNoo Run-FFT-Program V NFFT oc> Digital-Tone-Extractor V NDTE <x> Utility Commands Operator Command Enable-Recorder-LDO V LDO <x> Disable-Recorder-LDO V LDO <x> Idle-NCB-Prog ram V NIDLE <x> Figure 3.2 Plans for Configure-DSP and Coherence-Test sub-tasks 48 Figure 3.2 shows a sum m ary of two of the VLBI procedures: Configure- DSP and Coherence-Test. In the left-hand column of each procedure a list of the "operators" is given, and the corresponding commands are shown in the right-hand column. The O perator executes the procedure by issuing the commands for each of the operators. The sequence of commands will, under ideal circumstances, achieve the goals of the procedure. Each direc tive requires the O perator to supply one or m ore param eters, which m ust be determ ined from the resources provided (i.e., from system displays, support data, etc.). The procedural summaries shown above do not include the directives that activate the various system displays, of which there are m any. These two procedures will be used for all of the examples in the re m ainder of this chapter. Though it is not apparent from the figure above, the Configure-DSP procedure is supposed to be perform ed prior to the Coherence-Test. The utility commands are not members of any particular procedure, rather, they are used to control devices that are in an unex pected or undesirable state. 3.3 How tasks w ould be perform ed in an ideal w orld In an ideal w orld the O perator can rotely execute the procedures in the order shown in Figure 3.2. By rote execution, I mean that the Operator can mechanically execute the procedure by issuing the directives in the listed order w ithout giving heed to the meaning of the order. The m ain cognitive task involves searching for inform ation that can be used to param eterize the commands in the procedures. The search for a data item value occurs am ong the various inform ation resources at the disposal of the O perator, w hich includes the procedure m anuals them selves, dozens of different computer-based data displays that can be shown on dem and, mission-specific support data and w hiteboards in the opera tions room. For instance, the load-predicts operator uses the com mand NLOAD to load a file of mission-specific data into a configuration table. This data file is generically called a "predicts" set. Each mission has its ow n predicts set, and the NLOAD command requires a param eter specifying the name of the predicts set to be used for that mission. The Operator finds the name of the 49 predicts set in the support data printout that is provided as a mission re source. # Type Event Log Configure DSP Coherence Test Utility 1 OD >V NLOAD JK X 2 DR > COMPLETED. 3 OD > V NRMEDLDO X 4 DR > COMPLETED. 5 OD > V SAT 15 X 6 DR > COMPLETED. 7 OD > VXAT13 X 8 DR > COMPLETED. 9 OD > V NTOP 20.0 30.0 X 10 DR > COMPLETED. 1 1 OD > V OFST 2.7 X 12 DR > COMPLETED. 13 OD > V NPCG MAN X 14 DR > COMPLETED. 15 OD > V NRUN COLD X 16 DR > COMPLETED. 17 OD > V NDTE E X 18 DR > COMPLETED. 19 OD > VNFFTE X 20 DR > COMPLETED. Type Legend: OO = Operator Directive, DR = Directive Response Figure 3.3 Executing Configure-DSP and Coherence-Test in an ideal world Figure 3.3 shows how an event log w ould look if the Configure-DSP and Coherence-Test procedures were executed in an ideal world. A few w ords of explanation about the log are required. The first column on the left gives the line num ber of each event. The second colum n, labeled "Type," describes w hat kind of event is listed on each line. A legend of the event types is shown below the log. The log entries shown in Figure 3.3 fall into two categories: operator directives (OD) and directive responses (DR). A directive (i.e., command) appears in the log after the O perator issues it to the appropriate subsystem . The directive responses are messages sent by the subsystems to indicate the status of the directive; a directive that has been received and will be accepted by the subsystem will have a COMPLETED response. D irectives th at violate subsystem constraints are REJECTED. N ote that the directive response messages do not indicate whether the directive has actually been executed, only that the communications loop has been closed. The three columns to the right of 50 the Event Log are used to indicate to the reader which procedure each directive belongs to. N ote that in the ideal world the O perator issues the directives in the same order as show n in the sum m ary sheets in Figure 3.2. In addition, there is no overlap betw een the tw o procedures. Each directive is param eterized and there are no apparent problem s w ith the execution of the procedures. N o directives were rejected and there were no obvious procedural failures. Looking at this trace one m ight conclude that it is possible to act in a completely rote fashion w ith regard to the execution of these procedures. The procedure m anual acts as a default plan that requires no modification, and the state of the link devices plays no obvious role in the execution of the plan. 3.4 The effects of the real world on task perform ance N ow let us turn to an example that illustrates w hat can happen in the real world. The Event Log in Figure 3.4 shows how the same execution order that was used in the ideal world example will fail if the state of the devices is slightly different. In this case things didn’ t work out as smoothly as they previously did. Focusing on lines 1 and 2 of the Event Log, we observe that the first di rective was rejected in spite of the fact that the procedure was being exe cuted according to the defaults plans shown in Figure 3.2. The directive was rejected because one of the devices, the N arrow Channel Bandwidth recording program (NCB-Pgm) was not in a state that w ould allow the DSP subsystem to accept the NLOAD command. In this example, the Operator recognized the m eaning of the rejection message and responded on line 3 by issuing the command, NIDLE REC. This com mand changes the state of the NCB-Pgm from RUN to IDLE, which is positively indicated by the progress advisory message on line 5. Note that the NIDLE REC command does not belong to either of the standard procedures being executed, rather, it is a utility command. In this case the O perator knew about the NIDLE REC command, but a novice would not necessarily know about it. Once the NCB-Pgm was changed to the IDLE m ode, the Operator again attem pts to load the predicts set on line 6. Again the NLOAD command is 51 rejected, but this time for a different reason. The rejection notice in line 7 indicates that the predicts set that the Operator was trying to load is not pre sent in the system; a copy of the predicts set has to be transferred via the lo cal area network to the DSP subsystem before they can be loaded. This is a case where the Operator correctly param eterized the NLOAD directive with the predicts set nam ed JK, but a check was not m ade by the O perator to determ ine whether the desired predicts were available yet. To rem edy this situation, the Operator has to arrange for the transfer of the predicts to the DSP subsystem; their arrival is indicated by the progress advisory messages shown on lines 8 and 9. # Type Event Log Configure DSP Coherence Test Utility 1 OD >V NLOAD JK X 2 DR > REJECTED. NOTALLOWED IN RUN MODE. 3 OD >V NIDLE REC X 4 DR > COMPLETED. NIDLE REC INITIATED. 5 PA > N:PGM MODE: IDLE 6 OD >V NLOAD JK X 7 DR > REJECTED. NO PR FOUND FOR THIS SET 8 PA > PRED SET JK RECEIVED (008) 9 PA > S&L SET JK RECEIVED (008) 10 OD >V NLOAD JK X 11 DR > COMPLETED. 12 OD > V NRMED LD O X 13 DR > COMPLETED. 14 OD > V SAT 15 X 15 DR > COMPLETED. 16 OD > VXAT13 X 17 DR > COMPLETED. 18 OD > V NTOP 20.0 30.0 X 19 DR > COMPLETED. 20 OD > V OFST 2.5 X 21 DR > COMPLETED. 22 OD > VNPCG MAN X 23 DR > COMPLETED. 24 OD > V NRUN COLD X 25 DR > COMPLETED. 26 OD > V NDTE E X 27 DR > COMPLETED. 28 OD > VNFFTE X 29 DR > COMPLETED. Legend: OD = Operator Directive, DR = Directive Response, PA = Progress Advisory Figure 3.4 Executing Configure-DSP and Coherence-Test procedures in the "real world" 52 This example illustrates two points about this domain. First, rote execu tion of a procedure does not always work. Two cases of how an action may fail have been shown in Figure 3.4. In the first case the action failed be cause it violated a constraint having to do w ith the state of other devices in the link. In the second case, the Operator correctly param eterized the direc tive but did not check to see w hether the action being taken was valid un der the circumstances. The predicts set was not present, so a constraint was violated. Hence, the state of the w orld plays a role in the performance of the task, m eaning that the rote execution of a procedure will fail w hen the assum ptions behind the procedure are not true. The second point is related to the first: actions in this domain have pre conditions that must be satisfied before they can be successfully executed. Two of the preconditions of the NLOAD com m and have been illustrated in this example. The first precondition is that the NCB-Pgm m ust be in the IDLE m ode and the second precondition is that the predicts set being loaded m ust be present in the DSP subsystem. W hen a precondition is violated, the directive is rejected and the O perator has to attend to the situation in order to overcome the impasse that has been encountered. 3.5 O ther O perator behaviors in the real world In the last section we saw how the state of the w orld affects the perfor mance of tasks in the LMC domain, but there are other factors that also af fect the O perator's behavior, m any of which can be explained as a lack of understanding of the task components: the precedence relationship among procedures, procedure goals, and where to search for a data item to be used as a directive param eter. The example below will be used to illustrate some Operator behaviors that cannot be explained as sim ple action constraint vi olations. W hereas the examples in Figures 3.3 and 3.4 are representative of the kinds of behaviors that are observable in the mission event logs, the event log shown in Figure 3.5 comes from an actual mission. N ote that there are some directives in this log that are not shown in the procedural sum m ary in Figure 3.2. I have m arked which procedures each directive belongs to in the columns to the right of the directive. 53 Type Event Log Configure DSP Coherence Test Impasse 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 i n OD DR OD DR OD DR OD DR OD DR OD DR OD DR OD DR OD DR OD DR OD DR OD DR OD DR OD DR CA OD DR OD DR OD DR OD DR OD DR OD DR > VHI > V COMPLETED. HI. I AM DMV-5233-OP-E > V D PRF11 > COMPLETED. > V D NCNF12 > COMPLETED > VD NFFT11 > COMPLETED. > VNFFTE > REJECTED. NOT IN NRUN MODE > V NRUN ?? > COMPLETED. FMT: COLD/WARM <NOPLAY> > V NRMED NULL > COMPLETED. NRMED: NULL > V NRUN NOREC > REJECTED. UNDEFINED PARAMETER > V NRUN NOREC ?? > REJECTED. UNDEFINED PARAMETER >V NRUN NO REC. > REJECTED. UNDEFINED PARAMETER > NRUN NORTM > REJECTED. RECORDING MEDIUM - NULL >V NRMED LD O > COMPLETED. NRMED: LDO > V NRUN COLD > REJECTED. BAD RADIO SOURCE TABLE > V NLOAD KJ > COMPLETED. LOAD STARTED ... > SET KJ LOADED OK (031) >V NRUN COLD > REJECTED. SAT OR XAT UNDEFINED > V XAT12 > COMPLETED. X-BAND ATTENUATION = 12DB > V SAT 13 > COMPLETED. S-BAND ATTENUATION - 13DB >V NRUN COLD > COMPLETED. NCB MODE: INIT > V NTOP 21.1 17.7 > COMPLETED. NCB SYS TEMP - S 21.1 X 17.7 > V NDTE E > COMPLETED. NCB TMON ENAB, NPTS = 10 > ... > end of pass_______________________________ Legend: OD = Operator Directive, DR = Directive Response, PA = Progress Advisory, CA = Completion Advisory Figure 3.5 Actual event log illustrating impasse-based Operator behaviors 54 3909854 Scanning the log in Figure 3.5, one can im m ediately observe that the O perator chose to interleave the Configure-DSP and Coherence-Test proce dures. It is permissible and, in some instances, desirable to interleave pro cedures if there are no interdependencies am ong them; this can often cut dow n the overall am ount of time it takes to prepare the link for a mission. H ow ever, in this is instance it is not beneficial to interleave these two procedures, since there are m any interdependencies between them. Consequently, the Operator begins to run into difficulties beginning at lines 9 and 10 where the NFFT directive is rejected because the NCB-Pgm is not in the NRUN mode. W hat follows is a series of reactive steps taken by the O perator, who seems determ ined to proceed w ith the Coherence-Test in spite of the directive rejections that are incurred. Since the NFFT direc tive was rejected because the NCB-Pgm was not in the NRUN m ode, the O perator attem pts to issue the NRUN com m and on line 11. Though the NRUN com m and was accepted in line 12, the directive response also indi cates that the O perator needs to give the NRUN com mand a param eter (i.e., COLD or WARM or NOPLAY.) The O perator subsequently attem pts to issue the NRUN command on lines 15,17,19, 21, 25 and 30. In each case the directive is rejected, either because of an undefined param eter (lines 16, 18 and 20) or because a precondition was not satisfied (lines 22, 26 and 31). The unsatisfied preconditions (lines 22, 26 and 31) are a consequence of in terleaving the two procedures; if the Configure-DSP procedure had been com pleted prior to the Coherence-Test all of these preconditions w ould have been satisfied. Instead, the O perator ends up satisfying each of the preconditions of the NRUN com m and by reading the directive rejection messages and issuing the appropriate commands (lines 23, 27, 32 and 34), which all happen to be members of the Configure-DSP procedure. In this exam ple the O perator's behavior indicates a m isunderstanding about the relationship am ong the procedures. This m isunderstanding per sists throughout the log as the Operator attem pts to resolve all of the action constraint violations that result from it. At a reactive level the O perator responds to each of the directive rejections, though it does not appear that any of the preconditions were anticipated. If the Operator had known the preconditions associated w ith the NRUN com m and then we w ould expect 55 the appropriate Configure-DSP commands to be issued even if they were interleaved with the Coherence-Test procedure. 3.6 What Operators need to know In an ideal world a sufficient means of performing a task is to follow a pro cedure m anual. The ideal w orld is a place where a procedure can be exe cuted w ithout taking into account the preconditions of the individual ac tions, since the procedure's directive sequence can be counted on to take care of all preconditions. This makes it unnecessary for the O perator to know about preconditions since they are implicitly being satisfied by the di rective sequence. A nother assum ption of the ideal w orld is that directives have an im m ediate effect on the w orld, that is, there is no execution la tency. This reinforces the notion of depending on a directive sequence for the perform ance of a task since it becomes unnecessary to determ ine w hether an action has had its desired effects before carrying on with the next action in the sequence. Operators need to know directive preconditions and postconditions. W hat the real w orld examples tell us about tasks in the LMC dom ain is that the ideal world assum ptions are false. It is not sufficient to rotely exe cute a directive sequence. Rather, actions are situated in a world where the states of the communications link devices m atter and m ust be checked be fore each action is executed. Operators need to know the precedence relationships among proce dures/plans. In the example in Figure 3.5 the O perator interleaved the ac tions of two procedures that have a num ber of interdependencies. As a re sult, there were num erous action constraint violations. Of course, if the Operator knew the preconditions of all of the directives in these two proce dures, then an implicit order between the two procedures could have been achieved by only issuing those directives whose preconditions w ere satis fied. In the absence of the precondition knowledge, it w ould be helpful to know the order among procedures since it will help to avoid m any of the action constraint violations. Operators need to know each procedure's goals. A m ore insidious prob lem occurs when the Operator completes a procedure w ithout achieving its underlying goals. This type of error may go undetected until after the mis- 56 sion has been completed and it is too late to recover. By mis-parameteriz- ing a directive or by issuing a directive out of sequence, an Operator can fail a procedure w ithout perceiving any im m ediate evidence such as a direc tive rejection or device failure message. For instance, if the Operator sends the w rong clock offset value w ith the OFST com m and, the resulting problem will not show up until days after the mission has been completed and the data is being processed and correlated. Though this particular error is not fatal, it does make the correlation job m ore difficult. Operators need to recognize anomalous device states. The O perators need to be able to recognize when a device state is outside its norm al oper ating range. In m any cases there are built-in alarm s and w arnings that indicate that a subsystem or device has failed and requires attention. The O perator's response to the situation depends on the severity and type of anomaly. It can range from doing nothing to executing an emergency plan to work around the problem. This type of situation is outside of the scope of this thesis. There are plans to address these issues in the next version of REACT. 3.7 Tutoring w ith REACT Let us now consider how the REACT tutor interacts with a student who in correctly performs a task in the LMC domain. In the last section I identified three types of knowledge that the Operator m ust have in order to success fully perform tasks in the LMC domain: (1) knowledge of directive pre conditions and postconditions, (2) knowledge of plan dependency relation ships and (3) knowledge of plan goals. The Operator errors w e have seen in the previous sections can be accounted for as a lack of knowledge or a misconception in one or more of these categories. REACT helps students acquire this know ledge through the use of im passe-driven tutoring. Using this technique the tutor interacts only after it has detected that the student has reached an impasse while perform ing the task. The impasses detectable by the tutor are related to the types of knowl edge: (1) action constraint violations occur w hen a directive is issued w ithout its preconditions being satisfied, (2) plan dependency violations occur w hen the student begins executing a plan that depends on another plan th at has not yet been com pleted, and (3) goal failures occur 57 # Type Event Log REACT's Explanation 1 OD >V NLOAD JK 2 DR > REJECTED. NOT ALLOWED IN RUN MODE. The NLOAD command failed because one of its preconditions was unsatis fied: the NCB-program mode should be IDLE instead of RUN. First, Issue the Command: V NIDLE REC then Issue the Command: V NLOAD JK 3 OD > V NIDLE REC 4 DR > COMPLETED. 5 OD >V NLOAD JK 6 DR > COMPLETED. 7 OD >V NRMED LD O 8 DR > COMPLETED. 9 OD > V SAT 15 10 DR > COMPLETED. 11 OD > V XAT 13 12 DR > COMPLETED. 13 OD > V NTOP 20.0 30.0 14 DR > COMPLETED. 15 OD > V NPCG MAN 16 DR > COMPLETED. You started the Coherence-Test pro cedure before you finished the Configure-DSP procedure. Issue the Command: V OFST o 17 OD > V OFST 2.5 18 DR > COMPLETED. 19 OD >V NRUN COLD 20 DR > COMPLETED. You failed to achieve one of the goals of the Configure-DSP procedure: off set = 2.7 First, Issue the command: V NIDLE REC then Issue the command: V OFST 2.7 21 OD >V NIDLE REC 22 DR > COMPLETED. 23 OD >V OFST 2.7 24 DR > COMPLETED. 25 OD >V NRUN COLD 26 DR > COMPLETED. 27 OD > V NDTE E 28 DR > COMPLETED. 29 OD > VNFFTE 30 DR > COMPLETED. Figure 3.6 A tutored session w ith REACT w hen the student has completed a procedure w ithout also achieving all of its goals. Recognizing an impasse is only the first step, however. The im passe indicates the necessity of interacting w ith the student, but only after REACT has also explicated the impasse. The explication process includes determ ining the reason for the impasse as well as a means of resolving it, and the resulting explanation m ust be cast in terms that will help the stu dent to acquire the knowledge that is needed to avoid the im passe in the future. The following subsections give a m ore detailed description of how REACT explicates each of these im passe types. The discussion is based on the exam ple show n in Figure 3.6, which shows another event log. This event log was tailored for this example; though it is hypothetical it is plau sible, based on observations of actual event logs and Operators. As with the other exam ples, the stu d en t is perform ing the Configure-D SP and Coherence-Test procedures. In this case how ever, the event log also includes a column show ing w here and how REACT w ould explain the im passe during a task session. 3.7.1 Explaining an action constraint violation REACT's first interaction w ith the student is show n on line 2, after the NLOAD directive was rejected. The directive rejection message is imm edi ately recognizable as an action-constraint impasse. REACT did not have to evaluate the action itself to recognize it as an im passe, rath er this evaluation was done by the subsystem itself, which subsequently rejected the directive. Once REACT recognized the im passe, it generated an explanation, which is show n in the right column on line 2. Since the impasse was due to an action constraint violation, REACT explains the im passe's source in term s of a a precondition that w as violated, nam ely, the NCB-Program should have been in the IDLE m ode instead of the RUN mode. Hence, in the future the student should check w hether this condition is true before issuing the NLOAD command. Next, REACT helps the student to resolve the im passe by suggesting that the NIDLE REC com m and and then the NLOAD JK com m and be 59 issued. REACT does not actually take these actions: they are left for the student to perform. By explaining the actions that need to be taken to re solve the impasse, REACT is providing the student w ith some knowledge about directive postconditions. Specifically, the NIDLE REC com mand has the postconditions necessary to resolve the im passe associated w ith the precondition failure of the NLOAD command. This is essential knowledge since the NIDLE REC command is not a part of the Configure-DSP proce dure. In the untutored version of this example in Figure 3.4 we showed the Operator issuing the NIDLE REC command in response to the directive rejection. In this case the O perator knew the com m and and was able to recognize w hen to use it, but a novice w ould probably not know about it and we w ould expect a genuine impasse to occur: the Operator w ould un successfully try NLOAD over again and possibly experim ent w ith other commands to correct the situation. REACT dynamically derived the impasse explanation and resolution by putting itself into the student's problem solving context, w hich included looking at the current state of the devices, the param eterized directive is sued by the student, and the goals associated w ith the directive. By dynam ically deriving the solution, REACT avoids the need to store canned solu tions for each type of error that the student will potentially make. 3.7.2 Plan dependency violations example The explanation on line 16 is more subtle since the LMC sim ulator did not give any indication of a problem or impasse. REACT recognized that the two procedures, Configure-DSP and Coherence-Test, w ere being inter leaved, but it expected the Coherence-Test procedure to be perform ed after the Configure-DSP procedure had been completed. This is known as a plan dependency violation, and is used to teach the O perator about the interde pendencies that are defined among procedures. At line 16 REACT explicates the plan dependency violation by pointing out which plan had not been completed and then it shows the directive be longing to the Configure-DSP procedure that was yet unexecuted, which in this case was the OFST command. REACT does not param eterize the direc tive for the student since this is not the part of the task that the student 60 failed; the failure was in recognizing the interdependency am ong plans and the explanation is kept at this level. This is a fairly simple case of this type of impasse. If REACT w atched the O perator behavior shown in Figure 3.5 it w ould have interacted w ith the student very early in the directive sequence, beginning at the point w here the Coherence-Test directives were being sent long before they were due. 3.7.3 Goal failure example The final explanation, show n on line 20, was triggered by the fact that REACT recognized that the O perator had com pleted the Configure-DSP procedure w ithout achieving one of its goals. This im passe category is know n as a goal failure impasse, and it represents a potential rather than an actual im passe situation since it is likely that the student is not even aw are of a problem. In the instance shown in line 20, the student w ould have been able to complete the m ission w ithout evidence that there was a problem , since it w ould not have appeared until m uch later w hen the recorded data was being processed. For this reason, REACT chooses to interact as close to the impasse point as possible. In this w ay the student corrects the error near to its source, which gives the student the benefit of the error’ s context in assimilating the tutoring advice. REACT explains this goal failure impasse by first identifying the nature of the impasse, which in this case is a goal failure of the Configure-DSP procedure. In the same m anner that it explicated the action constraint vio lation at line 2, REACT dynamically derives a solution for working around the current impasse. As before, it is necessary to issue the NIDLE REC com m and to change the state of the NCB-Program before issuing the OFST com mand. In this instance, however, the im passe was not caused by the state of the NCB-Program. The OFST directive was not rejected as was the case w ith the NLOAD directive. REACT decides to issue the NIDLE REC com m and prior to re-issuing the OFST command because in the process of explicating the goal failure im passe, it also determ ined another im passe w aiting to happen. In this case it notes that before OFST can be issued again, one of its preconditions will have to be satisfied, and this can be done with the NIDLE REC command. 61 REACT could have interacted sooner than it did in this instance since it actually detected this potential impasse at line 18. It delayed its final inter pretation, however, until line 20 so as to give the student the opportunity to notice and correct the error on his or her own. By delaying the interven tion until after the student issued the NRUN COLD com mand, this m ade the explication som ew hat more difficult since this com m and changes the state of the NCB-Program into the RUN m ode, which violates one of the preconditions of the OFST command. Consequently, the explication had to take this into account in dynamically resolving the impasse. 3.8 Sum m ary This chapter has served two purposes. First, it describes the nature of skill and knowledge in an interactive dom ain, nam ely, the LMC system. The idealized view of task performance predicts that it is sufficient to execute the plans described in procedure manuals. As it turns out, rote execution of a procedure does not always work, and this is prim arily because actions in this dom ain have preconditions that m ust be satisfied before they can be successfully executed. The claim m ade in this chapter is that there are at least four kinds of knowledge that is required by Operators: (1) Operators need to know directive preconditions and postconditions; (2) O perators need to know the precedence relationships am ong procedures; (3) Operators need to know each procedure's goals; and (4) Operators need to recognize anom alous device states. The second purpose of this chapter was to illustrate how the REACT tutor assist students in the acquisition of the skills necessary to perform procedures in the "real world." REACT im plem ents a technique called "im passe-driven tutoring," whereby it watches the student perform the task and only interacts w ith the student w hen it detects an impasse. REACT recognizes three types of impasses: (1) action constraint impasses, (2) plan dependency im passes, and (3) goal failure im passes. Action constraint impasses are considered to be "actual" impasses in the sense, that the student took an action that failed. Plan dependency and goal failure impasses are considered to be "potential" impasses since they anticipate a failure due to violating a non-prim itive constraint on behavior. 62 Chapter 4 A Cognitive M odel of Reactive Skill Acquisition 4.1 Introduction This chapter gives the m otivation for the tutoring m ethodology im ple m ented in REACT. The last chapter m ade several claims about the nature of problem solving in the LMC dom ain and w hat Operators need to Iqiow. These claims w ere substantiated w ith some examples, b u t they do not in themselves lead to any particular conclusions about how the skill to per form these tasks should be acquired. The tutorial intervention question still remains: when and how should the tutor interact with the student? One tutoring m ethodology that has seen some success in a num ber of different ITS im plem entations is m odel tracing, which has its roots in a cognitive theory of skill acquisition, namely ACT* (Anderson, 1983) and its successor, PUPS (Anderson, 1989; A nderson, 1990). The m odel tracing paradigm was used to build tutors for Lisp program m ing (Reiser, 1985; A nderson, 1990), Geometry theorem -proving (Anderson, 1990), Algebra (Anderson, 1990), and Electric Fields (Ward, 1991). The cognitive m odeling work that inspired the model tracing approach in these ITS's has been aimed prim arily at static (i.e., non-interactive) tasks such as program m ing and m athem atical problem solving. Likewise, VanLehn's SIERRA model (VanLehn, 1987) of skill acquisition was applied to a static task, namely subtraction. Problem solving in static tasks typically does not involve interaction w ith an external environm ent; and in cases where it does, the environm ent is predictable. By internally sim ulating the effects of applying operators to a task, the problem solver in a static domain does not have to be concerned with the state of the w orld changing unex pectedly, nor does it have to be concerned w ith contingencies in the w orld that change an operator's applicability. These are the problem s w ith inter active tasks, and they are the reasons Suchman gives for w hy simple plan 63 execution is not sufficient for problem solving in these dom ains (Suchman, 1987). There has been relatively little w ork to date on modeling tasks th at in teract w ith an unpredictable external environm ent and using these models to m otivate tutoring system design. Since the m odel tracing approach to tutoring has been applied m ainly to static dom ains, it is appropriate to question its effectiveness in interactive task dom ains, particularly since its approach to tracking the student requires that the perform ance m odel be able to sim ulate the student's mental states in solving problems that have the ad d ed com plexity of interacting w ith an external environm ent. Though it seems that this should be theoretically possible, it does not ap pear to be the way that hum an tutors model the student (Galdes, 1990). To gain an understanding of the nature of skill and how it is acquired in an interactive task domain, a com putational m odel of skilled behavior and learning w as built for the LMC system (Hill&Johnson, 1993a). From this m odel a num ber of hypotheses about skill acquisition and tutoring were m ade that led to the design of REACT. The cognitive m odel was devel oped to account not only for problem solving behavior, but also the skill acquisition process: the m odel can im prove its perform ance over time, and it simulates the effects of tutoring on acquiring new knowledge and re covering from incorrect knowledge. The m odel is able to respond to dy nam ically changing situations and revise its problem solving procedures accordingly. 4.2 Overview of the Soar architecture The cognitive model described in this chapter was developed in Soar, an architecture that im plem ents a theory of hum an cognition (Laird et al., 1987; Newell, 1990; Laird at al., 1990). This section briefly describes Soar and w hat it brings to the modeling effort. This foundation will be used in the rem ainder of the chapter in the description of how cognitive m odel was im plem ented. 64 4.2.1 Soar cognitive architecture Soar is an integrated problem solving and learning architecture. Tasks in Soar are represented and perform ed in problem spaces. A problem space contains a set of operators and states, which at any given m om ent repre sents a situation. Problem solving involves applying operators to the cur rent state, where a new state is produced each tim e an operator is success fully applied. Goals guide the problem solving activity, influencing the de cisions involved with the selection of an operator or a state w ithin a prob lem space. W hen the search w ithin a problem space no longer makes progress tow ard a desired state, an impasse is said to occur. Impasses cause subgoals to arise, w here the problem solving is continued in another prob lem space in an effort to bypass the difficulty. Once the impasse is resolved, the subgoal can be term inated, and the problem solving may be resumed in the original problem space. In this way, the problem solving activity in Soar is a search through a hierarchy of problem spaces. Long-term knowledge in Soar is uniformly represented as a production system . Productions m anipulate declarative know ledge w ithin Soar’s short-term w orking m em ory, but all long-term know ledge is stored in a production form. Learning in Soar occurs w hen the results returned from a subgoal (i.e., a desired state) are converted into new productions that can be applied to achieve the same results under similar circumstances. These learned productions are called chunks, and they sum m arize both the goal and state context in which an operator applies. Chunks are added to the production m em ory as they are learned, which constitutes the recognition knowledge in the architecture. Whereas objects can be added to or deleted from the working memory, productions are never deleted. 4.2.2 Useful modeling features of Soar There are a num ber of features in Soar that have proved to be especially useful for m odeling problem solving and skill acquisition in the LMC do main. • Problem space task representation. The representation of tasks in problem spaces maps onto the concept of a procedure in the LMC domain. The mission procedure m anuals describe tasks in term s of com m and se- 65 quences, and to some extent they also describe the goals of each task. These com m ands can be thought of as Soar operator applications, and the task goals correspond to the Soar concept of a goal as a desired state. A com m and sequence is a prescribed path through a problem space, w here the de cisions of which operators to apply are based on order rather than on state. • Perception interleaved with processing. The processing in Soar is based on a decision cycle that is composed of an elaboration phase followed by a decision phase. During the elaboration phase, elements are added to and deleted from the working memory, but actual changes to context ob jects take place during the decision phase. The elaboration phase is actually com posed of any num ber of elaboration cycles where productions fire and add or delete working m emory elements. W hen a state of quiescence is reached w here changes are no longer being made, the decision phase is in voked. Inputs and outputs to the Soar problem solver are interleaved throughout the processing: input functions and o u tp u t functions are called before and after every elaboration cycle, respectively. The special significance of this Soar feature is that it supports the reactive style of prob lem solving that is observed am ong experts in the LMC dom ain. They need to be continually aware of the state of the world and changes that take place there in order to be able to respond. • Continuous goal monitoring. A nother benefit of the style of pro cessing just described is that it supports the continuous monitoring of goal com pletion. This is an im portant elem ent for a goal-oriented problem solver who needs to know when a task has been completed and it is okay to shift to another task. • Impasse-based subgoaling and learning. As will be show n later, the fundam ental view in Soar that learning takes place in a goal context has im plications for how skill and know ledge are acquired in the LMC domain. Refer to section 1.3 for a definition of an impasse. 4.3 M odels of problem solving behavior One of the goals of the cognitive m odel was to give an account of both novice and expert levels of skilled behavior. Chapter 3 presented several examples from the LMC domain of w hat these behaviors look like, particu- 66 larly w hen the task is being perform ed in the real w orld. The cognitive model described below is based on the same type of mission that was used in the previous examples: perform a VLBI m ission using the LMC system and the procedure m anuals for the mission's various subtasks. For refer ence purposes, the VLBI mission hierarchy is again included in Figure 4.1 below. Mission: Plans: Commands: V LBI Acquire-Data Configure-DSP Playback-Data Load-Predicts Coherence-Test Select-Recorder Set-SAT-Value Figure 4.1 The VLBI task w ith a representative set of plans and operators 4.3.1 Novice problem solving If action in interactive task domains such as the LMC system is truly situ ated, then it is reasonable to expect that novices will m ake errors when faced w ith a situation that dem ands a reactive adjustm ent to a standard procedure. Since the procedure itself may be the only resource available to the novice, situations not accounted for by the procedure will result in failures. H um an novice operators appear to perform tasks in the LMC dom ain by rotely following the procedure m anual. Since the novice has little or no knowledge of command preconditions and postconditions, fre quent problem solving impasses occur (e.g., Chapter 3 described the case where the NLOAD command was rejected because the predicts file was not present). Thus, an accurate model of a novice operator should produce the same kind of behavior, including errors, as have been observed in the analysis of Operator behavior. Novice problem solving behavior is modeled by im plem enting the task description shown in Figure 4.1 as a hierarchy of Soar problem spaces. In order to produce the rote behavior that is expected of the novice, the cogni 67 tive m odel is forced to execute the procedures and com m ands in a prede term ined (i.e., procedure m anual) order. The m odel’ s execution order is im plem ented through the use of Soar desirability preferences. Thus, the novice problem solver searches in the VLBI problem space for an operator and finds there are four to choose from: Configure-DSP > Coherence-Test > Acquire-Data > Playback-Data, where the ">" indicates a "better than" de sirability preference. Since the Configure-DSP operator is the m ost desir able choice of the set, it is selected, a sub-goal is created, and a Configure- DSP problem space is created. The problem solver searches the Configure-DSP problem space and finds that there are num erous applicable operators, along w ith another preference ordering: Load-Predicts > Set-Attenuation-Values > Select- Recording-Device. A subgoal is created for the Load-Predicts operator, and w ithin the Load-Predicts problem space the NLOAD com mand is applied to a device and the Load-Predicts goal terminates. This process is repeated for each of the command operators in the Configure-DSP problem space. Once all the com m ands have been applied, the Configure-DSP sub-goal term i nates, and the next subgoal, Coherence-Test, is created, and so on. The novice m odel does not consider preconditions in selecting proce dures and commands, thus the device states are irrelevant to the problem solver unless it reaches a problem solving impasse (i.e., the com m and is rejected by the device, which sends a message to the operator indicating it is unable to perform the action; or the procedure's goals are not achieved, as sum ing, that is, that the problem solver understands w hat the goals are!). As a result, the novice m odel produces the sam e behaviors, including problem solving impasses, that were observed in the examples in Chapter 3. Thus, the novice m odel has a "knowledge gap" w ith respect to the com m and preconditions and postconditions. This m issing know ledge ac counts for the knowledge level impasses that occur in non-standard situa tions. 4.3.2 Expert problem solving Expert operators have been observed to behave differently than novice op erators, w ho appear to perform tasks by rotely executing the procedures. In 68 contrast, expert operators appear to explicitly verify com m and precondi tions and postconditions, which often results in the com m ands being is sued in a different order than w hat is specified by the procedure manual. The expert cognitive model reflects this observation by building some addi tional problem solving knowledge into the novice model, and as a result the expert m odel performs the task correctly, avoiding m any of the prob lem solving im passes encountered by the novice. The expert cognitive m odel was im plem ented, first, by adding two problem spaces below each com m and problem space: V erify-C om m and-Preconditions and Verify- Com m and-Postconditions.1 These problem spaces are searched before and after applying a command to a device. Figure 4.2 shows an example of the problem space hierarchy for the Load-Predicts operator. NLOAD JK Device Operator-x Load-Predicts Verify-Operator- Preconditions Verify-Operator- Post conditions Repair-Unsatisfied- Precondition Attend-to-Unsatisfied- Postconditlon Figure 4.2 Verification and recovery problem spaces for Load-Predicts problem space W hereas the novice w ould im m ediately apply the NLOAD com m and to the device, the expert first verifies w hether the Load-Predicts precondi tions, shown in Figure 4.3, are satisfied. Though one m ight expect that the preconditions and postconditions w ould be em bedded as conditions on the "if" side of the Soar productions, they are m odeled declaratively here and are therefore m anipulated in working m em ory initially (i.e., before chunk ing). One of the reasons for modeling the preconditions this w ay is that it makes it easy to add them to the sim ulated student's m em ory as knowl edge as they are learned in the tutoring process. Thus, the Verify- 1 The explicit representation and tracking of preconditions and postconditions was inspired by Unruh et al. (1990). 69 Command-Precondition problem space contains an operator that compares the condition stated in the precondition (i.e., the expected device attribute- value) to the actual device state; if the two match, then the precondition is annotated as being satisfied (i.e., SAT), otherwise it is UNSAT. Once the problem solver verifies that all of the com m and’s preconditions are satis fied, it sends the NLOAD command. Operator Command Precondition Name Device Attribute | Value Load-Predicts V NLOAD <id> Load-Predicts-PC1 NCB-PROGRAM MODE I IDLE Load-Predicts-PC2 VLBI-PREDICTS RECEIVED? YES Load-Predicts-PC3 VLBI-PREDIGTS QUALITY I OK Figure 4.3 Preconditions for Load-Predicts operator Operator | Command Postcondition Name Device Attribute Value Load-Predicts [ V NLOAD <id> Load-Predicts-POl VLBI-PREDICTS LOADED? YES Load-Predicts-P02 CONFIG-TABLE NCOMB 100.0 • « . . . . Load-Predicts-POn CONFIG-TABLE TYPE DOR Figure 4.4 Postconditions for Load-Predicts operator After the com m and has been sent, the problem solver searches the Verify-Command-Postconditions problem space to verify that the postcon dition, shown in Figure 4.4, is satisfied. If it is satisfied, the Load-Predicts problem space terminates. The process repeats itself w hen the next com m and operator is chosen. Thus, the difference between the expert and the novice is that the expert explicitly verifies preconditions and postcondi tions before and after taking an action, meaning that the expert has some additional knowledge that the novice does not possess. In the model this is sim ulated by giving the novice a knowledge gap. The transition from novice to expert skill levels appears to involve ac quiring not only the know ledge about preconditions and postconditions, b u t it also involves having the goal to verify them. One of the reasons for the O perator to have the goal to verify the preconditions and postcondi tions can be accounted for by the design of the user interface: it provides the O perator w ith im m ediate feedback each tim e a com m and is sent. The feedback includes an indication of w hether the com m and was accepted or rejected, and if it was rejected it also gives a brief message explaining why. This feedback appears to m otivate the O perator to attend to the situation 70 before and after sending a command. The novice appears to learn to antic ipate w hen it is appropriate to send a com m and by attending to the re sponse of the devices being acted upon. Moreover, in addition to verifying the preconditions and postconditions, hum an O perators are observed to find ways to w ork around problem solving impasses. To account for this, the expert m odel includes tw o additional problem spaces: Repair- Unsatisfied-Precondition and Attend-to-Unsatisfied-Postcondition. These problem spaces, which are explained in a later section, are crucial for recov ering from failures and for learning. 4.4 Knowledge compilation Learning occurs continuously during problem solving in this m odel. Problem solvers, w hether novice or expert, compile their know ledge dur ing subgoaling, w here the Soar chunking m echanism builds productions that sum m arize the subgoal results. The kind of learning that is taking place here, based on the model that has been described thus far, is symbol level learning (Rosenbloom et al., 1987; Dietterich, 1986). N o new knowl edge is be acquired, rather, the problem solver is compiling the knowledge it already possesses into a more efficient form. Thus, given that the next task situation is sim ilar to the current one, the problem solver will apply the learned chunks instead of searching for an operator in the problem space hierarchy. Consequently, as know ledge is com piled the problem solver's procedural performance improves in terms of how rapidly it exe cutes a task.2 A novice's compiled knowledge differs from an expert's in that it lacks the know ledge of com m and preconditions and postconditions as well as the steps involved w ith verifying that they are satisfied. For instance, 2 This basically follows the account of knowledge compilation given by ACT* (Anderson, 1983) and PUPS (Anderson, 1989; Anderson, 1990) where new productions are built to summarize the computations performed while solving a problem. Knowledge compilation in this model differs from PUPS, however, in that the Soar chunking mechanism immediately generates new productions after a subgoal terminates, which contrasts with the analogy- based compilation performed in PUPS. In PUPS, productions are built from the induced implications formed during the analogy process or by composing two or more of these implications (Anderson, 1989). 71 Figure 4.5 shows a chunk that a novice learns while rotely performing the Configure-DSP procedure. This production applies the NLOAD command im m ediately after starting the Configure-DSP procedure, w ithout checking the state situation. The only state information in this chunk is that there is a device called DSP to which it can issue the NLOAD JK command. No other state conditions are checked prior to applying the operator. Since there are a num ber of other com m and operators in the Configure-DSP problem space, the chunks that are acquired for applying them look identi cal to the one in Figure 4.5, except that the com mands themselves are dif ferent. ___________________________________ IF goal is VLBI & operator is Configure-DSP & state has name(object, DSP) — > command(DSPf NLOAD JK) Figure 4.5 Pseudo-code novice chunk that applies the NLOAD com mand for the Load-Predicts Operator In the cognitive model, once the novice problem solver has compiled into chunks all of the know ledge about how to apply the Configure-DSP com m and operators, then the next time it has to perform this task all of the chunks will fire sim ultaneously, m eaning that any of the com mands can be sent, though they are ultim ately issued in a random ly determ ined sequence. This contrasts w ith the expert chunks shown in Figures 4.6 and 4.7. The chunk in Figure 4.6 verifies that the Load-Predicts-PCl precondition (from Figure 4.3) is satisfied. Observe that this production summ arizes the result of the Verify-Command-Preconditions problem-space show n in Figure 4.2, and marks the evaluation-result w ith a SAT (for satisfied) if the conditions are met. The left-hand side of the production in Figure 4.6 also contains in form ation about the goal and state context in which to take the action. A sim ilar chunk is built for each of the preconditions that m ust be verified for the Load-Predicts operator. In fact, verification chunks will be built for all com m and preconditions. Hence, for each precondition that the novice does not know, it will concomitantly lack a verification chunk. 72 IF goal is VLBI & operator is Configure-DSP & state has name(object, DSP) & Load-Predicts-PCl (Load-Predicts, Satisfied) & Load-Predicts-PC2(Load-Predicts, Satisfied) & Load-Predicts-PC3(Load-Predicts, Satisfied) —> command(DSP, NLOAD JK) Figure 4.7 Pseudocode expert chunk that verifies a precondition chunk that applies the com m and for the Load-Predicts Operator for the Load-Predicts Operator Figure 4.7 shows the expert's chunk that recognizes w hen to apply the NLOAD com m and to the device. This chunk can be applied w hen the goal and state context match the current situation and the listed preconditions (e.g., Load-Predicts-PCl) are all satisfied. Once this chunk exists, it is not necessary to subgoal to the Load-Predicts problem space again unless the ac tual goal or state context differs somehow from the chunk, or else one of the preconditions is not satisfied. Given the same task in the future, the problem solver will apply its compiled knowledge w ithout subgoaling. The fact that the model builds tw o types of chunks for this action in stead of one large chunk reflects w hat is observable in expert behavior. Experts appear to think through w hat state the device needs to be in prior to applying a command, and this is reflected by the use of working memory elements indicating whether each precondition is satisfied. 4.5 Failure recovery and learning This section addresses the question of how new knowledge is acquired by problem solvers in this task domain. The approach taken here incorpo rates two of the insights that have been discussed so far. First, the cognitive model indicates that one of the key pieces of knowledge that differentiates novice and expert skill levels is w hether the problem solver know s the preconditions for issuing a command. This knowledge is used for deter m ining the appropriateness of each action, and w ithout it errors arise due to situational factors. The preconditions show n in Figure 4.3 cited illus trate this point: the expert model will not attem pt to load the predicts file while the NCB-Program m ode is RUN. This is a precondition that, w hen 73 IF goal is VLBI & operator is Configure-DSP & state has name(object, NCP-PROGRAM) & state has mode(object, NIDLE) — > Load-Predicts-PCl (Load-Predicts, Satisfied) Figure 4.6 Pseudo-code expert overlooked, leads to erroneous behavior. Second, the effect of compiling a problem solver's existing knowledge into skilled behavior comes from ap plying this knowledge in the act of performing the task, and the end result is an im provem ent in performance. This second observation is a direct re sult of Soar's learning mechanism, and it represents w hat has been charac terized as symbol-level learning as opposed to knowledge-level learning (Rosenbloom et al., 1987; Rosenbloom et al., 1988). These tw o insights w ere used in the cognitive m odel in the following m anner. Knowing that the problem solver needs to acquire know ledge about com m and preconditions, the cognitive m odel sim ulates the mecha nisms for acquiring this knowledge (i.e., bringing the know ledge into the problem solver's memory) at strategic learning points during the perfor mance of a task. The issue of how data is chunked into m em ory is being addressed by a num ber of researchers, e.g., (Rosenbloom et al., 1987; Rosenbloom et al., 1988). The real issue here was in identifying the strate gic learning points w here the problem solver has the goal to acquire new knowledge. From a m odeling perspective, it was helpful to be able to just add knowledge to the problem solver's memory; it enabled us to test the behaviors of problem solvers with differing levels of skill. The strategy for learning in this dom ain focuses on the tim e during w hich the problem solver is recovering from failures, which are caused by know ledge level impasses that are a result of missing or incorrect knowledge. In this section I illustrate how the model recovers from these types of failures using an ex tended example. It is especially im portant to note how the cognitive model acquires new knowledge because it reveals how a tutor could interact to fa cilitate the learning. 4.5.1 Missing knowledge Let us assum e that the problem solver begins the VLBI sub-task called Configure-DSP, shown in Figure 4.2, by applying the Load-Predicts opera tor. Furtherm ore, let us also assume that the problem solver is missing knowledge about one of the Load-Predicts operator's preconditions, Load- Predicts-PCl, shown in Figure 4.3. The Load-Predicts-PCl precondition says 74 that the NCB-Program should be in the IDLE MODE prior to issuing the NLOAD command of the Load-Predicts operator. # Type Event Lop Impasse 1 OD >V NLOAD JK 2 DR > REJECTED. NOT ALLOWED IN NRUN MODE X Figure 4.8 NLOAD command rejected To illustrate w hat happens w hen the problem solver is m issing this precondition, the NCB-Program device is started in the RUN MODE. W hen the problem solver verifies each of its preconditions in the verify- com m and-preconditions problem space (refer to Figure 4.2), it will not no tice the m ode discrepancy in the NCB-Program device since it does know that it m atters. Consequently, it issues the NLOAD com mand, b u t it is re jected by the device as shown below in Figure 4.8. After issuing the NLOAD com m and, the problem solver subgoals into the Verify-Command-Postconditions problem space (Figure 4.2) to deter m ine w hether the command had its desired effects, i.e., it checks w hether the predicts set was loaded. The problem solver discovers that the predicts set was not loaded and marks the postcondition "unsatisfied"; it subgoals into the Attend-to-Unsatisfied-Postcondition problem space w here it selects an operator called attend-to-directive-rejection, which reads the rejection message show n on line 2 in Figure 4.8. This operator has the same func tions as a tutor: it transforms the directive rejection message into a precon dition for the Load-Predicts operator and adds it to the problem solver's knowledge in a declarative form.3 The end result of applying the attend-to- directive-rejection operator is that the problem solver ends u p w ith the Load-Predicts-PCl precondition as show n in Figure 4.3, and the problem solver term inates the Attend-to-Unsatisfied-Postcondition subgoal. W ith the newly acquired knowledge of the Load-Predicts-PCl precondi tion, the problem solver recognizes that it m ust check w hether this pre condition is satisfied, and it once again subgoals into the verify-command- p re c o n d itio n s problem space. Since the NCB-Program is in the RUN 3 The operator does not read the natural language version of the directive rejection, rather, it reads a structured version of the message describing the reason for the failure in terms of a device, an attribute and an expected value. 75 MODE and it is supposed to be in the IDLE MODE, the Load-Predicts-PCl precondition is m arked unsatisfied and the subgoal terminates. The prob lem solver has to deal with the unsatisfied precondition before it can issue the NLOAD command again, so it subgoals into the repair-unsatisfied-pre- conditions problem space, w here it searches for an operator that will change the NCB-Program device from RUN to IDLE MODE. It performs this search by com paring the attribute-value pair of the unsatisfied precon dition w ith the postconditions of other command-level operators it knows about. The success of the problem solver's search for a command operator depends a on how m uch it knows about the postconditions of the candi date operators; note that this is another natural point where a tutor may be able to assist the student. It is not reasonable to expect students to have all of the knowledge they need to conduct this search for a "repair" operator, and in fact, this is the type of situation w here students may potentially m ake faulty repairs, as VanLehn points out in his repair theory of bugs (VanLehn, 1982; VanLehn, 1983). In this case, the problem solver selects a com m and operator called Idle- NCB-Program, which will theoretically have the desired effect on the NCB- Program device's mode. The Idle-NCB-Program operator is analogous to the Load-Predicts operator: it is a command-level operator (see Figure 4.2) and it has the same problem spaces as show n in Figure 4.2 (i.e., verify- com mand-preconditions, etc.) Thus, the problem solver applies this opera tor by verifying that all of its preconditions are satisfied, issuing the NIDLE REC command, and verifying that its postconditions are met. Successfully applied, the Idle-NCB-Program operator terminates, and the problem solv ing activity continues for the Load-Predicts operator. Since the expectation is that the Load-Predicts-PCl precondition will now be satisfied, the problem solver verifies that this is the case in the ver- ify-com m and-preconditions problem space, and it re-issues the NLOAD command. This time the command is accepted and executes properly (see Figure 4.9), allowing the problem solver to m ove on to another command in the Configure-DSP plan. 76 # Type Event Loq Impasse 1 OD >V NLOAD JK > COMPLETED. LOAD OF NCB PREDICTS STARTED ... SEE NEXT... > SET JK LOADED OK (031) 2 DR 3 CA Figure 4.9 NLOAD command accepted and successfully term inated In conclusion, this example highlights the process by w hich the cogni tive m odel acquires new knowledge: (1) the m issing know ledge causes a know ledge level impasse while solving the problem, which is detectable as an error in the form of a directive rejection; (2) it determ ines the cause of the im passe and translates this inform ation into a new operator precondi tion; (3) it applies the new knowledge, detecting the cause of the unsatis fied precondition; (4) it reactively plans and applies an action to resolve the impasse; and (5) it re-applies the operator. 4.5.2 Incorrect knowledge The second w ay that the cognitive m odel acquires new know ledge is in cases where it begins with a misconception about an operator precondition. To illustrate how the cognitive m odel recovers from incorrect knowledge, consider again the case w here the problem solver is perform ing the Configure-DSP plan of the VLBI task (Figure 4.2), but instead of missing a precondition, it has a misconception about the Load-Predicts-PCl precondi tion (Figure 4.3). Instead of believing that the MODE attribute should have a value of IDLE, the problem solver expects the value to be RUN. Hence, w hen the task begins and the NCB-Program device is in the IDLE MODE, it m eans that though Load-Predicts-PCl is actually satisfied, the problem solver erroneously concludes that it is unsatisfied. As a result, it subgoals into the Repair-Unsatisfied-Precondition problem space (Figure 4.2) and is sues a com m and to change the NCB-Program to the RUN MODE to repair the situation. Once the NCB-Program device's MODE has changed to RUN, the prob lem solver believes that Load-Predicts-PCl is satisfied. Given that the other preconditions are satisfied, it issues the NLOAD com m and to the device. But because the device is in the RUN mode, it rejects the com mand since it only accepts this command when it is in the IDLE MODE. 77 W hen the problem solver tries to verify that the postcondition (shown in Figure 4.4) is satisfied and finds that it is not (i.e., it couldn't be since the com m and was never accepted), it subgoals into the Attend-to-Unsatisfied- Postcondition problem space. Here it attends to the rejection message from the device and tries to determ ine a reason for the failure. It reads the mes sage and determines that the NCB-Program w as supposed to be in the IDLE mode. It also notes that the information in the message conflicts w ith one of its ow n preconditions, Load-Predicts-PCl. As a result, the problem solver makes a declarative modification of this precondition, changing it from RUN to IDLE. The problem solver again verifies the Load-Predicts preconditions, this time using the modified version of Load-Predicts-PCl; this time it is unsat isfied, due to its earlier erroneous attem pt to correct the situation that put the NCB-Program into the RUN MODE. Once again the problem solver subgoals into the Repair-Unsatisfied-Preconditions problem space, and this tim e it selects and issues a command that puts the NCB-Program back into the IDLE MODE, which satisfies the corrected precondition and allows the NLOAD com mand to be re-issued. This time the com mand is accepted, the postconditions achieved, and the Load-Predicts goal is successfully term i nated. 4.5.3 Learning w hile recovering from failure These are prim e exam ples of how the cognitive m odel acquires new know ledge about com m and preconditions in this dom ain and integrates this know ledge with its existing skills through the knowledge compilation process.4 N ote that the new knowledge, which in these examples added a m issing precondition and corrected a misconception of a know n precondi tion, was acquired in the context of a knowledge level impasse. The miss ing knowledge was added as a new precondition for the operator. The mis conception w as corrected by providing a new declarative value for the MODE attribute. In both cases, the new knowledge was applied while recov 4 I did not model the acquisition of postconditions in this manner, but it would be a similar process. 78 ering from a problem solving failure; it was com piled into chunks, which w ere built each time the subgoaling occurred. An im portant consequence of this style of learning is that in cases where the problem solver is missing a precondition but the device happens to be in the correct state, the problem solver's skill will end u p being in complete. The problem solver will only acquire the w hole skill during subsequent problem solving episodes w here the hidden precondition causes a failure impasse. 4.6 Results and design desiderata The cognitive m odel just described gives an account of problem solving and skill acquisition in a task dom ain where the problem solver interacts w ith an unpredictable external environm ent. The difference in skill be tween expert and novice problem solvers can be explained in terms of the know ledge of command preconditions and postconditions, but this is only part of the story. ITS researchers have been modeling differences in prob lem solving behavior along these lines for years (e.g., Langley, 1984; Ohlsson, 1988; Burton, 1982). A smaller num ber have also m odeled skill acquisition (Anderson, 1983,1989,1990; VanLehn, 1987). Others have built, or are building, detailed cognitive m odels similar to the one described in this paper. Blake W ard developed a cognitive model in Soar of a student solving electrostatics problems (Ward, 1991). It carefully m odels m any de tails of the problem solving process, such as perception and focus of atten tion; how ever, it does not learn. Ralph M orelli5 is w orking to extend W ard's work, and his m odel does learn. H ow ever, it does not address questions of how misconceptions are acquired or recovered from, nor how tutorial intervention m ight influence the learning process. Conati and Lehman (1993a, 1993b) are modeling problem solving and learning in a mi croworld learning environm ent called "Electric Field Hockey." It can recall failed problem solving attem pts, in order to avoid previous mistakes. H owever, this dom ain is more lim ited than the LMC domain. It does not involve interaction w ith a com plex environm ent during an extended ^Trinity College, Hartford, CT, work in progress 79 problem solving episode. The learning issues that their w ork addresses are largely orthogonal to the issues addressed by this work; the problem solver in that dom ain learns sequences of operations to perform to achieve a goal, whereas in the LMC dom ain the focus is on learning the conditions under which operations are appropriate to be perform ed and the effects that they should be expected to achieve. Two things distinguish this cognitive modeling effort. First, it provides an account of both problem solving and skill acquisition in task domains of this type (i.e. unpredictable and reactive), and second, it uses the model to design and evaluate an intelligent tutoring system for this domain. The rest of this section sum m arizes how the cognitive m odeling results m oti vate a tutoring system design. 4.6.1 Learning by doing Learning involves solving problems. In this cognitive model "learning by doing" is partly a result of the w ay that the Soar chunking m echanism works (Newell, 1990) and partly in the w ay that the problem spaces were im plem ented for recovering from errors. The basic notion of learning in the process of problem solving agrees with Anderson's account of procedu- ralization and skill acquisition (Anderson, 1983). It is not sufficient to ac quire declarative know ledge about a task, rather, know ledge m ust be brought directly to bear on solving problems. The LMC cognitive model only acquires new knowledge in the process of recovering from errors, and its perform ance im proves only as it compiles this know ledge through per formance. This is not to say that the cognitive model could not learn by sim ply acquiring declarative task know ledge, but this know ledge is not compiled until it is applied. A prediction that can be m ade from these observations is that students will acquire the desired skills most effectively by applying their knowledge to realistic problems. In so doing, novices will compile their knowledge by building chunks that sum m arize their problem solving experiences. In addition, novices w ho have com piled their know ledge through practice will perform dom ain tasks better than ones w ho have only acquired declarative knowledge about the task. Practice alone is not expected to help 80 novices become experts, but a system design that supports interactive prob lem solving and realistic situations will likely have positive effects in im proving novice performance. 4.6.2 Learning about preconditions Novice problem solvers tend to follow a procedural script w ithout paying attention to the device situation, but experts appear to evaluate the situa tion before and after each action to ensure that action preconditions and postconditions are satisfied. The experts are observed to m onitor system displays before and after executing m any of the commands, and they always attend to the event log to determ ine w hether their com m ands have been accepted or rejected by the system. Furthermore, the cognitive model illus trated that a lack of know ledge of com m and preconditions leads to the same type of erroneous behavior that was observed in O perator logs, while knowledge of the command preconditions produced behavior that avoided these novice-like errors. One possible explanation for the apparent com pulsiveness of this type of behavior (i.e., explicitly checking for precondi tions and postconditions) is that the environm ent provides a lot of explicit feedback to the O perators, beginning w ith the event log that shows w hether command was accepted or not. Given the relative frequency of di rective rejections, it is not too surprising that the Operators give a lot of at tention to the context in which they take action. Thus, the know ledge of preconditions and postconditions m ust be ac quired in order to become an expert problem solver in the LMC domain. By focusing on how to assist novice problem solvers to perform procedures in the context of a dynamically changing situation, it is expected that the novices' overall perform ance will im prove in term s of m aking fewer er rors in a wide variety of situations. 4.6.3 Rote learning is incom plete As a corollary to the last two results, learning a procedure by rotely execut ing a sequence of actions will lead to incomplete problem solving knowl edge. W hen the cognitive model simulates a novice (i.e., no knowledge of preconditions), it solves problems by taking actions in the order specified in 81 the procedure m anual. W hen this know ledge is chunked the problem solver recognizes w hat step to take based only on the current goal, but it makes errors when the device properties are not correct. It is expected that novices w hose problem solving decisions are based only on procedure m anual action sequences to m ake errors w henever preconditions are not satisfied. 4.6.4 Learning occurs in a goal context Problem solving is a goal-driven activity, and learning occurs in a goal con text. This observation is em bodied in Soar as a cognitive architecture, and it is also reflected in the w ay that the cognitive m odel was constructed for this dom ain. Though the LMC dom ain dem ands reactive problem solv ing, goals drive the search for solutions. Recall that the chunks built dur ing problem solving include goal context inform ation, which enables the problem solver to recognize w hen to apply the chunk in future situations. The central role played by the goal context in learning leads to the fol low ing prediction: tutorial intervention will be m ost effective w hen the student has the goal to resolve a (knowledge level) impasse. This means that the student m ust be aware of an im passe (i.e., an inability to make progress) and then have the goal to resolve it. The tw o problem spaces w here this occurs in the LMC cognitive m odel are Repair-Unsatisfied- Precondition and Attend-to-Unsatisfied-Postcondition; these are the places where the problem solver actively seeks a w ay of resolving an impasse, and this is w here the student will most likely benefit from tutoring, since inter ruption provides information consistent w ith the student's current goal. Conversely, if the student does not have a goal to acquire im passe-re lated inform ation, then tutoring is not likely to be helpful since it may dis rupt the student’ s goal stack by turning attention away from the task. This argues against the frequent interruptions of some early model tracing sys tems caused by a need to inquire about the student's goals, and it also indi cates that w aiting until after the problem solving session to give tutorial feedback m ay be too late; it is preferable to have a tutorial interaction in the context of an impasse. REACT's design has taken these observations 82 into account by focusing on how to recognize and explicate problem solv ing impasses (Hill&Johnson, 1993b). 4.6.5 Learning via impasse Failures in problem solving provide strategic opportunities for learning new preconditions. Impasses change the problem solver's focus from try ing to achieve a task goal to trying to recover from a failure related to the impasse, which is a goal that makes the problem solver amenable to acquir ing new knowledge. Recall that the cognitive model commits errors when it lacks knowledge of preconditions or has misconceptions. Once the prob lem solver detects an anomaly, it recovers by first interpreting the situation and the causes for the failure. Next, it modifies its knowledge by acquiring a declarative precondition for the action, and then it com piles the new know ledge w ith chunking. Each of these recovery steps suggests tutoring intervention opportunities that w ould not disrupt the problem solver's goal context. The cognitive m odel's ability to recover from errors clearly represents an idealized self-tutoring capability. H um an students are not able to tutor themselves as easily as the cognitive model did in the examples described in this paper. Rather, the error recovery process can be viewed as a strate gic opportunity for an intelligent tutor to assist the novice in acquiring new knowledge. The errors that occur in this dom ain result from knowledge level impasses: a lack of knowledge about how to appropriately take an ac tion causes the Operator to fail. Consequently, the design of the intelligent tutor should focus on how to capitalize on error recovery situations. This includes the issues of how to create failure situations, how to recognize failures, how to recognize failure recovery, and how to assist the student in recovering from the error. In the LMC dom ain some errors may not cause an im passe for a signifi cant am ount of time. An action taken w hile perform ing one procedure may not cause an impasse until a m uch later procedure. If the tutor waits until the novice reaches an im passe under these circumstances, then the conditions for learning will not be favorable. The goal context of the origi nal error will be long gone, and the error recovery procedure will poten- 83 tially be convoluted. It may be better to force the student to deal w ith unde tected errors prior to completing a procedure, thus the tutor will have to be capable of detecting errors w ithin the goal context of a procedure rather than w aiting until after a failure impasse. A challenge from a tutoring de sign perspective is how to force the student to deal w ith the error w ithout unduly interfering w ith the student's problem solving activity. 4.6.6 Detecting hidden knowledge gaps Even as the novice's knowledge of preconditions begins to converge w ith the expert level, there m ay still be hidden knowledge gaps that are difficult to detect. This is a case w here the action sequence order elim inates the need to verify that a precondition is satisfied because the results of one ac tion satisfy the preconditions of the next. As long as the actions are always executed in the same order the precondition will be satisfied and the pre condition will remain hidden in the sense that problem solver will not act to verify that it is satisfied. Whereas some actions in a sequence have pre conditions that are independent of the other actions, it is difficult to create device situations that will force the student into a failure impasse. The tu tor has to find ways to detect hidden knowledge gaps, perhaps by forcing the student out of a rote action sequence and into situations where the ac tions are executed in a different order. 4.6.7 Recovery from incorrect know ledge W hen the cognitive m odel recovers from errors it builds new chunks sum m arizing w hat it learned during the failure impasse. If the error was caused by incorrect knowledge then there is a potential conflict between the correct and the incorrect chunks. The Soar cognitive architecture does not excise chunks from recognition memory, so w hat happens to the incorrect knowledge? The current cognitive model does not have a satisfactory an swer to this question. It does not have a problem recovering from incorrect knowledge, but this is mainly due to some fortuitous aspects of the imple m entation: an old chunk will no longer apply after the student has ac quired knowledge of a new precondition because it contains a flag stating th at the stu d en t does not know a precondition. Once the stu d en t 84 "acquires" the new precondition, the flag is set to "known" and a new chunk is built that matches the modified precondition. Building on Laird's approach to recovering from incorrect knowledge (Laird, 1988), it is possible to fix this so that the problem solver learns chunks that give reject prefer ences for incorrect operators; these w ould be learned w hile recovering from failure. H ow ever, to some extent the existing m odel has already served its function: it m ade clear the problem of counteracting incorrectly learned knowledge. It resembles Laird's approach to error recovery in that it disregards the "credit assignment" problem for the error and focuses on adding the know ledge needed to m ake the problem solver behave cor rectly. 4.7 Conclusions: Benefits and lim itations of the cognitive m odel The model described in this chapter makes some clear predictions about how novice Operators acquire skill in the LMC dom ain, and these predic tions have a direct bearing on the design of the tutor. The model has given some answers to the tutorial interaction question: it says that the best time to interact w ith the student is at an impasse point w here the student has a goal to acquire some additional knowledge. The model sim ulated the pro cess of acquiring new know ledge from the outside, though it could have also been acquired through search. The model also gives an indication of one of the types of knowledge that should be tutored, nam ely, the knowl edge of operator preconditions. In addition to addressing the tutorial interaction problem , the model also gives an indication that m odeling the student's behavior has to be flexible and efficient. The need for flexibility stems from the nature of problem solving in the domain. Since the situation may w arrant actions that are not part of the default plans, the tutor needs to be able to recognize w hen this is the case rather than classifying such actions as impasses. The need for efficiency is linked to the idea of interacting w ith the student as close to the im passe point as possible. This requirem ent extends beyond the recognition of the im passe to include the explication of the impasse also, since this is a necessary part of interacting w ith the student. 85 In spite of w hat has been gained from this cognitive m odel, however, there are som e lim itations that have not already been m entioned. The m odel assum es that the O perator has complete access to device state in form ation w hen evaluating preconditions and postconditions. W hile this is theoretically true for actual LMC Operators, the device state inform ation is m ade available only through a large set of system displays that can be in voked from the console. Thus, knowing where to find an item of inform a tion is an aspect of the skill that is partly overlooked by the model; the m odel takes into account w hat the inform ation is in terms of an object at- tribute-value pair, but it does not keep track of where the item is located. This w ould be an interesting extension to the current cognitive model; in fact, a recent study seems to indicate that knowing w here to find inform a tion can differentiate the skill efficiency levels in some dom ains (Vera et al., 1993). This lim itation of the cognitive model m ay not be significant, however, since the procedural descriptions used by the Operators normally include a list of which displays are relevant to the procedure. A nother assum ption m ade by the cognitive model is that the O perator can effectively attend to and use a resource such as a procedure m anual or a procedural sum m ary sheet. This is not a significant concern for the cogni tive model since I assume that all Operators have a basic ability to read and follow directions. These assum ptions are based on the prerequisites for be ing an Operator. Finally, the cognitive m odel does not cover the cases w here the O perator lacks knowledge about plan goals and plan interdependencies. In Chapter 3 I identified these two types of knowledge as necessary for skilled behavior in the LMC dom ain, so it w ould seem desirable to m odel how this knowledge is acquired also. This is not necessary, however, since this type of know ledge resembles w hat has already been m odeled, only at a higher level. In a Soar representation, plans are just higher level operators that have preconditions and postconditions. The preconditions of a plan, in this case, are derived from the precedence relations am ong plans; these were expressed as desirability preferences in the model. The plan postcon ditions are the goals; once a plan goal is achieved, the operator correspond ing to the plan is terminated. The impasses that occur with respect to these 86 types of knowledge have the following observed characteristics. Either they do not m anifest them selves until after the mission is com plete and they show up as a flaw in the scientific data, or they show up as directive rejec tions in other plans, as was the case in the actual event log shown in Figure 3.5. In the form er case the O perator w ould probably never acquire new know ledge related to the error since a noticeable im passe w as never reached. This is obviously not a desirable situation and suggests that the tutor should somehow get the Operator's attention and artificially create an im passe in w hich the m issing know ledge can be acquired. In the latter case, where an impasse does eventually occur, one can see that the student m ight eventually acquire the higher level knowledge corresponding to the plan precedence or the plan goal, but that it m ay require a trem endous am ount of backtracking, as was the case in Figure 3.5. A m uch more effi cient way of acquiring this knowledge, however, w ould be for the tutor to interact w ith the student prior to their becoming m ired in a long string of impasses. This w ould again involve having the tutor somehow indicate to the student that a potential im passe has occurred, thereby inducing the student to have the goal to attend to and resolve it. This is the basic strat egy for recognizing impasses and interacting w ith the student that was cho sen and will be described in the next chapter. 87 Chapter 5 Situated Plan Attribution The first step tow ard addressing the tutorial interaction problem is being able to recognize the best tim e to interact w ith a student during a problem solving session. One of the major conclusions draw n from the cognitive m odeling w ork in the last section is that it is best to interact with the stu dent near an impasse point. Consequently, the focus of the tutor's student m odeling efforts should be on detecting impasses. In this section I describe a new m ethod for student m odeling called sit uated plan attribution. This approach enables the tutor to recognize vari ous types of impasses w ithout attem pting to trace the mental states that led to the student's behavior, w hich is how the m odel tracing approach models the student (Anderson et al., 1990). Situated plan attribution also contrasts w ith plan recognition approaches to student m odeling, in which every action m ust be accounted for, either by plan m atching or through parsing w ith an action grammar. The situated plan attribution approach does not presuppose how a student will execute a particular plan; it allows for the need to reactively adjust a plan in response to the state of the world. Though plan execution requires reactivity, this approach also takes into account the goal-oriented nature of tasks in a real-time operations domain; plans have goals, regardless of how the actions are executed. The tutor attributes plans to the student based on a high-level description of the task called a tem poral dependency network (TDN), which defines a partial order am ong the plans. This section, which is organized into three parts, makes the following points: it (1) presents some assumptions about plans and actions and how they influenced the decision to use im passe recognition to drive the tutor ing intervention; (2) describes how situated plan attribution creates expec tations about behavior and provides a context for intervention; and (3) I describes how the tutor recognizes impasses. i I 88 5.1 A ssum ptions about plans and action Situated plan attribution is based on a key assum ption: plans do not de term ine behavior (Suchman, 1987). As was show n in the cognitive m odel ing work, plans influence behavior, but they act prim arily as a resource to help guide the performance of the task. Consequently, plans are useful for creating expectations about behavior, but they cannot always be used to understand and explain it. For instance, some situations call for reactive planning, resulting in actions that, w hen observed by a tutor, do not fit into the default plan. The tutor needs to be able to recognize w hether the student's actions are appropriate in these situations, but it m ay be very dif ficult to do so if the action has to be recognized and explained in terms of a known plan. A strict plan recognition approach to understanding situated behavior would require a library of all plans for all situations. For the LMC dom ain, there is a complete set of default plans for per form ing a m ission, b u t it w ould be im practical to represent all of the variations of the plans that w ould be needed to account for situational differences that arise from the dynamic nature of the problem domain. For instance, in the situation that was described in Chapter 3 for the example show n in Figure 3.4, a variation of the Configure-DSP plan w ould have to include the NIDLE REC command to cover the situations where the Load- Predicts-PCl precondition (Figure 4.3) is not satisfied (i.e., when the NCB- Program is in the RUN MODE instead of the NIDLE MODE.) The same w ould hold true for every precondition of every operator in the plan: the situation could potentially force the Operator to deviate from the standard plan every time a precondition was not satisfied, m eaning that there would potentially have to be a plan variant for each such situation. This viewpoint led to viewing plans as a w ay of providing only partial understanding of the student's behavior; more specifically, plans provide a fram ew ork for recognizing w hen the student has reached a potential impasse. Since impasses m ay result from a com bination of student m is conceptions and anom alous device states, we seek to understand the im passe using an attributed plan as a starting point, but not as the only means of understanding student behavior. Besides plans, the tutor also has access to several other resources for interpreting student behavior: it sees the state 89 of the devices (i.e., situation in which the action is taken), it tracks the state of the goals associated w ith the attributed plans, and it sees the student's actions and the device's response to them . This approach shifts the com putational burden aw ay from giving a plan-based account of a stu dent's behavior and tow ard using plans as one of several resources for the task of recognizing and explaining an impasse. Plans are a convenient way of organizing knowledge about goals and actions related to the task; they are a declarative description of how to perform the task as opposed to being an executable model, and they are m eant to orient the tutor rather than giving a complete account of how to behave in the current situation, which is consistent with Suchman's view of plans and situated action. An additional resource for detecting certain types of im passes is the sim ulator itself. The sim ulator rejects directives1 whenever they violate a system -im posed constraint.2 Directive rejections are cues used by the tutor in deciding w hether to intervene: they elim inate the need to guess w hether there is an impasse in these instances. Instead, the tutor only has to note that the directive was rejected and then determ ine the causes for the im passe and resolve it. This eliminates the need to understand every student action, which is norm ally a requirem ent for both plan recognition and model tracing approaches to tutoring. 5.2 Situated plan attribution The prim ary purpose of situated plan attribution is to generate expectations about O perator behavior so as to guide the impasse recognition process. The w ay that REACT performs situated plan attribution is explained in the follow ing m anner. First, I discuss how plans, goals, and action are represented. This discussion describes the roles of these interpretation re sources w ith a m inimal am ount of explanation telling how their attribute values are initialized and modified. One of the key features of this repre sentation is that the order am ong a plan's actions are not explicitly repre sented and considered w hen interpreting a student's behavior. 1 Directives are synonymous with commands. 2 The simulator is faithful to the actual link monitor and control system in the way that it accepts or rejects directives. 90 After discussing how individual plans are represented, I discuss how plans are collectively used to describe a mission through the use of a tem poral dependency network (TDN). The TDN drives the tutor's expectation about which plans the student should be executing at any given time. W ith this representational foundation, I describe how REACT im ple m ents the situated plan attribution approach to im passe recognition in a Soar problem space hierarchy. 5.2.1 The representation of plans, goals and action Each mission has a set of plans associated w ith it that will, under ideal cir cumstances, achieve the mission's goals. Figure 5.1 lists the plans for the VLBI Delta DOR mission. For the purpose of situated plan attribution, each plan has a nam e and three attributes: state, execution status and goal status. These three attributes are m onitored and updated by REACT as it observes the student performing the task. I will now discuss the m eaning of each of these attributes and their usage in situated plan attribution. VLBI Delta DOR Mission Plan Name State Execution Status Goal Status Confiqure-Microwave INACTIVE UNFINISHED NULL Initialize-PPM INACTIVE UNFINISHED NULL Initialize-Receiver-Exciter INACTIVE UNFINISHED NULL Confiqure-Antenna INACTIVE UNFINISHED NULL Conf iq u re-Receiver-Excit er INACTIVE UNFINISHED NULL Confiqure-DSP INACTIVE UNFINISHED NULL Confiqure-PPM INACTIVE UNFINISHED NULL Coherence-Test INACTIVE UNFINISHED NULL Figure 5.1 Plans and their attributes 5.2.1.1 Plan state attribute The State attribute indicates whether REACT expects the student to be exe cuting the plan or not. The plan's state is based on the current situation and a description of the overall m ission. A plan is ACTIVE w hen its execution is w arranted, and it is INACTIVE otherwise. Once its state is set, it influences how REACT interprets the student's actions. As will be show n later, for instance, if a student takes an action that is inconsistent w ith all of the active plans but is consistent w ith an inactive plan, then it m ay indicate that there is an impasse in progress. Likewise, the state of a 91 plan also influences whether its execution and goal statuses are monitored; the tutor only monitors and updates these plan attributes w hen the plan is in an active state. 5.2.1.2 Plan execution status attribute The plan attribute called Execution Status is used to keep track of w hether the student has com pleted the execution of a plan. Each plan contains a set of operators that the tutor expects to be used in the course of executing the plan. Figure 5.2 shows the list of operators for the Configure- DSP plan. Each of a plan's operators also has a set of attributes. Configure-DSP Plan Operators Operator Name Command Execution Status Load-Predicts NLOAD NULL Set-S AT-Value SAT NULL Set-X AT-Value XAT NULL Set-Temperature NTOP NULL Select-Recorder NRMED NULL Set-Offset OFST NULL Figure 5.2 Plans contain operators, which have attributes Besides its name, an operator has a command associated w ith it and an execution status. The com m and attribute value is the observable action that is matched with the active plans' operators. Once the com mand is ob served, the o p erato r’s execution status is changed from NULL to MATCHED. Thus, the meaning of a plan's execution status is based only on w hether all of its operator commands have been observed. Once they have been observed, the execution status is changed from UNFINISHED to COMPLETED. It is im portant to note that in this representation of a plan and its operators, no order is im posed or specified. Even though in terdependencies may exist am ong a plan's operators, they are not explicitly taken into consideration while m atching actions to plans. The ramifica tions of this representational feature is that it does impose an order on the tutor’ s expectation of how the actions will be executed. This is m ade feasi ble by the fact that each operator has a set of preconditions; the precondi tions define an implicit order on the execution of a plan. If a com m and’ s preconditions are not satisfied before the student issues it, then it is likely 92 that the sim ulator will reject the command and the tutor can then explicate it. In the cases w here the com mand is not rejected, the m is-ordering of commands, if it matters, will show up as a goal failure for the plan. These issues are discussed in greater detail in the section describing the REACT problem spaces that im plem ent the situated plan attribution approach to im passe recognition. 5.2.1.3 Plan goal status attribute The plan attribute called goal status, reflects w hether or not the student has achieved the plan's goals. This attribute is a high-level sum m ary of all of the plans goals and its value can be NULL, UNSAT and SAT. Note that in this scheme, a plan's goals can be satisfied w ithout completely executing the plan. Likewise, a completely executed plan may not achieve the goals of the plan. Configure-DSP Goals Device Type Device Name Attribute Value Status VLBI-PREDICTS JK LOADED? YES NULL SYNTH-CHANNEL DCS01 FREQUENCY 300.21 NULL SYNTH-CHANNEL DCS01 DWELL-TIME 0.4 NULL SYNTH-CHANNEL DCS01 LO 2000.0 NULL CON FIG-TABLE CONFIG-TABLE SAT-VALUE 30 NULL CON FIG-TABLE CONFIG-TABLE RECORDER LD O NULL CONFIG-TABLE CONFIG-TABLE OFST-VALUE 2.7 NULL RECORDER LDO MODE ENAB NULL Figure 5.3 Plans have goals The tutor continually m onitors each active plan's overall goal status. Once all of a plan's individual goals are achieved, it changes the plan’ s goal status from NULL or UNSAT to SAT. The goal status is set to UNSAT in cases where the plan's execution status is COMPLETE but the goals are not all individually satisfied. The goal status is initialized to NULL, w hich m eans that the plan's goals are not completely satisfied, though they may be partially satisfied. Figure 5.3 shows a sample of the goals associated w ith the Configure-DSP plan. Note that each goal is associated w ith a device, which has a type and nam e (i.e., a unique identifier.) The attribute in each goal description 93 corresponds to an attribute of the device in question, and the value is a value the device’ s attribute m ust have in order to achieve the goal. Each goal of an active plan is independently and continually m onitored for success by REACT. Once the tutor perceives the achievement of a goal, it changes the goal's status attribute value from NULL or UNSAT to SAT. Once a goal has been satisfied it is still m onitored by the tutor in case the student does something to the device to dissatisfy the goal. W hen the tutor detects such a state change, it marks the goal UNSAT. 5.2.2 O rder among plans: tem poral dependency netw ork In the last section I discussed how a plan's state, active or inactive, influ ences the tutor's interpretation of the student's actions. This attribute is used to express which plans the tutor expects the student to be performing at any given m oment. The source of inform ation for deciding a plan's state is a structure a Temporal Dependency Network (Cooper, 1991; Cooper et al., 1991; Fayyad&Cooper, 1992; Hill&Lee, 1992). A TDN describes a partial order am ong a mission's plans. The goal of the TDN is to express the order in a w ay that maximizes the concurrency of plan execution. Figure 5.4 shows the TDN that defines the partial order am ong the plans from Figure 5.1. Two plans can be executed concurrently if their com m ands may be interleaved in the command sequence w ithout harm ful interaction. Plans that have dependencies are placed in succession to one another. Figure 5.4 Partial Temporal Dependency Network for VLBI Task The TDN resembles a procedure net (Sacerdoti, 1975; Sacerdoti, 1977). From a mission-level perspective it describes a class of plans that w ould theoretically achieve the task goals. But unlike the procedure net approach Initialize' Initialize-PPM Exciter Configure-Receh— c —l ‘~" Configure-Microwave^-initialize-Receiver-Exciter -H;--Configure-DSP Coherence-Test Configure-PPM Configure-Antenna 94 to tutoring (Chen et al., 1991; Rickel, 1988; W arriner, 1990), w e do not use the TDN as the sole basis for interpreting student action and providing tutoring. One difference from the procedure net representation is the granularity of the descriptions: the TDN specifies the relationship am ong plans, w hile the procedure nets used in the aforem entioned tutoring systems represent these relationships and constraints at the action level. The usage of the TDN also loosens the procedure net assum ption that ac tions are plan-based. W hereas the procedure net provides an action gram m ar for deciding on whether an action is appropriate or not, the TDN description is used to orient the impasse recognition process, which will be described later. Given a m ission, the REACT uses the m ission's TDN to determ ine w hich plans are eligible for execution, and it m arks their state attribute w ith a value of active, as was previously discussed. A plan is eligible for execution (i.e., to be active) w hen all of its im m ediate predecessors in the TDN have a goal status of SAT, m eaning that the goals of these plans have all been achieved. Thus, the Configure-PPM plan show n in Figure 5.4 w ould be eligible for execution once the Initialize-PPM and Initialize- Receiver-Exciter plans were successfully completed. These plans are con sidered to be inactive once their goals have been achieved. Figure 5.4 also illustrates how m ultiple plans m ay be concurrently ac tive. N ote that the plans Initialize-PPM , Initialize-Receiver-Exciter and Configure-Antenna m ay all be active at the same time, m eaning that the tutor expects the O perator to interleave actions from these plans w hile perform ing the mission. Device: DCS01 Name I Type | Channel | Frequency | Dwell | LO DCSOI I SYNTH-CHANNEL I 1 I 303.20 I 0.2 I 2000.0 Figure 5.5 Device model example 5.2.3 Representing the situation Besides using resources like plans, goals and action, REACT also perceives the real w orld and uses this inform ation to reason about the student's behavior. The real world in this case is the LMC sim ulator w ith which the 95 student interacts through the user interface. REACT perceives all of the sim ulated devices, w hich it internally represents as objects having an identifier and a set of attribute values to represent state. Hence, the tutor's internal m odel of the w orld contains inform ation such as is show n in Figure 5.5. REACT's internal model of the w orld changes as quickly as it perceives the changes to device attribute values. Inputs to Soar are read at the be ginning of each Soar elaboration cycle. When an object is perceived for the first time, REACT has an operator nam ed perceive-object that creates an object in the internal model with the object's attribute values. Subsequent changes to any of the object’ s attributes values are autom atically perceived during the input cycle and changed in the internal model accordingly. 5.2.4 Problem space description of situated plan attribution N ow that I have described the knowledge representation used for situated plan attribution, I will describe the com putational mechanisms that actu ally perform this processing. Just as Soar (Laird et al., 1987; Newell, 1990; Laird et al., 1990) provided the problem solving architecture for the cognitive model described in Chapter 4, REACT is also im plem ented in Soar. There are three Soar problem spaces used to im plem ent REACT's situated plan attribution approach to im passe recognition: the top-level problem space, the analyze-action-and-response problem space, and the evaluate-plan problem space. The triangles in Figure 5.6 represent these problem spaces. The reader m ay recall the description of the Soar architecture that was given in Chapter 4, w hich highlighted a num ber of the useful m odeling features provided by Soar. For the purpose of the understanding the rem ainder of this chapter, it is useful to begin by reviewing some of the key concepts in Soar. A Soar problem space consists of a collection of operators and states. O perators are proposed, selected and applied to the current state; the application of an operator changes the state, w hich m ay result in other operators being proposed, selected and applied. A Soar im passe occurs w hen the problem solving stops m aking progress; the architecture 96 supports the creation of a subgoal to resolve the im passe by selecting another problem space w here there are operators that can continue to m ake progress on the task. W hen the subgoal problem solving is successful, its results are saved in new productions created by the Soar chunking m echanism , w hich also traces and saves all of the conditions that led to the impasse in the first place so that the next time the impasse conditions occur the chunk will be applied instead of having to subgoal to search for a solution. Analyze-Action-Response Problem Space Figure 5.6 Problem space hierarchy for situated plan attribution Each of the REACT problem spaces used to im plem ent situated plan attribution has a set of operators, which are denoted by bullets in Figure 5.6 and are show n to the right of the problem space triangles. The top-level problem space operators are used prim arily for perception processing. For instance, the perceive-object operator is used to add external objects to • wait • perceive-object • recognize-desired-results • recognize-undesired-results • recognize-goal-completion • recognize-plan-completion f analyze-action-response • evaluate-plan REACT Tutor: Top-level Problem Space for Situated Plan Attribution • detect Evaluate-Plan Problem Soace match-completed-directive-wrth-active-plan match-rejected-direct ive-with-active-plan match-completed-directive-with-inactive-plan match-rejected-directive-with-inactive-plan • deactivate-SAT-plan • identify-candidate-plans • cull-candidates • activate-plans • detect-complete&UNSAT-plan 97 o REACT's internal model of the world. Once added, any changes to the ob jects are automatically updated in REACT's internal model. The recognize-desired-results o p erato r m onitors the successful achievem ent of the individual goals of all active plans. Every tim e the student sends a com m and to the sim ulator and a device changes state, these state changes are observed by REACT. If the state of a device matches w ith one of an active plan's goal states, then this operator is selected and applied to the individual goal status attribute (see Figure 5.3) to update its value to SAT. Conversely, the recognize-undesired-results operator m on itors individual goals of active plans once they have been satisfied to in sure that they continue to be satisfied. If a goal that was previously satisfied becomes unsatisfied, this operator changes its evaluation to UNSAT. Both of these goal m onitoring operators are located in the top-level problem space so that they will continually monitor the student's task progress with respect to achieving the goals of each of the active plans. The recognize-goal-com pletion operator, w hich is also located in the top-level problem space, m onitors the goal achievem ent of entire active plans. As soon as all of a plan’s individual goals are satisfied, this operator updates the plan's goal status attribute w ith a SAT value. Likewise, the recognize-plan-completion operator perceives w hen all of an active plan's actions have been taken. This is the operator that sets the plan's execution status attribute to a value of COMPLETE. The next two operators, analyze-action-response and evaluate-com - pleted-plan, both create subgoals to problem spaces that are used for doing specialized processing related to impasse recognition. The analyze-action- response operator is selected w hen REACT perceives a directive and re sponse pair from the simulator, which means that the student has taken an action and the sim ulator has responded to it. Once this operator has been selected and a subgoal formed, the analyze-action-response problem space is selected, w hich has a num ber of operators that recognize different scenarios. The four operators in this problem space, show n in Figure 5.6, are used both for interpreting w hat the student is doing w ith respect to w hat is expected and for recognizing two different types of impasse: the ac tion constraint violation and the plan-dependency violation. These im 98 passes will be explained in greater detail in the next section, but for now it suffices to point out that the type of im passe that is detected depends on w hether the observed action matches w ith an active or inactive plan. Of course, this relates back to the issue of situated plan attribution and the ex pectations that it generates about the student at the time the actions are ob served. Regardless of w hether an im passe is detected, the action is also used to update its operator execution status in the plan to w hich it is m atched (refer to Figure 5.2.) The purpose of the evaluate-plan operator is to compare w hat the stu dent has done to w hat has actually been achieved. U p to this point the student's actions have been continually tracked by the analyze-action-re sponse and recognize-completed-plan operators. The achievement of goals has been tracked by the recognize-desired-results, recognized-undesired- results and recognize-goal-completion operators, which have m ade their evaluations w ith respect to the state of the world. Once the student has executed all of a plan's actions, it is necessary to also determ ine w hether the goals of the plan have also been achieved. If they have been achieved,then the tutor adjusts its expectations about w hat plan(s) the stu dent will perform next, and if they have not been achieved, then a poten tial impasse, called a goal failure impasse, is suspected. Thus, one way that the evaluate-plan operator is proposed is w hen it detects that a plan's execution status has just been updated to a value of COMPLETE by the recognize-plan-completion operator. Once selected, this operator creates a subgoal and the evaluate-plan problem space is selected, show n in Figure 5.6, w here the plan is evaluated w ith respect to its goal status. If the goal status is SAT, then the plan is deactivated (i.e., its state is set to inactive), and the TDN is evaluated to determ ine w hether any plans should be activated. On the other hand, if the plan's goal status is NULL at this point, then a goal failure im passe is suspected, and the plan's goal status is changed to a value of UNSAT. A second w ay that the evaluate-plan operator m ay be proposed is when a plan's goals have been completely satisfied, regardless of w hether all of the expected actions have been taken. Once in the evaluate-plan problem space, the deactivate-SAT-plan operator is used on the plan w hose goals 99 were satisfied to remove it from active plan set and place it in the inactive plans. The effect of deactivating the plan is that it changes the tutor's ex pectations about the student's behavior. The tutor no longer expects to ob serve actions that belong to the now inactive plan. The tutor's expectations are further changed w hen a plan's goals are satisfied in another way. Once a plan has been successfully terminated, the plan's successors m ay potentially become active, m eaning that the tutor could expect to see actions belonging to these plans if they are activated. Therefore, every tim e a plan's goals are satisfied, the identify-candidate- plans operator proposes a list of successor plans to the satisfied plan. For example, if the goals of the Initialize-Receiver-Exdter plan shown in Figure 5.4 were satisfied, the plans that w ould be proposed for activation w ould include Configure-Receiver-Exciter, Configure-DSP, and Configure-PPM. But since the TDN also describes a set of activation constraints, these plans w ould only be activated if the Initialize-PPM plan's goals w ere also satisfied. The cull-candidates operator removes candidates that have pre decessor plans w ith unsatisfied goals. The plans that rem ain are m ade ac tive by the activate-plans operator. 5.2.5 The processing cycle with no student impasses To further elucidate the process by which the tutor m onitors the student, let us trace through an example of how these problem spaces operate while observing the student and the simulator. 5.2.5.1 Waiting for input Soar runs in a two phase cycle: there is an elaboration phase followed by a decision phase. There is an operator called w ait in the top-level problem space that is repeatedly proposed and retracted while REACT is waiting for input from the sim ulator. This operator does not do anything, but it per m its Soar to continue cycling until som ething m ore interesting happens that will cause other operators to be proposed and applied. 100 5.2.5.2 Device state changes The sim ulator changes the state of its devices in response to com m ands sent by the student. These device state changes are detected by the Soar problem solver (i.e., REACT) at the beginning of its elaboration cycle as in put. The initial tim e that REACT perceives an object in the sim ulator, the perceive-object operator creates an internal representation of the object, as was previously described. Any changes to the external object's attribute values are automatically updated in the internal model, once the changes are perceived during an input cycle. In response to a change in the internal model, the tw o operators, rec- ognize-desired-results and recognize-goal-completion may be proposed and applied. The former is proposed if a device's state change matches a goal state; on some occasions this operator m ay be applied num erous times since a single action may result in a lot of state changes. The latter operator is proposed if all of the goals are now satisfied for a particular plan. 5.2.5.3 Processing a student action The sim ulator responds to the student's directive w ith a directive response message. This action-response pair is perceived by REACT and is added to an internal model of the event log. The analyze-action-response operator is proposed w hen there are new event log items, and it subgoals into the analyze-action-response problem space. Given that the student has taken an action that is correct, both in terms of w hat plan should be executed and w hat is allowable in the current situation, the match-completed-directive- with-active-plan operator will be proposed and applied. After annotating the appropriate plan directive, this operator and its supergoal terminates, and REACT goes back into a wait mode. 5.3 Im passe recognition Given this description of how situated plan attribution works, w e can now turn to the purposes for which it is being employed: impasse recognition. I have already introduced the notion of an impasse, both in Chapter 1 and Chapter 4, but now I will reify this concept in the context of the REACT tu tor. 101 You will recall that the notion of an im passe is not new: it has been used in repair theory to help explain procedural errors in subtraction [Brown and VanLehn, 1980] and as a m otivation for subgoaling during problem solving (e.g., the Soar architecture (Laird et al., 1987).) The defini tion of an im passe is consistent with these uses of the term: impasses are obstacles to successfully perform ing a procedure, w here the obstacle is caused by a lack of knowledge or misconception about w hat to do next in a task situation. The impasses recognized by REACT fall into three categories: action constraint violations, goal failure, and plan dependency violations. Impasses in the first category, action constraint violations, are usually easy to recognize, both by the student and the tutor, because the device simula tor gives an indication w hen a system constraint has been violated. Im passes in the other tw o categories are different from the action con straint violations in that they are potential rather than actual impasses. They do not become actual impasses until the student notices that there is a problem, which m ay not be for a significant am ount of tim e in these in stances. Since I hypothesize that it is desirable for the tutor to intervene as near to an impasse as possible, REACT can use this knowledge of potential impasses to create a tutoring opportunity, either by forcing an impasse by directly m anipulating the sim ulator, or through som e other indication in the user interface that will signal that there is a potential problem, while m inim izing the level of disruption to the student's problem solving. One of the significant aspects of this approach is that it does not attem pt to simulate the mental states that led to the impasse. Instead, it depends on the device to detect m any of the action constraint violations, while it uses it know ledge about plans, goals and the situation to detect the other impasses. The rem ainder of this section describes how the tutor recognizes im passes of each type. To facilitate this discussion, I will revisit some of the previously used examples from Chapter 3 and explain how REACT's prob lem spaces for situated plan attribution actually detect the impasses de scribed there. 102 5.3.1 Action constraint violations The tutor recognizes action constraint violations by w atching the com m unications link sim ulator. Though I have described several different operators that watch the simulator, the prim ary operator used for detecting action constraint violations is the analyze-action-response operator. Let consider how this operator detects this type of impasse in the context of an example. # Type E vent Log Configure DSP Coherence Test Utility 1 OD >V NLOAD JK X 2 DR > REJECTED. NO PR FOUND FOR THIS SET Figure 5.7 Example of an action constraint violation The exam ple in Figure 5.7 shows a case where the student has issued the NLOAD directive and it has been rejected by the simulator. N ote that the sim ulator is a faithful representation of the link m onitor and control system: it rejects com m ands when action constraints are violated. These constraints constitute a subset of the operator preconditions. Once the message on line 2 has been issued by the sim ulator, REACT will perceive it and the analyze-action-response operator will subgoal into the analyze-action-response problem space to process it. The tutor is, in ef fect, reading the same rejection message that the student receives, and de term ining how the student's action fits into its expectations for the current situation. Once in the analyze-action-response problem space, the operator called m atch-rejected-directive-with-active-plan is proposed since the directive w as rejected and because the directive m atches w ith an active plan, Configure-DSP. This operator sim ply annotates REACT's internal copy of the event log entry to indicate that there is an action constraint im passe at that point in REACT's internal copy of the event log. The subgoal is term inated at this point and processing continues at the top-level. The process involved w ith resolving this im passe will be explained in great detail in the next chapter, but it is w orth noting that there is an impasse explication operator called resolve-action-constraint-im passe that detects the event log annotation declaring there is an im passe. This operator 103 creates a subgoal to resolve and explain the im passe, all of w hich it attem pts to do before the student performs another action. In conclusion, recognizing an action constraint im passe is m ade very sim ple by the fact that the sim ulator detects m ost of these kinds of errors. Thus, recognizing this type of im passe does not depend on a highly de tailed cognitive model of the student or of the student’s plans. It isn't a straight case of recognizing a directive rejection, however, since there are instances w here the directive that is rejected belongs to an inactive plan; this im passe, though it is obviously an action constraint violation, is treated differently since it can also be categorized as a plan dependency vio lation. Situated plan attribution provides the right am ount of detail to categorize impasses of these types into different categories, w ithout having the b u rd en of having to determ ine th at an action violates a set of constraints, which is w hat the simulator does. At the same time it does not place the burden of im passe diagnosis and recovery on the sim ulator, which only needs to recognize w hether an action can be taken given the current state of the devices being simulated. 5.3.2 Goal failure The idea behind a goal failure impasse is that students m ay execute a pro cedure w ithout understanding the goals associated w ith it. A goal failure violation occurs when the student completes a plan (i.e., all of the expected actions have been observed for an active plan) but does not satisfy the plan's goals prior to beginning a successor plan. This type of error m ay not m anifest itself in an obvious form, such as w ith a directive rejection, therefore, a student may not recognize the im passe immediately. Instead, a goal failure impasse w ould likely show up later as another type of impasse in a subsequent procedure. Allowed to pass w ithout tutorial interaction, a goal failure im passe could be very difficult for the tutor to diagnose and resolve since the original context of the im passe may have been lost. O ur intuition is that it is better to deal w ith these types of errors as they are recognized by the tutor rather than w ait until it becomes an actual impasse at the action or task goal level (i.e., task failure). 104 Goal failure impasses are recognized using the expectations about ac tions, plans and goals generated by situated plan attribution. The tutor is initially cued to this type of impasse w hen the student takes an action be longing to an inactive plan. If the inactive plan is a successor of an active plan that is complete but whose goals are not satisfied, then the tutor flags the action as a goal failure impasse, which triggers the tutor's cognitive model to diagnose and recover from the impasse in the current situation. This idea is im plem ented in the problem spaces show n in Figure 5.5, which have already been described in this Chapter. The sequence of opera tors that leads to detecting a goal failure impasse can be illustrated with the example in Figure 5.8. The analyze-action-response operator annotates an active plan for each observed action matching an operator in that plan. In the example, the Configure-DSP plan is active and the Coherence-Test is inactive. Thus, each of the event log entries on lines 1 through 12 are pro cessed and m atched to the Configure-DSP plan by the analyze-action-re sponse operator. # Type Event Log Configure DSP Coherence Test Utility 1 OD >V NLOAD JK X 2 DR > COMPLETED. 3 OD > V NRMED L D O X 4 DR > COMPLETED. 5 OD > V SAT 15 X 6 DR > COMPLETED. 7 OD > VXAT13 X 8 DR > COMPLETED. 9 OD > V NTOP 20.0 30.0 X 10 DR > COMPLETED. 11 OD >V OFST 2.5 X 12 DR > COMPLETED. 13 OD > V NPCG MAN X 14 DR > COMPLETED. Figure 5.8 Example of a goal failure impasse Once the OFST command on line 11 was m atched to the plan, the rec- ognize-plan-completion operator is proposed because it observes that all of the actions in Configure-DSP have been taken. When applied, the operator annotates the plan’s execution status w ith a COMPLETE. This causes the evaluate-com pleted-plan operator to create a subgoal to analyze the 105 com pleted plan. Either the deactivate-complete&SAT-pIan operator or the detect-complete&UNSAT-plan operator will be proposed for a given plan, depending on whether or not the plan's goal status is SAT or NULL. In the example, the Configure-DSP plan's goal status is NULL because one of its goals was not achieved, nam ely, the OFST value should have been set to 2.8, not 2.5. (Note: This error was not detected as an action constraint violation because the device itself will accept an incorrect param eter and REACT does not evaluate w hether all of a directive's preconditions are satisfied unless the device rejects the directive first.) REACT applies the detect-complete&UNSAT-plan operator, w hich annotates the Configure- DSP plan’s goal status w ith an UNSAT value. Although the Configure-DSP plan’ s execution status is COMPLETE, its goal status is UNSAT, so the plan is not deactivated, which also means that none of the subsequent plans will be activated yet. The consequence of this is that when the student issues the NPCG command shown on line 13, this action is m atched w ith the Coherence-Test plan, w hich is still inactive. Thus, the NPCG command is initially interpreted to be a plan dependency violation, but this turns out to be the final step in recognizing a goal failure impasse. An operator nam ed resolve-goal-failure-impasse recognizes that w hen the student takes an action that belongs to a plan subsequent to an active plan whose goals are UNSAT, then there is a goal failure impasse. 5.3.3 Plan dependency violations The reason for classifying certain behaviors as plan dependency violations is to aid the student w ho has an incorrect understanding of the order in which plans m ay be executed and is therefore confused about which plans are applicable for a given situation. The TDN describes the precedence con straints am ong plans. Thus, if the tutor observes the student executing a plan that is clearly inappropriate for the situation, it m ust be capable of providing tutorial advice. Consider how this type of im passe is detected in the context of the fol lowing example, shown in Figure 5.9. In this case the student begins by is suing the NLOAD com mand, but imm ediately follows it w ith the NRUN com m and, w ithout first sending all of the other commands in Configure- 106 DSP. The NRUN com m and is rejected, since there are several commands in the Configure-DSP plan that work tow ard satisfying its preconditions. # Type Event Log Configure DSP Coherence Test Utility 1 OD > V NLOAD JK X 2 DR > COMPLETED. 3 OD V NRUN COLD X 4 DR > REJECTED. Figure 5.9 Example of a plan dependency violation impasse REACT determ ines that NRUN is a plan dependency violation in the analyze-action-response problem space. The match-rejected-directive-with- inactive-plan operator is proposed for the event log entries on lines 3 and 4. Even though this com m and can be interpreted as an action constraint violation, since it belongs to an inactive plan, this is treated as a plan dependency violation instead, and the operator annotates the event log as such. Due to the situated nature of tasks in this dom ain, however, the tutor does not autom atically assume a com m and that does not m atch an active plan is a plan dependency violation. The tutor evaluates the com m and with respect to the com mand preconditions and goals of active plans. If it appears that the student was attem pting to correct an anomaly or an unsat isfied precondition, then the tutor does not treat the action as a plan de pendency violation. # Type Event Log Configure DSP Coherence Test Utility 1 OD > V NIDLE REC X 2 DR > COMPLETED. 3. OD >V NLOAD JK X 4 DR > COMPLETED. Figure 5.10 Example of situated plan attribution allowing a plan de pendency violation A good example of this is shown in Figure 5.10. In this instance, the student issued the NIDLE REC as the first com m and of the sequence. In the analyze-action-response problem space the tutor proposes the match- com pleted-directive-with-inactive-plan operator, because it recognizes that 107 this com m and does not belong to the Configure-DSP plan, w hich is cur rently active. This operator m arks the event log entry as a plan-depen- dency-violation initially. Later, however, w hen the tutor is explicating this impasse, it will change its treatm ent of this event log entry. Since NIDLE, in this case, satisfies one of the NLOAD preconditions (i.e., it puts the NCB- program into the IDLE mode), the tutor accepts the action. Consequently, the tutor does not interact w ith the student over w hat could have been confused as an impasse because the command clearly served a purpose in the current situation and the O perator was able to avoid a rejection of the NLOAD command. This is the weakest category of impasse because it is the m ost difficult to correctly recognize. It requires being able to understand actions that do not fit into an expected behavior. The next extension to the tu to r will be to recognize and use severe device anomalies or failures to drive the model of expectation and situated plan attribution. The current effort focuses m ore on norm al operations w here the range of anom alies can be accom m odated by fairly simple reactive plans or knowledge. 5.4 Com parison to other approaches This chapter describes how REACT addresses the problem of deciding w hen to interact w ith the student during a problem solving session. I claim ed that being able to decide w hen to interact is the first step tow ard addressing the tutorial interaction problem. The approach that is used in REACT depends on being able to detect w hen an im passe has occurred. The approach I have described in this chapter for recognizing impasses is called situated plan attribution. The best w ay to compare situated plan at tribution to other approaches is on the basis of how each approach ad dresses the issue of recognizing an error. Situated plan attribution resembles elements of both model tracing and plan recognition, though it cannot be completely characterized as one or the other according to current definitions. In fact, the current conception of m odel tracing should be extended to allow for model tracing at different levels of abstraction. To clarify w hat this means, consider how model trac ing and plan recognition address the tutoring intervention problem. 108 Model tracing recognizes errors using an executable performance model of the student to search for production rule paths that account for the student's behavior. If an action can only be accounted for via a m al-rule application, then the tutor concludes that the student m ade an error and passes this interpretation to the pedagogical model (Anderson et al., 1990). This differs from the approach taken in REACT; it does not detect errors by running a cognitive sim ulation of the student. Rather, REACT focuses on recognizing impasses, which m ore closely resembles plan recognition than model tracing w hen deciding w hether to intervene. W hereas m odel tracing uses procedural know ledge to detect student errors, plan recognition approaches to student analysis typically m atch the observed behavior to a declarative description of action. This either in volves m atching and interpreting the student's action sequence using a li brary of plans (Johnson, 1986; Johnson, 1990; Calistri, 1990), or else in terp retin g th e action w ith an action gram m ar or p rocedure net description of the task (Burton, 1982; Rickel, 1988; W arriner et al., 1990). In a plan m atching approach, errors are detected w ith mal-plans or dif ference rules, w hile in the action gram m ar case an error is any action that can't be parsed by the grammar, or which is only parsed with a model that includes buggy rules. A situated plan attribution approach to im passe recognition bears some resemblance to these approaches in that the tutor initially matches student actions to declarative plan descriptions and rec ognizes potential impasses w hen actions do not fit the expectations gener ated by the plans and their goals. It differs in that the tutor does not use m anually encoded mal-plans or difference rules to recognize the impasse; as the tutor gains experience it effectively generates its own mal-plan rules in the form of recognition chunks. These chunks sum m arize the tu to r’ s experience in recognizing when a student's action does not fit the model of expected behavior. For example, when the tutor detects a plan dependency impasse, it creates a chunk while in the analyze-action-response problem space containing the com m and that was issued (e.g., NPCG) and the fact that the plan that it belongs to (i.e., Coherence-Test) was inactive at the time. In the future, w hen the chunk fires it autom atically classifies this com mand as a plan dependency violation w ithout the need to subgoal into 109 analyze-action-response again. Hence, REACT compiles its knowledge of w hen to interact through experience. At an abstract level it traces a per formance model, but the difference from standard model tracing is that it does not attem pt to generate m ental states to do so. It traces the student's progress in enough detail to m ake predictions of w hat plans the student m ight be following, and no more. 5.5 Scope and limitations of situated plan attribution Situated plan attribution provides a flexible and efficient m eans of in terpreting student behavior, but to some extent this is due to the assum p tions that are m ade and some fortuitous aspects of the approach to tutor ing. It benefits from these things in that they make it easier to understand w hat the student is doing. One of the assumptions at the outset is that the mission that the student is perform ing is known. This differs from the assum ption behind other approaches to plan recognition, e.g., (Kautz, 1991; Kautz&Allen, 1986), w here the goals of the agent are not know n in advance, and the purpose of the plan recognition process is to determ ine the agent's goals by matching actions to plans; the goals are inferred in these dom ains in a bottom -up fashion as the plans are matched. The difficulty w ith these approaches is that the plan recognizer has to attem pt to fit all of the actions into a single plan hierarchy in order to infer the goals and explain the behavior. Since the assum ption in REACT is that the student's goals are known, i.e., accomplish the mission, this allows the tutor to m ake predictions of w hat plans the students are likely to be following. All model tracing and m ost plan recognition systems used for student m odeling m ake sim ilar assum ptions. However, it should be possible to weaken this assum ption and retain the same basic approach. Device failures and anomalies can lead the O perator to carry out different or additional plans. O perators sometimes get confused about which plans are appropriate for which mis sions, and carry out inappropriate plans. These cases can sim ply be treated as alternative sources of expectations for plans. Tutorial intervention w ould continue to focus on the appropriateness of attributed plans to the situation at hand. 110 Another w ay that situated plan attribution is m ade easier is due to the fact that REACT interacts w ith the student each tim e an im passe is de tected. This helps w ith the problem of understanding the student's behav ior because some of the burden of interpreting intertw ined im passes is avoided. Each tim e an impasse is detected the tutor interacts w ith the stu dent by offering advice about the im passe and how to resolve it. If the student follows the tutor’ s advice then the possibility of having an impasse carry over between plans is reduced. I l l Chapter 6 Im passe Explication Recognizing an impasse is only the first step in solving the tutorial interac tion problem: it provides the tutor with a cue to shift its focus from moni toring the student's behavior to explicating a perceived im passe, which is the second step. Impasse explication is the process of deciding w hat to say to the student during an interaction. This process has two basic goals: First, explain the nature of the impasse in the context of the current situation. The explana tion m ust provide inform ation about the impasse at a level of detail that will theoretically enable the student to avoid it in the future, given a simi lar situation. In the case of an action constraint violation this means ex plaining how a precondition was violated in the current situation, both in term s of the expected and actual values of a device state. The sam e holds true for a goal failure im passe, w hich takes a sim ilar form. For both of these impasse types, the explanation provides knowledge of how to situate one's actions, which is one of the overarching goals of REACT. For a plan dependency violation, on the other hand, the explanation is m eant to show the student how a particular resource, in this case a plan, w as m isused. Since plans are m eant to orient the student's actions, it is im portant that the student stay w ithin the broad framework for action that is provided by the temporal dependency network (TDN). This framework is an abstract representation of action for a broad class of situations, and it is useful for providing general explanations of how to behave w hen perform ing a task. Thus, when the student takes an action that cannot be explained either in terms of reacting to the situation or in the more general term s of the plan framework, then the explanation that is given will start from a m ore general level, that is, from the plan level, since this is the resource that is m eant to guide the student's actions in the first place. It is I assum ed that the student will have access to the TDN itself in order to i 112 acquire a high level understanding of the relationships am ong the task's plans. The second goal of impasse explication is to help the student resolve the im passe as it exists in the current situation. W hereas the first goal of im passe explication is to help in understanding how the impasse came about, the second goal is m eant to enable the student to resolve the im passe so that problem solving m ay continue. W ithout this type of assistance the student can easily slip into a situation w here attem pting to resolve one impasse leads to others, as was the case in some of the examples in Chapter 3, which can be frustrating and disorienting for the student. Though there m ay be some benefit in allowing the student to flounder, i.e., m ore im passes leads to more learning, the approach I have adopted in this research is to deal w ith each im passe as close to its source as possible in order to avoid entangling the student in complex, interm ingled errors. Even if I later choose to limit the am ount of explanation given to the student at the im passe point, e.g., based on skill level, user-selected help level, etc., the tutor still needs to be capable of performing the explication of each impasse. The rem ainder of this chapter describes how the im passe explication process works. I begin by giving a brief overview of the problem space hierarchy that perform s im passe explication. The relationship of these problem spaces to the situated plan attribution problem spaces described in the last chapter is that they are linked at the top-level of the tutor, which provides a tight coupling between the impasse recognition and impasse ex plication. After describing the overall problem space hierarchy, I give a detailed account of how each of the three types of impasses are explicated: action constraint violation, goal failure and plan dependency. Action constraint violations and goal failure impasses share some common problem spaces, thus m any of the functional details are the sam e for these two impasses. Both types are resolved in a set of problem spaces that implements an exe cutable expert cognitive m odel of the dom ain. Plan failure impasses are treated som ewhat differently. Errors of this type do not require a situated account of how to solve the problem, rather, they are treated as errors that arise from misconceptions about the plan framework. Before this type of 113 REACT Tutor: Too-level Problem Space for Impasse Explication ^ resolve-goal-failure-impasse resolve-plan-dependency-impasse roGplve-action-constraint-impasse Resolve-Plan-Dependencv-lmpasse Erablsm..Spac,8 Resolve-Goal-Failure-lmpassa Problem Space Plan Problem Space Plan Operator" Problem Space Verifv-Preconditions Repair-UNSAT-precondition Verifv-Postconditions ElPblem-Space Problem Space Problem Space Figure 6.1 Problem space hierarchy for impasse explication 114 im passe explication is m ade, however, some effort is devoted to finding a situational reason for the student’ s behavior. 6.1 Problem space hierarchy for impasse explication The problem space hierarchy used for im passe explication is show n in Figure 6.1. This problem space hierarchy is linked to the problem spaces for im passe recognition at the top-level, m eaning that the operators show n in the figure as resolve-action-constraint-im passe, resolve-goal-failure-im passe, and resolve-plan-dependency-im passe, are all in the same problem space as the top-level impasse recognition operators shown in Figure 5.6. Recall that w hen REACT recognizes an im passe it annotates the event log to indicate a problem. The three top-level explication operators shown in the figure are proposed and applied w hen the event log annotation indi cates an im passe in one of the three categories. Each of these operators forms a subgoal to resolve the impasse and the problem spaces in the hier archy are searched for a solution. The approach taken by the resolve-goal-failure-impasse and resolve-ac- tion-constraint operators is very similar in that they both em ploy a set of problem spaces that im plem ents an expert cognitive model; the expert cognitive model starts at the level labeled Plan Problem Space in Figure 6.1 and includes all of the problem spaces below it in the hierarchy. The expert cognitive model has access to the state of the devices as well as the event log, which shows the actions the student took in arriving at the current si tuation. It is capable of reactively solving the problem from the impasse point, and it generates an explanation while it is in the process of recover ing from the conditions that caused the impasse. The resolve-plan-dependency-impasse operator subgoals into a problem space by the same name, where it determ ines w hether there is actually an impasse, and, if so, it generates an explanation. 6.2 Action constraint impasses Action constraint violations occur w hen a com m and is issued before its preconditions are all satisfied. The preconditions are w hat helps the prob lem solver to situate the action: taking the action is contingent on the state 115 of the w orld, regardless of the plan's command sequence, and each precon dition encapsulates a small piece of the knowledge required to take the ac tion. Thus, explicating an action constraint violation first requires being able to determ ine which of the com m and preconditions w as not satisfied w hen the student took the action. This information provides part of w hat the student needs to know about the com mand and its usage in the real world. The next step in the explication process is to resolve the impasse caused by the unsatisfied precondition and also resolve any other unsatisfied pre conditions. This serves two purposes. First, this explanation will assist the student in resolving the current im passe so that problem solving on the task can be continued, w hich is im portant for learning. But the second purpose of this inform ation is that it goes further in explaining how to apply the desired action in the given situation. It is not enough to know that a precondition is unsatisfied, it is also necessary to know how to satisfy it. As will be show n later, this m ay involve changing a param eter to the original command, or it m ay involve taking other actions that will satisfy the precondition. 6.2.1 Explication using the expert cognitive m odel The top-level operator called resolve-action-constraint-impasse is proposed and selected w hen it detects that an event in the log has been annotated stating that there is action constraint violation. This operator creates a sub goal and selects the problem space of the plan that contains the operator w hose com mand had the action constraint violation. This problem space, w hich is labeled as the "Plan Problem Space" in Figure 6.1, represents an expert cognitive model for the task. In actuality there is a separate problem space like the plan problem space in the figure for each mission plan. Each of these plan problem spaces contains the command operators that imple m ent that plan, and each com m and operator also has its ow n problem space containing a set of operators for taking the desired action. Many of the com m and operators are only a m ember of one plan, which m eans that there are not m any am biguities in choosing a plan to im plem ent a particular command. This is especially true in this dom ain since the choice 116 ' resolve-goal-failure-impasse 1 resolve-plan-dependency-impasse f resolve-action-constraint-impasse [ load-predicts • set-temperature • select-recorder • select-playback • set-ofst • set-SAT-value • set-XAT-value Confiaure-DSP (Plan) Problem Space Load-Predicts (Plan Operator) PrpfrtetnSpaQ S Repair-UNSAT-Precondition Problem Space REACT Tutor; Top-level Problem Space for Impasse Explication parameterize-precondition-from-directive parameterize-precondition-no-directive parameterize-pc-device-from-directive parameterize-pc-device-no-directive verify-preconditions repair-u nsatisf ied-precondition send-nioad-command • vverify-postconditions Verifv-Postconditioris* Problem Space ■ modify-a-task-state-object 1 verify-a-postcondition-SAT select-corrective-operator-active-plans select-corrective-operator-current-plan select-corrective-operator-inactive-plans select-corrective-parameter detect-desired-state-repair ■ verify-precondition-satisfied-for-known-device ' verify-precondition-satisfied-for-unknown-device Verifv-Precondition§ Problem Space Figure 6.2 Resolving an action constraint violation impasse 117 of a plan is usually m ade from am ong the active plans, w hich further reduces the num ber of potential ambiguities. Thus, the issue of am biguity is not a major concern in the LMC domain, but it is anticipated that this problem will arise and be addressed as this research progresses and more task knowledge is added and other task domains are considered. Figure 6.2 shows an instantiated version of the part of Figure 6.1 used for resolving action constraint violation impasses. Note that in Figure 6.2, the nam e of the plan problem space is Configure-DSP, and the nam e of the com mand operator problem space is Load-Predicts. The operators for each of the problem spaces in the hierarchy is also shown in Figure 6.2. To ex plain how these problem spaces work, I will use a familiar example. 6.2.2 Detecting and explaining the impasse The event log in Figure 6.3 shows that the student issued the NLOAD com mand and it was rejected. REACT recognizes the directive rejection as an action constraint violation and m arks this event as such. The resolve- action-constraint-im passe operator detects the annotation, and creates a subgoal to resolve the impasse. # Type Event Log Configure DSP Coherence Test Utility 1 OD >V NLOAD M K X 2 DR > REJECTED. NO PR FOUND FOR THIS SET Figure 6.3 Event log with action constraint violation impasse example It selects the Configure-DSP problem space, shown in Figure 6.2, to re solve the impasse. The reason for selecting this problem space is that the NLOAD com m and belongs to the Configure-D SP procedure; the Configure-DSP problem space contains all of the com m and operators that w ould be used by an expert executing this plan. The basic strategy for explicating the impasse is to use expert problem solving knowledge to solve the problem. These problem spaces are very sim ilar to the ones used to do the cognitive m odeling w ork, described in Chapter 4, w ith a few differences that have to do w ith taking into account the student's actions. Hence, to resolve this particular impasse, the opera tor associated w ith the NLOAD com m and is selected and a subgoal is 118 form ed that selects the Load-Predicts problem space. Once in the Load- Predicts problem space the real w ork of explicating the im passe begins. This is the level in the problem solving at which the situation is taken into account in deciding whether the action is appropriate or not. Though it is not shown in Figure 6.2, the first thing that REACT does in the Load-Predicts problem space is make an internal copy of all of the Load- Predicts operator preconditions. The reason for this is to enable the tutor to m anipulate the precondition copies so as to sim ulate the student, which is necessary in order to explain the cause of the impasse. The reader m ay re call that a precondition is stated in the form shown in Figure 6.4. W hen the student issued the command V NLOAD MK, the param eter, MK, de term ines which predicts set will be loaded. Since there m ay be m any dif ferent predicts sets besides the one nam ed MK, the preconditions have to be param eterized to reflect the predicts set specified in the student's com m and. The binding of the VLBI-PREDICTS device nam e to the identifier MK is show n in Figure 6.4. The parameterize-pc-device-from-directive op erator (Figure 6.2) makes this binding between the student's com mand pa ram eter and the precondition device name. Likewise, in cases w here the precondition value m ust be bound to the command param eter the parame- terize-precondition-from-directive operator is used. Device Type Device Name Precondition Name Attribute Value VLBI-PREDICTS MK Load-Predicts-PC1 RECEIVED? YES VLBI-PREDICTS MK Load-Pred icts-PC2 QUALITY OK NCB-PPROGRAM NCB-PROGRAM Load-Pred icts-PC3 MODE IDLE Figure 6.4 Student preconditions of load-predicts operator Device Type Device Name Attribute Value NCB-PPROGRAM NCB-PROGRAM MODE RUN VLBI-PREDICTS JK RECEIVED? YES VLBI-PREDICTS JK QUALITY OK Figure 6.5 Actual device states Once the preconditions have been param eterized from the student's com mand, the next proposed operator is verify-preconditions, which cre ates a subgoal into the Verify-Preconditions problem space. This operator subgoal is applied to each of the preconditions to determ ine w hether the precondition’ s attribute-value pair m atch w ith the state of the actual de- 119 vices; if they m atch, then the precondition is m arked SAT, and it is m arked UNSAT otherw ise. In the exam ple above, the precondition nam ed Load-Predicts-PCl is UNSAT because the predicts set specified by the student does not exist. This is where the first part of the explanation is built, using the student's version of the precondition that was param eter ized by the student's com mand, and the expert version of the precondi tion, which is param eterized from a mission-briefing statem ent. A short explanation of the form shown in Figure 6.6 is created for use in the inter action w ith the student. Impasse Explanation Device Type I Device Name I Attribute I Expected Value | Actual Value VLBI-PREDICTS I MK I RECEIVED I YES I NO Figure 6.6 Partial explanation of w hy NLOAD failed Impasse Explanation Student Command | Student Parameter I Correct Parameter NLOAD | M K | JK I Figure 6.7 Explanation of w hy NLOAD failed, continued As it turns out in this example, the reason that the precondition is un satisfied is that the student used the w rong predicts set nam e in the NLOAD command. There is no set nam ed MK, which is why the directive was rejected. To recover from this situation, the repair-unsatisfied-precon- dition operator is selected and a subgoal into the repair-UNSAT-precondi- tion problem space is created. This is where REACT begins to resolve the impasse, which it does in one of two ways: either it selects a new param e ter for the rejected command that will correct the error m ade by the stu dent, or else it selects a command operator that will bring about the desired state change that will satisfy one of the current com m and's preconditions. In this example, the operator select-corrective-parameter is selected since it notes that the student used a different param eter than the expert w ould have chosen. This operator changes the param eter bindings in the precon dition to the correct values and it also generates another piece of the expla nation, shown in Figure 6.7. Once the UNSAT p recondition’s param eter bindings have been changed, it is once again evaluated in the Verify-Preconditions problem 120 space, and this time it is satisfied, allowing the process of evaluating the other preconditions to be continued. 6.2.3 Resolving the impasse So far this example has show n how REACT detects, corrects and explains action constraint im passes resulting from an incorrectly param eterized com m and. But this exam ple also illustrates how REACT dynam ically plans an action that can correct an unsatisfied precondition. This is the form of reactivity that reflects the situated nature of action in the LMC do main. It so happens that in this example there is another precondition that is unsatisfied; the precondition nam ed Load-Predicts-PC3 is violated be cause the NCB-Program is in the RUN m ode instead of the IDLE mode. In this case, w hen REACT enters the repair-UNSAT-precondition problem space, it repairs the unsatisfied precondition by selecting a utility command operator called idle-NCB-program. This operator is chosen because its end effects will satisfy the precondition by putting the NCB-Program into the IDLE mode. In selecting this operator, REACT forms a subgoal and selects the idle-NCB-program problem space (not shown in Figure 6.2), which is nearly identical to the Load-Predicts problem space. It goes through the process of verifying preconditions, just as in the case of the Load-Predicts operator, and it simulates the effects of issuing the NIDLE REC com mand to the sim ulator devices. REACT keeps a copy of its model of the external w orld for just this purpose, and it changes the value of the NCB-Program from RUN to IDLE. At the same time that REACT simulates the execution of the NIDLE REC command, it adds this command to the explanation of how to resolve the current impasse, which is shown below. Impasse Explanation Corrective Command Sequence > V NIDLE REC > V NLOAD JK Figure 6.8 Impasse resolution for NLOAD, continued Once the corrective operator has been applied, the idle-NCB-program subgoal terminates and the Load-Predicts-PC3 precondition is now satisfied, which means that the command is now ready to be issued. The command 121 is param eterized from the corrected version of the Load-Predicts-PCl pre condition, and the command is added to the explanation of how to resolve the impasse. According to the model of skill acquisition described in Chapter 4 and (Hill&Johnson, 1993a), the inform ation contained in the explanation show n in Figure 6.8 is precisely w hat the student needs to learn about the situational constraints of the failed com mand in order to avoid the same im passe in the future. It is assum ed that as the student reads the explanation above that this knowledge will be acquired in a conceptual or declarative form and later incorporated into the student's procedural skill as it is applied in a problem solving context. The cognitive m odel’s account for the proceduralization of skill is a direct result of the Soar architecture's learning capability provided by the chunking mechanism (Rosenbloom&Newell, 1986; Laird et al., 1987). M oreover, this m odel of skill acquisition closely resem bles the account given b y . A nderson (Anderson, 1983; Anderson, 1989; Anderson et al., 1990). 6.3 Goal failure impasses A goal failure impasse occurs when the student completes a procedure (i.e., finishes executing a plan) w ithout achieving all of its goals. This can hap pen as a result of misparameterizing a command, or it can be caused by exe cuting a sequence of commands in a such a way that the devices do not end up in the goal state. For example, if the Operator selects a recording device that does not have any spare disk space for recording new data, then the goal of the procedure is not satisfied even though the O perator took all of the right actions. A third possibility is that a device m ay fail during the ex ecution of the plan, leading to a goal failure, but this case has not been ad dressed by this research. W hereas the action constraint impasses are rooted in a lack of knowl edge about individual actions and how they m ust be situated in the world, goal failure impasses may arise out of a lack of understanding of the effects of the plan’ s operators or from not know ing the ultim ate purposes of a sub-task in terms of a desired end state and how to recognize whether they have been achieved. 122 A goal failure impasse may not make itself obvious until long after the task w ith which it is associated has been completed. In the LMC domain, for example, some goal failure impasses m ay not be detected until weeks after they occurred, which makes it nearly im possible to give corrective feedback to the Operator (Resch, 1991). It is desirable, therefore, to detect and explicate these types of problems as soon as they occur. REACT Tutor: Top-level Problem Soace for Impasse Explication Resolve-Goal-Failure-lmpasse* Problem Space Configure-DSP (Plan! Prpbtem Space ^ resolve-goal-failure-impasse * resolve-pfan-dependency-impasse * resolve-action-constraint-impasse • create-explanation-unsat-task-impasse-source * resolve-coherence-test-impasse ,* resolve-configure-dsp-impasse ■ load-predicts • set-temperature • select-recorder • select-playback • set-ofst • set-SAT-value • set-XAT-value Figure 6.9 Resolving a goal failure impasse 6.3.1 Explicating a goal failure impasse REACT explicates goal failure im passes in a m anner very sim ilar to the w ay that it handles action constraint impasses. The prim ary difference is that in the case of an action constraint impasse the specific action that failed has already been pinpointed and it is simply a matter of subgoaling into the 123 com m and operator's problem space (e.g. the Load-Predicts problem space) and determ ine which preconditions were violated. The difference in the case of the goal failure im passe is that the com m and operators that w ere involved in the im passe are initially undeter mined. Hence, w hen the resolve-goal-failure-impasse operator, show n in Figure 6.9, is proposed, it forms a subgoal and selects the Resolve-Goal- Failure-Impasse problem space. Once in this problem space, an operator called create-explanation-unsat-task-impasse-source generates the first part of the explanation, detailing which plan and goals of the plan were not achieved. After this, the failed plan's operator is selected and REACT en ters the plan’ s problem space. Whereas in the case of the action constraint im passe the plan's problem space was entered merely as a step toward applying the command operator, the purpose here is different. A subset of the plan's goals have not been achieved and it is reasonable to believe that the operators that can achieve them are available in this problem space. The basic strategy that is used to resolve the impasse is to select and apply the com m and operators whose ef fects m atch the unachieved goals of the plan. This approach ends up im plem enting a sim ple backw ard chaining m echanism , since the problem solver will begin by selecting the operator that im m ediately achieves the goal. As the selected command operator is applied its preconditions will be checked w ith respect to the state of the devices. An unsatisfied precondi tion will lead to the selection of another com m and operator to repair the precondition, as was the case w ith one of the unsatisfied preconditions in the action constraint impasse. This process can go on as long as REACT is able to locate operators whose effects match the preconditions. Each step of the backw ard chaining will have the effect of adding more levels to the goal stack until all of the preconditions can be satisfied. Hence, the explica tion of a goal failure impasse is the same as an action constraint impasse at the com m and operator level, but the main difference is that the command operator(s) that are applied are selected on the basis of which goals they can achieve. 124 6.3.2 An example of goal failure explication To illustrate these principles, I give a brief example, leaving out the details covered in the section on explicating the action constraint im passe and plan dependency. The goal failure in Figure 6.10 is quite simple: the S- band attenuation value was set to 13 instead of 15 (see line 5). Though this is not a catastrophic goal failure, it will serve the purpose of illustrating how REACT would explicate it. ' ■ # Type Event Log Configure DSP Coherence Test Utility 1 OD >V NLOAD JK X 2 DR > COMPLETED. 3 OD > V NRMED LD O X 4 DR > COMPLETED. 5 CO > V SAT 13 X 6 DR > COMPLETED. 7 OD > V XAT13 X 8 DR > COMPLETED. 9 OD > V NTOP 20.0 30.0 X 10 DR > COMPLETED. 11 OD > V NPCG M AN X 12 DR > COMPLETED. 13 OD >V OFST 2.7 X 14 DR > COMPLETED. 15 OD > V NRUN COLD X 16 DR > COMPLETED. Figure 6.10 Example of a goal failure impasse N ote that at line 11 the student issued a command from the Coherence Test plan. This error w ould be detected and explicated as a plan depen dency impasse, which is described later. The impasse that interests us for this example occurs after the student issues the OFST com m and at line 13, because this is where the Configure-DSP plan ends. At this point the plan has been completely executed, but one of its goals is still unachieved. For this reason, the resolve-goal-failure-impasse operator will be selected and applied after line 16. This operator is applied here rather than at line 14 be cause the tutor waits to see whether the student will catch the error. Since the goal failure is in the Configure-DSP plan, the Configure-DSP problem space is selected for doing the explication. The tutor begins to backchain on its operators by selecting the set-ofst operator, w hich will achieve the failed goal, once properly parameterized. Since there is an un 125 satisfied precondition for the set-ofst operator, the tu to r continues to backchain by selecting the Idle-NCB-Program operator. Once its command is applied, the set-ofst operator can be applied, which in turn achieves the failed goal, thereby resolving the impasse. 6.3.3 The explanation of a goal failure The explanation of a goal failure impasse is structured in basically the same way as an action constraint impasse. The prim ary difference is that instead of identifying the preconditions that were unsatisfied, which is a command level explanation, it instead identifies the failed plan and its unachieved goals. The explanation elaborates each of the failed goals by showing the expected and actual device states associated with each goal. As in the case of the action constraint impasse, the explanation includes a sequence of command directives that will achieve the unsatisfied goals and resolve any im pending impasses, such as was the case in the example just given. 6.4 Plan dependency im passe This type of im passe is treated in a completely different m anner than the other two types. Impasses of this type arise from a m isunderstanding of the relationship and order among plans. Errors of this type are rooted iii mis conceptions about the abstract representations of action rather in how to execute an action in the real world. Consequently, the explanations pro vided for plan dependency impasses focus on giving the student an under standing of how an action conflicts w ith the high-level description of the task provided by the Temporal Dependency N etw ork (TDN). Of course, there is the possibility that even if the student's action ap pears to violate the plan order defined by the TDN, it may be acceptable if it is in response to som ething in the situation. For instance, several of the examples have used the idle-NCB-program operator, w hich violates the order defined by the TDN, but under the circum stances it w as the only com m and the student could use to satisfy the preconditions of another command. The possibility that a command categorized as a plan depen dency im passe is actually legal and even desirable is taken into account during the explication of a plan dependency impasse. 126 REACT Tutor: Top-level Prc for Impasse I Impasse * resolve-action-constraint-impasse f resolve-plan-dependency-impasse * resolve-goal-failure-impasse unexpected-directive-satisfies-precondition unexpected-directive-satisfies-active-plan-goal elaborate-unfinished-plans Resolve-Plan-Deoend encv- Im p a s s e ^ * ^ Problem Space ^ Figure 6.11 Resolving a plan dependency violation impasse Figure 6.11 shows the problem space hierarchy for explicating plan dependency impasses. The three operators shown in the Resolve-Plan- D ependency-Im passe problem space cover the scenarios just described. Actions that have been initially categorized as plan dependency violations are evaluated by two operators that can distinguish cases w here the situa tion called for the action: unexpected-directive-satisfies-precondition and unexpected-directive-satisfies-active-plan-goal. As the nam es suggest, the first operator recognizes when the action was taken to satisfy the precondi tion of some other expected com mand, while the second operator recog nizes cases where the action satisfies a goal of one of the active plans. If ei ther operator is applied to the command, then it is no longer treated as an im passe, and the student will never know that the tutor was considering an interaction The elaborate-unfinished-plans operator is applied w hen the aberrant action is not w arranted by the situation. It identifies the active plans (i.e., the plans the tutor expects the student to be executing) and the commands in these plans that have not yet been executed. The intent of this operator is to build a plan-level explanation of the impasse, resulting in a tutorial 127 interaction that is the least situated of the three impasse categories. The ra tionale for giving this type of explanation is based on the view that plans are resources for orienting action; if an action is w arranted by neither a plan nor the situation, then it makes sense to re-orient the student to the plan rather than giving a detailed explanation of w hy the situation did not w arrant the action. 6.5 The tutor im proves over tim e To conclude this discussion of how REACT works, it is im portant to point out a significant feature of the system that is derived from taking advan tage of Soar's learning capability: REACT's perform ance im proves w ith experience. To understand the source and nature of the tutor's im prove ment, one m ust look at the kind of learning that takes place in Soar. In the last three chapters I have described a cognitive model of skill ac quisition and a tutor which both perform their problem solving in the Soar architecture. As we have seen, Soar embodies the idea that problem solv ing is a goal-oriented activity involving the search for and application of operators to a state in order to attain some desired results (Laird et al., 1987). Search takes place in a hierarchy of problem spaces, where a problem space contains a set of operators and an initial state. The problem space hierarchy is traversed via subgoaling, w hich takes place w henever the problem solver cannot make any more progress tow ard a goal in the current prob lem space. Learning occurs when a subgoal yields a result; the result is stored in a production that summarizes the conditions for subgoaling and the result of the problem space search (Rosenbloom&Newell, 1986). The chunk can be applied the next time a similar situation arises and the same results can be achieved w ithout searching a subgoal problem space. The type of learning that takes place in REACT is w hat (Dietterich, 1986; Rosenbloom et al., 1987) calls symbol level learning. It does not learn new facts that are not implied by its extant knowledge, rather, it compiles w hat it know s into chunks, thereby avoiding the need to subgoal into the problem space hierarchy the next tim e the situation occurs. REACT chunks its knowledge of how to recognize an im passe using situated plan attribution, and it also chunks the know ledge of how to explicate the 128 im passe, once it is detected. O ne of the consequences of having an intelligent tutor that can com pile its know ledge is that it affects the characterization of REACT as an intelligent tutoring system architecture. W hereas the traditional intelligent tutoring system described in Chapter 2 has distinct student, expert and tutoring modules, the REACT architecture begins to look much more integrated than this approach. Though the tra ditional facets of an ITS are still identifiable w ithin the REACT problem space hierarchy, chunking compiles this know ledge into a m uch flatter space that requires subgoaling only w hen the tutor has not encountered a particular impasse. 6.5.1 Impasse recognition chunks To illustrate how the tutor learns to recognize im passes, consider w hat happens w hen the tutor sees an action-response pair (i.e., a student com m and and device response); it subgoals into a problem space called Analyze-Action-Response, w here it searches for a plan that contains the student's command. When the tutor finds a match, it m arks the command in the plan as "matched.’ ’ The tutor then decides whether there is a poten tial impasse, depending on w hether the plan containing the com mand was active or not and w hether the com mand was accepted or rejected by the device. Two types of potential im passes are identified in this problem space: plan-dependency-violation and action-constraint-violation. The Analyze-Action-Response problem space term inates once the tutor finishes analyzing the action-response pair. The chunks built w hile searching in the Analyze-Action-Response problem space automatically recognize w hether a specific action-response pair is a potential impasse for a given situation. Hence, the next tim e that the action-response pair is observed, the tutor will recognize w hether there is a potential impasse or not w ithout any further subgoaling or search. 6.5.2 Impasse explication chunks Once the tutor has resolved a particular impasse, the resulting explication chunks are general enough to solve the same problem again even though the param eter values may be different. In order for the tutor to completely 129 chunk all of its knowledge about im passe explication, however, it w ould need to resolve an impasse associated with each precondition (in the case of action constraint violations) and each plan goal (for goal failure im passes). This presents some issues for further research to determine how to train the tutor before fielding it. Command Recognize Impasse Explicate Impasse Total Ratio: Before/After (Total) Before After Before After Before After NLOAD 0.20 0.08 8.70 0.61 8.90 0.69 13::1 SAT 0.19 0.08 2.70 0.32 2.92 0.40 7::1 7 igure 6.12 Tutor’s performance on action constraint impasse before and after learning, m easured in seconds 6.5.3 Im provem ent There is a significant im provem ent in the tutor's perform ance after it has chunked the problem space hierarchies used to recognize and explicate stu dent impasses. Figure 6.12 shows the am ount of time it takes the tutor to handle action constraint violations for the NLOAD and SAT com mands before and after learning. N ote that the time it takes to recognize the impasse is constant among these commands, before and after learning; it takes roughly 0.20 seconds to recognize an im passe involving either com m and before learning, and it im proves to 0.08 seconds after learning. On the other hand, the am ount of tim e it takes to explicate an action constraint violation varies, depending on how m any preconditions and postconditions m ust be checked for the com m and. The NLOAD and SAT com m ands w ere chosen for this exam ple because they provide upper and lower bounds for this type of impasse. The NLOAD command has the greatest num ber of preconditions and postconditions and the SAT command has the fewest. 6.6 Conclusions There were several design requirements that had to be satisfied by the im passe recognition and explication functions in REACT. These design re quirements were form ulated as goals at the beginning of this chapter: First, explain the nature of the impasse in the student's problem solving context, and, second, help the student to resolve the impasse as it exists in the cur 130 rent situation. These two goals are satisfied by the impasse explication pro cess that I have just described. Action constraint violations and goal fail ures are both impasses that require a situated explanation. The expert cog nitive model is capable of giving such an explanation by putting itself into the situation and using the student's actions to generate an explanation and a solution that fits the problem solving context. Plan dependency vio lations are less situated in nature, since they appear to be caused by not un derstanding the relationship and dependencies of the plans themselves. C onsequently, the explication of this type of im passe focuses on plan knowledge rather than the situated application of its commands. A third goal, which is implicit to the impasse-driven approach to tutor ing is that the tutor m ust interact with the student as close to the impasse as possible. This goal also appears to be satisfied w ith the approach I have just described. Given that an impasse can be detected and explicated in less than a second w ith the explication technique described in this chapter, it seems likely that REACT can satisfy the requirem ent of interacting w ith the student as close to the impasse point as possible. It takes the O perator on the order of ten seconds to select and param eterize each directive, so it ap pears that the tutor can interact w ith the student before m uch problem solving can take place beyond the point where the impasse was detected. 6.7 Related work Im passe explication is how REACT decides "what to say" to the student. One of the strengths of model tracing is its ability to give a reasoned expla nation for an error, once it is detected. In this regard, REACT resembles a m odel tracer; it generates explanations using an executable cognitive model that diagnoses possible causes for the error and suggests how to re solve the current impasse. The difference between the two approaches is that model tracing relies partly on its knowledge of m al-rules to generate the explanation, w hile REACT learns to recognize the causes for errors while applying an ideal perform ance model to the impasse. The declara tive representations of operator preconditions used by the cognitive model end up being proceduralized, to a large extent, as the tutor gains experi ence. These chunks have the same effect as a m al-rule in that they can be used to explicate an impasse. 131 Chapter 7 Results 7.1 Introduction The purpose of this chapter is to describe the results of evaluating the hy potheses p u t forth by this thesis. Many of these hypotheses are based on the cognitive m odeling w ork described in C hapter 4. The m odel m ade several predictions about how people learn in the LMC dom ain, and from those predictions a num ber of design decisions w ere m ade for the LMC training system and REACT tutor. There are three hypotheses that sum m arize these decisions: Hypothesis 1: Impasse-driven tutoring is an effective way for choosing how and when to interact with the student. The term "impasse-driven tutoring" encompasses m any of the design fea tures of the LMC training system and the REACT tutor. It implies a certain style of tutoring, whereby the interactions between the student and the tu tor occur close to impasse points. This, of course, assumes that the student is actively engaged in solving a problem and the tutor is m onitoring the student’ s behavior. This hypothesis is based on three of the predictions m ade from the cognitive model: (1) students learn by doing, (2) learning occurs in a goal context, and (3) learning occurs in impasse situations, if the student is able to determine the cause and resolution of the impasse. This hypothesis is tested in two ways. The first w ay is intended to test whether students acquire skill more rapidly using the LMC training system w ith the REACT tutor than students w ho use the LMC training system w ithout the tutor. If the hypothesis is true, then we w ould expect, first of all, to see evidence that the students encountered an impasse; if learning takes place at the im passe point then there should be a concomitant im- j provem ent in perform ance the next tim e the same im passe situation oc curs. Furthermore, it is expected that students w ho have the tutor assisting j them will acquire the knowledge related to the im passe more rapidly than 132 students w ho have to resolve the impasse on their own. To test these pre dictions from the hypothesis, a prototype summative evaluation procedure was developed and applied to a small sample of novices in the domain to determ ine the learning rates w ith and w ithout the tutor. This evaluation is described later in this chapter. The second w ay of testing this hypothesis is by a form ative evaluation. A questionnaire was developed to obtain the test subject's reaction to the style of tutoring in REACT. Of particular interest was w hether the tutor was too disruptive or too passive, in terms of the frequency and style of in teraction. Hypothesis 2: Situated plan attribution is an effective way to interpret the student's behavior and to recognize impasses. Impasse-driven tutoring depends on the tutor's ability to rapidly recognize and explicate impasses. W ithout this ability the tutor cannot intervene close to the impasse point. The test for this capability is to determ ine how quickly the tutor can recognize and respond to an impasse situation. Some of these results were reported in Chapter 6, but the tutor's execution in a live exercise yielded som e m ore data that has been com piled and is re ported later in this chapter. The second test of this hypothesis is to determ ine how well situated plan attribution recognizes impasses. Recall that one of the predictions m ade from the cognitive m odel was that experts do not rotely execute plans. This results in part from the fact that the dom ain dem ands a reac tive form of plan execution; the O perator has to attend to the state of the devices while attem pting to achieve the goals of a plan, which sometimes necessitates deviating from the default command execution order. This re sult im plied that the tutor had to have a flexible approach to interpreting student behavior; it had to recognize when the student's deviation from a plan is based on situational contingency and w hen it is sim ply an error. Conversely, it m ust recognize w hen the situation dem ands a deviation from the default plan but the student does not take the appropriate action. Finally, it has to recognize w hen a plan's goals have not been achieved even though the plan m ay have been executed. 133 The test for these design requirem ents involves analyzing the event and tutoring logs from a population of students who have used the LMC training system w ith the REACT tutor. These logs contain inform ation about w hat actions the student took, the response by the simulator, and the interpretations by the tutor. The results of the test show how well the tutor perform ed its interpretation task. Hypothesis 3: To acquire skill in the LMC domain, students need to leam command preconditions and postconditions, plan goals and plan dependency relationships. The cognitive model predicted that students acquire skill by learning about com m and preconditions. This prediction was expanded to include learn ing about com m and postconditions, plan goals and plan dependencies; these are all knowledge components that are used to explain the Operator behaviors discussed in C hapter 3. The expert cognitive m odel was expanded to recognize im passes associated w ith these ad d itio n al knowledge components (Chapter 5) and to explicate them (Chapter 6). To test w hether this knowledge enhances the skill of the student involves m easuring w hether the students perform better after learning from REACT than they did before. This can be done using the same experim ent as is used to test Hypothesis 1; it involves selecting a population of students that can be divided into a control group and a test group. Both groups take a pre-test, receive some training, and take a post-test. The difference between the two groups is that the test group perform s its training on the LMC training system w ith the REACT tutor, w hile the control group uses the LMC training system w ithout REACT. In addition to the prototype sum m ative evaluation, data is collected from the subjects from the questionnaire. The questions focus on two aspects of th e know ledge th a t is com m unicated by th e tutor: understandability and helpfulness. If the tutoring is not understandable then it is likely that it will not be helpful, w hich does not necessarily invalidate the hypothesis. On the other hand, if it is understandable and not helpful, then there is something w rong w ith the hypothesis. 134 7.2 Prototype sum m ative evaluation A thorough sum m ative evaluation requires a fairly large sample popula tion that can be divided into test and control groups. To be credible, it also requires the LMC training system to provide a broader coverage of opera tional tasks than currently exists. There are plans to expand the training system's capabilities and perform a more thorough sum m ative evaluation, but this w ork is outside the scope of this thesis. As a precursor of the sum m ative evaluation that will be conducted, a prototype sum m ative evaluation was developed, w hich serves tw o pur poses. First, it provides indications of w hether the hypotheses stated in this thesis are valid by example. Second, in addition to providing some qualitative results, a prototype summative evaluation also guides the con struction of a m ore thorough evaluation after the system has been ex panded. It will help in identifying the data that needs to be collected and the procedure for the evaluation session itself. In the rest of this section I describe the prototype summ ative evaluation and how its results are expected to give an indication of whether the hy potheses are valid. The summ ative evaluation section is followed by the section describing the questionnaire that was used to evaluate each of these hypotheses. 7.3 M ethod The basic m ethod of evaluation and data collection is quite simple. Each test group subject will first be given a pre-test, followed by some training w ith the tutor. Once the training session is complete, a post-test will be adm inistered to evaluate changes in the student’s skill level. The control group subjects will receive the same pre-test and post-test as the test group, but their training will be limited to using the sim ulator w ithout the tutor. The following data items w ere collected during the various phases of the evaluation: the num ber of impasses, the num ber of commands issued to perform the assigned task, and the time elapsed, and w hether the stu dent successfully achieved the goals of the task. In cases where the student is unable to complete a task due to a lack of knowledge, the corresponding data item was m arked with an infinity symbol: 00. In addition, the event 135 and tutor logs were saved from each mission, and all of the entries are tim e-stam ped in order to use in evaluating the reaction times of the stu dent and of the tutor. 7.3.1 Task assignm ent The task used for perform ing this evaluation is essentially the same as the one described in Chapter 1. The evaluation involves having the student perform this task num erous times in the context of different m issions. The differences from one mission to the next are prim arily in the starting conditions of the devices themselves. T ask : Perform the pre-calibration procedures for a Very Long Baseline Interferom etry (VLBI) mission. Specifically, configure the DSCC Spectrum Processing (DSP) Subsystem and perform the coherence test on the configuration. C o n d itio n s: Perform the task on the LMC sim ulator, using its com mand m enu window to issue the directives, the event log to m onitor directive responses and event notice m essages, the NCNF (configuration table) display, which shows the configura tion status, and a W hiteboard display, which shows some key di rective param eters you will need. In addition, you will be given a h andout at the beginning of each phase show ing the initial NSUM (equipment status) and NCAL (calibration table) displays. If you are in G roup I you will also have a tutor w indow during the training phase that will be used to explain how to w ork around certain kinds of errors. If you are in Group II you will have to w ork around any errors during the training phase on your ow n using the resources provided. You will also be pro vided a procedural sum m ary sheet for the tasks you are to per form as well as a directive dictionary that describes the use of each directive. In cases where the training system displays do not provide inform ation about the state of a particular device in the link, you may query the controller for specific data items. For ex am ple, you m ight ask a question like "Is disk LDO on-line?", though this specific information is provided in a hand-out. G oals: Complete the procedures w ithout any directive rejections. Complete the task as quickly as you can safely do so. Correctly set the values for all of the Configuration Table entries. Com plete the initialization of the N arrow Channel B andw idth (NCB) 136 recording program (abbreviated NCB-Pgm) and determ ine w hether the configuration is coherent. If you reach a point dur ing the pre-test or post-test w here you do not feel that you can proceed, please inform the controller, w ho will allow you to m ove on to the next mission. 7.3.2 Pre-test phase The pre-test phase was designed to collect data on the subjects' ability to perform tasks prior to receiving any training. These skills were tested by requiring the subjects to perform the assigned tasks in tw o different mis sions. In the first mission (M l), the initial states of the devices reflect an ideal world where the task can potentially be performed by executing the default procedure. The expectation is that the prim ary source of impasses will be from m is-param eterizing the directives, but the m ission does not contain any device states requiring the student to deviate from the w ritten proce dure. The second mission (M2) starts one of the devices in a state that requires the student to take corrective actions, either to avoid or to resolve an action constraint impasse. Specifically, the disk drives LDO and LD1 were placed in the disabled mode; this violates a precondition of the NRMED directive, which requires that the selected recording m edium be enabled. By placing these devices into the disabled m ode the pre-test reveals the subject's knowledge of directive preconditions as well as the knowledge about how to resolve the specific impasse. In addition to planting a device state to force an action constraint im passe, M2 also presents a potential goal failure impasse. Mission M2 ini tializes the param eters for the S-band drift, m agnitude and jitter limits to the wrong values. Since one of the goals of the Coherence Test, as specified in the procedure guide used by the subject, is to insure that these limits are set correctly, this tests the subject's attentiveness to a portion of the goal state of the procedures. 137 7.3.3 Training phase During the training phase the student performs the same task that was as signed in the pre-test. One of the differences from the pre-test phase is that during the training phase the G roup I subjects interact w ith the REACT tu tor w hen it detects an impasse, while the G roup n subjects have to rely solely on the other resources to resolve the impasse. The other contrast w ith the pre-test phase is that an additional action constraint is introduced via the initial state of the devices. In addition, a variation on the potential goal failure impasse is also used. The first training mission, M3, is similar to M2 in that it contains a po tential action constraint impasse and goal failure impasse. It differs from M2 in the details of the initial state: M3 initializes one of the devices, the NCB-Pgm, to the NRUN m ode, which differs from its ideal starting state, the NIDLE mode. This device state means that one of the preconditions of the NLOAD directive is violated. In addition, the X-band limits for drift, m agnitude and jitter are also initialized to the incorrect values, m eaning that there is a potential goal failure. These limits are identical in use to the S-band limits that were corrupted in M2 The second training mission, M4, repeats the initial state of M3, but it also adds the potential action constraint impasse posed in M2 whereby both disks are disabled. Thus, M4 presents an initial device state containing two potential action constraint im passes (i.e., for NLOAD and NRMED) and one goal failure impasse (i.e., the S-band limits). 7.3.4 Post-test phase The post-test phase is identical in nature to the pre-test: the subjects in both groups have to perform the assigned task two more times w ithout the assistance of the tutor. The first post-test mission, M5, uses an identical starting state as was used in M l. Thus, im provem ents in performance on the task in the ideal w orld can be measured by comparing the elapsed time of M l and M5. The starting conditions of the second post-test mission, M6, are identical to the ones in mission M3. 138 7.3.5 Sum m ary of potential impasses by m ission The table show n in Figure 7.1 below summ arizes the missions that are to be perform ed in the pre-test, training and post-test phases. This sum m ary indicates the potential impasses that are created by varying the initial con ditions of the device state. The "None" entries indicate that there are no device states that w ould cause the subject to reach an action constraint im passe. The NRMED and NLOAD entries indicate initial device states that violate preconditions for these directives. The S-limits and X-limits entries correspond to cases where the values for these param eters are initialized to incorrect values and will cause a goal failure if the subject does not attend to them. Pre-test Training Post-test Ml M2 M3 M4 M5 M6 None NRMED NLOAD NLOAD None NLOAD S-limits X-limits NRMED S-limits S-limits Figure 7.1 Summary of potential mission impasses The rationale for the design of this suite of missions is as follows. The pre-test will be used to compare the starting know ledge of the subjects in the two groups. M l provides a baseline against which to measure how dif ficult the procedures are w ithout any action constraint violations present in the devices. M2 introduces potential action constraint and goal failure impasses. This not only tests the knowledge level of the two groups of sub jects, but it also gives an indication of the validity of Hypothesis 3, that is, skill requires know ledge of command preconditions, postconditions and plan goals. If the hypothesis is true, then we w ould expect two results in the data. First, we w ould expect to see the subjects struggle with the im passe, which w ould be m ade evident by the am ount of time elapsed, com m ands being rejected, and goals not being achieved. It is expected that the test subject w ho "struggles" w ith an im passe will spend a significant am ount of tim e resolving the im passe in com parison to the am ount of time that it takes to perform a routine action: a lower bound on this type of m easurem ent w ould be twice the am ount of tim e it takes for a routine action and there is no upper bound since the subject m ay sim ply quit. 139 Second, if the subject is able resolve the impasse, then we can expect to see im proved behavior the next time the action constraint violation occurs. It is expected that even the pre-test will be a learning experience. The training phase introduces another action constraint violation, asso ciated w ith the NLOAD command, from which we hope to glean some in sight about the difference between learning w ith the tutor and w ithout the tutor. Hypotheses 1 and 3 predict that the subjects in Group I will acquire knowledge about the NLOAD preconditions in M3 m ore rapidly than the subjects in Group II. M4 serves to reinforce this knowledge in both groups. G roup I should dem onstrate that they have acquired the new knowledge by im proving their performance. Members of G roup n w ho are able to re solve the NLOAD im passe should also show im provem ent in their per formance. M4 also contains the potential NRMED impasse. Hypotheses 1 and 3 predicts that subjects in G roup I who did not previously resolve this im passe in M2, will now be able to acquire the knowledge to resolve it in M4. Furtherm ore, Hypothesis 3 predicts that subjects in Groups I and n who did previously resolve the NRMED impasse will show im proved performance in this case. By the tim e that the subjects reach the post-test phase they will have perform ed the tasks four times. M5 is used as a basis of m easuring how m uch im provem ent has occurred in both groups since the first tim e that they perform ed this mission in M l. M6 measures how well the members of the two groups learned from the training with the NLOAD impasse in M3 and M4. The hypotheses predict that the subjects of either group who w ere able to resolve impasses that they previously encountered will show im proved performance. It is assum ed that the members of both groups learned from doing, and that learning in an impasse situation w ithout the aid of a tutor can potentially result in the same level of perform ance as tutor-aided training. The differences expected between these two groups will be the am ount of time that was necessary to learn in the impasse situa tion and the higher probability that the tutored students will learn from their impasses. 140 7.4 A nalysis of sum m ative evaluation results The data for the sum m ative evaluation was collected from a population of seven test subjects. Four subjects were in Group I, which interacted w ith the REACT tutor during the training phase. Three subjects were in Group II, which received no assistance during the training phase. All of the sub jects were members of the technical staff at the Jet Propulsion Laboratory (JPL), and they had a basic familiarity w ith the operational dom ain of the Deep Space Network. The experience level of the test subjects in the LMC domain varied somewhat: one test subject was previously an LMC opera tor, while another test subject is a college student working at JPL for the summ er. For the m ost part, however, the test participants appeared to have an equally low level of experience in perform ing the assigned tasks. The tests were conducted with one subject at a time, and each test took ap proximately an hour to administer. 7.4.1 Data collected on individuals Figure 7.2 shows an example of the data that was collected and collated on each participant in the evaluation. The columns labeled M l through M6 represent the results from missions 1 through 6. The rows are labeled w ith the type of data that was collected during each mission. Subject #2, Group I | Pre-test Training Post-test (Training with Tutor) | Ml M2 M3 M4 M5 M6 Total Number of Impasses 1 1 2 1 0 1 Action Constraint Impasses 0 0 1 1 0 0 Plan Dependency Impasses 0 0 0 0 0 0 Goal Failure Impasses 1 1 1 0 0 1 Number of Requests for Help? 1 1 0 0 0 0 Correspondence Impasse /Request 0 1 0 0 0 0 Number of Commands 13 19 14 18 14 13 Min. Number of Commands 11 13 13 14 11 13 Time Elapsed (seconds) 632 624 256 234 174 139 Mean Time per Command (seconds) 49 33 18 13 12 11 Successfully Achieved Goals? (Y/N) N N Y Y Y N Figure 7.2 Example of data collected on each subject The first four rows detail the num ber of impasses the subject encoun tered: the first row shows the total num ber of impasses, while the follow- 141 ing three row s show how m any of each type occurred. Below this is a record of the num ber of times the subject asked the controller a question related to the task. If the question had something to do w ith a current im passe, then it was added to the num ber for the Correspondence Impasse / Request. The total num ber of commands issued by the student during the task was recorded, and below this is the m inim um num ber of commands that could have accomplished the mission. The tim e elapsed was recorded in seconds for each mission, and the mean tim e per com m and was calcu lated by dividing the tim e elapsed by the num ber of com mands issued, rounded to the nearest second. Finally, a boolean value indicating whether the subject accom plished the goals of the task for each m ission w as recorded. It is interesting to note that this test subject failed the task goals during the last mission. In this instance the goal that was not satisfied had to do w ith setting the S-band limit values correctly. Since the subject had correctly satisfied this goal on three other occasions, it appears that the er ror was due to a mental slip. Pre-test Training (with Tutor) Post-test M2 M3 V[4 M6 NRMED NLOAD NRMED NLOAD NLOAD | Subject 2 | 173 29 13 24 11 Figure 7.3 Time (in seconds) required to resolve potential action constraint impasses The event and tutor logs were recorded while the subject performed the task in each mission. The subject's actions and the tutor's responses were given time-stamps w hen they were recorded. Using these logs, data was collected on the am ount of time required to resolve the potential impasses that were created in the various missions. Figure 7.3 shows the data col lected on one of the subjects. In this case the subject was able to resolve the NRMED impasse during in M2 in 173 seconds, w ithout the aid of the. tutor since it was during the pre-test. During M3 and M4 the subject trained with the help of the REACT tutor to achieve the results shown. N ote that with ! the tutor, the subject resolved the NLOAD im passe over five tim es as quickly as NRMED impasse. During M4, the NRMED impasse was avoided 142 or resolved quickly by the subject in 1 NLOAD impasse. 7.4.2 Summary and Let us begin by tak skills in term s of Figures 7.4 and 7. Figure 7.4 shows t m ean time per con test subjects imprc there is very little jects of the two grc averaged about 12 This contrasts with to perform the task Operator.) , probably due to the fact that it was previously resolved d2. In M6, the subject dem onstrates a m astery of the analysis of results ing a high level view of how the subjects im proved their the am ount of tim e required to perform the tasks. 5 sum m arize how long it took to perform the tasks, he total time per mission, while Figure 7.5. shows the tm and for each mission. It is quite clear that all of the >ved their perform ance over the six missions. In fact, difference in the end performance levels am ong the sub- >ups. W ith the exception of subject #4, all of the others seconds per command by the time they had reached M6. the approxim ated 60 seconds per com mand that it took in M l (excluding subject #1, w ho was previously a DSN Pre-test Training Post-test Ml M2 M3 M4 M5 M6 Group I: Subject 1 226 OO 185 210 98 147 Training with Subject 2 632 624 256 234 174 139 Tutor Subject 3 744 503 341 254 145 168 Subject 4 862 OO 654 418 201 285 Group II: Subject 5 928 OO 441 248 152 169 Training without Subject 6 632 488 854 234 144 188 Tutor Subject 7 810 OO 784 736 147 161 Figure 7.4 Total time required to perform tasks in seconds Pre-test Training Post-test Ml M2 M3 M4 M5 M6 Group I: Subject 1 19 n/a 12 13 9 10 Training with Subject 2 49 33 18 13 12 11 Tutor Subject 3 62 26 24 16 13 12 Subject 4 72 n/a 38 25 20 22 Group II: Subject 5 66 n/a i9 16 14 12 Training without Subject 6 45 26 36 14 13 13 Tutor Subject 7 68 n/a 52 33 13 12 Figure 7.5 Average time per command in seconds 143 In com paring the perform ance times of M l and M5, w hich w ere the missions w ith the ideal initial conditions, the subjects in both groups im proved their perform ance in terms of tim e by a factor of four or m ore in most cases. One of the exceptions to this was subject #1, but this can be ex plained by higher experience level that this individual had prior to the test. One conclusion that can be draw n from this data is that people do improve w ith practice; this was an expected result and is not a surprise. A second conclusion is that the experience level of the two groups seems to be fairly consistent w hen m easured in term s of the perform ance in the tw o m is sions, M l and M5. The results of M2 are also consistent w ith w hat was predicted. Four out of the seven subjects were unable to resolve the NRMED impasse, indicat ing that they were missing some key know ledge for perform ing the task. The three w ho w ere able to resolve the im passe took a considerable am ount of time to do so, relatively speaking. Figure 7.6 shows that it took these three subjects 173, 190, and 152 seconds respectively, to resolve the impasse. This is roughly three times am ount of tim e per command that w as taken to perform the task in M l, and roughly thirteen times the am ount of time per com mand in M6. All of this supports the notion that the subjects experienced an impasse and that it was due to a lack of knowl edge about the preconditions of the NRMED command. In addition, the data in Figure 7.6 confirms that the subjects in both groups lack this partic- Pre-test Training Post-test M2 M3 I V [4 M6 NRMED NLOAD NRMED NLOAD NLOAD Group I: Training with Tutor Subject 1 OO 8 11 8 2 Subject 2 173 39 13 24 11 Subject 3 190 29 17 11 12 Subject 4 OO 42 26 16 76 Group II: Training without Tutor Subject 5 OO 168 49 23 , 10 Subject 6 152 413 10 8 10 Subject 7 OO 444 449 13 8 Figure 7.6 Reaction time for potential action constraint impasses, with and w ithout the tutor. 144 The results from the training phase appear to confirm the predictions m ade by Hypotheses 1 and 3 that the tutored subjects (Group I) w ould ac quire knowledge about com m and preconditions and postconditions more rapidly than the untutored subjects (Group E Q . M3 shows the Group I sub jects resolving the NLOAD im passe roughly five to ten tim es faster than the G roup n subjects. This result is supported in M4 by the NRMED im passe. The subjects in G roup I w ho had not previously resolved the NRMED im passe clearly acquired this knowledge m ore rapidly than the G roup II subjects who had not previously acquired it. Subjects 1 and 4 re solved the impasse in 11 seconds and 26 seconds, respectively, while sub jects 5 and 7 took 49 seconds and 449 seconds to resolve it. One of the other conclusions that can be draw n from the data in the training phase and from M6 is that once the subjects had resolved an im passe they w ere able to avoid it or very quickly resolve it w hen it occurred again. In almost every case in Figure 7.6, the subject drastically decreased the am ount of time to resolve or avoid an im passe once it had been resolved the first time. It does not seem to make an end performance difference whether the new knowledge came from assistance with the tutor or from solving the problem unassisted; the m ain difference is that the tutor shortened the am ount of tim e required to acquire the knowledge. This is consistent with the predictions m ade from Hypotheses 1 and 3. I Pre-test Training Post-test | Ml M2 M3 M4 M5 M6 Group I: Subject 1 Y N Y Y Y Y Training with Subject 2 N N Y Y Y N Tutor Subject 3 Y Y Y Y Y Y Subject 4 N N Y Y Y Y Group II: Subject 5 Y Y Y Y Y Y Training without Subject 6 N N N N N N Tutor Subject 7 Y N Y Y Y N Figure 7.7 Did the subject successfully achieve the task goals? A nother benefit of tutor-assisted training can be seen in Figure 7.7, where two out of the three Group II subjects failed to achieve the task goals in M6. Only one out of the Group I subjects failed a task goal, and in this 145 case it appears to be attributable to a slip since the subject had previously satisfied the failed goal on three other occasions. One of the problems with unassisted learning is that there m ay be tim es w hen the student is not aware that there was an impasse to resolve, as in the case of achieving the goals of a procedure. Action constraint impasses are very obvious to the student in the LMC Training System: in m any cases the directive will be rejected by the simulator and the student can immediately observe this fact in the event log. Goal failures are usually less evident because the LMC system does not complain in obvious ways w hen they occur. All of this suggests that the claim by Hypothesis 3 that students need to learn plan goals may be valid, and it also seems to support Hypotheses 2 and 3 in that the was tutor evidently able to help the subjects in Group I to achieve the all of the task goals in M3 and M4, while only one task goal failed in M5 and M6. Due to the small size of the sample, however, it will be necessary to substantiate these hypotheses in a statistically m eaningful w ay in a future evaluation. 7.5 Evaluation by questionnaire Each of the Group I subjects completed a questionnaire after they had com pleted the suite of missions just described. The purpose of the question naire w as to provide a m eans of evaluating the hypotheses other than through empirical experimentation. The questionnaire was organized into three sections: background information, questions related to Hypothesis 1 and questions related to Hypothesis 3. The rem ainder of this section de scribes the rationale for each part of the questionnaire, lists the questions used, and then presents the results along w ith some conclusions. 7.5.1 Background inform ation The intent of this section was to gain an understanding of the am ount of experience and knowledge that each of the subjects was bringing to bear in answ ering the questions. This inform ation is also useful in explaining some of the results that were gathered in the empirical evaluation. 146 7.5.1.1 Q uestions for gathering background inform ation Q0.1: W hat is your current position? Q0.2: H ow m uch experience do you have in the Deep Space N etw ork? Q0.3: How much experience do you have in the operational details of the LMC? 7.5.1.2 Results A lthough the questionnaire was not adm inistered to the m em bers of Group II since they did not interact w ith the tutor, data was collected about their backgrounds. All of the subjects are m em bers of technical staff. Subject 1 has a great deal m ore experience in the Deep Space N etwork and the operations of the LMC. This m ay explain w hy this subject performed better initially than did the others. Subjects 2 through 7 had similar levels of experience, though subjects 4 and 7 claim to have no background in the operation of the LMC. I 0.1 0.2 0.3 Subject 1 Member Technical Staff 15 years 5 years Subject 2 Member Technical Staff 1.5 years 1.5 years Subject 3 Member Technical Staff .5 years 2 weeks Subject 4 Member Technical Staff .5 years none Subject 5 Member Technical Staff 3 years .5 years Subject 6 Member Technical Staff 2 years 1 year Subject 7 Member Technical Staff 2 years none Figure 7.8 Background inform ation 7.5.2 Formative evaluation of H ypothesis 1 H ypothesis 1 stated that im passe-driven tutoring is an effective way for choosing how and when to interact w ith the student. There are several underlying assum ptions to this hypothesis: (1) impasses are learning op portunities; (2) tutorial interaction should take place close to the impasse point; (3) the student has a goal to resolve an impasse, w hich includes learning w hat caused it in the first place; and (4) the tutorial interaction will not disrupt the student's learning (i.e., no more than the unresolved 147 im passe interrupts it.) To test these assum ptions, the following questions were devised. 7.5.2.1 Q uestions related to H ypothesis 1 Q l.l: In general, were you comfortable w ith the style of interaction w ith the tutor? 1 - not comfortable thru 5 - very comfortable Q1.2: W ould you have preferred m ore interaction or less interac tion? 1 - much more, 3 - about right, 5 - much less Q1.3: W ould you prefer that the interaction was more direct or less direct? For instance a m ore direct approach m ight halt the sim ulator and force the student to be tutored. A less direct ap proach m ight provide a subtle signal in the user interface to indicate that an error was committed or an impasse is pending. 1 - more direct thru 5 - less direct Q1.4: W ould you prefer to receive tutoring im m ediately after a mis take or after the session? 1 - interact immediately thru 5 - interact after session 7.5.2.2 Results Figure 7.9 shows the answers given for the portion of the questionnaire re lated to Hypothesis 1. 1 1.1 1.2 1.3 1.4 Subject 1 4 2 3 5 Subject 2 5 3 4 2 Subject 3 4 3 2 1 Subject 4 3 2 4 1 Figure 7.9 Results related to Hypothesis 1 The purpose of Question 1.1 was to determine, in very subjective terms, w hether the style of interaction w ith the tutor, which is im passe-driven, was comfortable or not. If they were not comfortable then it m ight indicate that the interactions were disruptive, irrelevant, or otherw ise incom pre 148 hensible. If the interactions were comfortable then there is evidence that there m ust have been some utility in the interaction style and content. Based on the results shown in Figure 7.9, it appears that the interactions were not uncomfortable. Subject 2 was very comfortable w ith the interac tions, while subject 4 was less comfortable. The other two appear to fairly comfortable. It appears that the impasse-driven style of interaction is not too disruptive for the subjects. Question 1.2 tested w hether the subjects thought the interactions with the tutor were frequent enough. The results in Figure 7.9 seem to indicate that the frequency of interaction was about right, and if anything, a little m ore interaction m ight have been useful. This supports the notion that the tutor was providing useful information in a timely fashion. Question 1.3 evaluated how the subjects felt about the w ay that the in teraction was initiated. The results here are som ewhat mixed. Two sub jects w ould have preferred the tutor to give a more subtle signal to begin the interaction, while one subject felt the current approach is fine and the other wished a more direct signal. The conclusion in this case is that there are no serious flaws with the way that the tutor currently signals the stu dent. Q uestion 1.4 also evaluates the appropriateness of an im passe-driven style of interaction with the tutor. Three out of the four subjects prefer that the tutor interact at the point where the impasse occurs. Subject 1, on the other hand, preferred to be tutored after the session. It is interesting to note that subject 1 is also the m ost experienced of the group, hence may not w ant or need to be helped by a tutor when perform ing a familiar task. On the other hand, it is interesting to note that Subject 1 indicated in question 1.2 the desire for more interaction with the tutor, which seems inconsistent with the desire to interact after the mission has been completed. The other subjects, however, seemed to prefer that the tutoring occur im m ediately when an impasse occurs. 7.5.3 Formative evaluation of Hypothesis 3 Hypothesis 3 states that to acquire skill in the LMC domain, students need to learn command preconditions and postconditions, plan goals and plan 149 dependency relationships. To evaluate this hypothesis requires first find ing out w hether the explanations provided by the tutor w ere understand able. If the student cannot relate the tutor's explanation to the problem solving impasse, then the hypothesis cannot be tested. Second, if the ex planations w ere understandable, then a determ ination has to be m ade of w hether they w ere actually helpful in term s of im proving the student's knowledge of how to perform the task. 7.5.3.1 Questions related to Hypothesis 2 Q2.1 In general, were the explanations offered during the interac tions w ith the tutor understandable? 1 - not understandable thru 5 - very understandable Q2.2 In general, were the explanations helpful? 1 - not helpful thru 5 - very helpful Q2.3 In general, did the explanations have the right am ount of de tail? 1- not enough 3 - just right 5 - too much Q2.4 W hat other inform ation w ould have been helpful? Q2.5 If you had any directives rejected while perform ing the task, was the explanation for the directive rejection understandable? 1 - not understandable thru 5 - very understandable, 6 - not applicable Q2.6 Given that such an explanation is understandable, is it helpful to find out w hy a directive was rejected? 1 - not helpful thru 5 - very helpful, 6 - not applicable Q2.7 Was it helpful to be told how to work around the directive re jection? 1 - not helpful thru 5 - very helpful, 6 - not applicable Q2.8 If you failed to achieve any plan goals w hile perform ing the task, was the explanation given by the tutor understandable? 1 - not understandable thru 5 - very understandable, 6 - not applicable 150 Q2.9 Given that such an explanation is understandable, is it helpful to find out that a procedure goal was not satisfied? 1 - not helpful thru 5 - very helpful, 6 - not applicable Q2.10 If you performed any procedures out of order, was the explana tion for this error understandable? 1 - not understandable thru 5 - very understandable, 6 - not applicable Q 2 .ll Given that such an explanation is understandable, is it helpful to find out that such an error had occurred? 1 - not helpful thru 5 - very helpful, 6 - not applicable 7.5.3.2 Results The results of the questions related to Hypothesis 3 are shown in Figure 7.10. 1 2.1 2.2 | 2.3 2.5 2.6 2.7 2.8 2.9 2.10 2.11 Subject 1 4 4 3 4 5 4 6 4 5 4 Subject 2 5 4 3 5 5 5 6 6 6 5 Subject 3 4 5 3 3 5 5 6 6 6 6 Subject 4 2 3 2 3 5 5 2 5 5 5 Figure 7.10 Results related to Hypothesis 3 Questions 2.1 through 2.3 tested the general impressions of the subjects of the understandability and helpfulness of the explanations given by the tutor. The results in Figure 7.10 indicate that the explanations were under standable, helpful, and the right am ount of detail. Subject 4 was least satis fied of the group and wanted more detail in the explanations; based on the background data this appears to be related to the fact that subject 4 was the only one who claims to completely lack experience w ith the operational details of the LMC. This also may explain w hy subject 4 perform ed the tasks at a m uch slow er rate than the others, even after practice. In discussing the explanation detail question w ith subject 4 it became clear th at p art of the problem w as that the explanations w ere not being thoroughly read. It also became evident that it w ould have been helpful for subject 4 to know the difference between the different kinds of impasses that were being detected and explicated by the tutor. 151 Question 2.4 prom pted the subjects for information that they w ould like to see included in the explanations that is not currently available. One sub ject indicated a desire for a reference in the Software O perator's M anual w hen giving an explanation. Another subject indicated that a m ore direct way of signaling the student w hen an impasse has occurred w ould be use ful. Questions 2.5 through 2.7 asked a more detailed version of question 2.1 through 2.3 as they relate to the interactions with the tutor in the case of an action constraint impasse. From the subjects' perspective this type of im passe occurred at a directive rejection. In general, the subjects appear to be lieve that the explanations were understandable and helpful. Subject 4 ap peared to have some difficulty understanding the explanations, as did sub ject 3 in one category. Again, this may be attributed to the lack of experi ence of these two subjects in the LMC domain, and it indicates that some of the terminology in the explanations could be m ade clearer. Q uestions 2.8 and 2.9 w ent into more detail about the explanations given in the case of a goal failure impasse. This question is revealing be cause of the num ber of responses indicating that the subject did not feel that the question applied to them. Of the four subjects, only subject 1 did not have a goal failure impasse during any of the missions. Yet subjects 2 and 3 did not think that the question applied in their case. In discussions w ith each of the subjects after they had taken the questionnaire I explained the theory behind the tutor and the kinds of impasses that it detects and explains. With this explanation the subjects seemed to have a better un derstanding of w hat was going on during the interactions w ith the tutor. It m ay be helpful to include this type of inform ation in the introduction to the training system. In any case, subject 4 again found the explanation to not as understandable as it could be while also stating that it is helpful. Questions 2.10 and 2.11 dealt w ith the explanations for plan dependency violations. Those who felt that the question applied to them indicated that the explanations were understandable and helpful. 152 7.5.3.3 Conclusions about questionnaire The data from the questionnaire was useful for getting a subjective im pres sion of the tutor from the test subjects. The questions could be im proved by rem oving am biguities about their m eaning in some cases by giving a more detailed example of w hat is meant. For instance, the questions where the subjects answered that it did not apply to them clearly did not make the connection between the type of tutoring they received and the nam e that was being used to describe it in the question. Likewise, there needs to be a way of clarifying inconsistent answers by an individual test subject, as was the case w ith subject 1 on questions 1.2 and 1.4. It may be helpful in the future to identify questions w hose an swers m ay conflict ahead of time, so that it the conflict occurs then it will be easier to just ask the subject w hat was meant. There are two questions that I would like to add for the next evaluation: (1) Was the tutor disruptive when it initiated an interaction w ith you?, and (2) Did you receive help w hen you w anted or needed it? These two questions will test whether the impasse point is the right place to initiate an interaction w ith the student. If the interaction is distracting, then a refinement of the w ay it chooses to interact will be necessary. If it is not giving the student help w hen it is w anted or needed, this could indicate that the im passe is occurring sooner than it is being detected. Given this situation, one alternative would be to give the student a way of indicating that there is an impasse so that the tutor can interact sooner. 7.6 Assessing the effectiveness of situated plan attribution Hypothesis 2 states that situated plan attribution is an effective w ay to in terpret the student's behavior and to recognize impasses. An im portant aspect of im passe-driven tutoring is being able to recognize an impasse. REACT uses the technique called situated plan attribution to interpret the student’ s behavior and to recognize impasses. The general test for evaluat ing this hypothesis is a follows: Was the tutor able to effectively interpret all of the student's actions as either fitting into an expected plan or as an impasse? This question empirically tests the completeness and accuracy of 153 the technique used in REACT to detect student impasses. It is answered by evaluating the tutor's performance by the following criteria. El: Did the tutor correctly interpret all of the "correct" actions under normal circumstances? (i.e., When there are no device states requiring a deviation from the default procedures) E2: Did the tutor correctly interpret student deviations from the default procedure when there are no situational factors requir ing the deviation? In other words, does REACT correctly rec ognize all plan dependency violations? E3: Did the tutor correctly interpret student deviations from the default procedures when they were in reaction to situational factors? E4: Did the tutor recognize w hen the student fails to achieve a goal? Was it able to explain the goal failure to the student? E5: Did the tutor recognize all action constraint violations? Was it able to explain the action constraint violation? Figure 7.11 Evaluation criteria for Hypothesis 2 7.6.1 Evaluation m ethod The event and tutor logs from each session w ere recorded and subse quently analyzed according to the criteria above. Figure 7.12 shows how m any commands were issued by each subject during each of the six mis sions. There were 604 total com mands that had to be interpreted during the evaluation, which provides a fairly large set of examples for applying the evaluation criteria listed in the previous section. Though the com m ands that were issued from one mission to the next were essentially the same, there was a fairly large set of possible ways for the subjects to param e terize and apply the commands to the simulator. The evidence for this is in the num ber of impasses that were detected and explicated by the tutor during the evaluation. 154 I Ml M2 M3 M4 M5 M6 Subject 1 12 13 15 16 11 14 Subject 2 13 19 14 18 14 13 Subject 3 12 19 14 16 ll 14 Subject 4 12 9 17 17 11 1$ Subject 5 14 13 15 16 11 14 Subject 6 14 19 24 17 11 14 Subject 7 12 11 15 22 11 14 Totals 89 103 114 122 80 96 Fif Grand Total: jure 7.12 Tot 604 al number of commands for tutor to interpret Action Constraint Impasses Plan Dependency Impasses Goal Failure Impasses Subject 1 3 0 0 Subject 2 2 0 4 Subject 3 8 0 0 Subject 4 5 2 6 Subject 5 4 3 0 Subject 6 6 0 6 Subject 7 8 0 1 Totals 36 5 17 Figure 7.13 Impasses detected by tutor by category 1 The tutor detected num erous impasses in each of the categories that have been previously described: action constraint, plan dependency and goal failure. Figure 7.13 shows how m any of each type of impasse was de tected for each subject. Action constraint violations were by far the most com mon type of impasse detected; there were 36 action constraint im passes, com pared to 17 goal failure impasses and only 5 plan dependency impasses. Again, this illustrates that there were many opportunities for the subjects to mis-apply the commands in the course of performing a task. Given this set of data, the evaluation criteria were applied to the event and tutor logs. The results of this analysis are reported in the following sections, beginning with El and finishing with E5. 7.6.2 Evaluation of El Did the tutor correctly interpret all of the "correct" actions under norm al circumstances? 155 This is a test to determ ine w hether the tutor recognized all of the ac tions that w ould norm ally be executed under ideal circumstances. These are the actions that are theoretically a part of the tutor’ s active plans if the student follows the procedures. During the evaluation there was a total of 604 com m ands issued by the seven subjects. The m ajority of the com m ands that w ere issued fell into this category (i.e., 604 commands m inus 58 impasses.) There were no instances where a correct action under norm al circumstances was m isinterpreted. 7.6.3 Evaluation of E2 Did the tutor correctly interpret student deviations from the default proce dure w hen there are no situational factors requiring the deviation? In other w ords, does REACT correctly recognize all plan dependency viola tions? Due in part to the structure of the task, there w ere not m any plan de pendency violations. Since the procedural instructions that were provided to the subjects were very sequential in nature, there were not m any oppor tunities to get the plans out of order. This aspect of the evaluation is illus trated in Figure 7.13, which shows that the tutor detected only five in stances where the subject took an action that would be categorized as a plan dependency violation. As it turns out, every one of these cases were re lated to the NRMED impasse: some subjects chose to resolve the impasse by skipping that step in the procedure. This m eant that they did not issue the NRMED and PMED com m ands because they noticed that the disks were disabled. In so doing, these subjects began the coherence test task w ithout finishing the configure DSP task. If the subjects had tried the NRMED and PMED directives then the im passe w ould have been identi- t fied as an action constraint impasse. Instead, it was interpreted as a plan dependency violation, and the explication m erely rem inded the subject that they had skipped the NRMED and PMED commands. This approach forces the student to deal directly with the impasse by not allowing it to be : bypassed. Once the student deals directly w ith the impasse, the tutor can j | give more detailed information that is relevant to the student’ s goals at the | m om ent. ! i 156 All of the actions that should have been interpreted as plan dependency impasses were interpreted correctly by the tutor. 7.6.4 Evaluation of E3 Did the tutor correctly interpret student deviations from the default proce dures when they were in reaction to situational factors? Recall from Figure 7.3 that each subject was faced with five different de vice situations and four goal failure situations requiring a deviation from the default plan. This means that out of the 604 commands that were is sued by the subjects during the evaluation, 63 of these commands were po tentially deviations from the default plans. There was one instance where the tutor interpreted an action as a goal failure im passe w hen it was actually a correct action being taken in re sponse to the situation. This led to the insight that before evaluating an ac tion as a goal failure impasse, it m ust first be determ ined w hether the ac tion is related to overcoming the goal failure. There are operators in the plan dependency impasse problem space, namely, unexpected-directive-sat- isfies-precondition and unexpected-directive-satisfies-active-plan-goal, that do this type of evaluation, but there were none in the goal failure problem space. Consequently, I have put copies of these operators in the goal failure problem space and they have corrected this problem. 7.6.5 Evaluation of E4 Did the tutor recognize w hen the student fails to achieve a goal? Was it able to explain the goal failure to the student? Each subject was given four different goal failures (see Figure 7.3), plus i m any other opportunities to fail. Figure 7.13 shows that the tutor detected 17 goal failures. Of these goal failures, roughly half were due to mis-pa- ram eterizing commands because the subject was unsure about how to read certain tables that were provided in the resources. There were no cases where the tutor failed to detect a goal failure. 157 7.6.6 Evaluation of E5 Did the tutor recognize all action constraint violations? W as it able to ex plain the action constraint violation? Action constraint violations were by far the m ost prevalent type of im passe that occurred during the evaluation. Most of the 36 action constraint impasses (Figure 7.13) that the tutor detected were due to the five abnormal device states that were planted in the missions (Figure 7.3). The tutor de tected and explicated all of these impasses in all cases. It is interesting to note that som e subjects also encountered action constraint impassies of their own making by mixing the order of the steps in the procedure or by issuing a com m and w ith an incorrect param eter. The tutor was able to interpret all of these situations. There was one case where the subject sent a com mand that was rejected by the simulator, which w ould normally be interpreted as an action con straint violation. In this instance, however, the tutor correctly interpreted the action as a goal failure since the subject had not achieved the goals of the preceding procedure. In so doing it forced the student to attend to the unachieved goals of the procedure first before dealing w ith the apparent im passe associated w ith the directive rejection. The unsatisfied goals in this instance were an indirect cause for the rejection of the com mand in the first place, so this interpretation made a great deal more sense than try ing to resolve the most recent impasse. 7.7 Summ ary Three hypotheses w ere p u t forth at the beginning of this chapter along w ith a set of evaluation criteria. The rem ainder of the chapter described the m ethod and results of perform ing the evaluation. The m ethod of evaluation had two aspects: (1) show that the REACT tutor accomplishes the claims put forth in this thesis by doing a prototype sum m ative evalua tion w ith a population of test subjects, and (2) by gathering the reactions of the test subjects to various aspects of their interactions w ith it. The summ ative evaluation had a num ber of interesting results. First of all, the notion of an impasse in the LMC domain was confirmed by the fact j that over half of the subjects were not able to resolve one of the situations | * ! 158 that was intentionally planted in the second mission. The other subjects resolved the impasse, but struggled to do so. The hypotheses about im passe-driven tutoring and nature of the skill required for the domain was supported by the fact that the Group I subjects acquired knowledge about how to resolve impasses m any times faster than the G roup II subjects. In addition, the questionnaire indicated that the tim ing and content of the interaction w ith the tutor w as about right. G roup II subjects also showed a tendency to have goal failure im passes w ithout realizing it, while Group I subjects seemed to be more cognizant of the goals of each of the procedures after receiving tutoring. The situated plan attribution technique used in the tutor perform ed well during the evaluation. The performance of the tutored students im proved as expected, and they were able to learn how to resolve impasses m ore quickly than the untutored students, which provides som e indica tion that the tutor was saying the right things at the right time. The ques tionnaire substantiated this conclusion in that the tutored subjects gener ally found the information timely, understandable and helpful. Finally, a detailed analysis of the event and tutor logs indicates that there were very few unexpected results. One of these, having to do with the interpretation of utility actions that are meant to help in satisfying a goal failure, has led to an im provem ent in the way that the goal failure problem space is im plemented. It is expected that other limitations of the situated plan attribu tion m ethod will appear as more complex procedures and devices are in troduced to the training system. Finally, there was another very subjective measure of success that came w ithout provocation or by design. The four test subjects w ho had inter acted w ith the tutor independently indicated that the learning experience was fun, and in some cases had suggestions on how it could be im proved j into a video game format. In contrast, the three subjects w ho had to re solve all of the impasses on their own did not give the same kind of indica tion as the Group I subjects. Though the control group appeared to be in- J terested in w hat the tutor had been doing behind the scenes w hile they I were encountering and resolving impasses, they ultimately seemed less en- j thusiastic at the end of the evaluation than did the test group. Again, this j 159 result is merely anecdotal, but it makes sense that the control group may have felt a little more frustrated in the end since they had to work so hard to gain the knowledge needed to successfully resolve the impasses they en countered. Chapter 8 Conclusions 8.1 Summary of thesis This thesis introduces a new m ethod for intelligent tutoring in task dom ains requiring reactive skills. The general approach, which is called im passe-driven tutoring, is based on a cognitive model of reactive skill acquisition, where the task domain is the Link Monitor and Control (LMC) system used in NASA's Deep Space N etwork (DSN). The cognitive model, im plem ented in Soar, gives an account of problem solving behaviors that have been observed in actual operations in the LMC dom ain, including behaviors that can only be explained in term s of an underlying Operator misconception or knowledge gap. The cognitive model was designed to im prove its behavior over time in the same way that a novice becomes an ’ expert: it acquires new knowledge about command preconditions, recovers from incorrect knowledge (by sim ulating the effects of tutoring), and it is able to revise its problem solving procedures to respond to dynamically changing situations. As a result of this modeling work, a theory of tutorial interaction was developed and im plem ented in a system called REACT. The REACT tutor is integrated w ith an LMC simulator; students perform missions on the LMC sim ulator and REACT interacts w ith the student w hen it observes that the student has reached an impasse. The decision to interact w ith the student is based on the ability of the tutor to recognize im passes; REACT uses a technique called situated plan attribution to understand the student's behavior and determine w hether an impasse has occurred. Once detected, the tutor explicates the impasse using its expert cognitive model; the explanation contains the same types of knowledge that the cognitive model predicts the novice requires to become an expert. A prototype sum m ative evaluation of the LMC training system w ith the REACT tutor has been conducted, and the results suggest that the theory of j tutorial interaction implemented in REACT is valid. 161 8.2 Contributions of this thesis • A theory of tutorial interaction in interactive domains. The goal of this thesis w as to develop an intelligent tutoring system for novice Operators in a task domain that requires reactive skill. Starting from the assum ption that hum ans learn such skills by "doing," the focus was on developing a training environm ent that w ould integrate "learning by doing" w ith "learning from a tutor." Consequently, the route that was taken to achieve the overarching goal w as shaped by the tutorial interaction question: when should the tutor interact and w hat should it say? To answer this question, a detailed cognitive model of the Operator was built that could be used as an artificial student. By sim ulating the skill acquisition process using the cognitive m odel, a num ber of key insights were gained that eventually led to the design and im plem entation of the REACT tutor itself. The cognitive model provided some key insights that contributed to an understanding of how to answer the tutorial interaction question: (1) it confirmed the notion of learning by doing; (2) it specifically showed one of the things that Operators need to learn, namely, command preconditions; (3) it showed the insufficiency of rote plan execution in the dom ain, thereby raising an aw areness that the training environm ent should contain the same situational obstacles as are present in the actual system; (4) it showed that learning occurs in a goal context, and that (5) learning occurs in im passe situations; (6) it revealed that there can be hidden gaps in the Operator's knowledge, based on the fortuitous ordering of actions in a procedure; and (7) it also showed that in addition to making errors as a result of knowledge gaps, it is necessary in some cases for the O perator to recover from incorrect know ledge. These results, in them selves, represent one of the prim ary contributions of this thesis because they give insight into the nature of problem solving and skill acquisition in a reactive task domain. Furtherm ore, these results and the predictions about tutoring that were m ade from them were confirmed in large part during the evaluation phase. ! The cognitive m odel led to a theory about tutorial interaction; this j theory, which is im plemented in REACT, predicts that it is best to interact i ! 162 w ith the student as dose to the impasse as possible; it is from this concept that the term "impasse-driven tutoring" is derived. Once the decision to interact has been made, the tutor has to know w hat to say to the student to help both in understanding the nature of the im passe as well as how to resolve it. This is the second part of the theory of tutorial interaction, and it is also informed by the cognitive model. The decisions about w hat to say to the O perator are m ade during the explication of the im passe by the expert cognitive model, which is itself derived from the cognitive model of reactive skill acquisition. Thus w e see that the cognitive m odel served not only to develop a theory of tutorial interaction but also to implement it. •Situated Plan Attribution: a new approach to student modeling. One of the keys to im plem enting an im passe-driven tutor, therefore, is to be able to recognize when the student has reached an impasse. Three impasse categories were identified in this work: action constraint violations, plan dependency violations and goal failure. The technique used to understand the behavior of the student and to recognize when an impasse has occurred is called "situated plan attribution." The purpose of situated plan attribution is to detect w hen the student deviates from the behavior predicted by the model; a deviation from the model is categorized as one of the three impasse types and the task of explicating the impasse is given to the expert cognitive model. This approach combines features of. both model tracing and plan recognition while avoiding some of the difficulties associated w ith each of these techniques, and its use is not lim ited to a particular task domain. Situated plan attribution takes into account the situated nature of action in reactive task dom ains, and it provides an efficient way of understanding student behavior w ithout the need for mal- plans or mal-rules, which are used by plan recognition and m odel tracing approaches to understand erroneous behavior. A prototype sum m ative evaluation showed that this technique was able to correctly recognize most student impasses. The only exception required a minor addition to one of the tutor's problem spaces. The other key to im plem enting REACT was to solve the im passe | explication problem, which has to do with deciding w hat to say during the 163 interaction with the student. The cognitive m odeling work predicted that students need to learn command preconditions in order to acquire skill in a reactive task dom ain. This hypothesis w as extended to include other types of knowledge as well: command postconditions, plan goals and plan dependency relationships. These know ledge categories, taken together, constitute the kind of inform ation that need to be provided during the interaction w ith the student, and it is the know ledge that is needed in order to avoid impasses in the future. The cognitive model also illustrated the need to be able to resolve the current impasse in order to continue with the problem solving. Hence, the second kind of explanation that REACT generates during the explication of an im passe is how to resolve the impasse; the knowledge that is provided in this part of the explanation ties back to the need for the Operator to acquire knowledge about com mand postconditions. The tutor's solution specifies the com m ands that will resolve impasse; if the im passe is associated w ith an action constraint violation or goal failure, then the solution represents a way of teaching the student how the postconditions of one or m ore com m ands satisfy the preconditions of some others. • Cognitive modeling: a methodology for designing tutoring systems. The theory of tutorial interaction that was implemented in REACT is based on a cognitive m odel of problem solving and skill acquisition in an interactive dom ain; the cognitive model accounts for the behaviors of novice and expert O perators and it also gives an explanation of how a novice acquires expertise. The cognitive m odel m ade a num ber of hypotheses about skill acquisition that were used in guiding the design of a tutor for an interactive domain. By sim ulating the student/O perator, the I cognitive model suggested ways that learning could be enhanced, given ! processes involved in perform ing tasks in the task dom ain. The | underlying m ethodology th at is suggested by this research is that I j constructing an executable cognitive m odel can inform the design of an [ intelligent tutor. In this research the cognitive model suggested a num ber | of hypotheses about skill acquisition that had a direct bearing on answering 164 the tutorial interaction question. These hypotheses, in turn, influenced the design requirements of the tutor, and they were implem ented in REACT. A prototype sum m ative evaluation w ith a lim ited num ber of test subjects gives som e positive indications that the theory of tutorial interaction im plem ented in REACT has validity. The evaluation m ade tw o contributions. First, it suggests that the predictions of the cognitive model have some validity, and, second, it dem onstrates that an impasse- driven approach to tutoring has the potential to aid students in acquiring reactive skills in the LMC domain. The cognitive model predicted that the stu d en ts w ould reach an im passe in situations w here com m and preconditions were not satisfied; this appeared to be true for every student in both the test and control groups. The cognitive model also predicted that students w ould learn w hile resolving impasses; the results of the evaluation suggest that this prediction is also true. Moreover, the tutored stu d en ts overcam e and learned from im passes m ore rap id ly than untutored students, w hich suggests that im passe-driven tutoring is an effective w ay of com m unicating know ledge in an interactive learning domain. Finally, besides giving some support to the hypotheses about tutorial interaction, the evaluation also provided insight into how the prototype evaluation could be im proved before conducting a full-scale evaluation. 8.3 G enerality of m ethods used for impasse-driven tutoring REACT was developed for tutoring in the LMC system dom ain, but the usefulness of the m ethods and theory of tutorial interaction im plem ented in REACT is not lim ited to this task dom ain. N or are situated plan attribution and the cognitive m odel em ployed for im passe explication lim ited to use in tutoring systems only. U nderstanding the behavior of other agents has broader applicability than tutoring, as does problem solving in interactive domains. The LMC dom ain has a num ber of characteristics in common w ith a \ ! general class of dom ains w here hum ans interact w ith com puters or com plex equipm ent to perform tasks or solve problem s: tasks are perform ed by taking actions on devices in the w orld that have a state; 165 devices change state in direct response to actions taken upon them , and they m ay also change state on their own; the applicability of an action is lim ited by device state; plans are used to prescribe a default action sequence; and plans have goals, which are m easured in terms of the state of the devices. The tutoring techniques implemented in REACT are broadly applicable to other dom ains because they are designed from a cognitive model that copes w ith problem solving in this class of dom ains, w hich generally requires situated action. Furtherm ore, since the problem spaces that are used in REACT for situated plan attribution and im passe explication are based on a theory of problem solving that is not dom ain specific, to im plem ent a tutor for another dom ain m ainly requires encoding the dom ain's action preconditions and postconditions, plan relationships and goals for use by the tutor. The main lim itations of transferring the approach taken in REACT to another tutoring dom ain are as follows. The student's actions m ust be observable by the tutor as well as the state of the devices being acted upon. This inform ation is necessary for detecting impasses. There should also be a way of determ ining when an action constraint has been violated. REACT had the benefit of a simulator that does much of this checking and rejects directives that violate state conditions. However, in the absence of such a command processor, it appears that this is a good use of an action grammar for a device, which could act as a pre-processor to the tutor or to the training device. Its main purpose w ould be to reject actions that violate a constraint of the system being operated. Beyond tu to rin g there are other applications for situated plan attribution and the basic expert cognitive m odel used for im passe explication. W ith some extensions situated plan attribution could be used in any dom ain where it is necessary to model another agent, especially in cases w here the m odeling involves interpreting the actions of the other ! agent. The assum ption that is m ade in situated plan attribution is that the j modeler knows the basic goals of the other agent, and is trying to interpret I the other agent's actions with respect to these goals. Hence, this technique I w ould be useful, for instance, in implementing an intelligent assistant in a 166 m onitor and control dom ain, w here the tasks being perform ed by the hum an are pre-specified or easily recognizable. The expert cognitive model that provides REACT's impasse explication capability also has the potential to act in other capacities. Given its ability to resolve impasses in this domain, including ones involving goal failure that require more extensive planning, the expert cognitive m odel could be extended to fill the role of the O perator in systems like the LMC. The cognitive model was built to perform tasks in the LMC dom ain, and it dem onstrated that it was capable of doing so to the extent that it was implemented. To extend the cognitive model for the LMC w ould require encoding m ore knowledge about the procedures, actions and goals of the dom ain. W hat is lacking at this point, however, is a reliable way of recovering from m ore serious device anom alies th a t require deeper diagnostic and planning knowledge. The current m odel is applicable to relatively routine situations that occur, including som e anom alous ones, but major device failures and other such contingencies are not covered. The point here, however, is that the expert cognitive m odel provides an architecture of problem spaces that can be applied to m ore than impasse explication for tutoring purposes. It provides a problem solving capability that can be used for intelligent control in its own right. 8.4 Direction of future work Though the evaluation showed some very prom ising results, it also gave some indications of the direction future work should take. In addition, there are some issues that were raised by the cognitive modeling work that have not been addressed by REACT yet. Let us now consider w hat additional research needs to be done to improve and evaluate REACT as an intelligent tutor. The cognitive m odel indicates th at O perators w ill have hidden know ledge gaps. It predicts that some knowledge will not be acquired because the Operator has not encountered an impasse that necessitates its acquisition. This problem can potentially be dealt with in two ways. The I first w ay is to provide the Operator w ith some declarative instruction on j the potentially missing knowledge. The assum ption then w ould be that I 167 the knowledge would be available the first tim e that it is needed, but the problem w ith this approach is that it goes against the idea of learning by doing. The second approach is to have the tutor generate the device state that will force an impasse to occur corresponding to the knowledge gap. This approach will require the tutor to have an accurate overlay m odel of w hat the student knows and doesn't know as well as a model of how to change the state of the devices to force the impasse. REACT already has some of the necessary command and device knowledge to do this, but there are other issues to consider such as w hether certain device states make sense w ith respect to an overall configuration. This is where model-based reasoning may have to be introduced in order to determine logical ways of setting a configuration into a reachable state. A nother issue that arose while observing the test subjects during the evaluation w as that they did not learn everything they should have learned during the impasse. In particular, I observed some students who learned how to resolve an impasse caused by a precondition violation, but not how to avoid it in the first place. In cases where the student had been tutored the precondition was explicitly stated, but in at least one case the student d id not know w here to find the inform ation related to the precondition in the data displays. It is interesting to note that the location of the state data was not an issue addressed by the cognitive model. The cognitive model had complete access to state data w ithout worrying about where it was located on a display. To im prove this situation the first step will be to include or highlight the specific data items that need to be attended to on the displays. One of the long term solutions in the LMC system will be to change the way that the displays are designed so that the data that is needed for a particular procedure is more localized and less distributed. This is w here the cognitive m odel can play a role in j influencing the operability of the actual system. I Thus one clear research direction that should be taken is to continue the process of extending, refining, and verifying the cognitive m odel itself. ] I | I This will likely involve performing a protocol analysis w ith the Operators ! I I | to further define the cognitive activities that are involved w ith tasks in the ; LMC domain. The protocol analysis can verify whether the problem spaces I 168 in the current cognitive model are accurate, and it can also suggest other problem spaces, such as a model of attention, that w ould have an impact on the design of the tutoring system itself. Besides continuing the analysis of the cognitive m odel, the REACT tutor m ust also be evaluated m ore fully. Prior to a full-scale evaluation, however, the num ber of plans, actions and devices covered by the system m ust be increased. This im provem ent and a larger test population will provide a m ore statistically sound basis for a full-scale evaluation. In addition, the hypotheses and questions being evaluated m ust also be refined. Finally, one of the long term goals of this project is to embed a tutor like REACT in an operational system. The concept of an em bedded training system would be to include the tutor and a sim ulator in the design of the system from the outset rather than trying to retrofit the system w ith a trainer after it has already been fielded. Before em bedded training systems become a routine part of all operational systems, the issues that have been raised here m ust first be resolved. i B ibliography (Agre, 1987) Philip E. A gre and D avid C hapm an. Pengi: An im plem entation of a theory of activity. Proceedings of AAAI-87: Sixth National Conference on Artificial Intelligence, Seattle, W ashington, July 13-17,1987: 268-272. (A nderson, 1982) John R. A nderson. Acquisition of cognitive skill. Psychological Review, 89, 4, 1982:369-406. (A nderson, 1983) John R. A nderson. The architecture of cognition. Cambridge, Massachusetts: Harvard University Press, 1983. (Anderson, 1988) John R. Anderson. The expert module. Foundations of Intelligent Tutoring Systems. Edited by M artha C. Poison and Jeffrey J. Richardson. H illsdale, N ew Jersey: Lawrence Erlbaum Associates Publishers, 1988: 21-53. (Anderson, 1989) John R. Anderson. Use of analogy in a production system architecture, Similarity and Analogical Reasoning. Edited by Stella V osniadou and A ndrew O rtony. N ew York: C am bridge University Press, 1989. (Anderson et al., 1990) John R. Anderson, C. Franklin Boyle, Albert T. Corbett and M atthew W. Lewis. Cognitive m odeling and intelligent tutoring. Artificial Intelligence, 42, 1990: 7-49. (Bloom, 1984) Benjamin S. Bloom. The 2-sigma problem: The search for m ethods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13,6, June/July 1984: 4-16. (Brown&VanLehn, 1982) John Seely Brown & Kurt VanLehn. Repair theory: A generative theory of bugs in procedural skills. Cognitive Science, 4, 1982:379-426. 170 (Burns&Capps, 1988) H ugh L. Burns and Charles G. Capps. Foundations of intelligent tu to rin g system s: An introduction. Foundations of Intelligent Tutoring Systems. Edited by M artha C. Poison and J. Jeffrey Richardson. H illsdale, N ew Jersey: Lawrence Erlbaum Associates Publishers, 1988. (Burns&Parlett, 1991) H ugh L. Burns and James W. Parlett. The evolution of intelligent tutoring systems: Dimensions of design. Intelligent Tutoring Systems: Evolutions in Design. Edited by H ugh L. Burns, James W. Parlett and Carol Luckhardt Redfield. Hillsdale, New Jersey: Lawrence Earlbaum Associates Publishers, 1991. (Burton, 1982) Richard R. Burton. Diagnosing bugs in a simple procedural skill. Intelligent Tutoring Systems. Edited by Derek Sleeman and John Seely Brown. New York: Academic Press, Inc., 1982. (Burton, 1988) Richard R. Burton. The environm ent m odule of intelligent tutoring systems. Foundations of Intelligent Tutoring Systems. Edited by M artha C. Poison and Jeffrey J. Richardson. Hillsdale, N ew Jersey: Lawrence Erlbaum Associates Publishers, 1988. (Calistri, 1990) Randall J. Calistri. Classifying and detecting plan-based m isconceptions for robust plan recognition. Ph.D. diss., Technical R eport No. CS-90-11, D epartm ent of C om puter Science, Brown University, 1990. (Chen et al., 1991) Thomas T. Chen and Diann Barbee. Integrating an intelligent tutoring system w ith an existing sim ulator. Proceedings of the 1991 Conference on Intelligent Computer-Aided Training. NASA/Johnson Space Center, Houston, Texas, November 20-22, 1991. (Clancey, 1986) W illiam J. Clancey. Q ualitative student m odeling. Annual Review of Computer Science, 1, 1986:381-450, A nnual Reviews, Inc. (Clancey, 1987) W illiam J. Clancey. Knowledge-Based Tutoring: The GUIDON Program. Cambridge, Massachusetts: The MIT Press, 1987. (Conati&Lehman, 1993a) Christina Conati and Jill Fain Lehman. Toward a m odel of student education in m icrow orlds. Proceedings of the Fifteenth Annual Conference of the Cognitive Science Society, 1993:353- 358. Hillsdale, New Jersey: Lawrence Erlbaum Associates Publishers. (Conati&Lehman, 1993b) Christina Conati and Jill Fain Lehman. EFH- Soar: m odeling education in highly interactive environments. Lecture Notes in Artificial Intelligence. New York: Springer-Verlag, 1993. (Cooper, 1991) Lynne P. Cooper. Operations automation using tem poral dependency networks. Technology 2001, San Jose, CA, December 3-5, 1991. (Cooper et al., 1991) Lynne P. Cooper, Rajiv Desai and Elmain Martinez. O perator assistant to support Deep Space N etw ork link m onitor and control. Proceedings of the SOAR Symposium, Houston, Texas, 1992. (Dewey, 1939) John Dewey. Fundam entals of educational process. Intelligence in the Modern World: John Dewey's Philosophy. Edited by Joseph Ratner. New York: Random House, Inc., 1939. (Dietterich, 1986) T. G. Dietterich. Learning at the knowledge-level. Machine Learning, 1, 1986: 287-315. (Doorenbos, 1993) Robert B. Doorenbos. M atching 100,000 learned rules. Proceedings of the Eleventh National Conference on Artificial Intelligence, July 11-15, 1993:290-296, W ashington, D.C. (DSN-OPS, 1991) N arrow and wide bandw idth VLBI data acquisition. Deep Space Network Operations Plan. Document 890-110 Revision C, July 1, 1991. JPL-D-1230. Pasadena, CA: Jet Propulsion Laboratory. (Dvorak, 1987) Daniel L. Dvorak. Expert systems for m onitoring and control. Technical Report AI 87-55. Departm ent of Com puter Sciences, The University of Texas at Austin, May, 1987. (Fayyad&Cooper, 1992) Kristina Fayyad and Lynne Cooper. Representing o p eratio n s p ro ced u res using tem p o ral d ep en d en cy netw orks. Proceedings of the Second International Symposium on Ground Data Systems for Space Mission Operations, SPACEOPS-92, Pasadena, CA, November 16-20, 1992. i (Fink, 1991) Pamela K. Fink. The role of dom ain knowledge in the design of an intelligent tutoring system . Intelligent Tutoring Systems: Evolutions in Design. Edited by H ugh Burns, James W. Parlett and Carol Luckhardt Redfield. Hillsdale, N ew Jersey: Lawrence Erlbaum Associates Publishers, 1991. 172 (Frederiksen&White, 1989) John R. Frederiksen and Barbara Y. White. An approach to training based upon principled task decomposition. Acta Psychologica, 71, 1989:89-146. (Galdes, 1990) Deborah K. Galdes. An empirical study of hum an tutors: The implications for intelligent tutoring systems. Ph.D. diss., The Ohio State U niversity, 1990. O rder N um ber 9031068, UMI D issertation Services, Ann Arbor, Michigan. (Genesereth, 1982) Michael R. Genesereth. The role of plans in intelligent teaching system s. Intelligent Tutoring Systems. E dited by Derek Sleeman and John Seely Brown. New York: Academic Press, 1982:137- 155. (Goldstein, 1982) Ira P. Goldstein. The genetic graph: A representation for the evolution of procedural knowledge. Intelligent Tutoring Systems. Edited by Derek Sleeman and John Seely Brown. N ew York: Academic Press, 1982:51-77. (Halff, 1988) Henry M. Halff. Curriculum and instruction in autom ated tutors. Foundations of Intelligent Tutoring Systems. Edited by M artha C. Poison and J. Jeffrey Richardson. Hillsdale, N ew Jersey: Lawrence Earlbaum Associates Publishers, 1988. (Hill&Johnson, 1993a) Randall W. Hill, Jr. and W. Lewis Johnson. Designing an intelligent tutoring system based on a reactive model of skill acquisition. Proceedings of the World Conference on Artificial Intelligence in Education (Al-ED 93), Edinburgh, Scotland, 1993. (Hill&Johnson, 1993b) Randall W. Hill, Jr. and W. Lewis Johnson. Im passe-driven tutoring for reactive skill acquisition. Proceedings of the 1993 Conference on Intelligent Computer-Aided Training and Virtual Environment Technology (ICAT-VET-93), N A S A /Jo h n so n Space Center, Houston, Texas, May 5-7,1993. (Hill&Lee, 1992) R andall W. H ill, Jr. and Lorrine Lee. Situation m anagem ent in the Link M onitor and Control O perator Assistant. Proceedings of the Second International Symposium on Ground Data Systems for Space Mission Operations, SPACEOPS-92, Pasadena, CA, November 16-20, 1992. (Johnson, 1986) W. Lewis Johnson. Intention-based diagnosis of novice programming errors. Los Altos, CA: M organ Kaufm ann Publishers, Inc., 1986. 173 (Johnson, 1990) W. Lewis Johnson. U nderstanding and debugging novice programs. Artificial Intelligence, 42, 1990:51-97. (Kautz&Allen, 1986) Henry A. Kautz and James F. Allen. Generalized plan recognition. Proceedings of the National Conference on Artificial Intelligence, Philadelphia, PA, 1986. (Kautz, 1991) Henry A Kautz. A formal theory of plan recognition and its im plem entation. Reasoning about Plans. San Mateo, CA: M organ Kaufmann Publishers, Inc., 1991:69-125. (Laird, 1987) John E. Laird, Allen Newell and Paul S. Rosenbloom. Soar: A n architecture for general intelligence. Artificial Intelligence, 33, 1987:1-64. (Laird, 1988) John E. Laird. Recovery from incorrect knowledge in Soar. Proceedings of AAAI-88. St. Paul, M innesota, 1988. Los Altos, CA: M organ Kaufmann Publishers, Inc., 1988. (Laird et al., 1990) John Laird, Clare Bates Congdon, Erik Altmann, and Kathy Swedlow. Soar User’s Manual: Version 5.2. Technical Report CSE-TR-72-90, E lectrical E ngineering an d C o m p u ter Science Departm ent, The University of Michigan, Ann Arbor, Michigan, 48109- 2122. (Langley&Ohlsson, 1984) Pat Langley and Stellan Ohlsson. A utom ated cognitive m odeling, Proceedings of the National Conference on Artificial Intelligence. Austin, Texas, 1984:193-197. (Langley et al., 1990) Pat Langley, James Wogulis and Stellan Ohlsson. Rules and principles in cognitive diagnosis. Diagnostic Monitoring of Skill and Knowledge Acquisition. H illsdale, N ew Jersey, Lawrence Erlbaum Associates Publishers, 1990. (Lee&Hill, 1992) Lorrine F. Lee and Randall W. Hill, Jr. Process control and recovery in the Link M onitor and Control O perator Assistant. Proceedings of the SOAR Symposium. H ouston, Texas: NASA Johnson Space Center, 1992. (Macmillan et al., 1988) Stuart A. Macmillan, David Emme and Melissa Berkowitz. Instructional planners: Lessons learned. Intelligent tutoring systems: Lessons Learned, Edited by Joseph Psotka, L. Dan Massey and Sharon A. M utter. H illsdale, N ew Jersey: Law rence Earlbaum Associates Publishers, 1988. 174 (Newell, 1990) Allen Newell. Unified Theories of Cognition. H arvard University Press, 1990. (Newell&Simon, 1972) Allen Newell and Herb Simon. Human problem solving. Englewood Cliffs, N ew Jersey: Prentice-Hall. (Ohlsson&Langley, 1988) Stellan Ohlsson and Pat Langley. Psychological evaluation of path hypotheses in cognitive diagnosis. Learning Issues for Intelligent Tutoring Systems,. Edited by H einz M andl and Alan Lesgold. New York: Springer-Verlag, 1988. (Regian, 1991) J. W esley Regian. Representing and teaching high perform ance tasks w ithin intelligent tutoring system s. Intelligent Tutoring Systems: Evolutions in Design. Edited by H ugh Bums, James W. Parlett and Carol Luckhardt Redfield. H illsdale, N ew Jersey: Lawrence Erlbaum Associates Publishers, 1991. (Reiser, 1985) Brian J. Reiser, John R. Anderson and Robert G. Farrell. D ynam ic stu d e n t m odelling in an in tellig en t tu to r for lisp programming. Proceedings of IJCAI-85. Los Angeles, CA, 1985. (Resch, 1991) George Resch. Interview of George Resch, M ember of Technical Staff and Engineer for Very Long Baseline Interferom etry (VLBI), Tracking Systems and Applications Section, Jet Propulsion Laboratory, August 12,1991. (Rickel, 1988) Jeff Rickel. An intelligent tutoring fram ew ork for task- oriented dom ains. Proceedings of the International Conference on Intelligent Tutoring Systems.. Montreal, June 1-3, 1988. (Ritter&Feurzeig, 1988) Frank Ritter and Wallace Feurzeig. Teaching real tim e tactical thinking. Intelligent Tutoring Systems: Lessons Learned. Edited by Joseph Psotka, L. Dan M urray and Sharon A. M utter. Hillsdale, New Jersey: Lawrence Erlbaum Associates Publishers, 1988. (Rosenbloom&Newell, 1986) Paul S. Rosenbloom and Allen Newell. The chunking of goal hierarchies: A generalized model of practice. Machine Learning, Volume II. Edited by Ryszard S. M ichalski, Jaime G. Carbonell, and Tom M. Mitchell. Los Altos, CA: M organ Kaufmann Publishers, Inc., 1986: 247- 288. I 7 5 j (Rosenbloom et al., 1987) Paul S. Rosenbloom, John E. Laird and Allen Newell. Knowledge level learning in Soar. Proceedings of AAAI-87, Sixth National Conference on A rtificial Intelligence, S eattle, Washington, July 13-17, 1987:499-504. (Rosenbloom et al., 1988) Paul S. Rosenbloom, John E. Laird and Allen Newell. The chunking of skill and knowledge. Working Models of Human Perception. Edited by H. Bouma and A.G. Elsendoorn, New York: Academic Press, Inc., 1988:391-410. (Sacerdoti, 1975) Earl D. Sacerdoti. The nonlinear nature of plans. Proceedings of the Fourth International Joint Conference on Artificial Intelligence, 1975:206-214. (Sacerdoti, 1977) Earl D. Sacerdoti. A Structure for Plans and Behavior. New York: Elsevier Computer Science Library, 1977. (Snoddy, 1926) G.S. Snoddy. Learning and stability. Journal of Applied Psychology, 10, 1926:1-36. (Suchman, 1987) Lucy A. Suchman. Plans and situated actions. N ew York: Cambridge University Press, 1987. (Unruh, 1990) Amy U nruh and Paul S. Rosenbloom. Abstraction in problem solving and learning. Proceedings of AAAI-90, 1990. (VanLehn, 1982) Kurt VanLehn. Bugs are not enough: Empirical studies of bugs, im passes and repairs in procedural skills. The Journal of Mathematical Behavior, 3, 1982:3-71. (VanLehn, 1983) K urt VanLehn. Felicity conditions for human skill acquisition: Validating an Al-based theory. Tech. Report CIS-21. Palo Alto, CA: Xerox Palo Alto Research Center. (VanLehn, 1987) Kurt VanLehn. Learning one subprocedure per lesson. Artificial Intelligence, 31, 1987: 1-40. (VanLehn, 1988a) K urt VanLehn. Student m odeling. Foundations of Intelligent Tutoring Systems. Edited by M artha C. Poison and Jeffrey J. Richardson. H illsdale, New Jersey: Lawrence Erlbaum Associates Publishers, 1988:55-78. 176 (VanLehn, 1988b) K urt VanLehn. Toward a theory of im passe-driven learning. Learning Issues for Intelligent Tutoring Systems. Edited by Heinz Mandl and Alan Lesgold. New York: Springer-Verlag, 1988:19-41. (Vera et al., 1993) Alonso Vera, Richard Lewis and F. Javier Lerch. Situated decision-making and recognition-based learning: A pplying symbolic theories to interactive tasks. Proceedings of the Fifteenth Annual Conference of the Cognitive Science Society. H illsdale, N ew Jersey: Lawrence Erlbaum Associates Publishers, 1993. (Vera&Simon, 1993) Alonso H. Vera and H erbert A. Simon. Situated action: A symbolic interpretation. Cognitive Science, 17, 1993:7-48. (W ard, 1991) Blake W ard. ET-Soar: Toward an ITS for theory-based representations. Ph.D. diss., CMU-CS-91-146, School of Com puter Science, Carnegie Mellon University, Pittsburgh, PA. (W arinner et al., 1990) A ndrew W arinner, Diann Barbee, Larry Brandt, Tom Chen, and John Maguire. Building an intelligent tutoring system for procedural dom ains. Proceedings of the First CLIPS Conference, NASA Johnson Space Center, Houston, TX, 1990:881-892. (Warren&Goodman, 1993) Kimberly C. W arren and Bradley A. Goodman. E ngineering in tellig en t tu to rin g system s. 1993 Conference on Intelligent Computer-Aided Training and Virtual Environment Technology, (ICAT-VET-93), NASA Johnson Space Center, H ouston, Texas, May 5-7,1993. (W enger, 1987) Etienne W enger. Artificial Intelligence and Tutoring System s: Computational and C ognitive Approaches to the Communication of Knowledge. Los Altos, CA: M organ K aufm ann Publishers, Inc., 1987. (W hite&Frederiksen, 1990) Barbara Y. W hite and John R. Frederiksen. C ausal m odel progressions as foundation for intelligent learning environments. Artificial Intelligence, 42, 1990:99-157. (Wilkinson, 1992) Belinda Wilkinson. Operability engineering in the deep space network. Proceedings of the Second International Symposium on Ground Data Systems for Space Mission Operations, SPACEOPS-92, Pasadena, CA, November 16-20,1992. 177 (Woolf, 1991) Beverly Woolf. Representing, acquiring, and reasoning about tutoring knowledge. Intelligent Tutoring Systems: Evolutions in Design. Edited by H ugh Burns, James W. Parlett and Carol Luckhardt, Hillsdale, N ew Jersey: Lawrence Earlbaum Associates Publishers, 1991. I i Appendix A: Summative Evaluation Data This appendix contains the data collected on each subject for the evaluation desribed in Chapter 7. Subject #1, Group I (Training with Tutor) Pre-test Training Post-test Ml* M2* M3 M4 M5 M6 Total Number of Impasses 0 1 1 0 0 1 Action Constraint Impasses 0 1 1 0 0 1 Plan Dependency Impasses 0 0 0 0 0 0 Goal Failure Impasses 0 0 0 0 0 0 Number of Requests for Help? 0 0 0 0 0 0 Correspondence Impasse /Request 0 0 0 0 0 0 Number of Commands 12 13 15 16 11 14 Min. Number of Commands 11 13 13 14 11 13 Time Elapsed (seconds) 226 oo 185 210 98 147 Mean Time per Command (seconds) 19 n/a 12 13 9 10 Successfully Achieved Goals? (Y/N) Y N Y Y Y Y * Subject had the opportunity to training the first two missions. Subject #2, Group I (Training with Tutor) Pre-test Training Post-test Ml M2 M3 M4 M5 M6 Total Number of Impasses 1 1 2 1 0 1 Action Constraint Impasses 0 0 1 1 0 0 Plan Dependency Impasses 0 0 0 0 0 0 Goal Failure Impasses 1 1 1 0 0 1 Number of Requests for Help? 1 1 0 0 0 0 Correspondence Impasse /Request 0 1 0 0 0 0 Number of Commands 13 19 14 18 14 13 Min. Number of Commands 11 13 13 14 11 13 Time Elapsed (seconds) 632 624 256 234 174 139 Mean Time per Command (seconds) 49 33 18 13 12 11 Successfully Achieved Goals? (Y/N) N N Y Y Y N Subject #3, Group I I Pre-test Training Post-test (Training with Tutor) | Ml M2 M3 M4 M5 M6 Total Number of Impasses 0 4 1 2 0 1 Action Constraint Impasses 0 4 1 2 0 1 Plan Dependency Impasses 0 0 0 0 0 0 Goal Failure Impasses 0 0 0 0 0 0 Number of Requests for Help? 3 2 1 0 0 0 Correspondence Impasse /Request 0 1 1* 0 0 0 Number of Commands 12 19 14 16 11 14 Min. Number of Commands 11 13 13 14 11 13 Time Elapsed (seconds) 744 503 341 254 145 168 Mean Time per Command (seconds) 62 26 24 16 13 12 Successfully Achieved Goals? (Y/N) Y Y Y Y Y Y I I Subject #4, Group I (Training with Tutor) Pre-test Training Post-test Ml M2 M3 M4 M5 M6 Total Number of Impasses 1 2 3 4 0 1 Action Constraint Impasses 0 1 1 2 0 1 Plan Dependency Impasses 0 2 0 0 0 0 Goal Failure Impasses 1 1 2 2 0 0 Number of Requests for Help? 2 1 0 0 0 0 Correspondence Impasse /Request 1 1 0 0 0 0 Number of Commands 12 9 17 17 11 13 Min. Number of Commands 11 13 13 14 11 13 Time Elapsed (seconds) 862 OO 654 418 201 285 Mean Time per Command (seconds) 72 n/a 38 25 20 22 Successfully Achieved Goals? (Y/N) N N Y Y Y Y Subject #5, Group II (Training without Tutor) Pre-test Training Post-test Ml M2 M3 M4 M5 M6 Total Number of Impasses 1 4 2 0 0 0 Action Constraint Impasses 1 1 2 0 0 0 Plan Dependency Impasses 0 3 0 0 0 0 Goal Failure Impasses 0 0 0 0 0 0 Number of Requests for Help? 2 1 1 0 0 0 Correspondence Impasse /Request 1 1 1 0 0 0 Number of Commands 14 13 15 16 11 14 Min. Number of Commands 11 13 13 14 11 13 Time Elapsed (seconds) 928 OO 441 248 152 169 Mean Time per Command (seconds) 66 n/a 29 16 14 12 Successfully Achieved Goals? (Y/N) Y Y Y Y Y Y 180 Subject #6, Group II (Training without Tutor) Pre-test Training Post-test Ml M2 M3 M4 M5 M6 Total Number of Impasses 1 3 2 3 1 2 Action Constraint Impasses 0 2 1 2 0 1 Plan Dependency Impasses 0 0 0 0 0 0 Goal Failure Impasses 1 1 1 1 1 1 Number of Requests for Help? 4 2 0 0 0 0 Correspondence Impasse /Request 0 1 0 0 0 0 Number of Commands 14 19 24 17 11 14 Min. Number of Commands 11 13 13 14 11 13 Time Elapsed (seconds) 632 488 854 234 144 188 Mean Time per Command (seconds) 45 26 36 14 13 13 Successfully Achieved Goals? (Y/N) N N N N N N Subject #7, Group II (Training without Tutor) Pre-test Training Post-test Ml M2 M3 M4 M5 M6 Total Number of Impasses 1 1 2 3 0 2 Action Constraint Impasses 1 1 2 3 0 1 Plan Dependency Impasses 0 0 0 0 0 0 Goal Failure Impasses 0 0 0 0 0 1 Number of Requests for Help? 4 3 3 4 0 0 Correspondence Impasse /Request 1 2 2 3 0 0 Number of Commands 12 11 15 22 11 14 Min. Number of Commands 11 13 13 14 11 13 Time Elapsed (seconds) 810 OO 784 736 147 161 Mean Time per Command (seconds) 68 n/a 52 33 13 12 Successfully Achieved Goals? (Y/N) Y N Y Y Y N Appendix B Acronyms DSCC - Deep Space Communications Complex DSN - Deep Space Network i DSP - DSCC Spectrum Processing (Subsystem) DTE - Digital Tone Extractor ITS - Intelligent Tutoring System JPL - Jet Propulsion Laboratory LMC - Link Monitor and Control NASA - National Aeronautics and Space Adm inistration NCB - N arrow Channel Bandwidth NOCC - Network Operations Control Center PCG - Phase Calibration Generator PPM - Precision Power Monitor TDN - Temporal Dependency Network VLBI - Very Long Baseline Interferometry

Asset Metadata

Core Title 00001.tif

Tag OAI-PMH Harvest

Permanent Link (DOI) https://doi.org/10.25549/usctheses-oUC11255711

Unique identifier UC11255711

Legacy Identifier DP22865

Linked assets

University of Southern California Dissertations and Theses