A script-based approach to modifying knowledge -based systems

PDF

Download

Open document

Flip pages

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content INFORMATION TO USERS This manuscript has been reproduced from the microfilm master. UM I films the text directly from the original or copy submitted. Thus, some thesis and dissertation copies are in typewriter face, while others may be from any type of computer printer. The quality of this reproduction is dependent upon the quality of the copy submitted. Broken or indistinct print, colored or poor quality illustrations and photographs, print bleedthrough, substandard margins, and improper alignment can adversely affect reproduction. in the unlikely event that the author did not send UMI a complete manuscript and there are missing pages, these will be noted. Also, if unauthorized copyright material had to be removed, a note will indicate the deletion. Oversize materials (e.g., maps, drawings, charts) are reproduced by sectioning the original, beginning at the upper left-hand corner and continuing from left to right in equal sections with small overlaps. Photographs included in the original manuscript have been reproduced xerographicaliy in this copy. Higher quality 6" x 9” black and white photographic prints are available for any photographs or illustrations appearing in this copy for an additional charge. Contact UMI directly to order. ProQuest Information and Learning 300 North Zeeb Road, Ann Arbor, M l 48106-1346 USA 800-521-0600 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. A SCRIPT-BASED APPROACH TO MODIFYING KNOWLEDGE-BASED SYSTEMS by Marcelo Tallis A Dissertation. Presented to the FACULTY OF TH E GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree D O CTO R OF PHILOSOPHY (Com puter Science) Decem ber 2000 Copyright © 2000 Marcelo Tallis Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. UMI Number: 3018037 __ ___ __< § ) UMI UMI Microform 3018037 Copyright 2001 by Bell & Howell Information and Learning Company. All rights reserved. This microform edition is protected against unauthorized copying under Title 17, United States Code. Bell & Howell Information and Learning Company 300 North Zeeb Road P.O. Box 1346 Ann Arbor, Ml 48106-1346 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. UNIVERSITY O F SOUTHERN CALIFORNIA T H E GRAD UATE SCHOOL UNIVERSITY PARK L O S A N G ELES. CALIFORNIA 90007 This dissertation, written by U D . ...... under the direction of V lmz.. Dissertation Committee, and approved by all its members, has been presented to and accepted by The Graduate School in partial fulfillment of re quirements for the degree of DOCTOR OF PHILOSOPHY Dean a) ite Studies Date ..A p .r il..2 .3 J .„1999. DISSERTATION COMMITTEE P&O ( — fio S g T V ) B C O Q if^ A X o U L ^ - e ^ A f t , ------- . . . S ( l - l - I X n > O ' L s s n O -V 0 f Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Abstract Modifying knowledge-based systems (KBS) is a complex activity. One important aspect of this complexity is th a t several related portions of the system m ust be changed in concordance in order to avoid leaving the system in an incoherent state. In addition, it is difficult for users to figure out on their own w hat m ust be changed and how to change it. Current knowledge acquisition (KA) approaches attem p t to avoid this difficulty by either lim iting the range of modifications th at users are allowed to make or providing the user with lim ited guidance in performing those modifications. This thesis presents an approach th at provides strong guidance for a wide range of modifications. In this approach, knowledge of prototypical procedures for modifying knowledge- based systems is used to guide users in changing all related portions of a system. These procedures, called knowledge acquisition scripts (or I\A Scripts), capture how related portions of a knowledge- based system can be changed in coordination. By using KA Scripts, a knowledge acquisition tool would be able to relate individual changes to a knowledge base, enabling the analysis of each individual change from the perspective of the overall modification. To design a script-based knowledge acquisition tool we needed to address some im portant research issues concerned with designing KA Scripts, coordinating the application of KA Scripts, and interacting flexibly with the user. Based on the analysis of several knowledge-based systems modification scenarios, we developed a system atic m ethod for building a KA Scripts library with good coverage. We also implemented ETM , a script-based KA tool th a t builds on the EX PECT framework for building knowledge-based systems. Empirical evaluations of ETM show th a t ETM effectively guided users through all of the required changes in realistic KBS modification scenarios, and suggest th a t ETM iii Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. users complete a KBS modification in less tim e than users of other KA approaches that offer lim ited help. Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Acknowledgements This Thesis would uot have been possible w ithout the dedication and support from several persons. First I would like to thank my advisor Yolanda Gil. Yolanda was an incredible teacher and an exceptional human being. She did not confine herself to merely guiding me through the realization of this dissertation. In fact, she was truly com m itted to make me a competent and conscious researcher th at can genuinely contribute to the advancement of science. She taught me innumerable lessons, from recognizing valuable research to communicating my research to others effectively. In addition, she has offered my family and me her valuable support during hard times. I am also thankful to the other members of m y thesis committee, Bill Swartout, Paul Rosenbloom, and Daniel O ’Leary, for the valuable comments th a t greatly contributed to the clarity of the thesis. I also would like to thank the other members of the EXPECT group, including Jihie Kim, Jim Blythe, Surya Ramachandran, Andre Valente, and Bing Leng, for contributing to the development of the EX PECT software and for their insightful discussions. Together with Ramesh Pattil and Eric Melz, they also collaborated with the evaluation of ETM. Many people contributed to my research by offering good advice, commenting on my papers, and improving my presentations. In addition to the members of the EXPECT group, Kevin Knight, Jose Luis Ambite, Ion Muslea, Richard Angros, Ali Erdem, Gal Kaminka, and Rogelio Adobbati were specially helpful. I was also glad to have met other people who have offered me their friendship and support, including Cristina, M artha, Vibhu, Adrian, Laura, Lori, Jose, Fojo, Alicia, Ariel. Laura, Ernesto, and Elisa. I am also thankful to the Organization of American States (OAS) who supported the first years of my Ph.D. I cannot thank enough my parents who were always affectively and physically available whenever I needed them . I hope that some day they will forgive me for the time I have kept them away from their grand children. I am also thankful to my mother-in-law Luci and to my uncles Victor and Mario who were also there when I needed them . Finally, I am grateful to my wife M arina and my children Federico and Melisa who have ventured with me on this endeavor and have patiently waited for me throughout all these years. Marina, I dedicate this dissertation to you. Your strength and courage will always amaze me. You supported me through out all these years, consuming precious time and health, for the love th at you have for me. Only you know how much all this has cost you. I hope to be able to pay it back to you from now on. This work was supported in part by a scholarship from the Organization of the American States (OAS), in part by the Defense Advanced Research Projects Agency (DARPA) with contract DABT63-95-C-0059 as part of the D A RPA /Rom e Laboratory Planning Initiative and with grant F30602-97-1-0195 as part of the DARPA High Performance Knowledge Bases Program. The views and conclusions contained in this thesis are the author’ s and should not be interpreted as repre senting the official opinion or policy of any of the above organizations or any person connected with them. v Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Contents Abstract iii Acknowledgements v List Of Tables x List Of Figures xi 1 Introduction 1 1.1 The Need for Modifying Knowledge-Based S y stem s............................................................ 2 1.2 C urrent Knowledge Acquisition Tools and their L im ita tio n s............................................ 4 1.3 Thesis and A pproach..................................................................................................................... 7 1.4 C o n trib u tio n s.................................................................................................................................. 8 1.5 Guide to the rest of the Thesis D o c u m e n t............................................................................ 9 2 A Script Based Approach to Modifying Knowledge-based systems 10 2.1 Difficulties of Knowledge-Based System Modifications..............................................................10 2.2 Proposed Approach: Script Based Knowledge A c q u isitio n ...................................................14 2.3 Research Issues of script-based knowledge-acquisition.............................................................16 3 Designing a Script-Based Knowledge Acquisition tool 19 3.1 Designing KA Scripts ..................................................................................................................... 19 3.2 Coordinating the Application of KA S c r ip ts .............................................................................24 3.3 Interacting Flexibly with U s e r s .................................................................................................... 26 4 ETM: A Script-Based Knowledge Acquisition Tool 28 4.1 Background: EXPECT KBS Framework and Baseline KA t o o l ..........................................28 4.1.1 Knowledge R epresentation................................................................................................. 29 4.1.2 Problem S o lv e r..................................................................................................................... 29 4.1.3 EX PEC T KA T o o l.............................................................................................................. 31 4.2 ETM : EX PE C T ’s Script-Based KA T o o l................................................................................... 32 4.2.1 Representing KA S c rip ts.....................................................................................................33 4.2.2 Coordinating the Application of KA S c rip ts................................................................. 36 4.2.3 Interacting Flexibly with the U s e r ................................................................................37 4.2.4 Using Context I n fo r m a tio n ............................................................................................. 38 4.3 An Exam ple Scenario for E T M .................................................................................................... 41 VI Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 5 Developing a KA Script Library for EXPECT 45 5.1 Developing a Library of KA S c rip ts..............................................................................................45 5.1.1 Analysis of Syntactic C hanges............................................................................................ 45 5.1.2 Analysis of KBS Modification S c e n a rio s ........................................................................47 5.1.2.1 Case study I: Initial prototype im p lem en tatio n ..........................................50 5.1.2.2 Case study II: Extending im plem ented p ro to ty p e .......................................51 5.1.2.3 C onclusions.............................................................................................................53 5.1.3 Analysis of Attributes of Knowledge Acquisition S c r ip ts ........................................... 54 5.2 An Im plem ented KA Script Library ...........................................................................................56 6 A Study of Users Modifying Knowledge-Based Systems: Evaluation of ETM 58 6.1 General Hypotheses .........................................................................................................................59 6.2 Overview o f the experiments ........................................................................................................60 6.3 Study I: Controlled E x p e rim e n ts ................................................................................................. 60 6.3.1 Results .................................................................................................................................... 60 6.3.2 P ro c e d u re .................................................................................................................................61 6.3.3 Analysis of the R e su lts......................................................................................................... 65 6.4 Study II: Realistic and Unseen Scenarios.................................................................................... 72 6.5 S u m m a r y .............................................................................................................................................74 7 Related Work 76 7.1 Support for KBS M o d ificatio n s.................................................................................................... 76 7.2 Support for KB R e fin e m e n t........................................................................................................... 78 7.3 Verification & V alidation..................................................................................................................79 7.4 Knowledge Based Software Engineering ....................................................................................80 8 Conclusions 82 8.1 Sum m ary of the Approach and R e su lts....................................................................................... 82 8.2 C o n trib u tio n s......................................................................................................................................83 8.3 Lim itations and Future W o r k ........................................................................................................83 Reference List 85 Appendix A Knowledge-Acquisition Scripts L ib ra ry ..................................................................................................89 A .l KA Script A ttrib u te s ........................................................................................................................ 89 A.2 Im plem ented Knowledge-Acquisition S c rip ts .............................................................................94 Appendix B Experim ent M aterials used in Study I: Controlled Experim ents - Round II ...........................101 B .l Problem -Solving Knowledge fra g m e n ts..................................................................................... 101 B.2 D om ain C o n c e p ts............................................................................................................................. 104 B.3 Task specification for RTT Scenario ........................................................................................104 B.4 Task Specification for PAE S c e n a r io ........................................................................................ 106 B.5 EX PE C T M ethods in Initial Knowledge B a s e ........................................................................108 B.6 D om ain Concepts in Initial Knowledge B a s e ...........................................................................112 B.7 LOOM Factual Dom ain Instances in Initial Knowledge B a s e ............................................ 115 Vll Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. A p p e n d ix C Logs From Study I: Controlled E xperim ents.................................................................................... 118 C .l Scenario RTT - Round I ..............................................................................................................118 C.2 Scenario RTT - Round II .......................................................................................................... 119 C.3 Scenario PAE - Round I ..............................................................................................................120 C.4 Scenario PAE - Round II .......................................................................................................... 122 A p p e n d ix D Logs From Study II: Realistic S c e n a rio s .......................................................................................... 124 v i i i Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. List O f Tables 4.1 Some inconsistencies th a t EX PECT can d e t e c t ....................................................................... 32 6.1 Study I: Results RTT-I scenario - Summary per s u b je c t......................................................67 6.2 Study I: Results R TT-II scenario - Summary per subject .................................................. 67 6.3 Study I: Results PAE-I scenario — Summary per s u b j e c t ......................................................70 6.4 Study I: Results PAE-II scenario - Summary per subject .................................................. 70 6.5 Study II: Number of follow-up modification commands .......................................................74 6.6 Study II: Base m ethod selection for new m e th o d s....................................................................74 IX Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. L ist O f F igu res 2.1 Exam ple scenario - Initial KBS: m o v e m e n t s ’s lift are s h i p s ...................................................11 2.2 Exam ple scenario continuation — Changes needed to make m o v e m e n t s ’s lift be any v e h i c l e ....................................................................................................................................................................................................12 2.3 An example of a very generic KA Script...................................................................................... 15 2.4 An example of a KA Script for a more specific situation........................................................ 16 2.5 The KA Script of Figure 2.4 applied to the example of Figure 2.2........................................16 4.1 Some definitions of concepts and problem solving m ethods in a simplified transporta tion dom ain............................................................................................................................................30 4.2 KA Script representation exam ple — I .................................................................. 33 4.3 KA Script representation exam ple — I I .................................................................. 34 4.4 KA Script representation exam ple — III ............................................................... 34 4.5 ETM ’s Control L o o p ........................................................................................................................ 37 4.6 Snapshot of ETM ’s user in te rfa c e .................................................................................................39 4.7 An exam ple of a KBS modification s c e n a rio ............................................................................ 42 5.1 Exam ple of cluster of changes between versions for a m e t h o d ........................................... 49 6.1 Study I: Detailed analysis of execution tim e for R T T - I ........................................................ 68 6.2 Study I: Detailed analysis of execution tim e for R T T - I ........................................................ 69 6.3 Study I: Detailed analysis of execution tim e for P A E - I ........................................................ 71 6.4 Study I: Detailed analysis of execution tim e for P A E -II........................................................ 72 B .l Study I: Domain concepts and relations — Part I ................................................................ 105 B.2 Study I: Domain concepts and relations — Part I I .............................................................106 B.3 Study I: Dom ain concepts and relations — Part I I I .............................................................107 x Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Chapter 1 Introduction Modifying knowledge based systems (KBSs) is a complex and common activity. Once a prototype knowledge-base (KB) is initially developed, users would like to maintain and extend it throughout its lifetime. One of the difficulties of modifying knowledge-based systems is that after changing portions of the system, other related portions might also need to be changed to conform to the former changes. Determ ining what other portions have to be changed and how to change them to conform to previous changes requires a deep understanding of how the elements within the KB interact. This requirement is especially hard to fulfill if the rationale behind the design of a system or the details of its implementation are unknown, which is often the case due to staff m igration or third party software development. Current knowledge-acquisition (KA) tools offer lim ited support for making changes to a KBS. This thesis presents a new approach called script- based knowledge acquisition (or SBKA) that uses a library of typical knowledge-based systems modification procedures (or I\A Scripts) to guide users through the modification of knowledge- based systems. The rest of this chapter highlights the importance of tools to support knowdedge- based system modifications, surveys current research on this topic, and describes our approach, finalizing with a guide to the rest of the dissertation. 1 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 1.1 The Need for Modifying Knowledge-Based Systems There are several reasons why knowledge-based systems need to be modified and updated repeatedly over tim e. First, to conform to the ch an g es in th e e n v iro n m e n t where the system operates. For example, financial systems that deal with tax laws (e.g., EXPERTax [Shpilberg et al., 1986]) have to be modified each tim e that the tax legislation changes significantly. Systems th at deal with product inform ation (e.g.. R1 [Bachant and M cDerm ott, 1984], Pimtool [Dev and Anderson, 1997]) have to be modified each time th at new products are developed. Second, modifications need to be m ade in order to d e b u g in a d e q u a te k n o w le d g e . Unlike conventional software systems, a knowledge-based system is both a piece of software and a model of knowledge and reasoning [O’Keefe and O ’Leary, 1993]. As a model, a KBS will never be perfect and will tend to evolve over tim e as the decisions th a t the system models become better understood. Finally, to conform to n e w re q u ire m e n ts for extending the system functionality and to c u sto m iz e the system to user preferences and institutional practices. Many of these modifications cannot be predicted before hand and can involve changes to any kind of knowledge represented in the system. The significance of this issue was very well docum ented in [Bachant and M cDerm ott, 1984] by the developers of the well-known and long-lived R1 knowledge-based system for configuring VAX com puter systems developed at Digital Equipm ent Corporation. When R1 was first put into production after one year of development, its developers believed that R1 already had most of the configuration knowledge th a t it would ever need. They expected that the only modifications th at R1 would undergo would be minor bug fixes and the addition of knowledge specific to new system types. However, these expectations turned out to be false. By the tim e th a t the cited article was w ritten after four years of R1 being operational, R l’s knowledge base was growing at a constant rate to the point of quadrupling its size (from 850 rules to 3300 rules). O f this added knowledge, over 65% was concerned with extending R l ’ s general configuration capabilities and only the remaining 35% was concerned with knowledge about specific system types. O f this 65%, 2 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. a t least 15% was added to correct or refine knowledge about how to perform some subtasks that supposedly the system already knew how to perform. In this article, the authors state (p. 28): “Though much of R l ’ s knowledge was added to correct o r complement existing knowl edge, a significant p art of the additions came as a result of R1 having to have the knowledge to perform new tasks. Some of these were the result of Digital introducing new com puter system types and the rest resulted from the user’s observations that things would be better if R1 could do one more thing. We believe all expert systems will be bound to continue to grow for both of these reasons. . .” The authors articulate eloquently the prominence of this process (p. 21): “At the beginning of R l ’s development, no clear expectations existed about how long it would take to collect enough knowledge to make R1 an expert. We did expect that at some point the rate at which R1 would acquire new knowledge would at least slow, if not stop. We even thought th a t R1 would be done eventually (that is, R1 would enter a m aintenance mode of well defined and minor additions, interspersed with occasional bug fixes.) It is difficult now to believe R1 will ever be done; we expect it to continue to grow and evolve for as long as there is a configuration task. It may be that if R l ’s dom ain were less volatile, R1 would not require perpetual development. But it is probably also true th at if the dom ain were less volatile, the task would not require a knowledge-based system .” From all the reasons exposed so far we can conclude th at there is a real need for KA tools that support users in modifying KBSs. Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 1.2 Current Knowledge Acquisition Tools and their Limitations The approach adopted by most knowledge-acquisition tools to manage the complexity inher ent with modifying knowledge-based systems is to restrict the type of modifications that users are allowed to perform (e.g., SALT [Marcus and M cDermott, 1989], PROTEGE II [Puerta et al., 1992], and DIDS [Runkel and Birmingham, 1993]). In this role-limiting approach, knowledge ac quisition tools assume th at the knowledge-based system follows a predefined inference structure or problem-solving m ethod (PSM ). They use expectations derived from this problem-solving method to support only the addition and modification of the domain-dependent knowledge that fills the knowledge roles of the problem-solving m ethod. For instance, SALT [Marcus and McDermott, 1989] is a knowledge acquisition tool tailored to the propose-and-revise problem-solving method for solv ing system-configuration problems. SALT supports only the acquisition of the domain-dependent knowledge used by propose-and-revise such as procedures that propose initial values to different configuration param eters. By using knowledge that is specific to the problem-solving methods, these knowledge acquisition tools can provide a very strong support for modifying knowledge- based systems. However, its applicability is severely limited since a) they can only be used for a particular PSM, and b) they can only support modifications to the domain-dependent knowl edge specified by the corresponding PSM. PRO TEG E-II [Puerta et al., 1992] adds flexibility to the role-limiting approach by allowing the assembly of a problem-solving m ethod from a library of smaller components called mechanisms. This library of mechanisms might include alternative implementations of the same mechanism in order to allow a system developer to choose the one th at best fits the requirements of the application dom ain. Although PROTEGE-II adds flexibility to standard role-limiting m ethod tools, it is still limited because it cannot support modifications to the mechanisms themselves. Another class of KA tools, such as TAQL [Yost, 1993, Yost, 1992] and EXPECT [Gil, 1994] have adopted an alternative approach th a t does not have the above lim itations. These tools do 4 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. not restrict the type of modifications th at users can perform and support modifications to any kind of KBS independently from its underlying problem-solving method. However, the trade-off for this gain in flexibility is a weaker support for modifying a KBS. TAQL is a KA tool that imple ments a high level problem-solving m ethod called Problem Space Computational Model (PSCM) [Newell et al., 1991], the PSM underlying the SOAR [Rosembloom et al., 1991] general-purpose intelligent architecture. This problem-solving method defines a few generic knowledge roles for directing the search performed by the system to solve a problem (e.g., propose and select prob lem spaces, initial states, and operators; apply operators; and recognize desired states). TAQL provides a language for expressing this knowledge and a compiler to translate TAQL statements into SOAR declarations and production rules. TAQL also provides a suite of utilities for adding and debugging TAQL statem ents, however these utilities concentrate only in superficial aspects of a KBS (e.g., syntax). Consequently, the kind of support th a t TAQL provides is still too weak to guide users through complex m odifications to knowledge-based systems. EX PECT [Gil, 1994, Sw artout and Gil, 1995] is a framework for building knowledge-based systems th at provides a se m antically rich knowledge representation language and a flexible KA tool. EX PECT’s KA tool derives expectations for supporting knowledge-based systems modifications based on a general un derstanding of the interactions am ong the system elements. In [Gil and Melz, 1996] the authors describe how EXPECT, based on these expectations, can determ ine what knowledge is missing in the knowledge base and acquire the sam e knowledge that SALT can acquire and even more, illus trating the power and flexibility o f this approach. EXPECT supports users in modifying KBSs by identifying inconsistencies in the KB (e.g., a goal th at cannot be achieved with the available knowl edge) and suggesting changes th a t will repair them (e.g., adding a method capable of achieving the unachieved goal). Although E X PE C T ’s analysis of the KBS can be considered more thor ough than the one that TAQL performs, E X PE C T ’s suggestions for fixing inconsistencies are the same regardless of the context in which these inconsistencies were produced. Consequently, these suggestions are unnecessarily more general th at the advise th a t could be offered to the user. 5 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Consider the following example of a KBS modification in EX PEC T . One type of element repre sented in EX PEC T knowledge-bases is a method (or procedure) capable of achieving goals posted by other m ethods. Each m ethod declares the type of goals th at the m ethod is capable to achieve. Suppose now th at a goal posted by some m ethod makes a reference to a variable, and that a user changes the declaration of th at variable to a more general type. This change produces an incon sistency because the posted goal now becomes more general and cannot be m atched anymore with the m ethod th a t used to achieve it before. Based on an analysis of the interactions among KBS elements, EX PECT would detect th at there is a goal th a t cannot be achieved. However, it can only suggest some general remedies for unachievable goals, like creating a new m ethod, modifying another m ethod’s declaration, modifying the posted goal, or removing the m ethod that posted the unachievable goal. EX PE C T ’s suggestions are not as good as they could be since they were generated w ithout considering the context in which the inconsistency was produced. First, these suggestions are too general. A better suggestion would include an indication of how to modify the m ethod th a t was able to achieve the posted goal before the change in order to agree with the changed goal now. Second, some suggestions might be inappropriate given their context, as for example the removal of the m ethod which the user has ju st added. In order to produce more precise and meaningful suggestions it would be necessary to analyze each individual change from the perspective of the overall m odification and to understand how these changes are related. In conclusion, current KA tools either provide strong guidance by lim iting the type of modifi cations th at users are allowed to perform, or allow any type of modification but provide a weaker support. However, the support th at the latter kind of tools provide could be made stronger without harm ing its flexibility by enabling these tools to analyze each individual change from the perspec tive of the overall modification and to relate changes from different parts of the KBS, and this is precisely the aim of the approach introduced in this thesis. 6 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 1.3 Thesis and Approach To build a knowledge acquisition tool th at both provides strong guidance to users and allows a wide range of modifications for any type of knowledge-based system, we investigated the use of an alternative source of expectations: the knowledge of prototypical procedures for modifying knowledge-based systems. These procedures, which we call Knowledge Acquisition Scripts (or KA Scripts), capture how related portions of a knowledge-based system can be changed coordinately. They provide a context for relating individual changes to different parts of a knowledge-based system and hence enable the analysis of each change from the perspective of the overall modification. In addition, because KA Scripts are dom ain independent they can be applied to a variety of KBS types. The thesis of this work is that knowledge o f prototypical Knowledge-based systems modification procedures (KA Scripts) can be used to provide strong guidance fo r a wide range o f modifications and system types. This approach, which we named Script-Based Knowledge-Acquisition (or SBKA), uses a library of KA Scripts to guide a user in changing all related portions of the knowledge-based system in a consistent way. In developing this approach we have investigated several issues concerning the design of KA Scripts, how to arbitrate between the different KA Scripts th at apply to a given situation, and how to flexibly interact with users. To develop a library of KA Scripts we have performed a series of analyses of real and hypothet ical knowledge-based system modification scenarios to gain insight on procedures for modifying knowledge-based systems. Based on these analyses, v ve have developed a m ethod for building system atically a KA Scripts library with good coverage. Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Finally, we have implemented a script-based knowledge acquisition tool called ETM (for EX PEC T Transaction M anager1) that extends EX PECT’s support for modifying of knowledge-based systems. We have carried out several empirical studies th at tested subjects using ETM in com plex and realistic scenarios and compared the subjects performance in modifying knowledge-based systems with and without ETM . These studies have shown that ETM guides users throughout all the related changes of the KBS modification scenarios and suggest that ETM reduces the time required to complete KBS modifications. 1.4 Contributions The m ain contributions of this thesis are the following: • A new approach to guide users in completing a knowledge-based system modification using KA Scripts. The issues addressed include designing KA Scripts, arbitrating among alternative KA Scripts, and interacting flexibly with the user. • A methodology for system atically developing a KA Script library with good coverage of the typical procedures used in knowledge-based system modification. • An implementation of a script-based KA tool th a t contains a KA Scripts library, a suitable user interface and KA Script m anager. • An analysis and evaluation of the implemented script-based KA tool to demonstrate the advantages of the approach. 1 U sing a n a n a lo g y w ith d a ta b a s e s , we c a n view th e p rocess o f m o d ify in g th e k now ledge base a s a sequence o f tra n sa c tio n s, w h ere K A S c rip ts s u p p o r t u sers b y en fo rcin g t h a t tra n s a c tio n s a re c o m p le te d so th a t th e know ledge b a se is n o t left in c o h e re n t 8 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 1.5 Guide to the rest of the Thesis Document T he rem ainder of this thesis is organized as follows. Chapter 1.5 discusses some of the difficulties involved in modifying KBSs using an exam ple th at will be used throughout the rest of the thesis. Then it introduces script-based knowledge-acquisition, our proposed approach for supporting KBS modifications. C hapter 3 discusses how we have approached several research issues related to the design of an SBKA tool. Chapter 4 describes ETM , our im plem entation of a script-based KA tool, including how KA Scripts are represented, the algorithms used for coordinating the application of KA Scripts, and the user interface. Then we illustrate the use of ETM w ith a detailed example. T h at chapter also includes an introduction to the EX PECT framework for developing knowledge- based systems, the baseline KA tool th a t we extended with ETM . C hapter 5 describes our method for developing a library of KA Scripts. It also describes the analysis of the knowledge-based systems modification scenarios th at we carried out to get insight into w hat are typical procedures for modifying KBSs. C hapter 5.2 describes the empirical studies th at we carried out to evaluate ETM . Finally, Chapters 7 and 8 discuss related work and conclusions respectively. 9 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Chapter 2 A Script Based Approach to Modifying Knowledge-based systems Modifying a knowledge-based system (KBS) is a complex task. One of its difficulties is th at after changing portions of the KBS, other related portions m ight also need to be changed to conform to the form er changes. Determining what other portions have to be changed and how to change them requires a deep understanding of how the elements of the KBS interact and how these elements have to be changed coordinately. Supporting users in dealing with this kind of difficulties is precisely the aim of the approach th at this chapter introduces. We first describe some of the difficulties involved in modifying KBSs. Then we introduce the Script-Based Knowledge-Acquisition (SBKA) approach and explain how this approach would support users in dealing with those difficulties. Finally we will enumerate some of the research issues th at needed to be addressed in building a SBKA tool and th at are going to be elaborated further through out the rest of this document. 2.1 Difficulties of Knowledge-Based System Modifications Consider the following scenario concerning a KBS in a transportation planning domain (Figure 2.1). For this example we assume that a knowledge-based system consists of a) a model of a dom ain and b) problem-solving methods. The dom ain model describes concepts, relations, and their instances. Problem solving methods (or methods for short) describe procedures for achieving 10 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. M1: capability: estimate (RTT, MOVEMENT ?m) body: begin maximum ( com pute (RTT, m ? -> LIFT, ?m -> ROUTE)) end M2: capability: compute ( RTT, SHIP ?s, ROUTE ?r) body: begin divide (find (SAILING-DISTANCE,?r), ?s -> SPEED) end M3: capability: find (SAILING-DISTANCE, ROUTE ?r) body: begin end speed VEHICLE MOVEMENT SHIP route AIRCRAFT Figure 2.1: Example scenario - Initial KBS: m o v e m e n t s ’s lift are SH IPS goals in th at domain. Each m ethod consists of a capability description and a body. The capability description indicates the type of goal th at the m ethod is able to achieve and the parameters of the method, both stated in term s of the concepts included in the domain model. The m ethod’ s body describes a procedure for achieving the goals specified in the m ethod’s capability. The body may include subgoal expressions, that have to be achieved by other methods, and relational expressions th at refer to other elements of the dom ain model. The domain model of Figure 2.1 describes v e h i c l e s which can be specialized into s h i p s and a i r c r a f t . It also describes t r a n s p o r t a t i o n m o v e m e n t s whose l i f t s consist of s h i p s . This figure also shows three problem-solving m ethods. M ethod M l for estimating the round-trip time (R T T ) o f a M O VEM EN T which requires to compute the R T T o f the L IF T o f the M OVEM ENT fo r the R O U TE o f the M OVEM ENT. Because the lift of a movement consist of ships, this goal can be achieved by m ethod M2 which computes the R T T o f SHIPS. M ethod M2, in turns, needs to find the SAILIN G D ISTAN C E o f a R O U T E , which can in turn be achieved by m ethod M3. 11 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. M2: capability: compute ( RTT, SH body: begin M1: capability: estimate (RTT, MO body: begin maximum ( compute (RTT, m? -> U end MOVEMENT ?m -> ROUTE)) AIRCRAFT M|dwrdfe(frncfX&YING-OlSTANCE>?r), ^ ^ ;c^abfn^raira2(FliyiNGiTDISTANCE|?ROUTE ?r) r timiffiiiM'r - t R xG divide (find (SAILING-OISTANCI ?s -> SPEED) end M3: I capability: find (SAILING-DISTANCEj body: begin end Figure 2.2: Exam ple scenario continuation — Changes needed to make m o v e m e n t s ’ s lift be any V E H IC L E Suppose now th at due to advances in the transportation technology now the lift of a trans portation movement can include aircraft in addition to ships. This change can be implemented by generalizing the range of the l i f t relation to any kind v e h i c l e (Fig. 2.2, change (1))- Because the elements of this range are an argum ent of the goal c o m p u t e posted by M l, this goal becomes gen eralized too. T he original KB included methods to com pute the RTT of ships but did not include any method to com pute the RTT of aircraft. Consequently, the modified KBS will not be able to solve some problems. Therefore, additional changes are needed to complete the modification and leave the KBS in a coherent state. One possible follow-up for this change is to add a new m ethod (M2’) th at computes the R T T o f an aircraft (Fig. 2.2, change (2))1. This m ethod can be created by copying the m ethod th at applies to s h i p s (M2) and adapting this copy to a i r c r a f t . One of the differences between M2 'a n o th e r p o s s ib ility c o u ld b e to g e n eralize th e m e th o d t h a t c o m p u te s th e R T T o f sh ip s 1 2 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. and M2’ is th at instead o f the sailing distance, M2’ needs to find the FLYING DISTANCE o f a RO UTE. Because there is no m ethod capable of achieving this goal we need to provide one. One possibility is to create a new m ethod (M3’) based on the m ethod th at applied to the sailing case (M3). (Fig. 2.2, change (3)). In summ ary, we needed to m ake three different changes in order to complete the modification. If some of these changes would have been om itted, the KBS would have been left incoherent. There are several reasons th a t make it hard for a user to complete a modification such as the one ju st described: • C han g es in o n e e le m e n t o f th e K B S im p a c t to o th e r e le m e n ts: Due to interdepen dencies among KBS elements, modifications to one elem ent may require that other elements be modified, too. For example, the m ethod for estim ating the round-trip tim e of a move ment depends on the description of the vehicles used. W hen this description is modified, this method has to be modified too. • H id d e n in te rd e p e n d e n c ie s : Interdependencies am ong KBS elements may not be explicit in the system. They are often dynamically determined by a problem solver (e.g., through a search process) and supported by system-made inferences (e.g., class inheritance). In our example, the only reason for the interdependency between the method for computing the round trip time of a ship and the description of the vehicles used in movements is that this method was used for achieving an intermediate goal concerning round-trip time of movements. A user unaware of this fact will not be able to recognize this interdependency. • C ascad in g o f c h a n g e s: After changing a KBS elem ent, other elements might need to be changed too. Furtherm ore, each of these additional changes can in turn originate the need for yet other additional changes. It is hard for a user to track down and to keep in mind all the changes that are pending. 13 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. • I n c o m p a tib ility b e tw e e n m o d ifie d a n d e x is tin g know ledge: Changes and additions of knowledge should be made com patible with existing knowledge (i.e., should avoid redun dancies or contradictions). One of the changes in our example was to include a new method for computing the round trip tim e of an aircraft. However, the existence of a method for com puting the round trip time of a ship m ight suggest that generalizing this m ethod (to cover aircraft) is a better solution than creating a slightly different and almost redundant method specifically for aircraft. In summ ary, modifying a knowledge-based system often requires several individual changes to various elements, and all those changes m ust be carefully coordinated. A good starting point is to build knowledge acquisition (KA) tools th a t find problems with the knowledge-based system and alert users about them . And in fact, this is pretty much the kind of support th at a conventional compiler provides to programmers when they change their code. But helping users notice the problems only partially addresses the issue. Ideally, KA tools should also support the user in resolving these problems by making suggestions about what additional changes m ay be needed in the knowledge base. Current KA approaches either lim it the range of modifications th at users are allowed to make or can only provide very general and hence limited guidance in performing those modifications. To provide a stronger guidance for a wide range of modifications a KA tool needs to take into account the context in which changes are performed and the task that the user is trying to accomplish. 2.2 Proposed Approach: Script Based Knowledge Acquisition To develop a knowledge acquisition tool th at both provides strong guidance to users and allows a wide range of modifications for any type of knowledge-based system, we exploit knowledge of prototypical procedures for modifying knowledge-based systems. These procedures, which we call knowledge acquisition scripts (or I\A Scripts), capture how related portions of a knowledge-based 14 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. if a posted goal is m ade more general then generalize the m ethod th a t achieved the goal before being changed. Figure ‘ 2.3: An exam ple o f a very generic KA Script. system can be changed coordinately. They provide a context for relating individual changes to different parts of a knowledge-based system and hence enable the analysis of each change from the perspective of the overall modification. By focusing in the overall modification, a KA tool would be able to ensure th a t a user performs all the changes required to complete the modification [Gil and Tallis, 1997, Tallis, 1998]. A script-based KA tool (SBKA) supports users in modifying a KBS by executing KA Scripts th a t follow up an ongoing modification. Figure 2.3 describes an example of a fairly simple KA Script. T his KA Script indicates that a generalization of a goal can be followed by the generalization of the m ethod th at used to achieve it. Even a K A Script as general and simple as this one provides two im portant sources of guidance: a) which elem ent of the KBS could be changed to follow up the initial change (the method that achieved the goal before) and b) how to change th at method (generalizing it). Figure 2.4 describes another KA Script th at can also be applied to follow up a goal generalization but in a more specific situation. It indicates how to follow up a generalization of a goal that can be decomposed into subgoals by creating m ethods for each one of the subgoals. T he new methods can be created using as a model the m ethod th a t used to achieve the goal before being changed. This KA Script could be applied in the KBS m odification scenario of the previous section as shown in Figure 2.5. 2.3 Research Issues of script-based knowledge-acquisition Four m ain research issues of script-based knowledge-acquisition are addressed by this dissertation: 15 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. if a) a posted goal is made more general and b) the generalized goal now can be decomposed into two disjunctive subgoals, and c) one of these subgoals is equivalent to the goal before being gener alized then 1) duplicate the method th at achieved the goal before being changed (this m ethod can still achieve one of the subgoals), and 2) adapt this copy to the other subgoal. This way, the two methods combined would achieve the modified goal. Figure 2.4: An example of a KA Script for a more specific situation. M1: method: estimate (..., MOVEMENT ?m) body: M2: method: compute (..., SHIP ?s,...) body: find (SAILING-DISTANCE,...) MOVEMENT AIRCRAFT M2’: i method: compute (..., AIRCRAFT ?a,...) i body: find (FLYING-DISTANCE, ...), l ? c o R p r § t h < ^ |t h a t ^ c h i e \ ^ ^ •substitute itypestomatchi theiother subgoal X„ ^ j |§ H I P A I R C R A F T ; •ohange method body Figure 2.5: The KA Script of Figure 2.4 applied to the example of Figure 2.2. • T h e D e s ig n o f K A S c rip ts: We needed to decide how to organize knowledge about pro totypical KBS m odification procedures into practical KA Scripts that can be m anipulated efficiently by a SBKA tool. Some of the issues concerning the design of KA Scripts are the 1 6 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. type of modification procedure to be represented in KA Scripts, their appropriate level of generality, and their structure. • C o o rd in a tio n : The tool needs a mechanism to prioritize and arbitrate among all the KA Scripts that apply to a given situation. Several KA Scripts could be applicable not only because there can be alternative ways to follow up changes but also because a change can have several effects and each one of them might need to be followed up by a different KA Script. A script-based KA tool also needs to determine which problem to attend first since these problems might be related and the order in which they are handled may simplify or complicate the task. • F le x ib le in te ra c tio n w ith u sers: The tool should allow users to modify a KBS either by using KA Scripts or by performing isolated changes. It is unlikely that a KA Scripts library contains all possible strategies for following up a modification and users should have the liberty to follow strategies th at are not included in the library. Additionally, the tool should help the user to understand and follow the tool’s guidance avoiding that the user get lost in the ramifications of the modification. • D e v e lo p m e n t o f a K A S c rip t L ib ra ry : An effective KA Script library should include a set of non-redundant KA Scripts that cover m ost of the situations in which a user might get during a knowledge-based system modification. In addition, KA Scripts should be at an appropriate level of generality. Overly general KA Scripts do not provide an adequate guidance for users to follow. For example, a KA Script that indicates “Modify some m ethod” will not be as useful as a KA Script that indicates “Generalize the method for computing the round trip tim e of a ship to make it applicable for vehicles.” The following chapters elaborate our approach further. Chapter 3 discusses some of the design decisions that led us to the construction of a successful script-based KA tool. Chapter 4 describes the implementation of ETM, our SBKA tool. ETM is a tool for supporting modifications of KBSs 17 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. build in EX PEC T ([Swartout and Gil, 1995, Gil, 1994]). Finally, C hapter 5 describes ETM ’s KA Script library and the methodology th at we used to build it. 1 8 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Chapter 3 Designing a Script-Based Knowledge Acquisition tool The previous chapter introduced some of the research issues for developing script-based knowledge- acquisition (SBKA) tools: the design of KA Scripts, coordinating the execution of KA Scripts, interacting flexibly w ith the user, and the development of a comprehensive KA Script library. In this chapter we present our approach to those issues, except for the development of a KA Script library which will be described in C hapter 5. C hapter 4 provides more details regarding our im plem entation of a script-based KA tool. 3.1 Designing KA Scripts We needed to decide how to organize the knowledge about prototypical KBS modification proce dures into practical KA Scripts th at can be efficiently m anipulated by a SBKA tool. Some of the issues concerning the design of KA Scripts were: • Modification procedures So far the SBKA approach does not impose any constraints on the kind of KBS modification procedures th a t can be represented by KA Scripts. However, we believe that it would be better to concentrate on one particular kind of modification procedure in order to simplify 19 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. the development of a KA Script library and the assessment of the benefits of the proposed approach. W ith that respect, it is useful to distinguish the following types o f procedures: 1. Macros. Procedures for autom ating common sequences of changes in a knowledge-based system . The purpose of these procedures is to simplify and speed-up modifications to a KBS. An example of a procedure of this type might be a macro for splitting a problem solving method. This would be useful, for example, when a method is too complex or when one wants to reuse a portion of it. This procedure substitutes a fragment of the original method by an invocation to a new method, and creates a new method that corresponds to the substituted code. 2. Procedures for fixing errors. Procedures th at implement common remedies to known inconsistency or error types in the KB. The purpose of these procedures is to assist a naive user in fixing the error. An example might be a procedure that, to remedy the absence of a method for achieving a goal, adapts another method to th at purpose. This procedure would guide a user in choosing and adapting an existing m ethod to solve the unachieved goal. 3. Procedures for following up changes. Procedures for propagating the changes performed to one KB element to other related elements in the KB. The purpose of these procedures is to help a user to deal w ith the complexity of the interactions among elements of a KBS. An example of a procedure of this type might be a procedure th at modifies a m ethod to conform to a change of the goal th a t the method is required to achieve. These categories are not m utually exclusive. Some KB modification procedures may belong to more than one of these classes. For example, a macro can be used to fix an error, and this fix m ight correspond to the following up a previous change. We decided to focus on the procedures fo r following up changes because they would address one of the more difficult tasks 20 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. for users and because it would provide us with a good scheme for taking into consideration the context of a m odification, which was our m ain concern. • Following up changes versus following up their effects KA Scripts describe procedures for following up changes to KB-elements. However, we have found more practical to define these procedures based on the effects th a t these changes have upon the problem-solving process rather than on the kind of changes that were performed. T hat is, KA Scripts describe procedures for following-up effects of changes. The reason behind this decision is th at the actions required to follow up a change depend more on the effects produced by th at change than on the nature of the change itself. First, because the same change to a KB element will have different effects depending on how this element interacts with others, and the actions to follow up this change depend on those effects. For example, a change to generalize the range of a relation,1 like the first change in our example scenario of Figure 2.2 in C hapter 1.5, will have different effects whether the elements in this range were used as argum ents of a goal or as a dom ain for a relational expression.2 In the case of the goal, this change will cause a generalization of the goal. In the case of the relational expression, this change will cause a generalization of the domain of the relational expression. Depending on the case, this change should have to be followed up differently. In the goal case the continuation m ight change the method th a t used to achieve th a t goal, while in the case of the relational expression the continuation m ight change the definition of the referred relation. Second, because different changes may produce the same effect and this effect is followed up independently of its cause. 1A re la tio n h as a n a m e , a d o m a in , a n d a ra n g e . It defines re la tio n s b etw een elem en ts o f its d o m ain a n d its ra n g e . F o r e x a m p le , in th e K B S o f C h a p te r 1.5, F ig u re 2.1, th e re la tio n L IFT re la te s in stan c es o f TRANSPORTATION-M OVEM ENT ( its d o m a in ) w ith in sta n c e s o f SH IP (its ra n g e ) 2 A re la tio n a l e x p ressio n h a s tw o a r g u m e n ts , a re la tio n n a m e a n d a n e le m en t in th e re la tio n ’s d o m ain . I t re tu rn s th e e le m en ts from th e ra n g e o f th e re la tio n t h a t a re re la te d to th e g iv en d o m a in e le m en t. F o r ex am p le, th e re la tio n a l e x p re ssio n (LIFT c o a l-m v m n t-2 ) w ill r e tu r n a list o f a ll th e sh ip s (its ra n g e ) th a t a re re la te d to th e d o m a in in sta n c e c o a l-m v m n t-2 by th e re la tio n L IFT 21 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Our first attem pt was to define KA Scripts based on the type of change th at was performed in the KB element, b u t this produced very redundant and cumbersome KA Scripts. The problem was th a t each KA Script had to include provisions for every possible consequence that the change to be followed up can have. However, because most of these consequences are independent of the change itself, the same provisions have to be repeated in many KA Scripts. • Generality The more general a KA Script procedure is, the more situations it will cover but the weaker the guidance it will provide to users. Conversely, the more specific a KA Script is, the stronger the guidance it will provide to users but the less the opportunities for applying it. To design KA Scripts a t an appropriate level of generality both factors should have to be balanced. The generality of a KA Script procedure is in direct relation with the generality of the situations th at the KA Script procedure is intended to follow up. If the situation is very general the follow up procedure is going to be general too. The situation to be followed up is determined by two aspects, the change (or effect of change) to be followed up, and the context in which th at change was produced. Changes to a KBS can be described at different levels, from a purely syntactic level (e.g., change a goal argum ent from X to Y) to a very abstracted level (e.g., modify a problem -solving method so instead of considering trips exclusively with ships, it now considers aircraft too). While a low-level description would make it easier to describe and formulate changes, it fails to capture the intention behind each change, and KA Scripts for following them up turnout so general th at they are typically not very useful. For example, consider a KA Script th a t propagates any change in a goal argum ent to the m ethod that is supposed to achieve th a t goal. This KA Script cannot provide too much guidance in modifying th a t m ethod because this modification would be very different depending on the specifics of the change in the goal (e.g., whether the goal was m ade more general or more 22 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. specific). Another drawback of using low level descriptions is that descriptions at this level m ight constitute only a sm all fraction of the change that has to be followed up. For example, several param eters might have changed in a goal expression, and it would be better to follow the whole goal change at once and not each change to a parameter in isolation. Therefore, we believe it is more useful to describe changes at a more conceptual level. However, we have found difficult to systematically enumerating changes at this higher level, while enumerating syntactic changes is a relatively easy task. Descriptions of the context in which changes are performed usually make reference to the content of the KBS (e.g., that a domain concept can be partitioned by a set of subconcepts) and to previous changes (e.g., a particular method had been created from a copy of another m ethod). • Scope of KA Scripts There can be many alternative ways of following up a change. Several of these alternatives might be similar and have some steps in common. This seems to indicate that it might be more convenient to define one single KA Script to cover all similar alternatives and to let the final direction that that KA Script takes to be determined during its application. However, in our experience we have found more practical to define one KA Script per single alternative. T hat is, different alternatives are represented by different KA Scripts. The reason behind this decision is that KA Scripts that im plem ent a single alternative are easier for users to understand and to follow up. This turned out to be very important because it is the user who indicates which one of the applicable KA Script is the most appropriate for a given situation. And in order to effectively choose one, the user needs to understand what is going to be the effect of applying a KA Script. • Size The sequence of steps of a KA Script has to be short and simple, and not complicated with provisions for handling all possible consequences of its own changes. It is better to 23 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. attend these consequences in separate KA Scripts. The rationale behind this is that m ost consequences of a change are independent from the other changes in the KA Script, hence it does not help to include provisions for handling all of them as part of the sam e KA Script. In addition, shorter KA Scripts are easier for users to understand and follow up. • Order of steps The sequence of steps in a KA Script has to be easy for users to follow and understand. In addition, the sequence of steps in a KA Script should be ordered to maximize the support that the tool can provide in executing these steps. Sometimes, the way a user executes a step sheds some light on how she may go about executing further steps. Consequently it is preferable to place earlier any step th a t can be analyzed to guide subsequent steps. For example, it is preferable to perform the changes that affect the values assigned to a variable before the changes th at concern the use of that variable. 3.2 Coordinating the Application of KA Scripts Since several KA Scripts may be relevant to a given situation, there needs to be a policy th a t prioritizes and arbitrates among them . Several KA Scripts could be relevant not only because there are several alternatives to follow up a modification but also because a single change can produce several effects and each of them might require the execution of a different KA Script. We also needed to determine when to execute a KA Script. Changes performed during the application of a KA Script can require the execution of additional KA Scripts. We had to decide whether to execute the KA Scripts th at follow up on those changes imm ediately after the execution of those changes or to defer their execution to the future. 24 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. The following are some of the determ inations concerning these issues: • When to launch a KA Script Changes performed by KA Scripts are not followed up by other KA Scripts until the whole execution of the KA Script is com pleted. Only after the whole KA Script is executed can other KA Scripts be launched to attend the effects of the changes th a t still need to be taken care of. Not every change executed by a KA Script will need to be followed up by other KA Scripts. Some of these changes m ight have already been taken care by the other changes of the same KA Script. To help us to identify which of the changes performed by a KA Script still need to be addressed, first we look at the inconsistencies left in the KB after the execution of a KA Script is over, and then we determine which changes have produced those inconsistencies. The changes th a t produced those inconsistencies are the ones th at still need to be followed up. In the past, we used to launch KA Scripts immediately after the execution of a change. However, this practice was counterproductive for two reasons. First, because it produced a high degree of nesting th at m ade it hard for users to follow the thread of the modification. And second, because some of the problems introduced during the execution of a KA Script m ight get autom atically fixed by the successive changes of the sam e KA Script. • Which KA Script should be executed first At any given tim e there m ight be several changes th at have not been addressed yet, and the order in which they are addressed m ay simplify or com plicate the task. W hen several changes are pending, the ones concerning KB elements used earlier during problem solving are addressed first. The rationale behind this decision is th at fixing these problems usually causes the problem solving process to continue in a different way and eventually to cancel problems produced by the changes th a t were not addressed yet. W hen different KA Scripts apply to follow up on the same change we leave the choice up to the user, since the appropriateness 25 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. of a choice may depend on inform ation th at is not readily available to the tool (e.g., user’ s preferred strategy to modify knowledge bases). 3.3 Interacting Flexibly with Users The m ain issue regarding user interaction is how to guide users through out an intricate KBS modification without lim iting or annoying them . Users should have the flexibility to modify a KBS either by following predefined KA Scripts or by performing their own changes. Sometimes the user already knows the changes that she wants to perform and there is no point in forcing her to look for the KA Scripts th at would perform them for herself. On the other hand, it is unlikely that a KA Scripts library contains all possible strategies for fixing problems, and the tool should let the user follow strategies th at are not in the library. The tool should also refrain from interrupting the user excessively, which may cause the user to loose the thread of the modification that she is carrying out. Some of the decisions described earlier were already concerned with these issues, like adopting short and easier to follow KA Scripts and deferring the execution of new KA Scripts until the user or another KA Script finishes the sequence of changes being performed. The following are other considerations specific to the way th at the tool interacts with the user. • User should be in control To avoid that the user looses the thread of the modification th at she is carrying out, and to give her more control over the process, the user takes the initiative of starting the execution of a KA Script when she considers appropriate. When the user desires to be helped, she requests to the SBKA tool a list of KA Scripts applicable to the current situation and indicates which one she wants to execute. The user is also in control of starting the execution of each one of the steps in the KA Script. The KA tool displays a checklist with all the steps in the KA Script and the user starts the execution of each step by clicking on it. 26 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. This mode of interaction was an improvement over the one that we tried in a previous implementation. In th a t version, the KA tool was in control of the process. The execution of the KA Scripts and their steps were chained one after the other in a continuous stream of changes to the KBS. The user participation was lim ited to fill in the d ata requested by the KA Scripts when the KA tool asked the user to do that. This mode of interaction m ade users to get lost in a flow of changes and to misunderstand the purpose of each change in the sequence. • Interface with a general purpose KBS Editor To allow a user to make any desired change to a KBS, a SBKA tool needs to be integrated with a more general purpose KA tool that supports unrestricted changes to a KBS. The integration should be implemented in such a way th a t the SBKA is aware of the changes th at the user performed with this general KBS editor. This requirement is necessary so the SBKA tool would be able to integrate these changes with other changes performed by KA Scripts and to follow them up with the appropriate KA Scripts. Throughout this chapter we have presented our approach to some of the research issues con cerned with the design of a SBKA tool. The approach described is independent of any particular framework for constructing KBSs and hence should be of help for anyone interested in building a SBKA tool. In the next chapter we present our implementation of a SBKA tool for one particular KBS construction framework. 27 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Chapter 4 ETM: A Script-Based Knowledge Acquisition Tool We have implemented a Script-based Knowledge-acquisition (SBKA) tool, called E X P E C T Trans action Managerx or (ETM ), that supports modifications of EXPECT KBSs [Gil and Melz, 1996, Sw artout and Gil, 1995, Gil, 1994]. E X PE C T is an environment for building knowledge-based system s th at takes advantage of a sem antically rich knowledge-representation language to enable a reflective analysis of the KB content. E X P E C T ’s built-in analytical capabilities greatly supported ET M ’s functionality. We first introduce some background about EX PEC T . Then we describe ETM ’s implem entation centering this description around the m ain design issues presented in the previous chapter. Finally, we describe a KBS modification scenario in detail and show how ETM would help in carrying it out. 4.1 Background: EXPECT KBS Framework and Baseline KA tool EX PEC T is an environment for building and modifying knowledge-based systems. EXPECT provides three key capabilities: a KBS representation framework, a problem solving environment, 1 U sin g a n a n a lo g y w ith d a ta b a s e s , we c a n view th e p ro c e ss o f m o d ify in g th e k n o w led g e b a se a s a se q u e n ce o f tra n sa c tio n s, w h ere K A S c rip ts s u p p o rt u se rs b y e n fo rc in g t h a t tra n s a c tio n s a r e c o m p le te d so th a t th e k now ledge b a s e is n o t left in co h eren t. 28 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. and a KA tool. We introduce some aspects of EX PEC T as we present an example knowledge base th at is going to be used throughout the rest of the chapter. More details about EX PECT can be found in [Gil and Melz, 1996, Swartout and Gil, 1995, Gil, 1994]. 4 .1 .1 K n o w led g e R ep resen ta tio n EX PEC T’s knowledge bases contain factual dom ain knowledge and problem solving knowledge. The factual dom ain knowledge represents concepts, instances, relations, and the constraints among them . It is represented in Loom [MacGregor, 1991], a knowledge representation system from the KL-ONE family. Problem solving knowledge is represented in methods which are procedural descriptions for achieving goals. They consist of 1) a capability that represents the generic goal th at the m ethod can achieve, expressed by an action followed by several param eters (or action roles), 2) a method body th a t describes the procedure for achieving the goal specified in the capability, and 3) a result type th at specifies the type returned by the method body. Figure 4.1 shows an example extracted from a simplified transportation dom ain. A vehicle is defined as a kind of m ajor equipment th at has a speed and can be either a ship or an aircraft. The method M2 specifies th at in order to calculate the round-trip-tim e for a ship between two locations we have to find the sailing distance between these locations (a subgoal th a t has to be achieved by other methods) and divide it by the speed of the ship (a relational expression). 4 .1 .2 P r o b le m S olver Given a generic goal (referred to as the top-level goal) such as (e stim a te (obj (spec-of T R IP -D U R A T IO N j) (4-1) (of (inst-of TRA N SPO R TA TIO N -M O V EM EN T))) estim ate the trip duration o f a tran sportation m ovem en t EX PE C T ’s problem solver autom atically generates a dom ain specific KBS for solving instances of this goal, such us: 29 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. (defco n cep t VEHICLE :is - p r im itiv e (ran d KAJOR-EQUIPHENT (rtrhe HAS-SPEED SPEED) r d is jo in t-c o v e rin g (SHIP AIRCRAFT))) A vehicle is a kind o f m ajo r equipm ent and has a speed. A vehicle is either a ship o r an aircraft. (defco n cep t TRANSPORTATION-HOVEHENT :is - p r im itiv e (:an d ( :th e HAS-ORIGIN LOCATION) ( :th e HAS-DESTINATIOH LOCATION) ( :th e HAS-CARGO-WEIGHT-TO-KOVE TONS) (.-some HAS-AVAILABLE-LIFT SHIP))) A tran sp o rtatio n m ovem ent has an origin and a destination location, a cargo to move expressed in tons, and som e lift available consisting in ships. (d ef-ex p ect-m eth o d Hi (c a p a b ility (estim ate (obj ( ? t i s (sp e c -o f TRIP-DURATION))) (o f (?m i s ( i n s t - o f TRANSPQRTATION-HOVEHEHT)) ) ) ) ( re s u lt-ty p e ( i n s t - o f ELAPSED-TIHE)) (body (pick (obj (sp e c -o f M A X IM U M )) (o f ( c a lc u la te (o b j (sp e c -o f ROUND-TRIP-TIME)) ( f o r (HAS-AVAILABLE-LIFT ?m)) (from (HAS-ORIGIN ?m)) (to (HAS-DESTINATIQN ?m)) ) ) ) ) ) To estim ate the trip du ratio n o f a tran sp o rta tio n movement pick the largest of th e ro und trip tim e between th e origin and the destination of th e m ovem ent fo r each one o f the lift available for th a t movement. (d ef-ex p ect-m eth o d M 2 (c a p a b ility (c a lc u la te (obj (Tt is (sp e c -o f ROUND-TRIP-TIME))) ( f o r (?s i s ( in s t- o f SHIP))) (from (711 i s ( in s t- o f LOCATION))) (to (?12 i s ( in s t- o f LOCATION))))) ( re s u lt-ty p e ( i n s t - o f ELAPSED-TIHE)) (body (d iv id e (obj (fin d (obj (sp e c -o f SAILING-DISTANCE)) (from ? l l ) (to ?1 2 ))) (by (HAS-SPEED ? s )))> ) To calculate th e ro und trip tim e between two locations for a ship divide th e sailing distan ce betw een th e two locations by th e speed of th e ship. (d ef-ex p ect-m eth o d M 3 (c a p a b ility (fin d (obj (?d i s (sp e c -o f SAILING-DISTANCE))) (from (711 is ( in s t- o f LOCATION))) (to (712 i s ( in s t- o f LOCATION))))) ( re s u lt-ty p e ( i n s t - o f LENGTH)) (body ( i f (o r (unknown (obj 711)) (unknown (obj 712))) then (a s k -u s e r (obj SAILING-DISTANCE) (from 711) (to 712)) e ls e (loo k -u p (obj (append 711 712)) ( in SAILING-DISTANCES-TABLE)) ) ) ) To find th e sailing distan ce between two locations, if any o f th e locations is unknown then ask th e user for th a t distance, else look up the table of sailing distances. Figure 4.1: Some definitions of concepts and problem solving methods in a simplified transportation domain. 30 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. (estimate (obj (spec-of TRIP-DURATIONj) (of FIRST-MOVEMENT-FT-BRAGG-TO-RYAD)) estim ate th e trip duration o f the first m ovem ent from F ort Bragg to Rtjad Problem solving proceeds as follows: the top-level goal is expanded by m atching it with a m ethod capability and then expanding the subgoals in its m ethod body. This process is iterated for each of the subgoals and is recorded as a derivation tree. Throughout this process, EX PECT propagates the constraints on the goal param eters performing an elaborate form of partial evaluation which is supported by Loom ’s reasoning capabilities. The derivation tree is then used by the EX PECT KA tool to support knowledge acquisition. One strong capability of EX PECT is th at problem-solving is performed w ith generic goals such as Form (4.1) above. Hence, if EXPECT is able to solve this generic goal then it would be able to solve any instance o f it. Conversely, if EX PECT fails to solve th at generic goal, it will be able to identify what knowledge is missing in the KB. This ability is fundam ental for driving knowledge acquisition. In our example, the top level goal (Form (4.1)) can be m atched with m ethod M l. Method Ml needs to calculate the round trip tim e for each one of the lift available for a transportation movement (subgoal c a l c u l a t e ) . EX PECT will look at the role h a s - a v a i l a b l e - l i f t in the definition of t r a n s p o r t a t i o n - m o v e m e n t and will determine th a t the lift of a movement are instances of ships. Hence, the subgoal c a l c u l a t e of M l can be achieved using m ethod M2. In turn, method M2 would need to find the sailing distance between two locations (subgoal f i n d ) . This goal can be achieved using m ethod M3. 4 .1 .3 E X P E C T K A T ool Using the derivation tree, EX PEC T finds the interdependencies between domain facts and problem solving m ethods, which are used to guide knowledge acquisition. For example, the derivation tree will annotate th a t in expanding M2 the speed of a ship is used. If a new ship is entered in the 31 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Inconsistency Types u n m a tc h e d G o a l u n u se d m e th o d u n u se d r e s u it o f e x p re ss io n inv alid a r g u m e n t o f re la tio n less th a n tw o o p e r a n d s in lo g ical form u n n e c e ssa ry g r o u p in g in lo g ical fo rm e m p ty T H E N b lo c k e m p ty E L S E b lo c k e m p ty T H E N a n d E L S E b lo ck n o n b o o le a n c o n d itio n in IF fo rm in c o m p a tib le T H E N a n d E L S E ty p e s e m p ty W H E N b lo ck n o n b o o lea n c o n d itio n in W H E N form n o n S E T a rg u m e n t in F IL T E R form n o n b o o lea n c o n d itio n in F IL T E R form less th a t tw o a rg s in A P P E N D form in co m p a tib le ty p e o f a rg s in A P P E N D u n u sed v a ria b le u n d e clare d v a ria b le m e th o d to o c o m p le x d isa g ree m e n t in re s u lt ty p e o f m e th o d Table 4.1: Some inconsistencies th at EXPECT can detect knowledge base and its speed is unknown, the knowledge acquisition tool will ask the user to specify its speed. The derivation tree also points out to the EX PECT’s KA tool the inconsistencies or potential problems th a t axe present in the KB, such as goals that cannot be m atched by any method, undefined param eter types, and result types th a t do not agree with the actual type returned by the m ethod expansion. Table 4.1 lists some of the inconsistencies that EX PEC T can detect. All KB inconsistencies found by EX PECT are recorded in an agenda th at reminds the user of the inconsistencies th a t still need to be fixed. EX PECT supports users in resolving these agenda items by describing the details of these inconsistencies and by suggesting fixes to them . The EX PEC T KA tool can also display different graphical presentations and natural-language descriptions of the KB and provides a menu interface for executing comm ands th at modify the KB. 4.2 ETM: EXPECT’s Script-Based KA Tool ETM was built integrated to the E X PE C T ’ s architecture for two reasons: to take advantage of EX PEC T’s built-in analytical capabilities, in particular determ ination of KBS interdependencies and inconsistencies, and to allow users to use EX PEC T’s conventional KA tool if none of the available KA Scripts im plem ent the changes th at the user has in m ind. This last feature is essential because a user m ay always come up with new strategies to modify the KB th at are not contained in the KA Script library. 32 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. In c o n siste n c y to be fix ed : "Goal G-new cannot be matched" A p p lic a b ility C onditions: (a) P a st change has caused goal G to become more g e n e ra l, r e s u ltin g in g o a l G-new (b) Goal G eas achieved by method HI b e fo re b e in g changed (c) G-new can be decomposed in to subgoals ( G l,..,G n ) (d) Gl i s e q u iv a le n t to G K A S c rip t S te p s: For each Gi in (G 2 ,.. Gn) (1) Choose a method to be used as b a s is . E T M proposes (H I,. . ,H i - l ) . User may a c c e p t i t o r choose a n o th e r. (2) C reate method Mi-new analogous to th e b a s i s . ETK c re a te s Hi-new as a copy o f th e b a s is and proposes s u b s titu tio n s to make Hi-new to m atch G i. U ser may in d icate a d d itio n a l s u b s t i t u t io n s needed in the body o f Hi-new. (3) R evise body o f Ki-new i f m o d ific a tio n s o th e r th a n s u b s titu tio n s a re needed. U ser e d its body of Hi-new. D e s c rip tio n o f what th is K A S c rip t does: C reate methods th a t achieve goals YCG2,. . ,G n\> b ased on method M l E x p la n atio n o f why i t is re le v a n t to the c u r re n t s i t u a t i o n : Method HI used to achieve go al G, which now was g e n e ra liz e d to become the unmatched g o a l G-new. Hi can be used to achieve one of th e su b g o a ls i n th e d ecom position VCG l,..,G n\> of G-new . Ml may a ls o be used as a b a s is f o r c re a tin g new methods th a t achieve th e o th e r su bgoals in t h is d ecom p o sitio n . Figure 4.2: KA Script representation example - I ETM ’s implementation addressed the three main issues discussed in the previous chapter as follows. 4 .2 .1 R ep resen tin g K A S crip ts Figures 4.2 to 4.4 show several different examples of KA Scripts. In ETM, KA Scripts are repre sented using the following kinds of inform ation: • In c o n siste n c y to b e fixed: A type of inconsistency or error in the KB introduced by the kind of changes that this KA Script is intended to follow up. This information is used to index the KA Scripts in the library. Because the inconsistencies in the KB are readily accessible through the Expect Agenda, this inform ation can be efficiently used to retrieve relevant KA Scripts from the library. 33 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. In c o n siste n c y to be fix e d : "Goal G-new cannot be m atched" A p p lic a b ility C o n d itio n s: (a) P ast change h as m o d ified G-new, added G-new in to a m ethod, o r c re a te d a new method c o n ta in in g G-new (b) G-new i s in method N-new. H-new was c re a te d by analogy b a se d on method N (c) G-new in H-new co rre sp o n d s to G in N. (d) G-new i s s im ila r to G b u t n e ith e r one o f them subsumes th e o th e r . (e) G i s ach iev ed by method ( 1 K A S c rip t S tep s: (1) C reate method H-new analogous to H ETK c re a te s K-new as a copy o f H and proposes s u b s titu tio n s to make H-new to match G-new. U ser may in d ic a te a d d itio n a l s u b s titu tio n s needed in th e body o f H-new. (2) Revise body o f H-new i f m o d ific a tio n s o th e r th an s u b s t i t u t io n s are needed. U ser e d its body o f H-new. D e sc rip tio n o f what t h i s K A S c r ip t does: C re a te s a method b ased on H t h a t achieves g o a l G-new E x p la n atio n o f why i t is re le v a n t to the c u rre n t s itu a tio n : Kethod N, once u sed a s a b a s is f o r method N-new, p o s ts g o al G w hich corresp o n d s and i s s im ila r to G-new. G is ach iev ed by method K .T his same method can be u sed as a b a s is f o r c re a tin g a new method t h a t ach iev es G-new. Figure 4.3: KA Script representation example - II In c o n siste n c y to be fix e d : "Goal G-new cannot be m atched" A p p lic a b ility C o n d itio n s: (a) P a st change h as m od ified G-new, added G-new in to a m ethod, o r c re a te d a new method c o n ta in in g G-new K A S c r ip t S te p s: (1) C reate H-new from s c ra tc h ETH c r e a te s method H-new w ith a c a p a b ility th a t m atches G-new and an empty body. U ser e d i t s body of H-new. D e sc rip tio n o f what t h i s K A S c r ip t does: C reates a method from s c ra tc h th a t achieves G-new E x p la n atio n o f why i t i s r e le v a n t to th e c u rre n t s itu a tio n : Goal G-new cannot be a ch iev ed . A new method can be c re a te d from s c ra tc h to achieve G-new. Figure 4.4: KA Script representation exam ple - III • A p p lic a b ility c o n d itio n s: Conditions th at specify appropriate situations for applying this KA Script. There are two roles for these conditions: 1) as preconditions required to achieve the KA Script purpose (e.g., conditions (b), (c) and (d) in Figure 4.2), and 2) as conditions for ensuring th at the KA Script is a sound continuation for the ongoing modification (e.g., 34 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. condition (a)). T he applicability conditions also bind K A Script variables to KB elements, which result in an instantiated KA Script. The applicability conditions can make reference to: the content and the derivation tree of the current KB, the content and the derivation tree of previous states of the KB, and to the past changes th a t have been performed during the ongoing m odification. Some uses for these conditions are: to compare the initial and the current version of a KB element in order to determine w hat has been changed and in which way (e.g., the range of a relation has been generalized); to com pare a node of the derivation tree against its equivalent in the Derivation tree of the original KB in order to determine the im pact of changes upon the problem-solving process (e.g., a goal posted during problem solving was m ade m ore general); to look at how KB elem ents used to interact in the past in order to determ ine which elem ents in the current KB should have to be changed to follow up a change; to look a t how a sim ilar (or analogous) KB elem ent relates with other elements in order to use those relations as a reference when changing and creating KB elements (e.g., to create a m ethod for achieving an unmatched goal by looking at the m ethod that achieves an analogous goal); and to search in the history of KB changes for changes th at created new KB elements based on existing ones in order to identify analogous elements. • K A S c rip t S te p s: T he sequence of changes to be perform ed. ETM supports the user in executing these changes by giving step by step instructions and suggesting changes. Toward this end, the sequence of steps in a KA Script is ordered so th a t the execution of the preceding steps will provide useful inform ation to help the user to execute the successive steps. In our current im plem entation of ETM , each KA Script step is im plem ented with a LISP function that interacts with the user and applies the changes in the KB. • S h o rt D e s c rip tio n o f w h a t th e K A S c rip t does: A text tem plate th at describes what the KA Script does. This description is shown to the user to let her decide whether she wants to apply a suggested KA Script or not. The tem plate will be filled with concrete references 35 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. to elements of the KBS. This turns out to be very im portant because it is only after knowing these elements that the user can m ake such a decision. • E x p la n a tio n of w hy th e K A S c r ip t is re le v a n t to th e c u rr e n t situ a tio n : An ex planation th at relates this KA Script to the ongoing modification. This explanation can be requested by the user if she wants to know why this KA Script is suggested. It is a text tem plate that is filled with concrete references to the elements of the KBS. 4 .2 .2 C oord in atin g th e A p p lica tio n o f K A Scripts The overall control loop for the execution of KA Scripts is shown in Figure 4.5. The loop begins after the user has performed some changes to the KBS and requested the help of ETM to follow up the consequences of those changes. T he KBS is assumed to be consistent before beginning its modification. In each iteration of the loop ETM picks from the EXPECT Agenda the inconsistency th a t occurred earlier during problem-solving, and proposes KA Scripts that apply to that inconsistency given the current situation. This inconsistency is attended first because its solution might change the way th at problem-solving continues, possibly invalidating inconsistencies that were produced later. If the user agrees with one of the I<A Scripts proposed by ETM, then ETM, in collaboration with user, executes it. If not, the user fixes the inconsistency using EX PECT’s basic KA tool. To complete the overall modification to the KB it is usually required the execution of several KA Scripts. This is because some changes can introduce several inconsistencies that have to be followed up by different KA Scripts, and also because of the cascading of changes (i.e., changes that follow up previous changes require being followed up too). O ur goal was to automate as much as possible the application of KA Scripts. Although full autom ation is not possible, the tool could take care of significant parts of the process, leaving im portant high-level decisions to the user. Detecting which KA Scripts are applicable (including often several instantiations of a same KA Script) is a task that can be done by the tool. At any 36 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. User makes change(s) in the knowledge base E X P E C T posts in its Agenda the inconsistencies found in th e KB W hile there are item s left in the A genda ETM picks inconsistency i from the Agenda (the one that occurs earlier in problem solving) and generates set K o f K A Script candidates that can fix i If the user does not like any K A Script then user can quit ETM and fix i w ith E X PE C T , ETM can be invoked again anytim e later else user chooses k from the set K . ETM and user apply k E X P E C T updates its Agenda Figure 4.5: ETM ’s Control Loop given tim e, there can be many inconsistencies in the knowledge base and several KA Scripts may apply for each one of them . ETM guides the user by deciding which inconsistency to attend first and suggesting KA Scripts that will fix it. W hen several KA Scripts are applicable for the same inconsistency, we leave the choice up to the user, since the appropriateness of a choice may depend on information that is not readily available to the tool (e.g., user’s preferred strategy to modify knowledge bases). Finally, the execution of a KA Script is shared between the KA tool and the user. 4 .2 .3 In tera ctin g F le x ib ly w ith th e U ser The main concern regarding the interaction with users is how to guide a user through out a series of predefined sequences of changes (i.e., the KA Scripts) w ithout getting her lost in a stream of KB changes and without limiting her from performing different or additional changes if necessary. ETM achieves these requirements by letting the user be in control of the process. The user takes the initiative of executing a KA Script when she considers it convenient. Ad ditionally, between the execution of KA Scripts the user is free to perform with EX PECT any modification th at she considers appropriate w ithout being interrupted by ETM. To allow ETM to reason about the changes performed by the user with EX PEC T, EXPECT keeps a record of changes performed to the KB which includes also the changes performed by the KA Scripts. When 37 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. ETM analyzes the history of past changes, it treats changes performed by the user or by ETM indistinctly. W hen the user desires to be assisted by ETM , she asks ETM to present a m enu of applicable KA Scripts. Each menu entry summarizes the KA Script procedure, and the user can request a more elaborated description of any one of them . Once the user chooses one of the proposed KA Scripts, ETM displays its steps as a checklist. Then the user starts the execution of each one of those steps by clicking on it. Because the user explicitly started the execution of each step, she always knows which step is being executed. Nevertheless, ETM indicates which steps have already been executed, which one is currently being executed, and which ones axe still pending so the user would know where she is in the execution of the KA Script and what to expect next. Figure 4.6 shows a snapshot of ETM ’s user interface during the execution of a KA Script. 4 .2 .4 U sin g C o n te x t In fo rm a tio n To better support users to follow up a KBS m odification, ETM makes use of the following kinds of inform ation about the context of the modification: • In c o n siste n c ie s w ith in th e c u r r e n t k n o w le d g e -b a se d s y s te m introduced by changes to the KB. Examples of such inconsistencies are goals that cannot be achieved and references to non-existing attributes of domain concepts. These inconsistencies do not mean that past KB changes were incorrect but rather th a t the ongoing modification is still incomplete. ETM uses this information in several ways. First, as an indication th a t there might be effects of previous changes th a t have not been attended yet, which prom pts the execution of KA Scripts to follow up on those changes. Second, to focus the search for the specific effects of changes th at have to be followed up. Individualizing those effects requires to compare current and previous versions of the KB and the derivation tree and to analyze the history of KB changes. Once these effects are individualized, it is still necessary to determ ine whether they have already been addressed or they still need to be followed up. An exhaustive search for all 38 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. VttWIMMttSBBBI ii i i i irh i w iirfriiip M H B n s H a s n C r « * e # • * * t t a o d c h a c * c b l « w « s : (CALCULATE (O B J (S P E C -O F R O U N D -T R IP -T IM E )) (O F (IN S T -O F A IR C R A F T )) (FROM (IN S T -O F L O C A TIO N )) (TO (IN S T -O F LO C A T IO N )>) B a s * d o r o n * o f c h * f o l l o w i n g * * e h o d s : [ c a l e u l a t * c h * r o u n d c r i p C l * * o f a a h i p f r o * a l o c a d o n Co a l o e a d o n l ( o r a n y o c h * r * * c h o d o r f r o * a c r a C c h ) S e * p s ( * z « c u c * c h « * i n o r d « r > * 0 0 0 0 * * * * 1 . C h o o s o a « c h o d C o b * u a a d . a s b a s i s ( " c a l e u J L a c * c h # r o u n d c r i p c I m o f a E » * c u c * | 2 . C r « # c * M a c h o d A n a l o g o u s C o c h * B a a l s s h i p f r o * a l o c a d o n Co a l o c a d o n " ! E***5uc* I 3 . E d l c B o d y o f A n a l o g o u s M * ch o d Don* A b o r t , 1 . W r i t * * * c h o d f r o * a c r a C c h C r « a c i n g H « c h o d A n a l o g o u s c o c a l e u l a c * C h * r o u n d c r i p c l a * o f a s h i p f r o a o a c n i « o * G o a l CALCULATE (O B J (S P E C -O F ROOND-TRIP-TIM E} J (O F (IN S T -O F a i r c r a f t : 1 (FROM (IK S T -O F LOCATION}) _______________ (TO (IN S T -O F LOCATION}>)______________ a l o c a t i o n c o a l o c a d o n ] E x * c u C * c « p s ( * x * c u t * c h * * i n o r d * r ] 1 . S p « c l f y G l o b a l S u b s t i t u t i o n * 2 . S p a c i f y G l o b a l S u b s c i c u c i o n s 3 . S p a c i f y N*w N s* « E x o c i iC * f o r C a p a b i l i t y f o r e h o B * s c E x a c u C * E x o c u t * 1 4 . i n c o r p o r a c * A n a l o g o u s K a e h o d I n t o t b a KB D on# A b o r c , | ( C A L C O I t A T X (0| I m l | I m 3 | m 4 I m S I m 7 | m 8 | m 2 B a s # K « ch o d * s C a p a b i l i t y : (CALCULATE (O B J (7 T I S (S P E C -O F R O U N D -T R IP-T IK E }) 1 (OF (7 S I S (IN S T -O F S H IP } )} (PROM ( ? 0 I S (IN S T -O F LO CA TIO N :)) (TO (7D I S (IN S T -O F LOCA TIO N )])____________ |P r o p o a * d H ach o d * s C a p a b i l i t y : (CALCULATE (O B J (? T I S (S P E C -O F R O U N D -T R IP -T IM E ))) (OF ((?Aj i , K IN S T -O F AIRCRAFT')]} (FROM (7 0 I S (IN S T -O F LOCATICU)) ) (TO ( ?D I S (IN S T -O F LOCA TIO N )))____________ C a n c * l . D on* Figure 4.6: Snapshot of E T M ’s user interface. The EX PEC T’s KA tool window is in the background overlapped by three other windows displayed by ETM during the execution of the KA Script of Figure 4.2. The ETM ’s back-most window shows the changes sequence of the KA Script. This snapshot was taken during the execution of change 2 (create a m ethod analogous to the basis). On top of this window there is a window that guides the user step by step through the execution of this change. Finally, the front-most window shows ETM ’s suggestion for the first substep: to substitute SHIP by A IRC RA FT and ?S by ?A. theses indications m ight be too costly. ETM uses inconsistency information to restrict the search to only those KB elements that are related to the inconsistency. Finally, ETM uses this information to retrieve from the library the KA Scripts that are relevant in following up the changes th at introduced this inconsistency.2 In our sample scenario of Chapter 1.5 Section 2.1, after the user changed the range of the l i f t relation from s h i p to v e h i c l e (Fig. 2.2 Change 1) a new inconsistency is introduced: there is no method capable o f calculating the round-trip-time o f an aircraft. ETM will use 2 R e m e m b er th a t K A S c rip ts a re in d ex e d b y th e ty p e s o f in co n siste n c ie s in tro d u c e d by th e effects o f ch an g es th a t th e K A S crip t is in te n d e d to follow u p . 39 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. this inconsistency to infer th at some effects of past changes still need to be followed up, in our case the pending effect is the generalization of the goal calculate the round-trip-time o f a ship. ETM will search for those effects focused on the KB elements related to that inconsistency, in our example, the unachieved goal and the method th at used to achieve that goal in the past. Finally, it would consider the KA Scripts in the library intended to follow up changes that produce an unm atched goal, like the KA Script of Figure 4.3 which follows up the generalization of a goal. • A h is to ry o f th e ch an g es m a d e to th e k n o w led g e-b ased s y s te m which is used to infer what the user is trying to accomplish with the modification and to avoid considering KA Scripts that interfere with those intentions. Continuing with our example, ETM could observe th at the definition of vehicles has been recently changed and hence avoid suggesting any KA Script that attem pts to change this definition back to its original form. In our implementation of ETM, this information is taken from a log of executed modification commands maintained by EXPECT. This log was one of the extensions to EX PECT th at we had to implement in order to support ETM. • A re c o rd o f p a s t v e rsio n s o f th e k n o w le d g e -b a se d s y s te m a n d d e riv a tio n tre e s which is used to understand how KBS elements used to fit together and to identify how changes affected the problem-solving process. For example, observing which method had initially been used to calculate the round trip time of ships allows ETM to consider a KA Script that manipulates this same m ethod in order to calculate the round trip time of aircraft. In our implementation of ETM , this information consists of a snapshot of the KB and its corresponding derivation tree at the beginning of the modification. This snapshot can be updated to a later state if the KB is stable (i.e., there are not inconsistencies or pending follow up changes). 40 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. EX PEC T required only slight extensions in order to support this integration. Namely, to keep a record of past versions of the KB and the history of changes to the KB. 4.3 An Example Scenario for ETM Suppose th at the KBS of Figure 4.1 has to be modified because the available lift for transportation movements can now include aircraft besides ships. This modification requires five changes as depicted by Figure 4.7. First, in the definition of t r a n s p o r t a t i o n - m o v e m e n t , the range of the role h a s - a v a i l a b l e - l i f t has to be generalized to v e h i c l e , which already includes both s h i p and a i r c r a f t . This is achieved with Change 1 (see Figure 4.7). The fillers of this role are one of the argum ents of the goal c a l c u l a t e in m ethod M l, hence this argument changes autom atically from SH IP to v e h i c l e too. After this change, the goal c a l c u l a t e in M l cannot be m atched any more with M2 (M2 applies only to ships). However, the new goal can be decomposed into two disjunctive subgoads, one for ships and the other for aircraft. In order to achieve the ship subgoal, the m ethod that achieved it originally (M2) can still be used. This sam e method can also serve as a basis for constructing a new method for the aircraft subgoal. W ith Changes 2 and 3 the user creates this new m ethod (M2-PRIME) by duplicating M2 and then modifying its copy (substituting s a i l i n g - d i s t a n c e by f l y i n g - d i s t a n c e and adding the com putation of the time that the aircraft spends in en-route stops). The new m ethod contains two goals th at cannot be achieved with the current knowledge: a modification of the goal F IN D , whose argum ent s a i l i n g - d i s t a n c e was changed to f l y i n g - d i s t a n c e , and the new goal C O M P U T E . The user, with Change 4, creates a m ethod (MS- PRIM E) for achieving the modified f i n d based on the m ethod that achieved the original FIN D (M3), and with Change 5 a new m ethod (M4) for achieving c o m p u t e . In sum, five changes in different parts of the KB were needed to modify the KBS. If some of these changes were om itted, the KBS would have been left incoherent. It is im portant to realize th at these changes are difficult for a user to determ ine. For example, to determine th at after 41 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. (def-expect-method Mi (capability (calculate (obj (?t is (spec-of TRIP-DURATION)) (of (?m is (inst-of TRANSPORTATION-MOVEMENT))))) (result-type (inst-of ELA PS ED-TIME)) (body (pick (obj (spec-of MAXIMUM)) (of (calculate (obj (spec-of ROUND-TRIP-TIME)) (for (HAS-AVAILABLE-UFT ?rn)) (from (HAS-ORIGIN ?m)) (to (HAS-QgSTlNATION ?m))))))) 1 1 KB element EH1 Modified KB element d > KB elements interaction KB Changes Retrieve (HAS-AVAILABLE-UFT TRANS PORTATION-MVNT) Goal (calculate (obj (spec-ofROUND-TRIP-TIME)) (for (inst-of nreHici r )) (from (tnst-of LOCATION)) (to (inst-of LOCATION))) (defH exp^-tiK fbbd £ p M ; = & E H I C C E £ (dcfconcept TRANSPORTATION-MOVEME ds-primitive (rand DOMAIN-CONCEPT (the HAS-ORIGIN LOCATION (the HAS-DESTINATION L0< (some HAS-A V AILAB LE-LIF VI 4) < 4 : a t io n ) T|VEHICEE|)» J© (def-expect-method M2 (capability (calculate (obj (?t is (spec-of ROUND-TRIP-TIME)) (for (?s is (inst-of SHIP))) (from (711 is (inst-of LOCATION))) (to (712 is (inst-of LOCATION))))) (result-type (inst-of ELAPS ED-TIME)) (body (divide (obj (find (obj (spec-of SAIUNG-DISTANCE)) (from 711) (to 712))) (by (HAS-SPEED ?s))))) SSnI*nioiessSHIE=^>;AIRCRAEE4 n a O N jffD IS T X N C E l 0) (S nS j^lp (by?(HASSPEED?aTO)i Goal (find (obj (spec-ofSAIUNG- DISTANCE)) (from (inst-of LOCATION)) (to (inst-of LOCATION)))) AddnovGoalrf (def-expect-method M3 (capabUity (find (obj (?d is (spcc-ofSAILING-DISTANCE))) (from (711 is (inst-of LOCATION))) (to (712 is (inst-of LOCATION))))) (result-type (inst-of LENGTH)) (body (look-up (obj (append 1112)) (in SAIUNG-DISTANCES-TABLE)))) O ns^o fA IR C R A ^))^ ;'CuiSjrf LOCATION)) (Ins\^C LOCATION))] (find (obj (spcc-ofE^TKmpigFAW ^Pl)) (from(inst-of l UCAILUN)) (to (inst-of LOCATION)))) : 3(defScp^i^ C?@ Sbll»5S<»n^*tet(obj^i<^ls(swi^CSPENT?nM ^)>^^; iS ^ S S S S S i F ro f^ S c ratcttf (tdnX?12,is (inst-of LOCATIO ' ' t-o |Substiaite;^ ^SfAl^Gl^^YTNG TobjXdly (firid(6bj £(7dis(spec ps(obj.(fih4(o6jl(< ;(tc K 'a 2 )))V | M fJ;|^ ^^ (jP F g ^G a)lS T A N C T S ^T A B L a)))) (Inst-of ^ ip C o b Figure 4.7: An example of a KBS modification scenario. Five changes are required to allow the lift of movements to include aircraft. 42 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. generalizing the range of the role h a s - a v a i l a b l e - l i f t (Change 1), a m ethod for calculating the round trip tim e for an aircraft has to be created (Change 2), the user needs to determine the following facts: 1) The fillers of the role HAS-AVAILABLE-LIFT are used in one param eter of the goal C A L C U L A T E of m ethod M l, 2) C A LC U LA TE was achieved by M2, 3) M2 cannot achieve the variation of c a l c u l a t e , and 4) c a l c u l a t e can be decomposed into a subgoal for ships and another for aircraft. Many of these facts are supported by logic inferences like goal subsumption (e.g., whether goal c a l c u l a t e can be decomposed into disjunctive subcases or not), which makes it even harder for users to figure it out. The KA Script of Figure 4.2 could be applied to follow up the first change of this scenario (i.e., the generalization from SH IP to v e h i c l e , of the range of h a s - a v a i l a b l e - l i f t (Change 1)). It is applicable because this change caused the inconsistency “Goal c a l c u l a t e (of Ml) cannot be m atched” , the argum ent SH IP of c a l c u l a t e changed to v e h i c l e (condition (a)), this goal was achieved by M2 before (condition (b)), and now it can be decomposed into two subgoals, one for SH IP and another for a i r c r a f t (conditions (c) and (d)). This KA Script would perform Changes 2 and 3 of th a t scenario (i.e., create method c a l c u l a t e for a i r c r a f t (M2-PRIME), based on the m ethod c a l c u l a t e for s h i p s (M2)). Figure 4.6 shows ETM ’s user interface during the application of this KA Script. Similarly, the changes performed by this KA Script (Changes 2 and 3) can be followed up by the KA Script of Figure 4.3. It is applicable because the m ethod M2-PRIME containing the unm atched goal FIN D for f l y i n g distances was created based on m ethod M2 ( conditions (a) and (b)), the goal FIN D for f l y i n g distances corresponds to goal FIN D for s a i l i n g distance achieved by M3 (conditions (c) thru (e)). This second KA Script would perform Change 4 of the example (i.e., create m ethod FIN D for FLYING distances (M3-PRIME), based on the method f i n d for s a i l i n g distances (M3)). 43 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. After executing this second KA Script there still would be an inconsistency left produced by the application of the first KA Script. T his KA Script had added goal c o m p u t e to m ethod NO PRIM E (Change 3) which could not be m atched to any existing method. This inconsistency can be fixed by the KA Script of Figure 4.4 which would guide the user in creating a new m ethod (M4) from scratch (Change 5). This example also illustrates how ETM was able to suggest specific changes by relating these changes to previous ones, one of the m ain features of SBKA. For example, to remedy the absence of a m ethod to achieve FIN D for f l y i n g distances posted by M2-PRIME, ETM suggested the creation of a new m ethod based on the one th at achieved FIN D for s a i l i n g distances (M3) (i.e., the second KA Script). To generate this suggestion, it was necessary to trace back the origin of this inconsistency to a particular change, the creation of M2-PRIME by analogy to M2, and determine how th at change needed to be followed up. 44 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Chapter 5 Developing a KA Script Library for EXPECT A KA Script library is the core of a script-based knowledge-acquisition (SBKA) tool. To maximize its utility, this library should not merely be a repository of isolated KA Scripts. Rather, this library should combine KA Scripts such th at as a whole it covers m ost possible situations with minimum overlapping. This chapter describes our efforts to develop a KA Script library methodically. It presents our methodology, discusses the m ajor research issues th at we faced, and describes the KA Scripts that we have developed using this method. It also describes the analyses of KBS m odification scenarios th a t we have carried out to gain insight into the kinds of procedures used in modifying KBSs. 5.1 Developing a Library of KA Scripts We carried out three different kinds of analyses in order to develop a KA Script library [Tallis and Gil, 1999]. 5 .1 .1 A n alysis o f S y n ta c tic C h an ges O ur first analysis looked a t the kinds of syntactic changes th a t can be done to a knowledge-based system , their possible consequences, and the successive changes that could follow up on those consequences. For example, we analyzed different types of changes to goals (e.g., modify one 45 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. param eter), their possible consequences (it m ight not be possible to match th at goal to a m ethod), and their follow-up changes (e.g., change that same param eter in the method th at achieved that goal before, and then change the body of that m ethod accordingly). The set of all possible changes was generated from the gram m ar of the language used to represent the knowledge base elements of the systems (in our case this was EX PECT). The result of this analysis was a complete set of KA Scripts for following up all possible changes. These KA Scripts cover all the situations in which a user can get when modifying a knowledge base. However, tests with our initial implem entation showed that the guidance they provide to users was too general to be very useful. The m ain problem was that they do not make good use of the context available, like more specific characteristics of the change (e.g., the param eter was changed to a more general type), existing knowledge (e.g., the modified goal can be decomposed into two subcases), or the changes performed to other parts of the knowledge base (e.g., a similar change was performed to a related elem ent). Another problem of this approach was that it produced somewhat cumbersome and redundant KA Scripts. As we explained in Chapter 3, changes to KBS elements have different effects depending on how these elements interact with each other, and we have found that the procedures for following changes are more dependent on the effect of the changes than on the change itself. Especially because different changes can produce the same effect and this effect is followed up independently of its cause. By structuring KA Scripts around the type of change to be followed up, each KA Script had to include provisions for every possible consequence that the change to be followed up can have. However, because these consequences are independent of the change itself, these same provisions have to be repeated in every related KA Script. Nevertheless, this analysis allowed us to follow a systematic procedure for enumerating KA Scripts and generate a complete set. 46 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 5 .1 .2 A n alysis o f K B S M o d ifica tio n S cen arios This analysis was aim ed to generate KA Scripts th a t were less general (and thus more helpful) than those generated in our previous analysis. O ur hypothesis was th at a detailed analysis of KBS modification scenarios would allow us to identify patterns of related changes more specific to the context in which those changes were performed. For this analysis, we compared a num ber of subsequent versions of KBSs th a t were saved by users as they were developing them . A nother possibility for doing this analysis is to record the changes while users are editing the KB. The problem with that approach is th a t the analysis would include changes that are later undone by users, or partially undone and then completed in a way th at has more of an error recovery flavor and where it would be best if the user went back to the original KBS and started the change again. We did not want to design KA Scripts th at captured and sympathized with this kind of backtracking in the KB design. Com paring versions of the KBS does not have this problem, because typically users save versions when they reach a sound interm ediate point in the KB development process. We generated our data by comparing different versions of KBSs and reconstructed the changes done across versions. To reconstruct the changes performed between versions A and B of the same KBS and to identify KA Scripts th at would be useful to make those changes, we followed the procedure below: 1. C o rre la tin g K B v e rsio n s: Before initiating the comparison between versions A and B we need to find out the correspondences between the m ethods of A and the methods of B. These correspondences should include also m ethods th at have changed their name or capability and new methods that m ight have been created based on existing ones (hypothetically). In order to find out these correspondences, we built the trees o f method invocation for A and B and correlated their nodes. A tree of m ethod invocation is a tree in which each node represents a method and the children of that node correspond to the m ethods that achieve the goals 47 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. posted by the parent m ethod. To correlate the nodes of the m ethod invocation trees we used the following algorithm: To correlate the nodes of the trees Ta and Tjg whose roots Ia and t b represent methods M a and M b respectively (and assuming th at M a corresponds to M b , i.e., either M a has been modified into M b or M b was created based on M a ) do: (a) Find the set of correspondences C = {(Gf,Gf)...(Gf,Gf)} between the goals {Gf .. -Gf} posted by Ma and the goals { G f ..• G f } posted by Mb - (b) For each correspondence (Gf, Gf) in G do i. Look in T a and Tb for the children C f and C f th a t correspond to the methods that achieve Gf and G f respectively. ii. Mark that C f corresponds to C f iii. Correlate the nodes of the subtrees C f and C f recursively. 2. C o m p a rin g v ersio n s: For each m ethod in the KB, enum erate the differences between the two versions of the KBS. 3. Id e n tify in g c o n c e p tu a l ch a n g es: Not all changes perform ed in a method share their purpose. For each m ethod in the KB, hypothesize the purpose of each observed change and then cluster the changes with related purposes. Figure 5.1 shows an example of two clusters. Each cluster would constitute a conceptual change. In this figure there are two conceptual changes: rephrasing o f a method capability and specialization o f a method capability. 4. R e la tin g c o n c e p tu a l ch a n g es: Find sets of related conceptual changes. One way to accomplish this is by hypothesizing what changes should have been necessary to follow up an observed conceptual change and then trying to locate them . We use the tree of method invocation to find out relations am ong m ethods. For example, rephrasing a method capability m ight be followed up by rephrasing the goals th a t that m ethod achieved. These goals can be 48 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Before: (FIN D -SP L IT -R O U T E ___________ ( o b j ( ? p i s ( s p e c - o f PA SSABLE))) ( f o r ( ? o i s ( i n s t - o f MILITARY-ORGANIZATION) ) ) ( f r o m ( ? f i s ( i n s t - o f LOCATION))) ( t o ( ? t i s ( i n s t - o f L O C A T IO N )))) A f t e r : m (FIN D ( o b j ( ? p i s ( s p e c - o f PA SSA B L E-SPL IT-R O U TE )) ) ( f o r ( ? o i s ( i n s t - o f M ILITARY-UNIT) ) ) ( f r o m ( ? f i s ( i n s t - o f LOCA TION ))) C > ( t o ( ? t i s ( i n s t - o f L O C A T IO N )))) Figure 5.1: Two independent clusters of related changes between two versions of a m ethod capabil ity description. Cluster 1) corresponds to a rephrasing o f a capability to enhance readability, while cluster 2) corresponds to a specialization o f a capability (military-unit is a subtype of military- organization) located by looking at the tree of m ethod invocation, and once they are located, we can relate the observed changes. 5. G e n e ra liz in g se q u en ces o f c o n c e p tu a l ch an g es: Generalize the observed sequences of changes and determine the features from the scenario that would make these sequences pos sible and meaningful. 6. P ro p o s in g o th e r sequences: Propose other sequences of changes by perm uting the order of the changes in the sequence. We carried out the above analysis on two knowledge-based systems: • Case Study I: 4 successive versions of a trafficability KBS where the user was developing an initial prototype. • Case Study II: 6 successive versions of an air campaign plan evaluation KBS where the user was extending an already implemented prototype. 49 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 5 .1 .2 .1 C ase s tu d y I: I n itia l p r o to ty p e im p le m e n ta tio n In this scenario we identified 42 conceptual changes. The following is a list of the different concep tual changes observed and the num ber of times th at they occurred: 1. re p h r a s e m e th o d c a p a b ility . The capability of a m ethod has been rephrased by renaming, adding or deleting constant term s, usually to enhance readability. However, the m ethod’s procedure remains the sam e. (13 occurrences) 2. re p h r a s e g o al. Like the previous conceptual change but with goals (13 occurrences) 3. r e s tr ic t a p p lic a b ility o f a m e th o d . A m ethod capability is specialized but the m ethod’s procedure is not changed. (9 occurrences) 4. a d d e x c e p tio n . A fragm ent of the m ethod’s procedure is embedded inside the then (or else) clause of a new //statem en t. (1 occurrence) 5. r e s tr ic t r e s u lt ty p e. Specialize result type declaration. (1 occurrence) 6. p a ss a n a d d itio n a l a rg u m e n t in a m e th o d in v o c a tio n . (1 occurrence) 7. re q u ir e a n a d d itio n a l p a r a m e te r in a m e th o d c a p a b ility . (1 occurrence) 8. s p lit a m e th o d . Extract a fragm ent from a m ethod into a newly created method. (1 oc currence) 9. re o rg a n iz e access p a th to d o m a in e le m e n ts. Reorganize the sequence of dom ain rela tions used to retrieve dom ain elements from the KB (probably because the dom ain model has been reorganized too). (1 occurrence) The following relations between these conceptual changes were observed: 1. rephrase goal / rephrase capability of the method th at achieves that goal 2. rephrase capability / rephrase internal goals th at refer to the changed capability param eters 50 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 3. restrict applicability o f a method / restrict result type of method 4. restrict applicability of a m ethod / restrict applicability of the m ethods th a t achieve the internal goals th at refer to the restricted param eter (i.e., propagates capability restrictions to other m ethods) 5. rename concept in dom ain model / rename concept reference in methods 6. reorganize dom ain model / reorganize access path to dom ain elements 7. rephrase goal in procedure / rephrase sim ilar goals in same method 8. change body expression / replicate change to identical expressions in sam e m ethod 9. pass an additional argum ent in a m ethod invocation / require an additional param eter in a m ethod capability 5.1.2.2 Case study II: Extending implemented prototype In this scenario we identified 27 conceptual changes. The following is a list of the different concep tual changes observed and the number of times th a t they occurred: 1. rename a relational expression. Renam e a relational expression because the referred relation was also renam ed in the dom ain m odel. (1 occurrence) 2. add goal to a dynamically generated set of goals. The analyzed KBS used to dynam ically generate a set of goals corresponding to a set of hypotheses that needed to be checked. After adding an instance to the set of hypotheses, a new goal was autom atically generated and added to the dynam ic set of goals. (1 occurrence) 3. create a method to achieve a new dynamically generated goal. (1 occurrence) 4. create m ethod to achieve a new goal added to a method procedure. (3 occurrences) 51 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 5. r e s tr ic t a s e t o f e le m e n ts b e p ro ce sse d . Filter the elements of a set before processing them . (1 occurrence) 6. re m o v e a n append o p e ra n d . Append is a construct of the knowledge representation lan guage. (1 occurrence) 7. re m o v e a n u n u s e d m e th o d . (3 occurrences). 8. m e rg e tw o m e th o d s in to o n e . (‘ 2 occurrences) 9. re m o v e su p e rflu o u s g o al a rg u m e n t. (2 occurrences). 10. re m o v e su p e rflu o u s c a p a b ility a rg u m e n t. (2 occurrences). 11. r e p h r a s e goal. (2 occurrences) 12. rephrase capability. (2 occurrences) 13. split a method. (2 occurrences). 14. re g ro u p o p e ra tio n s p e rfo rm e d b y a set o f m e th o d s in to a d iffe re n t s e t o f m eth o d s. The operations performed by a chain of methods invocations is regrouped into a different set of m ethods to enhance m ethod reusability. (2 occurrences) The following relations between these conceptual changes were observed: 1. rename relation in dom ain model / rename relational expression 2. add goal to a dynamically generated set of goals / create a m ethod to achieve a new dynam ically generated goal 3. add goal expression to m ethod procedure / create m ethod to achieve new goal added to a method 4. remove goal expression from m ethod procedure / remove unused m ethod 52 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 5. remove superfluous goal argum ent / remove superfluous capability argument 6. regroup operations from a chain of m ethods invocations into a different set of methods / repeat the same changes for a parallel branch of method invocations. 7. rephrase goal / rephrase capability invoked method 5.1.2.3 C o n c lu sio n s This analysis perm itted us to recognize a num ber of additional interesting characteristics of KBS modifications and KA Scripts: • C o n c e p tu a l ch an g es: Conceptual changes describe changes at a level that captures the user’s intention behind the changes. For example, generalizing the type of a goal parameter conceptually corresponds to generalizing a goal. Note that a conceptual change has several realizations. For example, another way of generalizing a goal is by generalizing its verb (e.g., move is more general than fly) or by eliminating a goal param eter Referring to changes at a conceptual level has the following advantages: — Enhance user comprehension of the KA Script procedure because it is closer to the level in which users reason about changes. — Develop a KA Scripts library at an appropriate level of abstraction. — Factorize KA Scripts because referring to a conceptual change is equivalent to referring to all possible realizations of it. — Design a KA Script library th at is not so specific to any particular knowledge- representation language because most conceptual changes have corresponding changes in other KB frameworks. Referring to changes at a conceptual level has the problem th at it makes it more difficult to systematically enumerate all possible types of changes. 53 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. • In te rd e p e n d e n c ie s b e tw e e n K B S e le m e n ts ; The observed sequences of changes allowed us to identify types o f interdependencies between elements o f a KBS. For example, from two related conceptual changes, one in a m ethod capability and the other in a method subgoal that shared the type of one of their param eters, we inferred a possible interdependency between elements of this kind. Determ ining the possible types of interdependencies between elements of a KBS would help us to identify new KA Scripts by analyzing how the interdependent elements should be m odifying in coordination. It was not possible to establish the necessary conditions for all observed interdependencies between KBS elements. Hence, some identified KA Scripts were based on the hypothetical existence of such interdependencies. • R e c u rre n t K A S c rip ts : Several detected KA Scripts were observed repeatedly in different parts and in different KBSs. W hether the generation of KA Scripts based on specific scenarios would produce KA Scripts general enough to be reused was an issue that had concerned us before carrying out this analysis. Although the analysis of KBS m odification scenarios allowed us to identify KA Scripts that were more specific to their context than our initial analysis based on syntactic changes, it did not help in generating a comprehensive library of KA Scripts. Nevertheless, this analysis perm itted us to identify im portant attributes of the KA Scripts that constitute the base for our final analysis. 5 .1 .3 A n a ly sis o f A ttr ib u te s o f K n ow led ge A c q u isitio n Scripts In this analysis we extended the m ethod used in the first analysis to a full set of attributes. Starting from this set of attributes together w ith an enumeration of their range values, we can develop a comprehensive KA Script library by defining a KA Script for any feasible combination o f attribute values. The attributes th at we considered were as follows: 54 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 1. C h a n g e to follow u p : We designed a typology of conceptual changes based on. (a) changed element. A KB element (or a sub-element) that describes a conceptual unit (e.g., goal expression, m ethod capability). (b) type of change. W hether the change created, modified, copied, or deleted an element. (c) transform ation. The relationship between the changed element before and after the change (e.g., generalization, addition of an argument). 2. In te rd e p e n d e n c y : We listed the different types of KB elements th a t could be related to the changed element referred by the previous attribute (e.g., goal / m ethod for achieving it). A KA Script procedure would be concerned with propagating the change described by the previous attribute to the KB elements indicated by this attribute. 3. S tra te g y : This dimension enumerates some general strategies for propagating the change indicated in the first attribute along the interdependency indicated in the second attribute. For example, a strategy for propagating the addition of a new posted goal to the m ethod that will achieve it is to generalize an existing problem-solving method th at achieves a goal similar to the one recently added, so th a t it would be applicable to both goals. We have included several strategies th at either modify existing KB elements or create new KB elements based on an existing one to make it easier for users to perform the modification and to encourage knowledge reuse. The indication of which existing element should be modified or used as a basis for the modification is the purpose of the next attribute. 4. B a se e le m e n t: This attribute enumerates good candidates to be used as a basis for strategies th at modify an existing element or create a new one based on an existing one (e.g., the m ethod th at used to achieve a changed goal before starting the modification, a method used for achieving a similar goal posted in an analogous m ethod). 55 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Besides guiding the development of a KA Scripts Library, using a set of KA Script attributes to generate a KA Script library offered us some additional benefits. It allowed us to describe the content of the library by indicating the range of attribute values covered by the KA Scripts in the library. It also perm itted us to estim ate the size of the library during its development. 5.2 An Implemented KA Script Library We implemented a KA Script library for ETM . The following are the regions of the KA Script library th at we have implemented: 1. KA Scripts to follow any type of c h a n g e to a goal e x p re ssio n , th at take care of its in te r d e p e n d e n c y w ith th e p ro b le m -s o lv in g m e th o d that achieves that goal. We considered all the strategies that consisted in creating or modifying the problem-solving method that will achieve the changed goal. O ther possible strategies, not considered in our current im plementation, include modifying the changed goal expression further to m atch an existing problem-solving m ethod in its current state. 2. KA Scripts to follow ch an g es to a m e th o d c a p ab ility , th at take care of its in te rd e p e n d e n c y w ith th e goal e x p re ssio n s th at the method used to achieve. We considered all types of changes to method capabilities, and all strategies that consisted in creating or changing goal expressions to be m atched by the changed capabilities. Other possible strategies, not considered in our current im plem entation, include creating or modifying further the same or other problem-solving methods to m atch the existing goal expressions. These two regions of the KA Scripts space were chosen because they address the interdepen dencies between goal and m ethod capability which are one of the most prevalent and harder to solve problems that arise during knowledge-based system modifications. These two regions contain up to 100 KA Scripts altogether, from which we have implemented 37. We wanted to have enough 56 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. KA Scripts implemented to test our approach, but implementing the entire library is beyond the scope of this work. We were able to identify a few general operators that we used to implement steps of the KA Scripts (e.g., generalize a m ethod, create analogous m ethod to achieve similar goal). These operators could be reused to implement most of the KA Scripts steps in our KA Scripts library. For example, the operator to Generalize a method was used by a KA Script that, in order to achieve a new goal, generalizes the problem-solving m ethod that achieves an existing goal sim ilar to the one ju st added, and was also used by a KA Script th at, in order to achieve a goal that was generalized, generalizes the method th a t used to achieve th a t goal before being modified. Appendix A describes in detail the attributes used to generate this library and the KA Scripts that were implemented. 57 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Chapter 6 A Study of Users Modifying Knowledge-Based Systems: Evaluation of ETM This chapter describes the design of the experiments th a t we conducted to evaluate ETM, our hypotheses, and the results th at we obtained. This evaluation is one of the contributions of this dissertation. The empirical evaluation of KA technology is difficult and infrequent. Subjects are a limited resource. They need to have some fam iliarity with knowledge representation and knowledge engineering techniques and have to be fam iliar with the application domain or else they have to be trained. Knowledge acquisition experiments are tim e consuming, what limits even further the num ber of subjects willing to participate. Furthermore, KA tasks are complicated, which increases the influence of external factors th at can affect the outcome of the experiments and it is not clear how to control them. Due to the above reasons, very few empirical evaluations that measure the effectiveness of KA technology with controlled conditions have been conducted to date (e.g., [Yost, 1992, Webb et al., 1999]). In order to conduct this evaluation we have developed an experim entation methodology th at we hope be useful to others too. Our experiences in applying this methodology have also taught us about the problems th at can be present during its application, which are im portant to know in order to design more robust experimental procedures [Tallis et al., 1999]. This chapter describes the experim ental procedures and summarizes the result of the experiments, Appendix A .2 includes some of the m aterials used during the experiments, and Appendices C and D includes some detailed d a ta captured during the execution of the experiments. 58 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 6.1 General Hypotheses Script-based knowledge acquisition (SBKA) has the following features: • guides users step by step in completing a knowledge-based system modification, suggesting values for some modification comm and parameters (e.g., substitutions) and semiautomatically executing some of them. • identifies which knowledge base elements already existing in the KBS can be adapted when following up a modification • suggests several alternative strategies to follow-up a modification • indicates which of the errors simultaneously present in the KBS should be addressed first Based on these features, we formulated the following hypotheses regarding SBKA: 1. Users will be able to complete a KBS modification in less tim e . R a tio n a le : A SBKA tool would analyze for them how to follow up a modification and would semi-automatically execute some modification commands. 2. Users will make less m istak e s during a KBS modification. R a tio n a le : Same as item 1. 3. T he reduction in completion tim e and number of mistakes will be more noticeable for less e x p e rie n c e d u sers. R a tio n a le : Less experienced users should be the most benefited from the tool’s thorough guidance and should be able to use strategies th a t they would not be aware of otherwise. 4. The reduction in time will also be more noticeable for u se rs la c k in g a d e ta ile d know ledge of the KBS implementation. R a tio n a le : ETM reveals existing KB elements th a t can be reused or adapted, ones that users m ay not be aware of. This should be particularly noticeable when the KBS is large. 59 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. We also expected th at our KA Script library would be useful for a broad range o f dom ains and knowledge acquisition scenarios. 6.2 Overview of the experiments We have carried out two different studies to evaluate ETM . In the first one we conducted a series of controlled experiments to measure the com parative performance for subjects m odifying KBSs with ETM (plus EXPECT) against subjects using EX PECT alone (hypotheses 1 thru 3). This study involved subjects that were already fam iliar w ith EX PECT although their experience using EX PEC T was varied. None of the subjects was fam iliar with ETM . To evaluate the adequacy of ETM and our KA Scripts library to unanticipated and more realistic tasks, we carried out a second study in which subjects used ETM to accomplish scenarios which had been especially designed by a third party to evaluate a broad range of knowledge base technologies. These scenarios were developed independently from the developers of ETM and were concealed to them prior to the execution of the experim ent. 6.3 Study I: Controlled Experiments This study consisted in a series of experiments th a t com pared the performance of subjects modifying KBSs using ETM (plus EXPECT) vs. subjects using EX PECT only (the control group). The different experiments tested subjects already fam iliar with EX PECT but with different level of experience in using it. 6 .3 .1 R e su lts The d a ta obtained from the experiments suggest the following general results about ETM: • ETM was able to le a d s u b je c ts th r o u g h o u t a ll th e r e q u ir e d changes o f t h e sc en a rio s for m ost of the subjects that used ETM, including one complex and realistic scenario. In 60 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. the few cases in which subjects had to perform changes without ETM, these changes were outside of the scope of ETM . For example, some subjects needed to perform extra changes to revert previous changes th a t although were internally consistent with the KB did not correctly follow the task specifications. • S u b je c ts u s in g E T M to o k less tim e to complete the modification except when they m ade mistakes. Reverting their own m istakes required a considerable amount of tim e for some subjects in some cases. In particular, when a subject had misunderstood the dom ain and m ade wrong changes to the KB or when a subject had misunderstood the assigned task and oversaw some of the changes specified by the instructions. Both cases are outside of the scope of ETM. • The differences in tim e were m o re e v id e n t fo r th e s u b je c ts w ith less e x p e rie n c e u s in g E X P E C T . • The experiments did. n o t sh o w c le a r e v id e n c e t h a t E T M w o u ld red u c e th e n u m b e r o f u s e r m is ta k e s . However, we believe th a t some of the mistakes made by subjects in the control group would have been avoided w ith ETM . Section 6.3.3 will discuss the experiment results in more detail. 6 .3 .2 P ro ced u re We carried out a series of experim ents th at com pared the performance of subjects using EX PECT vs. subjects using EX PEC T plus ETM. Each subject performed two different scenarios, one with EX PEC T and the other with EX PECT plus ETM , so the results were independent of differences in subject’s background and skills. Each scenario wets performed with both tools so the results were independent of the complexity of the scenario. Some subjects used EXPECT first and others used EX PEC T plus ETM first so the results were independent of the order in which the tools were 61 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. used. All these factors were balanced such th a t they would even out:1 the number of subjects and their qualifications for each one of the EX PECT and ETM groups, the number of times that each tool was used for each scenario, and the num ber of times that each tool was used in the first place. We chose scenarios that were simple enough to be learned and executed in one session but com plex enough to challenge the subjects in the control group. To facilitate the subjects acquisition of the knowledge required to perform the scenarios, both scenarios were based in the same applica tion domain which was explained to all the subjects during a presentation. The subjects and the dom ain were especially chosen to avoid m arkedly differences in the subjects previous exposure to the domain. For this study we used an extract of a KBS from a transportation planning evaluation system developed as part of a DARPA funded project [Tate, 1996, Burstein, 1994]. We defined two modification scenarios of different level of complexity. The simpler of the scenarios (RTT)2 consisted in extending the KBS to handle transportation movements done initially only by ships to handle any kind of vehicles (i.e., both ships and aircraft). The more complex scenario (PAE)3 consisted in adding more detail to an existing evaluation. The subjects were divided into two categories: • E x p e rie n c e d E X P E C T u s e rs with previous experience in developing KBSs in EXPECT • N o n -E x p e rie n c e d E X P E C T u s e rs who were familiar with EXPECT and its principles, but who had not developed a KBS with EX PECT before. None of the subjects had used ETM before or knew about it in any detail. We carried out this experiment twice with slight modifications to the tool and the scenarios. For the first round we only had implemented a subset of the KA Script library (7 KA Scripts). To put subjects in a context in which the KA Scripts in the library would be applicable, we indicated 1 W ith o n e e x c e p tio n becau se th e n u m b e r o f s u b je c ts av ailab le w as uneven 2 R T T s ta n d s fo r R o u n d -T rip -T im e 3 P A E s ta n d s fo r P e rso n n e l-A irp o rts E v a lu a tio n 62 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. (to both groups, EX PEC T and ETM) the first change th a t had to be performed for each scenario. For the second round we had a more complete set o f KA Scripts (37 KA scripts) and we did not indicate or restrict the initial changes. In this round we also slightly complicated both scenarios. The num ber of subjects th at participated in this study was seven, four in Round I, and three in Round II. None of the subjects that participated in the first round participated in the second round. One of our hypotheses is that ETM would be especially helpful when the details of the KBS im plem entation are unknown. To explore this hypothesis, we concealed the details of the KBS im plem entation in the initial presentation of the dom ain to the subjects, and those details were disclosed only in the beginning of the execution of the scenario. We also described the scenarios in an im plem entation independent way. The m aterials given to subjects are described in Ap pendix A.2. All experiments followed the same general procedure distributed along two days. Day 1: Domain and. tools presentation In the first day, subjects attended a presentation th a t introduced EX PECT, ETM, and the appli cation dom ain. Day 2: Practice and execution In the second day, subjects performed the following activities: 1. Executed a practice scenario comparable to the ones to be used during the test. This scenario was performed once using EX PECT and repeated later using ETM . The purpose of this practice was to get subjects familiar with the tools, the dom ain, and the procedures of the experiment. 63 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 2. Executed two test scenarios, one using E X PE C T and the other using EX PECT plus ETM, alternating the order o f the tool that was used first. The sim pler scenario (RTT) was executed first. 3. Answered a feedback questionnaire regarding their impressions and difficulties in using both tools. Scenario execution Subjects were given the following materials: • General instructions for the experiment • Tool descriptions and m anuals (EXPECT and ETM) • Description of the dom ain used in the experim ents, as shown in Appendix A .2, including: — diagram s illustrating dom ain relations — English description of the problem-solving knowledge for the scenario to be performed. This description combined the knowledge represented in the initial KBS and the knowl edge required to perform the extensions indicated by the scenarios. • An initial version of the KBS (without errors) along with a printout of its content. • A high-level description of the modifications th a t the subjects had to perform. • A sam ple problem to test the KBS after the m odification. First, subjects analyzed the dom ain and the specifications. Then, they performed the given modification scenario until the KBS gave the correct results in the sample problem. During the execution of the scenarios, the experimenter only occasionally assisted subjects th at had problems interpreting the instructions, using the KA tools, or th a t got stuck or confused. In the first round of the experiment, this procedure was slightly different. We interrupted the subjects ju st before they were about to perform the first modification to the KBS, we took note of 64 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. the change that they intended to perform, and then we indicated them which change we wanted them to perform first (to both groups, EXPECT and ETM ). This was necessary because we had implemented only a small fraction of the KA Script library and we needed to bring the modification into a context in which these KA Scripts were applicable. The following d ata was collected during the execution of the scenario: • time spent analyzing the dom ain and the scenario specifications before starting to change the KBS • time to complete the whole modification • log of changes performed to the KBS • log of errors posted in the EX PECT Agenda after each change to the KBS • log of KA Scripts executed • detailed notes of the actions performed by the subjects, including how they approached the problem and what m aterials they consulted. For this purpose we asked the users to voice what they were thinking and doing during the execution of the scenario. 6.3 .3 A n a ly sis o f th e R e su lts This study consisted of four sets of experiments corresponding to the two scenarios: RTT (simpler) and PAE (more complicated), with Round I and Round II for each. We refer to these four sets of experiments as RTT-I, RTT-II, PAE-I, and PAE-II, and to the subjects participating in them Sl-I (Experienced), S2-I (Not Experienced), S3-I (Experienced), and S4-I (Not Experienced) for Round I, and Si-II (Experienced), S2-II (Experienced), and S3-II (Not Experienced) for Round II. Because in the second round we slightly changed the KA scenarios, the KA tools, and the experiment procedure (as described in the previous section), we will not combine data from different rounds. 65 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. In the simpler scenario for both rounds (RTT-I and RTT-II) the d a ta obtained were more homogeneous and allowed a more straightforward comparison. However, in the more complex scenarios (PAE-I and PAE-II) the d a ta had considerable levels of noise th at complicated their comparison. We will discuss the results of these scenarios separately. Simpler Scenarios (RTT-I and RTT-II) In RTT-I, all the subjects solved the assigned KA scenario in the same way. This solution consisted in modifying one existing m ethod and creating two new methods based on two existing ones (i.e., the overall modification involved five related methods out of a total of 12). This modification was performed with three changes (one initial and two follow-up) which all the subjects performed following the same order. Although the first of these changes was imposed by the experimenters, all the subjects except one had decided to start the modification by this same change (or a very sim ilar one). In RTT-II, all the subjects solved the assigned KA scenario in the same way although not in the same order. The solution to the scenario consisted in modifying one existing m ethod, creating two new methods based on two existing ones, and modifying one of the new m ethods (i.e., the overall modification involved five related m ethods out of a total of 11). This m odification was performed with four changes (one initial and three follow-up). From all the subjects, S l-II and S2-II followed the same order in performing these changes while S3-II swapped the order of the first two changes and the last two changes. Although in this round subjects were free to sta rt the modification with whatever change they considered appropriate, Sl-II and S2-II chose to sta rt with the same (or a very similar) change th at we had required in the previous round. The results of these experiments, summ arized in Tables 6.1 and 6.2, were positive and agreed with our hypotheses. In these tables we can see th at the subjects using ETM were able to suc cessfully execute all the follow-up changes guided by ETM (the number of modification commands executed by KA Scripts equals executed modification commands). These tables also show that the 66 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. E X P E C T E T M S l - I (E ) S2-I (N) S 3 -I (E ) S 4 -I (N ) T im e in itia l a n a ly s is p lu s first c h an g e (m in ) 16.00 15.17 10.07 8.98 T im e c o m p le tin g m o d ific a tio n (m in) 1 1 .0 5 1 5 .72 9 .0 0 6 .0 8 T o ta l tim e (m in ) 2 7.05 30.89 19.07 15.06 # e x e c u te d m o d ific a tio n c o m m a n d s 2 2 2 2 # m o d ific a tio n c o m m a n d s e x e c u te d by K A S c r ip ts N /A N /A 2 2 # e x e c u te d K A S c rip ts N /A N /A 2 2 # u s e r m ista k e s 0 0 0 0 # m ista k e s t h a t E T M w ould p rev en t 0 0 0 0 # c re a te d o r m o d ifie d m e th o d s 3 3 3 3 Table 6.1: Study I: Results RTT-I scenario - Summary per subject E X P E C T E T M S2-I1 (E ) S3-I I (N) S l - I I (E ) T im e in itia l a n a ly sis p lu s first c h a n g e (m in ) 4.39 10.32 2 7 .38 T im e c o m p le tin g m o d ificatio n (m in ) 1 3 .7 5 2 8 .4 2 1 1 .6 3 T o ta l tim e (m in ) 18.14 38.74 39.01 # e x e c u te d m o d ific a tio n c o m m a n d s 3 4 3 # m o d ific a tio n c o m m a n d s e x ec u te d b y K A S c r ip ts N /A N /A 3 # e x e c u te d K A S c rip ts N /A N /A 2 # u se r m is ta k e s 0 2 0 # m is ta k e s t h a t E T M w ould p re v e n t 0 0 0 # c re a te d o r m o d ifie d m e th o d s 3 3 3 Table 6.2: Study I: Results R T T-II scenario — Summary per subject time in completing the modification with ETM was shorter ( 44% shorter in RTT-I and 45% shorter in RTT-II), and th a t the tim e saved was more significant for the subjects with no previous expe rience with EX PEC T (19% saving for the experienced vs. 61% for the non-experienced in RTT-I and 15% saving for the experienced in RTT-II). We can also observe that the only subject th a t made mistakes (subject S3-II) was not using ETM (however, the number of mistakes made by the subjects seems not to be related to the tool, as we will see in the more complex scenarios). These mistakes were m ade while editing a m ethod body and would not have been avoided by using ETM . Figures 6.1 and 6.2 compare the subjects perform ance at each step of the m odification. A detailed analysis of these figures reveais some anom alies th at might have affected these results. In Figure 6.1 we can see th a t subject S2-I in the control group spent a disproportionate am ount of time to perform the second change (12 m in com pared to 6 m in maximum for the rest). The reason why S2-I took longer is th at S2-I was extra cautious and spent some time to confirm th at the first 67 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 1 2 3 » — S1-I (E) - A -S4-I (N - with ETM) —K -S 2-I (N) - ■ -S3-I (E - with ETM) Figure 6.1: Time increments at each step for RTT-I. Time increments are measured as the elapsed tim e between the finalization of two consecutive commands change had the intended effect in the KBS. Based on our experiment notes we estimate that the tim e taken by this extra check was 5 min. Even taking this tim e off, the time taken by this subject does not come close to the tim e of the subjects that used ETM. In Figure 6.2 we can see th at subject S3-II in the control group spent some time analyzing incorrect results and fixing errors caused by his/her own mistakes (7 minutes in Change 4 and 1 m inute in Change 5). Recovering from mistakes takes a significant am ount of time. Hence, it seems unfair to compare the performance of subjects when some of them m ade mistakes that were not related to the use of ETM . However, even without considering this tim e the time spent by S3-II does not come close to the tim e of the subjects that used ETM . 68 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 30.0 fixing mistake 25.0 analyzing incorrect results 20.0 15.0 10.0 5.0 0.0 i 2 4 5 3 I—»— S2-II (E) - ■ -S1-1I (E-w ith ETM) —* — S3-II (N) Figure 6.2: Time increments at each step for RTT-II M o re c o m p le x sc e n a rio s (P A E -I a n d P A E -II) Unlike the sim pler scenarios, there was considerable variance among the way that subjects solved the assigned scenario, which can be shown by the wide range of number o f created methods and number o f modified methods in Table 6.3 and Table 6.4 (from 2 methods for subject S2-II to 7 m ethods and 2 dom ain concepts for subject S3-I). The differences in the solutions were because while some subjects preferred to make a few general m ethods that cover several cases, others preferred to make several specific methods that cover a few particular cases each. Also, while some subjects preferred to modify the dom ain descriptions and let the knowledge representation subsystem make autom atic inferences for them , others preferred a more procedural approach. A third reason was th at while some subjects preferred to adapt or reuse existing m ethods, others preferred to ignore them . Another difference in the outcome of this more complex scenario was 69 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. E X P E C T E T M S 3 -I (E ) S4-I (N) S l- I (E ) S 2 -I (N ) T im e in itia l a n a ly sis p lu s first c h a n g e (m in ) 21.53 21.17 25.02 4.53 T im e c o m p le tin g m o d ific atio n (m in ) 4 8 .5 7 3 1 .7 2 1 4 .7 5 2 0 .3 0 T o ta l tim e (m in ) 70.10 52.89 3 9 .77 24.83 # e x e c u te d m o d ific a tio n c o m m a n d s 10 7 5 8 # m o d ific a tio n c o m m a n d s e x e c u te d b y K A S c rip ts N /A N /A 5 8 # e x e c u te d K A S c rip ts N /A N /A 3 4 # u s e r m ista k e s 3 3 0 0 # m ista k e s th a t E T M w ould p re v e n t 1 1 0 0 # c re a te d o r m o d ified m e th o d s 7 ( + 2 co n c.) 3 4 5 Table 6.3: Study I: Results PAE-I scenario — Summary per subject E X P E C T E T M S l - I I (E ) S 2 -II (E ) S 3 -II (N ) T im e in itia l a n aly sis p lu s first c h a n g e (m in ) 3 9 .9 7 11.62 73.55 T im e c o m p le tin g m o d ific a tio n (m in ) 2 5 .6 7 4 6 .2 5 3 2 .2 2 T o ta l tim e (m in ) 6 5 .6 4 57.87 105.77 # e x e c u te d m o d ific atio n c o m m a n d s 9 10 9 # m o d ific a tio n c o m m a n d s e x e c u te d b y K A S c rip ts N /A 3 4 # e x e c u te d K A S c rip ts N /A 2 2 i f u s e r m ista k e s 0 6 3 # m ista k e s t h a t E T M w o u ld p re v e n t 0 0 0 # c re a te d o r m o dified m e th o d s 7 2 3 Table 6.4: Study I: Results PAE-II scenario - Sum mary per subject the num ber of subjects that m ade costly mistakes (all the subjects in the control group in PAE-I and all the subjects using ETM in PAE-II). Most of those mistakes cannot be blamed to the usage or not of ETM . Although the presence of both circumstances (i.e., the differences in the solutions and the user mistakes) prevents us to perform a fair comparison between the performance of the subjects using ETM and the subjects in the control group, we still can draw some conclusions by analyzing the collected data carefully. In PAE-I, both subjects using ETM were able to successfully execute all the follow-up changes guided by ETM (the number of modification commands executed by KA Scripts equals executed modification commands in Table 6.3). Subjects using ETM m ade no mistakes while subjects in the control group m ade 6 mistakes. However, only 2 of those mistakes might have been prevented by using ETM . It would be unfair to make direct comparisons of completion time between subjects using ETM and subjects in the control group because both subjects in the control group spent 70 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 50.0 45.0 analyzing incorrect results + fixing mistake 40.0 35.0 fixing mistake 30.0 25.0 20.0 fixing mistake 15.0 10.0 5.0 . -X 0.0 9 11 5 6 7 8 1 2 3 4 j - ♦ *S1-I (E-with ETM) — S3-I (E) — S4-I(N) ■ M "S2-I (N - with ETM) j Figure 6.3: Tim e increments at each step for PAE-I extra tim e to correct their mistakes. Subject S3-I spent 32 minutes (Fig. 6.3 changes 3, 9, and II) while subject S4-I spent 7 minutes (changes 4, 6, and 8). From these two subjects, subject S4-I spent the least time recovering from m istakes and is the most suitable for comparisons. Subtracting the tim e it took S4-I to recover from his/her own mistakes, the tim e to complete the modification is still greater than the longer of the tim es for the subjects that used ETM (S4-I, discounting fixes, took 17% longer than S2-I). It is interesting to note that the m ajor difficulty that S4-I had during the modification was not being able to figure out which method needed to be changed in following up a change in a goal (change 5). This is exactly the kind of problems th at ETM aims to solve. In PAE-II, unlike the other scenarios, the two subjects using ETM were able to use ETM only for a fraction of the modification because they spent most of the m odification trying to fix their own mistakes (see Table 6.4). There are still other kind of changes th a t the ETM subjects were not 71 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 50.0 cleaning mess Failed attempts to correct mistake 45.0 40.0 35.0 30.0 fixing mistake- i f analyzing incorrect results - 25.0 20.0 15.0 10.0 5.0 0.0 10 12 3 8 9 11 4 5 7 1 2 6 —* — S1-1I (E) —• — S2-M (E) - ♦ -(with ETM) — A— S3-I1 (N) - A -(with ETM) - Figure 6.4: Time increments a t each step for PAE-II able to execute with ETM, and which were not concerned with correcting mistakes. These changes were needed to im plem ent modifications indicated in the task specifications that the users oversaw (Changes 9 for subjects S2-II and S3-II in Figure 6.4). All the changes that were not performed with ETM were actually outside of the scope of ETM . Subjects using ETM made 9 mistakes while subjects in the control group made none. Most of the mistakes were made by subject S2-II due to a misconception of the domain. 6.4 Study II: Realistic and Unseen Scenarios For this study we used the knowledge-bases and scenarios of the Challenge Problems from the High Performance Knowledge Bases (HPKB) DARPA program [Cohen et al., 1998]. These problems 72 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. were developed independently and with the specific purpose of testing the technologies being de veloped within the program. In particular, both the domain and the modification scenarios were unknown to the designer of ETM before this experiment. The modification scenarios consisted of performing five extensions to a large KBS concerned with evaluating enemy workarounds for a targeted obstacle and containing 62 problem -solving methods. Each extension required creating and modifying several problem-solving m ethods. In this experiment, we had a subject use ETM to implement the KBS extensions requested by the scenarios. Our subject was one o f the developers of the KBS who had also performed these sam e modifications m onths before w ithout ETM . During this experiment, the subject was allowed to consult the KBS th a t has resulted from th a t previous episode, and to copy and paste fragments of th at KBS in order to speed-up typing. However, we asked the subject to follow the guidance of ETM if appropriate. We expected th at our library would cover all the situations w ithin the scope of the current implem entation, th at is the situations in which m ethods should be created or modified to follow up additions or modifications of goals. Therefore, the following analysis concentrates on those situations. ETM was able to guide the subjects through most of the changes performed in the scenario (See Table 6.5). ETM guided the execution of 20 from a total of 22 modification comm ands (91%). The two remaining modification com m ands not executed with ETM were executed to correct user mistakes which were outside of the scope of ETM . ETM also helped to identify m ethods that already existed in the KB and which were used as a basis for newly created methods (See Table 6.6). From a total of 12 created m ethods, 7 were created based on specific methods suggested by ETM . From the remaining 5, 4 were created based on methods th a t the subject had picked up him /herself, and one was created from scratch. All of the methods were created with ETM . If the implem ented KA Script library would have included all the KA Scripts th at we had designed, the num ber of basis m ethods suggested by ETM would have been increased to 8. 73 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Follow-up Com m ands With ETM 20 91% W ith EX PEC T 2 9% Table 6.5: Study II: Number of follow-up m odification commands Base Method M ethods C reated with ETM Proposed by ETM 7 (8 ) 58% (67% ) Picked by the user 4 (3) 34% (25%) None (from scratch) 1 8% Table 6.6: Study II: Base m ethod selection for new m ethods (number in parenthesis would have been the result if all the designed KA Scripts would have been implemented) It is interesting to note th at when the subject wrongly thought that the modification was complete, ETM detected the need to perform additional changes. These changes were om itted the first tim e th at the subject had m ade these modifications, suggesting th at KA Scripts axe useful not only to make changes faster but also as checklists to ensure th at the changes are completed well. 6.5 Summary We have carried out two different studies to evaluate ETM . In the first study we compared the performance of subjects modifying KBSs using ETM (plus EX PECT) vs. subjects using EX PECT alone (control group). For this study, all the subjects were already familiar with EX PEC T but not with ETM . The results of this study show th at the subjects th at used ETM were able to perform all the required changes of the modification under the guidance of ETM, except for a few cases th at were out of the scope of ETM . The results also show th at when the subjects did not make mistakes, the subjects that used ETM completed the m odification in less time than the subjects in the control group, and th a t this difference in tim e was more evident when the subjects had less experience in using EX PECT. However, due to the sm all num ber of subjects and the high proportion of cases in which subjects made mistakes, these results cannot be considered conclusive. 74 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Although we expected to see a reduction in number of mistakes for the subjects using ETM, this hypothesis was not confirmed. In the second study, a subject used ETM to accomplish a realistic scenario. The scenarios had been developed specifically to evaluate knowledge acquisition technologies and were concealed to the developer of ETM prior to the beginning of the experiment in order to avoid biasing the development of ETM or the content of the KA Script library. The results of this study showed that ETM was able to guide our subject throughout almost all required changes of the scenario, and that it helped identify m ethods already existing in the KB that could be used as a basis for new methods. The execution of these experiments have illustrated some problems that can be present during the empirical evaluation of KA technology and which have to be considered when designing experimental procedures. 75 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Chapter 7 Related Work Our Script-based approach for modifying knowledge-based systems is related to other approaches for supporting modifications, refinement, and validation and verification of knowledge-based sys tems as well as some approaches to knowledge-based software engineering. 7.1 Support for KBS Modifications In role-limiting approaches ([M cDermott, 1988]), KA tools assume a fixed problem solving m ethod (or strategy) in the underlying KBS. This method identifies the roles that domain-dependent knowledge plays in problem solving. The KA tool helps users in filling these roles. For example, SALT ([Marcus and M cDerm ott, 1989]) is a KA tool for the propose and revise problem-solving method, a m ethod for solving configuration problems by proposing new values to configuration parameters and revising the configuration by fixing any constraint violation found. SALT supports users in entering the dom ain knowledge needed by this m ethod, which is (a) procedures for assigning initial values to the configuration param eters, (b) constraints that these param eters m ust satisfy, and (c) fixes th at revise these param eters if a constraint is violated. Like ETM, role-limiting tools support users in accomplishing KBS modifications that preserve the KBS consistency. C ontrarily to ETM, this support is dependent on the problem-solving method th at they implement and is hard-coded in the tool. This dependency enables the tool to provide a very strong support. For 76 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. example, SALT is able to detect sets of conflicting fixes using techniques specific to the propose-and- revise problem-solving m ethod. However, it cannot permit modifications to the problem-solving m ethod. This is a serious lim itation because problem-solving m ethods usually require modifications to adjust to different applications ([Musen, 1992]). Most recent work addresses these lim itation by composing a KBS from a library of smaller- size problem solving m ethods ([Puerta et al., 1992, Runkel and Birm ingham , 1993, Klinker et al., 199l]). This framework acknowledges th at a problem-solving m ethod is not a m onolithic procedure. A PSM has to solve different subproblem s, and the methods used to solve these subproblems may depend on the application. KA tools support knowledge engineers to construct the control structure of a KBS by combining fine-grained PSMs (called mechanisms) from a library. They can also generate knowledge-base editors for end users to fill in the knowledge roles needed by the PSMs. The tools for knowledge engineer are different from ETM in th at they support the creation of KBSs from reusable components (the PSMs) but do not support PSM modifications. Conversely, ETM does not support creation of KBSs but it supports modifications to PSMs. The end-user knowledge-editors are sim ilar to ETM in th at they exploit knowledge of the relationships between knowledge-base elements, in particular relations between domain concepts. However these editors use this knowledge only to support users to modify instances of dom ain concepts, while ETM can also support modifications to problem -solving methods and to definitions of domain concepts. In ([Yost, 1993, Yost, 1992]), Yost proposes an alternative way to overcome the limitations of role-limiting methods: to extend the ideas th a t underlay role-limiting methods to a general- purpose problem-solving m ethod. T he general-purpose problem-solving m ethod th at he uses is the Problem Space Com putational M odel (PSCM) [Newell et al., 1991]), a problem-solving method that manipulates problem-spaces, states, and operators. This m ethod defines a few general knowledge- roles concerned with search control decisions which are filled using a KA tool called TAQL. One of TAQL’s main contribution is the definition of a language for expressing these knowledge-roles. 77 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. However, its guidance for modifying the knowledge bases is weak. This contrasts with ETM , whose m ain focus is in guiding users in modifying KBSs. KI ([Murray, 1996]) is a machine-learning program th at support users to integrate new knowl edge into a large KB th at contains general statem ents about a domain. KI checks if the addition of the new knowledge allows the derivation of contradictions. In such a case it interacts with end-users to correct the faulty rules that caused the contradictions. KI also detects opportunities for embel lishing the KB. For example, it detects missing attributes and statements based on generalizations. Like ETM, KI supports users in modifying a KB preserving its consistency. But instead of relying on knowledge of modification procedures, as ETM does, it uses machine learning techniques. The use of these techniques is possible due to characteristics of the KBs content: simple horn-clauses describing a dom ain and no problem-solving knowledge a t all. This enables KI to infer how these KB elements can be modified to resolve a conflict. On the other hand, ETM’s knowledge bases contain complex elements (e.g., problem-solving methods), which makes impossible to infer how to modify them. 7.2 Support for KB Refinement Knowledge refinement is a stage in the life cycle of KBS development in which an initially imperfect KBS is corrected and completed. Given an imperfect KBS and sample problems accompanied with their solutions (provided by a dom ain expert), knowledge refinement tools autom atic or semi- autom atically determ ine how to correct the KBS so th at the solutions of the problems computed by the KBS agree with those of the expert. Several approaches have been explored for solving this task. Traditional KA tools like TEIRE- SIAS ([Davis, 1979]) or PROTOS ([Bareiss et al., 1989]) interact exhaustively with the domain expert to localize and correct the flaws in the KBS. In the other hand, theory revision tools like 78 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. EITH ER ([Ourston and Mooney, 1994]) or FOCL ([Pazzani and Brunk, 1991]), autom atically cor rect a KBS given a set of sam ple problems and their solution. They accomplish this task by using a combination of machine learning techniques. However, these techniques require th at the KBSs be rule-based classification system s. Finally, apprenticeship systems like ODYSSEUS ([Wilkins, 1990]), LEAP ([Mitchell, 1990]), and NeoDISCIPLE ([Tecuci, 1992]) receive as input a domain ex pert’ s problem-solving episode. This detailed inform ation permits them to more efficiently localize the faulty elements of the KBS. The problem that ETM seeks to solve is a different one. Rather than to correct existing errors, ETM has to determine how to modify a KBS so th a t the KBS is left free of inconsistencies. In other words, the inconsistencies that ETM has to fix were introduced during the modification of the KBS. To fix them, ETM uses knowledge of typical modification procedures to understand how these inconsistencies were originated and how they must be fixed. Another difference is that knowledge-refinement tools aim to fix all the errors in the KB and will not be satisfied until all sample problems were correctly solved. On the other hand, ETM only aims to resolve a subset of all the errors, namely internal inconsistencies. It might happen that a KBS is consistent and still produces wrong results (e.g, because the dom ain knowledge is incorrect). Ideally, after being modified with ETM, a KBS should be refined. 7.3 Verification & Validation Verification and Validation (V&V) is concerned with assuring the quality of a KBS ([O’Keefe and O ’Leary, 1993]). Like ETM , some verification techniques also aim to correct inconsistencies in the KBS. These techniques have focused mainly on rule-based systems, and address anomalies like redundant rules (i.e., a rule conditions implies the conditions of other rule with the same conclusion), unreachable rules (i.e., rule conditions are never satisfied), ambivalent rules (i.e., rule conditions imply conditions of other rules with contradictory conclusion), rule cycles, and predicates 79 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. with wrong arity. Although verification and ETM share sim ilar goals, there is a fundam ental distinction th a t enables ETM to use its particular technique. W hile verification has to correct inconsistencies th a t were already present in the KBS, ETM has to correct inconsistencies th at were introduced during the KBS modification process. This distinction allows ETM to anticipate the occurrence of the inconsistencies and to use inform ation about how they were generated to help users in fixing them . 7.4 Knowledge Based Software Engineering ETM utilizes a library of typical KBS modification procedures in order to support modifications of KBSs. A sim ilar idea has also been used in some knowledge-based software engineering tools. Knowledge-based Software Assistant (KBSA) and its successor ARIES ([Johnson and Feather, 1991, Johnson et al., 1992]) are two software engineering tools th at provide integrated support for require ments analysis and specifications development. They utilize a library of evolution transformations (or ET) to support the incremental evolution of requirem ents into specifications. These ETs are similar in spirit to ETM 's KA Scripts. However, differences in the nature of the underlying task make these tools very different from ETM. KBSA and ARIES m anipulate semi-formal specifica tions of system s while ETM modifies actual im plem entations of KBSs. An example of a system change for KBSA or ARIES is to split a semi-formal specification of a process (i.e., a description of the d a ta flowing in and out of a process) into two more specific subprocess. As a consequence of this change all references to that process have to be split too. A sim ilar change for ETM involves actually modifying the PSM procedure and the consequences of this change are more complex to determine. To handle this added complexity, ETM has to resort to novel techniques like keeping a history of modifications, past versions of the KB, and selecting KA Scripts based on emerging inconsistencies rather than on type of change. 80 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. KBEmacs ([Waters, 1985]). is another example of a tool th at utilizes a library of procedures to support users. KBEmacs is a knowledge-based program editor th a t perm its the construction of a program rapidly and reliably by combining algorithmic fragments (called cliches) from a library. KBEmacs cliches are equivalent to ETM KA Scripts except that cliches are used to generate pro gram s while KA Scripts are used to modify KBSs. Another difference is that with KBEmacs the program m er has to indicate when, which and how to use a cliche. ETM supports users by auto m atically detecting opportunities for using KA scripts (after observing inconsistencies in the KBS), suggesting what KA Scripts can be applied (the ones capable of fixing that inconsistency) and then supporting its application. A considerable part of our research was geared to the implementation of such support. An interesting point worth noticing is that both KBEmacs and ETM can resort to a general purpose tool (Emacs and EX PECT respectively) to let a user perform modifications not supported by their procedure libraries. 8 1 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Chapter 8 Conclusions 8.1 Summary of the Approach and Results Our goal is to develop an approach for building knowledge-acquisition tools that provide strong support for a wide range of modifications and knowledge-based system types. To achieve this goal, we equipped a knowledge-acquisition tool with a library of knowledge-acquisition scripts (or KA Scripts) which represent prototypical procedures for modifying knowledge-based systems. A knowledge-acquisition tool uses these KA Scripts to guide users in following up the consequences of changes performed to a KBS. The advantage of KA Scripts is th at they provide a context for relating individual changes to different parts of the knowledge-based system enabling the tool to analyze each change from the perspective of the overall modification. This kind of analysis complements previous approaches for interpreting changes to a KBS and enables a knowledge-acquisition tool to provide a more precise guidance. Because KA Scripts are problem-solving method independent, they can be used to support modifications of any kind of knowledge-based system. Furthermore, because KA Scripts represent varied procedures for modifying different aspects of a KBS, they can support a wide range modifications. We have implemented a script-based knowledge-acquisition tool called ETM that supports mod ification to knowledge-based systems developed within the EX PECT framework. In implementing 82 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. ETM we addressed several research issues th a t concerned the design of KA Scripts, the coordina tion of the execution of KA Scripts, a flexible interaction with the user, and development of a KA Script library. To evaluate ETM , we carried out two studies: Study I compared the performance in modifying KBSs for subjects using ETM vs. subjects using EXPECT, and Study II assessed the adequacy of ETM for complex and realistic scenarios. These studies have shown th at ETM guided users throughout all the related changes of the KBS modification scenarios and suggest th at ETM reduces the tim e required to complete KBS modifications. 8.2 Contributions T he m ain contributions of this thesis axe the following: • A new approach th at uses KA Scripts to guide users in completing a knowledge-based system modification. The approach addressed issues of designing KA Scripts, arbitrating among alternative KA Scripts, and interacting flexibly with the user. • A methodology for system atically developing a KA Script library with good coverage of the typical procedures used in knowledge-based system modification scenarios. • Im plem entation of a script-based KA tool including a KA Scripts library, a suitable user interface and a KA script m anager. • Analysis and evaluation of the im plem ented script-based KA tool to dem onstrate the advan tages of the approach. 8.3 Limitations and Future Work O ne lim itation of current incarnation of the Script-Based Knowledge-Acquisition approach is that the KA Scripts have a lim ited scope. Each KA Script procedure concerns only a couple of KB 83 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. elements. However, extended KBS modifications involve larger sets of related KB elements. The reason why current KA Scripts handle only a few KB elements each is that these KA Scripts were conceived to handle the low level details of the modification. A planned extension to this approach is to combine these KA Scripts with m ore abstract KA Scripts that can look at the KBS m odification from a higher-level perspective. The more abstract KA Scripts can be used to plan the modification at a m ore abstract level and the lower-level KA Scripts could take care of the details of the modification. T he current im plem entation of ETM is based on two assumptions th at do not always hold in reality. T he first one is th a t all incomplete KBS modifications will introduce inconsistencies in the KB. However, there are KB changes th at need to be followed up but that do not introduce inconsistencies. For exam ple, if both, a m ethod and the goal achieved by that method, have to be m ade more general, and the m ethod is changed first, then this change will not introduce an inconsistency because a more general m ethod can still achieve the same goal than before. Consequently th at change would not be continued. Another example is when the relation between two KB elements cannot be expressed in the EX PECT language and hence not inferred by the EX PEC T KA tool (e.g., the relation between two goals posted by the same method). In this case, changing only one of these elements and leaving the other unchanged would not introduce any inconsistency. The second assum ption is th at all inconsistencies in the KBS are due to incomplete KBS modifications. T his assum ption is ignoring the fact th at users indeed sometimes make mistakes which introduce inconsistencies. ETM will wrongly treat these mistakes as changes that have to be followed up. T he first of the assum ptions was needed because inconsistencies were used to recognize situations th at required of the application of a KA Script and to locate the KA Scripts th at would take care of those situations (because KA Scripts are indexed by the inconsistency th at they would repair). We can get rid of this assum ption if we add to the set of KB inconsistencies detected by EXPECT other indicators of an incom plete modifications, for example th at a m ethod was generalized but a 84 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. goal was Dot, or that a posted goal was changed but a sim ilar goal posted by the same method was not. Because EXPECT cannot infer whether those changes really have to be followed up, the user would have to confirm this fact. The second assum ption was introduced because ETM could not discern whether a user change was a m istake or not. Although ETM cannot autom atically resolve this conflict it can support a user to recover from a mistake by taking advantage of the the same historical information kept in the tool for other purposes. This information can be used by ETM to go back the modification to the state previous to the user mistake. Current KA Scripts and their steps are represented as LISP functions. One possible extension to this work is to implement a special-purpose language for describing KA Scripts and a basic set of operators that implement KA Script steps. Besides aiding in the maintenance of the KA Script library, a more expressive language for describing KA Scripts would enable the use of meta-tools th at can reason about KA Scripts themselves. For example, we could im port some concepts from general-purpose planners, like HTNs, and generate more flexible KA Scripts. We can also attem pt to learn new KA Scripts by observing how users cope with situations not covered in our library, or we might teach ETM new KA Scripts by dem onstration. An issue that was left over in this dissertation and which requires further research is how to guide users in modifying method bodies (e.g., how to generalize the body of a m ethod) Finally, we can investigate the com bination of the SBKA approach and the role-limiting ap proach. Theoretically, the role-limiting m ethods approach provides “off the shelf “ implementations of general problem-solving methods(PSM s) th at only need to be filled in with domain knowledge in order to be used. However, in practice, this PSM s need to be adapted to the peculiarities of the application domain. SBKA could be used to support users in achieving this adaptation. 85 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Reference List [Bachant and McDermott, 1984] Ju d ith Bachant and John M cDerm ott. R1 revisted: Four years in the trenches. A l Magazine, Fall 1984. [Bareiss et al., 1989] Eay Bareiss, Bruce W . Porter, and K enneth S. Murray. Supporting start-to- finish development of knowledge bases. Machine Learning, 4:259— 283, 1989. [Burstein, 1994] Mark H. Burstein, editor. A R P A /R om e Laboratory, Knowledge-Based. Planning and Scheduling Initiative, Workshop Proceedings, Tucson, Arizona, February 1994. [Cohen et al., 1998] Paul Cohen, Robert Schrag, Eric Jones, Adam Pease, Albert Lin, Barbara Starr, David Gunning, and M urray Burke. The darpa high-performance knowledge bases project. A l Magazine, 19(4), 1998. [Davis, 1979] Randall Davis. Interactive transfer of expertise: Acquisition of new inference rules. Artificial Intelligence, 12(2):121— 157, 1979. [Dev and Anderson, 1997] Narendra Dev and B art Anderson. Pim tool, an expert system to trou bleshoot computer hardware failures. In Proceedings o f the Fourteenth National Conference on Artificial Intelligence, Providence, RI, July 1997. [Gil and Melz, 1996] Yolanda Gil and Eric Melz. Explicit representations of problem-solving strategies to support knowledge acquisition. In Proceedings o f the Thirteenth National Con ference on Artificial Intelligence, Portland, OR, August 1996. [Gil and Tallis, 1997] Yolanda Gil and Marcelo Tallis. A script-based approach to modifying knowledge-based systems. In Proceedings o f the Fourteenth National Conference on Artificial Intelligence, Providence, RI, July 1997. [Gil, 1994] Yolanda Gil. Knowledge refinement in a reflective architecture. In Proceedings o f the Twelfth National Conference on Artificial Intelligence, Seattle, WA, 1994. [Johnson and Feather, 1991] W. Lewis Johnson and M artin S. Feather. Using evolution transfor m ations to construct specifications. In Automating Software Design, pages 65-92. AAAI Press, 1991. [Johnson et al., 1992] W. Lewis Johnson, M artin S. Feather, and David R. Harris. Representa tion and presentation of requirements knowledge. IE E E Transactions on Software Engineering, 18(10):853-869, October 1992. [Klinker et al., 1991] G. Klinker, C. Bhola, G. Dallemagne, D. Marques, and J M cDerm ott. Usable and reusable programming constructs. Knowledge Acquisition, 3(2):117-135, 1991. [MacGregor, 1991] R. MacGregor. The evolving technology o f classification-based knowledge rep resentation systems. In J. Sowa, editor, Principles o f Sem antic Networks: Explorations in the Representation o f Knowledge. Morgan Kaufm ann, San M ateo, CA, 1991. 86 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. [Marcus and M cD erm ott, 1989] S. Marcus and J. M cDermott. SALT: A knowledge acquisition language for propose-and-revise systems. Artificial Intelligence, 3 9 (l):l-3 7 , May 1989. [McDermott, 1988] J. M cDerm ott. Prelim inary steps towards a taxonom y of problem-solving methods. In S. Marcus, editor, Automating Knowledge Acquisition fo r Knowledge-Based Sys tems. Kluwer Academic Publishers, Boston, MA, 1988. [Mitchell, 1990] Tom M. Mitchell. LEAP: A learning Apprentice for VLSI Design. In Machine Learning: An Artificial Intelligence Approach, volume 3, pages 271— 301. Morgan Kaufm ann, San Mateo, CA, 1990. [Murray, 1996] Kenneth S. M urray. KI: A Tool for Knowledge Integration. In Proceedings o f the Thirteenth National Conference on Artificial Intelligence, Portland, OR, August 1996. [Musen, 1992] M. A. Musen. Overcoming the lim itations of role-limiting methods. Knowledge Acquisition, 4(2):165— 170, 1992. [Newell et al., 1991] A. Newell, G. R. Yost, J. E. Laird, P. S. Rosenbloom, and E. Altm ann. For mulating the problem space com putational model. In R. F. Rashid, editor, Carnegie Mellon Computer Science: A 25-Year Commerative. ACM Press/Addison-W esley, Reading, PA, 1991. [O’Keefe and O ’Leary, 1993] Robert M. O ’Keefe and Daniel E. O ’Leary. Expert System Verifica tion and Validation: A Survey and Tutorial. Artificial Intelligence Review, 7:3-42, 1993. [Ourston and Mooney, 1994] Dirk Ourston and Raymond J. Mooney. Theory refinement combin ing analytical and empirical m ethods. Artificial Intelligence, 66:311-344, 1994. [Pazzani and Brunk, 1991] Michael J. Pazzani and Clifford A. Brunk. Detecting and correcting errors in rule-based expert systems: an integration of empirical and explanation-based learning. Knowledge Acquisition, 3(2):157— 173, June 1991. [Puerta et al., 1992] A. R. P uerta, J. W . Egar, S. W . Tu, and M. A Musen. A multiple-method knowledge-acquisition shell for the autom atic generation of knowledge-acquisition tools. Knowl edge Acquisition, 4(2):171-196, 1992. [Rosembloom et al., 1991] P. S. Rosembloom, J. E. Laird, A. Newell, and R. McCarl. A prelimi nary analysis of the Soar architecture as a basis for general intelligence. Artificial Intelligence, 47:289-325, 1991. [Runkel and Birm ingham , 1993] J. T. Runkel and W . P. Birmingham. Knowledge acquisition in the small: Building knowledge-acquisition tools from pieces. Knowledge Acquisition, 5(2):221- 243,1993. [Shpilberg et al., 1986] David Shpilberg, Lynford E. Graham , and Harry Schatz. Expertax: An expert system for corporste tax planning. Expert Systems, 3(3):136— 151, July 1986. [Swartout and Gil, 1995] Bill Sw artout and Yolanda Gil. EX PECT: Explicit Representations for Flexible Acquisition. In Proceedings o f the N inth Knowledge-Acquisition fo r Knowledge-Based Systems Workshop, Banff, A lberta, Canada, 1995. [Tallis and Gil, 1999] Marcelo Tallis and Yolanda Gil. Designing scripts to guide users in modifying knowledge-based systems. In Proceedings o f the Sixteenth National Conference on Artificial Intelligence, Orlando, FL, July 1999. [Tallis et al., 1999] Marcelo Tallis, Jihie Kim, and Yolanda Gil. User Studies of Knowledge Acqui sition Tools: M ethodology and Lessons Learned. To appear in the Journal o f Experimental and Theoretical Artificial Intelligence, 1999. 87 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. [Tallis, 1998] Marcelo Tallis. A Script-Based Approach to Modifying Knowledge-Based Systems. To appear in the International Journal o f Human Computer Studies, 1998. [Tate, 1996] Austin Tate, editor. Advanced Planning Technology / Technological Achievements o f the A R P A /R om e Laboratory Planning Initiative. AAAI Press, Menlo Park, California, 1996. [Tecuci, 1992] G. D. Tecuci. A utom ating Knowledge Acquisition as Extending, Updating, and Improving a Knowledge Base. IE E E transactions on Systems, Man, and Cybernetics, 22(6):1444— 1460, 1992. [Waters, 1985] R. Waters. The program m ers apprentice: A session with kbemacs. IEEE Trans actions on Software Engineering, 11(11) :1296— 1320, November 1985. [Webb et al., 1999] Geoffrey I. Webb, Jason Wells, and Zijian Zheng. An experimental evaluation of integrating machine learning w ith knowledge acquisition. Machine Learning, 35:5-23, 1999. [Wilkins, 1990] David C. Wilkins. Knowledge base refinement as improving an incorrect and in complete dom ain theory. In Machine Learning: An Artificial Intelligence Approach, volume 3, pages 493-513. Morgan Kaufmann, San Mateo, CA, 1990. [Yost, 1992] Gregg R. Yost. TAQL: A Problem Space Tool fo r Expert System Development. PhD thesis, Carnegie Mellon University, School of Com puter Science, Pittsburgh, PA, 1992. [Yost, 1993] Gregg R. Yost. Acquiring Knowledge in Soar. IEEE Expert, 8(3):26-34, June 1993. 8 8 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Appendix A Knowledge-Acquisition Scripts Library This appendix describes the library of Knowledge-Acquisition (KA) Scripts that was developed for ETM , our implementation of a Script-Based Knowledge-Acquisition (SBKA) tool for EXPECT. The library was generated from a set of attributes for characterizing KA Scripts, as described in C hapter 5, Section 5.1.3. We concentrated our analysis on KA Scripts that take care of the interdependency between goads and the m e th o d s th at achieve them . We have considered two sets of KA Scripts, a) KA Scripts that follow up changes to a goal by modifying or creating the m ethods that would achieve the changed goal, and b) KA Scripts th at follow up changes to a m ethod capability by modifying or creating the goals to be achieved by the changed method. A fraction of approximately 40% of this library has been implemented in ETM . The appendix starts describing the set of KA Script attributes used to generate the library and then it describes the KA Scripts for following changes to goals th a t have been implemented for ETM. A .l KA Script Attributes The following are the attribute values used to generate the KA Scripts th at follow up changes to goals. 89 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. A ttr ib u te V alues 1. C h a n g e a c tio n t h a t p r o d u c e d t h e c h a n g e in th e g o al: (with respect to the state of the KB a t the beginning of the m odification.) M — Modification: The goal has been modified. N — Addition: The goal has been added to a method body. C — Creation: The goal is in a new m ethod. D — Deletion: The goal has been deleted from a m ethod body. R — Removal: The goal was in a m ethod th a t has been removed. I — Expansion: The goal has N O T been changed but its decomposition has included a new subgoal. 2. R e f e re n t (b a se ) case: A KA Script follows up the change in the goal by modifying (or creating a new m ethod based on) the m ethod that achieved the goal of the re f e re n t case. O — Original: Same goal at the beginning of the m odification. A — Analogous: An analogous goal.1 AO — Analogous Original: An analogous goal at the beginning of the m odification.2 S — Similar: A similar goal in the sam e m ethod to the changed goal. N — None: The follow up m odification is not based on any referent case. T he m ethod to be modified (or imitated) is chosen by the user or created from scratch. l G o a l G in m e th o d M is analogous to g o al G ’ in m e th o d M ’ if M is an alo g o u s to M ’ a n d G c o rre sp o n d s to G ’. M e th o d M is a nalogous to m e th o d M ’ if M w as c r e a te d b a s e d o n M ’ o r v ice v e rsa , (e.g., in th e e x a m p le o f F ig u re 2.2 in C h a p te r 1.5, g o al fin d (F L Y IN G -D IS T A N C E ) in m e th o d M 2 ’ is a n a lo g o u s to goal fin d ( S A I L I N G - D I S T A N C E ) in m e th o d M 2 ). 2 S o m e tim e s, in s te a d o f m o d ify in g a n e x is tin g m e th o d , a u s e r d e cid es to rep lace th e e x is tin g m e th o d by a new o n e c re a te d b y a n a lo g y b a se d o n th e e x is tin g o n e . I n th o s e c a se s, th e b a se m e th o d o f th e a n a lo g y c a n o n ly be found in th e o rig in a l K B . F u rth e rm o re , th e in d ic a tio n o f w h ic h m e th o d s a ch ie v e d th e goals p o ste d b y th e b a se m e th o d can o n ly b e fo u n d in th e o rig in a l K B to o . In th e g iv en s itu a tio n , a n a n a lo g o u s o rig in al goal is a c o rre sp o n d in g goal in a n a n a lo g o u s m e th o d t h a t c a n only b e fo u n d in th e o rig in a l K B . 90 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 3. Difference between the modified goal to be followed up (MG) and the goal of the refereed case (RG): G - Generalized: The modified goal includes the referred goal. For example, M G = (get (obj (spec-of location)) (of (INST-OF PORT))) and RG = (get (obj (spec-of location)) (of (INST-OF SEAPORT))). S — Specialized: The modified goal is a subcase of the referred goal. For example, MG = (get (obj (spec-of location)) (of (INST-OF SEAPORT))) and RG = (get (obj (spec-of location)) (of (INST-OF PO RT))). B — Broaden: The scope of the modified goal is broader than the scope of the referred goal. For example, MG = (count (obj (DESC PORT)) (of (inst-of location))) and RG = (count (obj (DESC SEAPORT)) (of (inst-of location))). N — Narrow: The scope of the modified goal is narrower than the scope of the referred goal. For example, MG = (count (obj (DESC SEAPORT)) (of (inst-of location))) and RG = (count (obj (DESC PORT)) (of (inst-of location))). M — Mutated: The modified goal is a sibling of the referred goal. For example, MG = (get (obj (spec-of location)) (of (INST-OF SEAPORT))) and RG = (get (obj (spec-of location)) (of (INST-OF AIRPORT))). R — Rephrased: The modified goal is a rephrasing of the referred goal. For example, MG = (FIND-ROUTE (obj (inst-of OBJECTIVE))) and RG = (FIND (obj (spec-of ROUTE)) (for (inst-of OBJECTIVE))). A - Added Parameter: The modified goal has more param eters than the referred goal. For example, MG = (find (obj (spec-of route)) (BY (SPEC-OF SEA)) (for (inst-of OBJECTIVE))) Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. and RG = (find (obj (spec-of route)) (for (inst-of OBJECTIVE))). D — Deleted Parameter: The modified goal deleted some param eters compared with the referred goal. For example, MG = (find (obj (spec-of route)) (for (inst-of OBJECTIVE))) and RG— (find (obj (spec-of route)) (BY (SPEC-OF SEA)) (for (inst-of OBJECTIVE)))). 4. G e n e ra l follow -up s tra te g ie s : General strategies for following up a change in a goal. C — Create: Create a new m ethod th at achieves the changed goal. A - Adapt: A dapt a method to the changed goal (after being modified this method might not be able to achieve the goals th a t it achieved before). For example, adapt a method with capability: (find (obj (spec-of RTT)) (of (inst-of SHIP))) to the goal (find (obj (spec-of RTT)) (of (inst-of A IRCRA FT))). G - Generalize: Generalize a m ethod to cover the changed goal (after being modified this method will still be able to achieve the same goals than before). For example, generalize a method with capability: (find (obj (spec-of RTT)) (of (inst-of SHIP))) to (find (obj (spec-of RTT)) (of (inst-of VEHICLE))) in order to achieve the goal (find (obj (spec-of RTT)) (of (inst-of AIRCRAFT))) as well. A particular case of the generalization is to change a constant param eter of the m ethod by a variable. D - Decompose: Instead of creating or modifying methods to achieve the changed goal, create methods to achieve the subgoals included in a decomposition o f the changed goal. For example, create methods with capabilities (find (obj (spec-of RTT)) (of (inst-of SHIP))) and (find (obj (spec-of RTT)) (of (inst-of AIRCRAFT))) in order to achieve the goal (find (obj (spec-of RTT)) (of (inst-of VEHICLE))). 92 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. C om b in in g A ttrib u te V alu es We generated a KA Script library by defining the KA Scripts corresponding to the meaningful combinations of attribute values. T he following rules were used to indicate invalid combinations of attribute values. 1. Change Action = (Addition or Creation) = > Referent Case ^ Original Rationale: A new goal did not exist in the original KB. 2. Referent case = Analogous Original = > (Change Action = (Addition or Creation)) and (Referent m ethod is not being used in the current state of the KB). Rationale: If Change Action = Modification, then the changed goal existed at the beginning of the modification. If the changed goal existed at the beginning of the modification, then it seems strange to prefer as a basis a goal matched in an analogous m ethod rather than in the original m ethod itself. An exception to this rule is when a goal in an analogous method has also been modified (or created) and already been resolved. B ut in th at case, we should look at the analogous method in the current state rather than at the beginning of the modification. If the analogous method is still being used in the current state, it seems strange to prefer to look at the method m atched at the beginning of the modification rather than the one at the current state of the KB . 3. Strategy = Adapt m ethod =>- (Change Action = Modification AND Referent Case = Orig inal) OR (Change Action = Creation AND Referent Case = Analogous Original AND the Goal that Referent M ethod used to achieve does not exist in current state) Rationale: We do not want to modify a m ethod that is still needed to achieve other goals. 4. Strategy = Decompose = > (Referent Difference = Broaden or Generalized) or Change Action = Expansion Rationale: This strategy does not apply to other cases. 93 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 5. Strategy = Generalize = > Referent Difference = M utate or Broaden or Generalized or Narrow Rationale: This strategy does not apply to other cases. 6. W hen Referent Difference = Generalized = > Strategy = Adapt is equivalent to Strategy = Generalize Rationale: In this case both strategies produce the same result. A.2 Implemented Knowledge-Acquisition Scripts Based on the valid combinations of a subset of attribute values, we defined and implemented the following KA Scripts (attribute values considered: Change Action = M ,N,C,I, Difference Referent Case = G,M ,A, and Strategy = C,A ,G ,D ). 94 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. R eferen t C h a n g e S tra K A S c rip t Rel | D iff A ctio n te g y 1 O M M A M u ta te th e m e th o d th a t achieved th e o rig in a l form o f a m u c a te d goal. 2 O M M C C re a te a m e th o d sim ila r to th e m e th o d t h a t achieved th e o rig in a l form o f a m u ta te d g o al. 3 O M M G G e n e ra liz e th e m e th o d th a t achieved th e o rig in a l form o f a m u ta te d g o al in o rd e r to e n a b le th e m e th o d to achieve b o th form s o f th e g o al. 4 O G M G G e n e ra liz e th e m e th o d th a t achieved th e o rig in a l form o f a g en eralized goal. 5 O G M C C re a te a m e th o d m o re g en eral th a n th e m e th o d th a t ach ieved th e o rig in al form o f a g en e ra liz e d g o al. 6 O G M D C re a te a m e th o d sim ila r to th e m e th o d t h a t achiev ed th e o rig in a l form o f a g e n e ra liz e d g o al in o rd e r to provide th e m e th o d s c a p a b le o f ach iev in g a decom p o sitio n o f th e g en eralized goal. 7 o A M A A d d a p a r a m e te r to th e m eth o d th a t ach iev ed th e o rig in a l form o f a goal to w hich a new p a ra m e te r h as been added. 8 o A M C C re a te a m e th o d sim ilar, b u t including an e x tr a p a ra m e te r, to th e m e th o d th a t achiev ed th e o rig in a l form o f a goal to w hich a new p a ra m e te r h as b een ad d ed . 9 o M I D C re a te a m e th o d sim ila r to th e m e th o d t h a t achieved o n e o f th e m em bers o f th e o rig in a l d e c o m p o sitio n o f a goal w hose c u rre n t d e c o m p o sitio n now includes a new m em b er. 10 AO M C ,N A M u ta te th e m e th o d c u rre n tly unused b u t t h a t o rig in a lly achieved a g o al a n a l o g o u s to a g o al a d d e d in an existin g o r re c e n tly c re a te d m e th o d .(l) 11 AO M C ,N C C re a te a m e th o d sim ila r to th e m eth o d c u rre n tly u n u se d b u t th a t o riginally achiev ed a g o al an alo g o u s to a goal a d d e d in a n ex istin g o r recen tly c re a te d m e th o d .(1) 12 AO M C ,N G G e n e ra liz e th e m e th o d c u rre n tly unused b u t th a t o rig in a lly achieved a goal a n a lo g o u s to a goal a d d e d in an ex istin g o r re c e n tly c re a te d m e th o d in o rd e r to e n a b le th e m e th o d to achieve b o th g o a ls .(l) 13 AO G C ,N G G e n e ra liz e th e m e th o d c u rre n tly unused b u t t h a t o rig in a lly achieved a go al antil o gous to a m o re g e n e ra l go al ad ded in a n e x is tin g o r re c e n tly c re a te d m e th o d .(1) 14 AO G C ,N C C re a te a m e th o d m o re gen eral th a n th e m e th o d c u rre n tly unused b u t th a t o rig in a lly achiev ed a g o al analogous to a m o re g en eral g oal a d d e d in a n ex istin g o r re c e n tly c re a te d m e th o d .(1) 15 AO G C ,N D C re a te a m e th o d sim ila r to th e m eth o d c u rre n tly u n u se d b u t th a t o riginally achiev ed a g o al an alo g o u s to a m ore g e n e ra l g oal a d d e d in a n ex istin g o r re ce n tly c re a te d m e th o d in o rd e r to provide th e m e th o d s c a p a b le o f achieving a d ec o m p o sitio n o f th e m o re g o a l.(l) 16 AO A C .N A A d d a p a ra m e te r to th e m eth o d cu rre n tly u n u se d b u t t h a t o rig in a lly achieved a g o al a n a lo g o u s to a g o al th a t includes a n e x tr a p a ra m e te r a n d th a t w as a d d ed in a n e x is tin g o r recen tly c re a te d m e th o d .(l) 17 AO A C ,N C C re a te a m e th o d sim ilar, b u t including a n e x tr a p a ra m e te r, to th e m eth o d c u rre n tly u n u se d b u t th a t originally achiev ed a goal a n a lo g o u s to a g o al th a t in clu d es a n e x tr a p a ra m e te r an d th a t w as a d d ed in an ex istin g o r re c e n tly c re a te d m e th o d .(1) 18 A M \1 ,C ,N C C re a te a m e th o d sim ila r to th e m eth o d th a t achieves a g o al a n alo g o u s to a goal e ith e r m u ta te d o r a d d e d in an existin g o r re c e n tly c re a te d m e th o d . 19 A M M ,C ,N G G e n e ra liz e th e m e th o d th a t achieves a g o al a n a lo g o u s to a g o a l e ith e r m u ta te d o r a d d e d in a n e x istin g o r recently c re a te d m e th o d in o rd e r to en a b le th e m e th o d to ach iev e b o th g o als. 20 A G M ,C ,N C C re a te a m e th o d m o re gen eral th a n th e m e th o d th a t achieves a goal an alo g o u s to a m o re g e n e ra l g o al e ith e r m odified o r a d d e d in an ex istin g o r recen tly c re a te d m e th o d . 21 A G M ,C ,N D C re a te a m e th o d sim ila r to th e m eth o d th a t achieves a go al antilogous to a m ore g e n e ra l g o al e ith e r m odified o r added in a n e x is tin g o r re c e n tly c re a te d m e th o d in o rd e r to p ro v id e th e m eth o d s c a p a b le o f ach iev in g a d eco m p o sitio n o f th e m o re g o al. 22 A A M ,C ,N C C re a te a m e th o d sim ilar, b u t including a n e x tr a p a ra m e te r, to th e m e th o d th a t achieves a g o al an alo g o u s to a goal th a t in clu d es an e x tr a p a ra m e te r a n d th a t w as e ith e r m odified o r a d d e d in a n e x istin g o r recen tly c re a te d m e th o d . 95 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. R e fe re n t C h a n g e S tra K A S c rip t R et D iff A ctio n te g y 23 S M M ,C ,N C C r e a te a m e th o d s im ila r to th e m e th o d t h a t achieves a go al sim ilar to a go al e ith e r m u ta te d o r a d d e d in a n ex istin g o r re c e n tly c re a te d m eth o d . 24 S M M ,C ,N G G e n e ra liz e th e m e th o d t h a t achieves a g o a l sim ila r to a go al e ith e r m u ta te d o r a d d e d in a n e x is tin g o r re c e n tly c re a te d m e th o d in o rd e r to en ab le th e m e th o d to a c h ie v e b o th g o a ls. 25 S G M ,C ,N C C r e a te a m e th o d m o re g e n e ra l th a n th e m e th o d th a t achieves a go al s im ila r to a m o re g e n e ra l g o a l e ith e r m odified o r a d d e d in a n e x istin g o r recen tly c re a te d m e th o d . 26 S G M ,C ,N D C r e a te a m e th o d s im ila r to th e m e th o d t h a t achieves a g oal sim ilar to a m o re g e n e ra l g o a l e ith e r m o d ified o r a d d ed in a n e x is tin g o r recen tly c re a te d m e th o d in o r d e r to p ro v id e th e m e th o d s c a p a b le o f a ch iev in g a d eco m p o sitio n o f th e m o re g o a l. 27 s A M ,C ,N C C r e a te a m e th o d sim ila r, b u t in clu d in g a n e x tr a p a ra m e te r, to th e m e th o d th a t ach ie v e s a g o a l s im ila r to a g o al th a t in clu d es a n e x tr a p a ra m e te r a n d t h a t w as e ith e r m o d ified o r a d d e d in a n e x istin g o r re c e n tly c re a te d m eth o d . 28 - - - C c r e a te a m e th o d fro m s c ra tc h in o rd e r to a ch iev e a go al F o o tn o te s (1) A d d itio n a l C o n d itio n : A n alo g o u s O rig in a l m e th o d sh o u ld n o t b e in u se in th e c u rre n t s ta te . T h is c o n d itio n also im p lies t h a t th e goal o rig in a lly a ch iev e d b y th e b a se m e th o d it is n o t p o s te d a n y lo n g er a n d hence th e o rig in al m e th o d c a n b e ch an g ed safely. C a n o n ica l O perators We identified a few canonical operators th a t could be used to implement most of the steps in our KA Scripts library. For example, the operator to Generalize a method was used by a KA Script th at, in order to achieve a new goal, generalizes the m ethod that achieves a goal similar to the one ju st added (KA Script 19), and was also used by a KA Script th at, in order to achieve a goal that was generalized, generalizes the m ethod th a t used to achieve th at goal before being modified (KA Script 4). Most of these operators cannot be executed autom atically w ithout the user participation be cause they require the introduction of dom ain knowledge not represented in the KB and that cannot be inferred. For example, the generalize method operator cannot autom atically generalize a m ethod th a t computes the duration o f a trip for a ship into a m ethod th at computes the duration of a trip for any kind of vehicle (i.e., ships and aircraft) because this modification would depend on how trips for ships are different from the trips for other kind of vehicles and might not be possible 96 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. to inferred it from the knowledge already contained in the KB. However, these operators are able to guide a user through all the steps th at have to be followed. Operators • (M utate-method M G): M utates m ethod M in order to achieve goal G (used by KA Scripts 1 and 10). • (Create-mutated-method M G): Creates a m ethod sim ilar to method M in order to achieve goal G (used by KA Scripts 2, 11, 18, and 23, and by the operator Create-methods- for-decomposition described below). • (Generalize-method M G): Generalizes m ethod M in order to cover goal G (used by KA Scripts 3, 4, 12, 13, 19, and 24). • (Create-generalized-method M G ) : Creates a m ethod more general than m ethod M in order to cover goal G (used by KA Scripts 5, 14, 20, and 25). • (Add-Parameter-to-method M G): Add a param eter to method M in order to achieve goal G (used by KA Scripts 7 and 16). • (Create-method-with-additional-parameter M G ): Creates a method sim ilar to m ethod M with an additional param eter in order to achieve goal G (used by KA Scripts 8, 17, 22, and 27). • (Create-method-from-scratch G): Creates a m ethod from scratch in order to achieve goal G (used by KA Script 28 and by operator Create-methods-for-decomposition described below). • (Create-methods-for-decomposition {G l, G2, ..,Gn}): Create methods to achieve the unm atched goals of the goal decomposition {G l, G2,..,Gn} based on the m ethods th at achieve the m atched goals of th at decomposition (used by KA Scripts 6, 9, 15, 21, and 26). 97 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Operator Steps The above Operators were implemented with the following procedural steps (described in the following section). • (Mutate-method M G) 1. (Apply-global-substitutions-to-method M G) 2. (Check-and-revise-method M) • (Create-mutated-method M G) 1. (Create-analogous-to-method M-PRIME M G) 2. (Check-and-revise-method M-PRIME) • (Generalize-method M G) 1. (Apply-global-substitutions-to-method M G) 2. (Check-and-revise-method M) • (Create-generalized-method M G ) 1. (Create-analogous-to-method M-PRIME M G) 2. (Check-and-revise-method M-PRIME) • (Add-Parameter-to-method M G) 1. (Add-parameter-to-capability M G) 2. (Check-and-revise-method M) • (Create-method-with-additional-parameter M G) 1. Create the m ethod M-PRIME by duplicating the m ethod M 2. (Add-parameter-to-capability M-PRIME G) 98 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 3. (Check-and-revise-method M -PRIM E) • (Create-method-from-scratch G) 1. Autom atically generate a C apability Description C th at matches G 2. Ask the user for a Result Type R 3. Ask the user for a method Body B 4. Generate m ethod M using C, R, and B 5. (Check-and-revise-method M) • (Create-methods-for-decomposition. ({G l, G2, ..,Gn}) 1. For each unm atched goal G in {G l, G2, Gn} (a) MS = {M l, M2, Mm} the m ethods th at m atch the m atched goals in {G l, G2, Gn} (b) ask the user to choose a m ethod from MS to be used as a basis for an analogy or to reject all of them (c) if the user chooses m ethod M then (Create-m utated-m ethod M G) (d) else (Create-method-from-scratch G) Step Descriptions The above steps were implemented by the following procedures. • (Apply-global-substitutions-to-metliod M G) 1. Infer the m apping P of terms th at would transform the capability of M into a capability th at would m atch G 2. Ask the user to enter an additional m apping Q of term s to apply to M if desired 99 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 3. Substitute the term s in m ethod M accordingly to the mappings P and Q • (Create-analogous-to-method M-PRIME M G) 1. Infer the m apping P of terms th at would transform the capability of M into a capability th at would m atch G 2. Ask the user to enter an additional m apping Q of term s to apply to M 3. Create m ethod M2-PRIME by duplicating m ethod M and applying the term substitu tions given in P and Q • (Add-parameter-to-capability M G ) 1. Infer the extra param eter P to be added to the capability of M to make it m atch G 2. Add P to M • (Check-and-revise-method M) 1. Ask the user to check the correctness of the procedure of the body of M 2. Ask the user to check that the body of M corresponds (i.e., will achieve) the capability of M 3. Ask the user to check whet the body of M will produce a result th at agrees with the result type of M 4. Ask the user to edit the m ethod M if it needs to be revised 1 0 0 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Appendix B Experiment Materials used in Study I: Controlled Experiments — Round II B .l Problem-Solving Knowledge fragments This is a set of knowledge fragments about the dom ain. This is not a description of the EXPECT m ethods contained in the KB but of the knowledge th a t is represented in those methods. Includes knowledge th at is not contained in the initial KB but th at it is going to be needed for the scenarios. C O A E valu ation A Course-Of-Action (COA) evaluation is a summ arization of different aspects of the COA which will serve to analyze the COA from different perspectives. 1. To compute the evaluation of a COA you have to compute all aspects for all evaluation perspectives of that COA. 2. To compute the Round-Trip-Time Aspect of a COA you have to sum the Round-trip-time of each COA-movement 3. To com pute the Seaport Personnel Aspect of a COA you have to estim ate the unloading- personnel and the seaport-support-personnel needed for the COA and add those results. 101 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. F in d in g L ocation s an d P o r ts for C O A s 4. To find the seaports available for a COA you have to find the seaports of the locations participating in the COA. 5. To find the airports available for a COA you have to find the airports of the locations participating in the COA. 6. To find the locations participating in a COA you have to find all the locations where COA- operations will be held. 7. To find the seaports available for a COA belonging to allied-countries you have to find the seaports of the locations participating in the COA that belong to allied-countries. 8. To find the airports available for a COA belonging to allied-countries you have to find the airports of the locations participating in the COA th at belong to allied-countries. 9. To find the locations participating in a COA belonging to allied-countries you have to find all the locations where coa-operations will be held and return only those locations th a t belong to an allied-country. 10. To find the locations participating in a COA belonging to non-allied-countries you have to find all the locations where COA-operations will be held and return only those locations that belong to a non-allied-country. 11. To determine if a location belongs to an allied-country you have to check if the country of the location is an instance of allied-country. 12. To determine if a location belongs to a non-allied-country you have to check if the country of the location is an instance of non-allied-country. 102 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. R o u n d Trip T im e o f C O A M o v em en ts 13. To compute the Round-trip-tim e of a COA-movement you have to com pute the Round-trip- tim e between the origin and the destination of the movement of each one of the ships available for the movement and pick up the shortest. 14. To find the origin of a movement you have to find the origin of the route of the movement. 15. To find the destination of a movement you have to find the destination of the route of the movement. 16. To compute the Round-trip-tim e of a ship between two locations you have to divide the double of the m aritim e distance between the two locations by the speed of the ship. 17. To compute the Round-trip-tim e of an aircraft between two locations you have to find the aerial distance between the two locations. As a simplification assum e th at if such distance is over 10,000 miles then the round-trip-tim e is 48 hours, otherwise it is 24 hours. 18. To find the m aritim e distance between two locations you have to look up the SEA-ROUTE- DISTANCES table with the pair of locations as a key. 19. To find the aerial distance between two locations you have to look up the AIR-ROUTE- DISTANCES table with the pair of locations as a key. E stim a tio n o f p erso n n el fo r a C O A 20. To estimate the unloading-personnel needed for a COA you have to take the 10% of the total troops involved in the COA. 21. To estimate the seaport-support-personnel needed for a COA you have to m ultiply = = > 5% < = = of the total troops involved in the COA by the num ber o f seaports available for the COA. 103 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 22. To estimate the seaport-support-personnel needed for a COA in allied-countries you have to multiply = = > 2% < = = of the total troops involved in the COA by the number of seaports available for the COA belonging to allied countries. 23. To estimate the seaport-support-personnel needed for a COA in non-allied-countries you have to multiply = = > 7% < = = of the total troops involved in the COA by the num ber of seaports available for the COA belonging to non-allied countries. 24. To compute the to tal troops involved in a COA you have to sum the pax-count of all force modules participating in all COA-operation. B.2 Domain Concepts Diagrams that depict dom ain concepts, the relations am ong those concepts, and their instances (See Figures B .l, B .l, and B .l). Section B.6 lists their codification in LOOM. B.3 Task specification for RTT Scenario Task Modify the com putation of the Round-trip-time of a COA-movement to consider also the aircraft available for the movement (in addition to the ships). U p d a ted P r o b lem S o lv in g K n ow led ge fragm en t 13. To compute the R ound-trip-tim e of a COA-movement you have to compute the time between the origin and the destination of the movement of each one of available for the movement and pick up the largest. 104 Round-trip- the vehicles Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. r-vehicles COA LIFT AVAILABLE VEHCLE AIRCRAFT COA MOVEMENTS SHIP MEDIA ROUTE ROUTE LOCATION SEA ROUTE ( 1 +) AIR ROUTE Figure B .l: Study I: Domain concepts and relations — P art I T est C ase Top Level Problem: (compute (obj (spec-of evaluation)) (for coa-1)) Result after entire modification Round-trip-time: 424 hours Seaport Support Personnel: 625 persons 105 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. COUNT VALUE FORCE MODULE COA COA o p e r a t io n , 'ERSONE UNIT VALUE LOCATION, GENERAL | t~&e1 PORT j ( I +' COUNTRY NON ALLIED .COUNTRY, SEAPORT ALLIED COUNTRY, AIRPORT Figure B.2: Study I: Domain concepts and relations — Part II B.4 Task Specification for PAE Scenario T ask Refine the com putation of the Seaport Personal Aspect Evaluation of a COA by dis crim inating between allied and non-allied countries in the com putation of the seaport- support-personnel. U p d a te d P r o b le m S o lv in g K n o w led g e fragm en ts 3. To com pute the Seaport Personnel Aspect of a COA you have to estim ate the following cate gories of personnel needed for the COA and add their results: unloading-personnel, seaport- support-personnel in allied-countries, and seaport-support-personnel in non-allied-countries. 106 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. (COA) evaluation perspecti /es logistic perspective .evaluation/ location perspective (.evaluation/ performance perspective • evaluation/ ion :cts lances logistics /aspects, ;pects RTT aspect evaluation. seaport aspect evaluation. seaport ' perspective .evaluation/ Figure B.3: Study I: Domain concepts and relations — P art III T est Case: Top Level Problem (compute (obj (spec-of evaluation)) (for coa-1)) Result after entire modification R o u n d -trip -tim e : 400 h o u rs Seaport Support Personnel: 525 persons Note: M ethod M12 is provided for your convenience, but you do not have to use it. 107 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. B.5 EXPECT Methods in Initial Knowledge Base D o m a in M eth o d s (compute (obj (spec-of evaluation)) (for (inst-of coa) ))) D o m a in M eth o d s ((name HI) (capability (calculate (obj (?t is (spec-of round-trip-time))) (of (?s is (inst-of ship))) (from (?o is (inst-of location))) (to (?d is (inst-of location))))) (result-type (inst-of time-value)) (method (value-divide (obj (value-multiply (obj (find (obj (spec-of distance)) (from To) (to ?d) (by (spec-of sea-route)))) (by 2-dimensionless))) (by (r-speed ?s))))) ((name H2) (capability (compute (obj (?t is (spec-of round-trip-time))) (of (?ra is (inst-of coa-movement) ) ) ) ) (result-type (inst-of time-value)) (method (value-min (obj (calculate (obj (spec-of round-trip-time)) (of (r-ships (r-mvt-lift-available ?ra))) (from (r-origin (r-mvt-route ?m))) (to (r-destination (r-mvt-route ?m)))))))) ((name M3) (capability (compute (obj (?t is (spec-of round-trip-time-aspect-evaluation))) (for (?c is (inst-of coa))))) (result-type (inst-of time-value)) (method (value-sum (obj (compute (obj (spec-of round-trip-time)) (of (r-coa-movements ?c))))))) ((name H4) (capability (compute (obj (?x is (spec-of seaport-personnel-aspect-evaluation))) (for (?c is (inst-of coa))))) (result-type (inst-of Cpersonnel-unit]-value) ) (method (value-sum (obj (append (estimate (obj (spec—of unloading—personnel)) (needed-for ?c)) (estimate (obj (spec—of seaport-support-personnel)) (needed-for ?c))))))) ((name H5) (capability (compute (obj (?t is (spec-of total-troops))) (involved-in (?c is (inst-of coa))))) 108 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. (result-type (inst-of [personnel-unit]-value)) (method (value-sum (obj (r-pax-count (r-aho (r-coa-operations ?c))))))) ((name H6) (capability (estimate (obj (?x is (spec-of seaport-support-personnel))) (needed-for (?c is (inst-of coa))))) (result-type (inst-of Cpersonnel-unit]-value)) (method (value-multiply (obj (take (obj 5-%) (of (compute (obj (spec-of total-troops)) (involved-in ?c))))) (by (count-items (obj (find (obj (set-of (spec-of seaport))) (available-in ?c)))))))) ((name H7) (capability (estimate (obj (?x is (spec-of unloading—personnel))) (needed-for (?c is (inst-of coa))))) (result-type (inst-of [personnel-unit]-value)) (method (take (obj 10-X) (of (compute (obj (spec-of total-troops)) (involved-in 7c)))))) ((name H8) (capability (find (obj (?d is (spec-of distance))) (from (?locl is (inst-of location))) (to (?loc2 is (inst-of location))) (by (?r is (spec-of sea-route))))) (result-type (inst-of distance-value)) (method (look-for (obj (append ?locl 71oc2)) (in sea-route-distances)))) ((name H9) (capability (find (obj (?p is (set-of (spec-of general-port)))) (available-in (?c is (inst-of coa))))) (result-type (set-of (inst-of general-port))) (method (find (obj ?p) (of (find (obj (set-of (spec-of location))) (of ?c)))))) ((name M10) (capability (find (obj (71 is (set-of (spec-of location)))) (of (?c is (inst-of coa))))) (result-type (set-of (inst-of location) ) ) (method (r-ohere (r—coa—operations ?c)))) ((name Hll) (capability (find (obj (?s is (set-of (spec-of seaport)))) (of (71 is (inst-of location))))) (result-type (set-of (inst-of seaport) ) ) (method (r-seaports 71))) ;; Seeded for scenario PAE ((name H12) (capability (determine-whether (obj (71 is (inst-of location))) (belongs-to (?a is (spec-of country))))) (result-type (inst-of boolean)) (method (is-it-a (obj (r-country 71)) Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. (of ?a)>>) S y ste m M eth o d s T he following dom ain-independent low-level methods implement basic primitive procedures. Their body is not written in Expect b ut in Lisp and they are not available for modification. (is-it-a (obj (?i is (inst-of top-domain-concept))) (of (?c is (spec-of top-domain-concept)))) (inst-of boolean) (is-it-a (obj (?cl is (spec-of top-domain-concept))) Cof (?c2 is (spec-of top-domain-concept)))) (inst-of boolean) (is-it-a (obj (?cl is (set-of (spec-of top-domain-concept)))) (of (?c2 is (set-of (spec-of top-domain-concept))))) (inst—of boolean) (look-for (obj (?k is (set-of (inst-of domain-concept) ) ) ) (in (?t is (inst-of table)))) (inst—of top-domain-concept) (value-sum (obj (?1 is (set-of (inst-of value))))) (inst-of value) (value-multiply (obj (?vl is (inst-of value))) (by (?v2 is (inst-of value)))) (inst—of value) (value-subtract (obj (?vl is (inst-of value))) (from (?v2 is (inst-of value)))) (inst-of value) (value-divide (obj (?vl is (inst-of value))) (by (?v2 is (inst-of value)))) (inst-of value) (value-max (obj (?1 is (set-of (inst-of value))))) 110 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. (inst-of value) ; ; * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * (value-min (obj (?1 is (set-of (inst-of value))))) (inst-of value) (value-equal (obj (?vl is (inst-of value))) (to (?v2 is (inst-of value)))) (inst-of boolean) (value-less-equal (obj (?vl is (inst-of value))) (than (?v2 is (inst-of value)))) (inst-of boolean) (value-greater-equal (obj (?vl is (inst-of value))) (than (?v2 is (inst-of value)))) (inst-of boolean) (value-decr-1 (obj (?v is (inst-of value)))) (inst-of value) (value-double (obj (?v is (inst-of value)))) (inst-of value) (take (obj (?p is (inst-of percent-value))) (of (?v is (inst-of value)))) (inst-of value) (find (obj (?t is (spec-of unit))) (of (?v is (inst-of count-value)))) (spec-of domain-concept) (count-items (obj (?1 is (set-of (inst-of top-domain-concept))))) (inst-of dimensionless-value) (is-equal (obj (?nl is (inst-of number))) (to (?n2 is (inst-of number)))) (inst-of boolean) (is-less-or-equal (obj (?nl is (inst-of number))) (than (?n2 is (inst-of number)))) (inst-of boolean) 111 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. (is-greater-or-equal (obj (?nl is (inst-of number))) (than (?n2 is (inst-of number)))) (inst-of boolean) B.6 Domain Concepts in Initial Knowledge Base (defconcept coa :is-primitive (rand domain-concept (rsome r—coa-operations coa—operation) (rsome r-coa-movements coa-movement))) (defrelation r-coa-operations :domain coa :range coa-operation) (defconcept coa-operation :is-primitive (rand domain-concept (:exactly 1 r-begin) (:exactly 1 r-end) (:exactly 1 r-what) (:exactly 1 r-vho) (:exactly 1 r-uhere) )) (defrelation r-uhere :domain coa-operation :range location) (defrelation r-who :domain coa-operation :range force-module) (defconcept location :is-primitive (rand domain-concept (rail r-general-ports general-port) (rthe r-country country))) (defrelation r-general-ports rdomain location rrange general-port) (defrelation r-seaports ris (rand r-general-ports (rrange seaport))) (defrelation r-airports ris (rand r-general-ports (rrange airport))) (defrelation r-country rdomain location rrange country rattributes rsingle-valued) (defconcept general-port 112 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. :is-primitive domain-concept :partitions $general-port$) (defconcept seaport :is-primitive general-port :in-partition $general-port$) (defconcept airport :is-primitive general-port :in-partition $general-port$) (defconcept country :is-primitive domain-concept) (defconcept force-module :is-primitive (:and domain-concept (:exactly 1 r-pax-count))) (defrelation r-pax-count :domain force-module :range [personnel-unit]-value) (defrelation r—coa-movements :domain coa :range coa-movement) (defconcept coa-movement : is-primitive domain-concept :constraints (rand (rthe r-mvt-route route) (:exactly 1 r-mvt-lift-available) ) ) (defrelation r-mvt-lift-available :domain coa-movement :range lift-available) (defconcept lift-available ris rprimitive rconstraints (rat-least 1 r-vehicles)) (defrelation r-vehicles rdomain lift-available rrange vehicle) (defrelation r-ships ris (and r-vehicles (range ship))) (defrelation r-aircraft ris (and r-vehicles (range aircraft))) (defrelation r-mvt-route :domain coa-movement rrange route rattributes rsingle-valued) (defconcept route ris (rand domain-concept (rexactly 1 r-origin) (rexactly 1 r-destination))) 113 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. (defrelation r-origin :domain route :range location) (defrelation r-destination :domain route :range location) (defconcept major-equipment :is-primitive domain-concept) (defconcept vehicle :is-primitive major-equipment :constraints (:exactly 1 r-speed) :partitions $vehicle$) (defconcept aircraft :is-primitive vehicle :in-partition $vehicle$ ) (defconcept ship :is-primitive vehicle :in-partition $vehicle$ ) (defrelation r-speed :domain vehicle :range speed-value) ; ; * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * (defconcept evaluation :is-primitive domain-concept :partition $perspectives$) (defconcept locations-perspective-evaluation :is-primitive evaluation :partitions $locations-aspects$ ) (defconcept seaports-aspect-evaluation :is-primitive locations-perspective-evaluation : in-partition $locations-aspects$) (defconcept logistics-perspective-evaluation :is-primitive evaluation :partitions $logistics-aspects$ :in-partition $perspectives$) (defconcept seaport-personnel-aspect-evaluation :is-primitive logistics-perspective-evaluation :in-partition $logistics-aspects$) (defconcept performance-perspective-evaluation :is-primitive evaluation :partitions $performance-aspects$ : in-partition $perspectives$) (defconcept round-trip-time-aspect-evaluation :is-primitive performance-perspective-evaluation :in-partition $performance-aspects$) (defconcept personnel 114 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. :is-primitive domain-concept) (defconcept unloading—personnel :is-primitive personnel) (defconcept seaport-support-personnel :is-primitive personnel) (defconcept total-troops : is-primitive domain-concept) (defconcept round-trip-time :is-primitive domain-concept) (defconcept media—route :is-primitive domain-concept :partitions $media-route$) (defconcept sea-route :is-primitive media-route :in-partition $media-route$) (defconcept air-route : is-primitive media-route : in-partition $media-route$) (create-value-type ’((personnel) ())) (create-value ’(10 X)) (create-value ’(5 X)) (create-value ’(2)) (tell (distances-table sea-route-distances)) (tell (distances-table air-route-distances)) ;; Seeded by modifications (defconcept allied-country :is-primitive country) (defconcept non-allied-country :is-primitive country) (create-value ’ (2 X) ) (create-value ’ (7 X) ) (create-value ’(24 (hour) ) ) (create-value ’ (48 (hour))) (create-value ’(10000 (mile))) B.7 LOOM Factual Domain Instances in Initial Knowledge Base ;; The following is the Top-Level Problem to be solved: (compute (obj (spec-of evaluation)) (for coa-1))) 115 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Factual Knowledge (tell (about coa-1 coa (filled-by r-coa-operations action—1—1 action—1-2) (filled-by r-coa-movements movement—1-1 movement-1-2))) (tell (about action-1-1 coa-operation (r-who fml) (r-where la))) (tell (about action—1-2 coa-operation (r-who fra2) (r-where ba))) (tell (about la location (r-country usa) (r-airports lax) (r-seaports long-beach) (r-seaports san-pedro))) (tell (about ba location (r-country argentina) (r-seaports mar—del-plata) (r-a_irports eze))) (tell (seaport long-beach) (seaport san-pedro) (seaport mar—del-plata)) (tell (airport lax) (airport eze)) (tell (allied-country usa)) (tell (non-allied-country argentina) ) (let ((?c (create-value ’(1000 (personnel) ())))) (tell (about fml force-module (r-pax-count ?c)))) (let ((?c (create-value ’(1500 (personnel) ())))) (tell (about fm2 force-module (r-pax-count ?c)))) (tell (about movement-1-1 coa-movement (r-mvt-route route-1-1) (r-mvt-lift-available lift-1-1)) (about movement-1—2 coa-movement (r-mvt-route route—1-2) (r-mvt-lift-available lift-1-2))) (tell (about route-1-1 route (r-origin la) Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. (r-destination ba)) (about route-1-2 route (r-origin la) (r-destination ba))) (tell (about lift-1-1 lift-available (filled-by r-vehicles ssl ss2 lsl lal)) (about lift-1-2 lift-available (filled-by r-vehicles lsl))) (let ((?sspl (create-value ’(40 (mile) (hour)))) (?ssp2 (create-value ’(70 (mile) (hour)))) (?aspl (create-value ’(600 (mile) (hour)))) (?asp2 (create-value ’(750 (mile) (hour))))) (tell (about ssl ship (r-speed ?sspl)) (about ss2 ship (r-speed ?sspl)) (about lsl ship (r-speed ?ssp2)) (about sal aircraft (r-speed ?aspl)) (about lal aircraft (r-speed ?asp2)))) 117 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Appendix C Logs From Study I: Controlled Experiments Detailed log of the actions performed by the subjects during the controlled studies described in C hapter 5.2. This log was generated from the Tool’ s instrum entation and the notes taken by the examiners during the execution of the experiment. T h e L ogs a r e a n n o ta te d to m a rk th e follow ing: ♦♦E R R O R **: S u b je c t m a d e a m is ta k e w hile d o in g th e c o rre c t c h an g e •♦IN C O R R E C T **: S u b je c t m a d e a w ro n g c h an g e (b e ca u se co n fu sed o r m isu n d e rsto o d do m ain ) **IN C O M P L E T E **: S u b je c t fo rg o t o r d id n o t realize so m e ch an g es (d e te c te d w hen ex ecu ted sa m p le p ro b lem s) ♦ ♦D E B U G **: S u b je ct e x e c u te d a sa m p le p ro b le m , g o t w ro n g re su lts, a n a ly z e d tra c e to iden tify p ro b lem . C .l Scenario RTT - Round I Subject: Sl-I (Experienced with EXPECT) - Tool: EXPECT time Method Modif 00:00:00 M2 Substs: R-VEHICLES 00:05:48 Ml’ Creates Ml’ - subst: AIRCRAFT and AIR-ROUTE 00:11:03 M8’ Creates H8’ - subst: AIR-ROUTE and AIR-ROUTE-DISTAHCES Subject: S3-I (Experienced with EXPECT) - Tool: ETM KA time Script 00:00:00 00:06:08 X 00:09:00 X 118 Method Modif M2 Substs: R-VEHICLES Ml ’ Creates Ml ’ - subst: AIRCRAFT and AIR-ROUTE M8> Creates M8’ - subst: AIR-ROUTE and AIR-ROUTE-DISTAHCES Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Subject: S4-I (lot Experienced with EXPECT) - Tool: ETH KA time Script Hethod Modif 00:00:00 H2 Substs: R-VEHICLES 00:03:49 X HI’ Creates HI’ - subst: AIRCRAFT and AIR-ROUTE 00:06:05 X H8’ Creates H8’ - subst: AIR-ROUTE and AIR-ROUTE-DISTAHCES Subject: S2-I (Hot Experienced with EXPECT) - Tool; EXPECT time Hethod Hodif 00:00:00 H2 Edit - changes R-VEHICLES 00:12:16 HI’ Creates HI’ - subst: AIRCRAFT and AIR-ROUTE 00:15:43 H8’ Creates H8’ - subst: AIR-ROUTE and AIR-ROUTE-DISTAHCES C.2 Scenario RTT - Round II Subject: Sl-II (Experienced with EXPECT) - Tool: - ETH KA time Script Hethod Hodif 00:00:00 H2 Edit - Appends goal (CALCULATE (R-AIRCRAFT)) (1) 00:03:54 X HI’ Creates HI ’ - subst: AIRCRAFT and AIR-ROUTE 00:08:43 X HI’ Edits body 00:11:38 X H8 ’ Creates H8’ - subst: AIR-ROUTE and AIR-ROUTE-DISTAHCES Hotes: (1) Because syntax error in the first change, start counting from second change instead of first one. (2) Heeded to leave ETH twice to dismiss warning errors (unused var and missing filler). I did not deduct this time. Subject: S2-II (Experienced with EXPECT) - Tool: EXPECT time Hethod Hodif 00:00:00 H2 Subst: R-VEHICLES 00:04:00 HI’ Creates HI’ - subst: AIRCRAFT and AIR-ROUTE 00:10:55 HI’ Edits body 00:13:45 H8 ’ Creates H8’ - subst: AIR-ROUTE and AIR-ROUTE-1 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Subject: S3-II (Hot Experienced with EXPECT) - Tool: EXPECT (Subject not familiar with ENACS) time Hethod Hodif 00:00:00 Ml’ Creates Hl> - subst: AIRCRAFT and AIR-ROUTE (1) 00:07:58 M2 Edit - changes R-VEHICLES 00:12:01 H8’ Creates H8’ - subst: AIR-ROUTE and AIR-ROUTE-DISTAHCES (2) * * IHCOHPLETE **: Didn’t change body of Ml’ »* DEBUG **: Forgot change body of Ml’ 00:27:09 Ml’ Edits body ** ERROR »*: Syntax error: Misspelled one goal action 00:28:25 Ml’ Fix syntax error Hotes: (1) Had problems executing first change. Start counting from the time the first modification was correct and the KBS was cleaned. C.3 Scenario PAE - Round I Subject: Sl-I (Experienced with EXPECT) - Tool: ETM KA time Script Method Hodif 00:00:00 M4 add param ALLIED-COUHTRY to estimate and copy goal for HOI-ALLIED-COUHTRY (1) 00:03:33 X M6 Add param ALLIED-COUHTRY to capability 00:06:33 X M6 Edit body — change 2% and add parameter to goal FIHD 00:08:32 X M9 Add param COUHTRY to capab 00:09:52 X M9 Edit body - add parameter to goal FIHD L0CATI0H 00:14:45 X H6’ Create H6’ - subst: HOH-ALLIED-COUHTRY and 7% (2) Hotes: (1) Made mistake in first modif and fixes it. (2) Clicked in a wrong KAS option twice. Had to come back manually and reload factual domain (actions 65 - 123) - Time discounted Subject: S3-I (Experienced with EXPECT) - Tool: EXPECT time Method Hodif 00:00:00 M4 add param ALLIED-COUHTRY to estimate and copy goal for HOH-ALLIED-COUHTRY 00:03:37 H6 Edit - Add param ALLIED-COUHTRY to capab, changes 2%, add ALLIED-COUHTRY to goal FIHD *» ERROR »*: syntax error in capab param. 00:08:59 H6 Fix syntax capability param 00:09:44 M6’ Creates H6’ - subst: HOH-ALLIED-COUHTRY and 7% 00:12:40 M9 Add param ALLIED-COUHTRY to capab, change param of goal FIHD L0CATI0H to ALLIED-LOCATIOH (1) 00:13:40 Defined concept ALLIED-LOCATIOH ** ERROR **: Defined as primitive 120 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 00:14:40 Defined concept HOH-ALLIED-LOCATIOH (2) ** ERROR «*: Defined as primitive 00:17:11 H9’ create H9> - Subst: HOH-ALLIED-COUHTRY and HOH-ALLIED-LOCATIOH (3) * * DEBUGGIHG **: Wrong results because missing method FIHD-ALLIED-LOCATIOH (Ho errors in Agenda because matched more general method) 00:28:35 H10 Edit - Change capab param L0CATI0H to ALLIED-LOCATIOH, FILTERS body 00:33:16 H10’ Created H10’ - subst: HOH—ALLIED-LOCATIOH ** DEBUGGIHG **: Wrong results because nee concepts wrongly defined as primitives. 00:48:34 Fixes concepts definition. Hotes: (1) Unbalanced parens and lost body and result-type (action 63) . Had to type vhole body (time discounted) (2) LOOK error. Cannot be recovered and had to load domain and repeat changes. I didn’t have a record of the time it took the last two commands. Being very conservatieve (i.e., favoring EXPECT) I assign 1 minute each. (3) Discounted time for Loading domain and repeating changes. Subject: S4-I (Hot Experienced with EXPECT) - Tool: EXPECT time Hethod Hodif 00:00:00 H4 Edit - add param ALLIED-COUHTRY to estimate and copy goal for HOH-ALLIED-COUHTRY 00:04:16 H6’ Created H6’ (no subst) 00:11:37 H6’ Edit - Add capab param ALLIED-COUHTRY, change 2%, add param to goal FIHD ** ERROR **: wrong type in capab (forgot SET) 00:14:05 H6’ Edit - correct type capab param 00:24:42 H6’ Edit - replace goal FIHD-SEAPORT—in-COA for FIHD-SEAPORT-in-(FIHD-LOCATIOH-in-COA) , This change should have done in body of H9, but WAS HOT ABLE TO RELATE H9 with changed goal because H9 was more general. »» ERROR **: Syntax error in body 00:26:07 H6’ Edit - Corrects syntax error 00:28:26 H6” Create H6” - Subst: HOH-ALLIED-COUHTRY *» IHCOHPLETE **: Hissing change 77 . * * DEBUG ♦*: forgot change 75C 00:31:43 H6” Edit - Change 77. Subject: S2-I (Hot Experienced with EXPECT) - Tool: ETH KA time Script Hethod Hodif 00:00:00 H4 Edit — add param ALLIED-COUHTRY to estimate and goal for HOH-ALLIED-COUHTRY 00:01:30 X H6> Created H6 ’ (no subst) 00:02:30 X H6 ’ Add Capab Param ALLIED-COUHTRY 00:10:23 X H6 ’ Edit - change 2%, add parameter to goal FIHD 00:12:11 X H9 ’ Create H9’ (no subst) 00:12:45 X H9 ’ Add Capab param ALLIED-COUHTRY 00:14:00 X H9 ’ Add param to goal FIHD 121 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 00:17:42 X H6” Create H6” - Subst: HOD-ALLIED-COUHTRY and 7% 00:20:18 X H9’ ’ Created H9” - Subst: HOH-ALLIED-COUHTRY C.4 Scenario PAE - Round II Subject: Sl-II (Experienced with EXPECT) - Tool: EXPECT - didn’t know how to use FILTER - spent a lot of time analyzing how to modify it before starting modification time Hethod Hodif 00:00:00 H10 ’ Create H10’ - Add capab param ALLIED-COUHTRY, FILTER locations 00:01:02 H9> Creates H9’ 00:04:42 H9 ’ Add capab param ALLIED-COUHTRY, add param to FIHD goal 00:16:02 H6 ’ Created H6>(1) 00:18:43 H6 ’ Added capab param ALLIED-COUHTRY, changed 255, added param to goal FIHD 00:20:38 H4 Added ALLIED-COUHTRY to goal ESTIHATE, copy ESTIHATE and change HOH-ALLIED-COUHTRY 00:22:42 H6 Add capab param HOH-ALLIED_C0UHTRY, change 7)5, add param to goal FIHD 00:23:29 H9 Add capab param HOH-ALLIED-COUHTRY, add param to FIHD goal 00:24:17 H10 Removes unused method 00:25:40 H10” Creates H10 ’ ’ - Substs: HOH-ALLIED-COUHTRY Dotes: (1) Strange syntax error in the Agenda. Done a fake edit to H101 to make it disappear (time discounted) Subject: S2-II (Experienced with EXPECT) - Tool: ETH KA time Script Hethod Hodif 00:00:00 H9' Creates H9’ - Add capab param COUHTRY, FILTER result of FIHD PORTS with COUHTRY »* IHCORRECT »*: Hissplaced FILTER. Ports are not subtypes of COUHTRY 00:02:15 X H6 Edit - Add parameter ALLIED-COUHTRY to goal FIHD (forgot to change 250 00:10:09 H6 Edit - Copied whole body and changed goal FIHD to HOH-ALLIED-COUHTRY, changed 255 and 755 too (1) ** DEBUG **: wrong FILTER 00:21:28 H9’ Edit - Inserts R-COUHTRY to condition of FILTER «» IHCORRECT »*: there is no R-COUHTRY relation and wrong use of FILTER 00:25:29 H9’ Edit - changes condition of FILTER to DH PORT belongs to COUHTRY ** IHCORRECT *«: There sure no roles of SEAPORTS that leads to COUHTRY 00:32:21 H9’ Edit - Inserts R-L0CATI0H to FILTER condition ** IHCORRECT »*: there is no role R-L0CATI0H for ports and wrong use of FILTER 122 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 00:33:51 H9’ 00:36:11 X H12 00:39:07 X H12 00:45:37 X H9 ’ 00:46:15 X H12 Edit - Removes R-LOCATIOH in FILTER ** IHCORRECT *»: There are no roles of SEAPORTS that leads to COUHTRY Creates H12’ - Subst: LOCATIOH by GEHERAL-PORT Edit - insert relation R-LOCATIOH to PORT ** IHCORRECT **: there is no role R-LOCATIOH for ports Edit - Moves FILTER to result of goal FIHD L0CATI0H instead of FIHD PORT Removes Hethod Hotes: Cl) LISP aborted while adding new step in CLIH. I recovered and he did it again with EMACS. Time (01:18) discounted based on information from "uic" data log file. Subject: S3—II (Hot Experienced with EXPECT) — Tool: ETH KA time Script Hethod Hodif 00:00:00 H6 Edit - adds ALLIED-COUHTRY argument to FIHD goal, changes 2%, copied whole body and changed 7'/. and HOH-ALLIED-COUHTRY (1) 00:01:13 X M9’ Creates H95 - Ho substs 00:01:46 X K9’ Add param ALLIED-COUHTRY to capab 00:03:32 X H9’ Edit - Saves with no changes (forgot to add FILTER or param to goal FIHD) 00:08:27 X M9” Creates H9> ’ - Subst HOH-ALLIED-COUHTRY (forgot to add FILTER or param to goal FIHD) *» IHCOHPLETE **: Forgot to FILTER locations in H9’ and K9” ** DEBUG »» 00:21:29 H9 ” Edit - FILTER (FIHD LOCATIOH) with DW HOH-ALLIED-COUHTRY ** ERROR **: Syntax error: forgot parameter name OBJ and wrong use of condition of FILTER (instead of repeating set expr put its type) 00:22:21 H9’’ Edit - Fix one syntax error 00:26:02 H9’’ Edit - Fix other syntax error but made another one ** ERROR »*: wrong parameter type in DW. IHST-OF instead of SPEC-OF in HOH-ALLIED-COUHTRY 00:29:00 H9’’ Edit - Fixed syntax error 00:32:13 M9’ Edit - FILTER (FIHD LOCATIOH) with DW ALLIED-COUHTRY Hotes: (1) He performed several wrong successive modifications to this method before succeeding. I’m considering the starting time the last modification of this series. Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Appendix D Logs From Study II: Realistic Scenarios This appendix shows the logs of the modifications to the KBS performed during Study II described in Section 6.4. The modification scenario consisted of performing five extensions to a large KBS concerned with evaluating enemy workarounds for a targeted obstacle. The KBS containing 62 problem-solving methods and was integrated with an extract of the CYC knowledge base th at contained hundreds concepts. Each extension to the KBS required creating and modifying several problem-solving methods. The extensions to be implemented were: 1. Include a new kind of bridge (TMM) and the procedures needed to employ it in workaround plans. 2. Define a new kind of workaround step (cut-down-bank) and implement the procedures needed to handle them . 3. Define two new kinds of workaround steps (fill-crater and move-asset-around-crater) and implement the procedures needed to handle it. 4. Modify the procedure for estim ating the time needed to perform one workaround step (hasty- breach). 124 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 5. Define two new kinds of workaround steps (assemble-raft and move-asset-across-raft) and implement the procedures needed to handle them . Due to space lim itation we were not been to include the KBS implementation in this docum ent. Extension 1 The Notes shown to the right are described at the end of the table. Tool Modification Actions Cl) lotes * Add member "tmm-bridge" to "military-fixed-bridge" (2) partitions Define concept "tmm-bridge” C2) Got two unmatched-goal errors: 1 Cfind (obj Cspec-of time)) (for Cinst-of emplace-bridge)) (with (inst-of military-bridge))) 2 (estimate (obj (spec-of assemble-time)) (for (inst-of assemble-bridge)) (with (inst-of military-bridge))) Fixing unmatched-goal 1 with KAS: SG-CREATE—AIALOGOUS-TO-ORIGIIAL-HETHOD—TO—ACHIEVE— IEH-HEHBER-OF-GOAL-REFORHULATIOI> * Create analogous to ESTIHATE-TIHE-TO-EMPLACE-AVLB (4) Hame: emplace-tmm-bridge Substitute: (((IIST-OF AVLB-BRIDGE) (IIST-OF TMM-BRIDGE) ) ) (3) Mo further editing of the new method. Fixing unmatched-goal 2 with KAS: SG-CREATE-AMALOGOUS-TO-ORIGIKAL-HETHOD—TO—ACHIEVE- IEH-MEMBER-OF-GOAL-REFORHULATIOI> * Create analogous to ESTIMATE-TIHE-TO-ASSEHBLE-AVLB (4) fame: ASSEHBLE-THM-BRIDGE Substitute: (((IIST-OF AVLB-BRIDGE) (IIST-OF TMM-BRIDGE))) (3) lo further editing of the new method. lotes: (1) Tool: L - Loom, E - Expect, K - ETH. P - Build PS Tree (2) inserted patch with initial changes (prepared off-line) (3) Substitutions were proposed by ETH 125 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. (4) Base method was one of the few methods proposed by ETH Extension 2 The Notes shown to the right are described at the end of the table. Tool Hodification Actions lotes ( 1 ) * Add member "cut-down-bank" to "real-wg-step" (2) partitions Defined concept "cut-down-bank" and its relations desired-aidth, far-bank—of, near-bank-of (2) Got unmatched-goal error: (estimate (obj (spec-of time)) (for (inst-of workaround-step))) Fixing unmatched-goal with KAS: SG-CREATE-AIALOGOUS-TO-ORIGIIAL-HETHOD-TO-ACHIEVE- (3) IEW-HEHBER-OF-GO AL-REFORHULATI OH * Create analogous to ESTIHATE-TIHE-TO-DESLOPE-BAHK (S) Same: estimate-cut-dosn-bank Substitute: (((IIST-OF DESLOPE—BAIK) (IIST-OF CTJT-DOWI-BAIK) ) (4) (REGIOI—OF IEAR-BAIK-OF)) * Edit: Delete argument (WITH (IEAR-BAIK-OF ?S)) to goal FIHD Got unmatched-goal error: (find (obj (spec-of length)) (for (inst-of cut-down-bank))) Fixing unmatched-goal with KAS: SG-CREATE-METHOD-FROH-SCRATCH-TO-ACHIEVE-GQAL (6) Create analogous to GET-HEIGHT—TO—REDUCE (7,8) lame: get-height-to-cut-down-bank Substitute: (((IIST-OF DESLOPE-BAHK) (IIST-OF CUT-DQWH-BAHK))(4) Edit: Remove extra capability parameter and several other parts of the body including the addition of the goals: (find (obj (spec-of height) (for (inst-of CUT-DOWH-BAHK)) (find (obj (spec-of value)) (of avlb-bank-height-rq)) Got unmatched-goal error: (find (obj (spec-of hight)) Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. (for (inst-of cut-down-bank) ) ) K Fixing unmatched-goal with KAS: SG-CREATE-HETHOD-FROH-SCRATCH-TO-ACHIEVE-GOAL * Create analogous to GET-KEIGHT-TO-CUT-DOWH-BAHK (7) lame: get-height-for-cut-down-bank Substitute: (((spec-of length) (spec-of height)) (4) * Edit: several parts of the body including the addition of the goals: (find (obj (spec-of value)) (of avlb-bank-length)) (find (obj (spec-of nuraber-from-percent)) (from (inst-of number) )) Rotes: (1) Tool: L — Loom, E - Expect, K - ETH. P — Build PS Tree (2) inserted patch with initial changes (prepared off-line) (3) When the subject was doing this, this KAS didn’t come out because of a limitation in the isomorphism between PS Tress that I have corrected now. She had chosen "create a method from scratch, but because it showed how to reformulated the goal and suggested her to base the new method in the methods for the other subgoals, she could use similar guidance from ETH than I have now. (4) Substitutions sere proposed by ETH (5) Base method was one of the few methods proposed by ETH (6) The KA Scripts that follow up the deletion of a goal argument were not implemented and hence were not offered. (7) Did not suggest a specific base method. Subject choose from a complete list of methods. (8) The not implemented KA Script would have suggested the right base method. Extension 3 The Notes shown to the right are described at the end of the table. Tool Hodification Actions Rotes ( 1 ) L * Add member "fill-crater" and "move-asset-around-crater" to "real-wg-step" partitions (2) Defined concepts "move-asset-around-crater" and "fill-crater" and its relation "crater-of" (2) P Got unmatched-goal error: (estimate (obj (spec-of time)) (for (inst-of workaround-step))) Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Fixing unmatched-goal oith KAS: SG-CREATE-AHALOGOUS-TO-ORIGIHAL-HETHOD-TO-ACHIEVE- (3) HEW-HEHBER-OF-GOAL-REFORHULATIOH * Create analogous to estimate-time-to-move—asset-around—tunnel (5) Same: time-to-raove-asset-around-crater Substitute: (((IHST-OF move-asset-around-tunnel) (IHST-OF move-asset-around-crater)) (4) Cl.5 1/12)) Ho further editing * Create analogous to estimate-time-to-narrou-gap C5) Hame: time-to-fill-crater Substitute: ( ( (IHST-OF narroB-gap-srith-bulldozer) CIIST-QF fill-crater))) C4) * Edit: Write new body, including goals: (FIHD (OBJ (SPEC-OF DIRT-VOLUHE)) (FOR (CRATER-OF ?S)))) (FIID (OBJ (SPEC-OF EARTHHOVIHG—RATE)) (FOR RUBBLE-IHSTAHCE) (WITH (DOZER-OF ?S)>) Got unmatched-goal error: (find (obj (spec-of dirt-volume)) (for (inst-of crater))) Fixing unmatched-goal oith KAS: SG-CREATE—HETHOD-FROH-SCRATCH—TO-ACHIEVE-GOAL * Create analogous to get-dirt-volume (7) lame: dirt-volume-of-crater Substitute: (((inst-of earth-stuff) (inst-of crater))) (4) » Edit: several parts of the body including the addition of the goals: (FIHD (OBJ (SPEC-OF VALUE)) (OF CODE-FACTOR)) (FIHD (OBJ (SPEC-OF AREA-TO-CLEAR)) (FOR ?C)) Got unmatched-goal error: (FIHD (OBJ (SPEC-OF AREA-TO-CLEAR)) (FOR (inst-of crater))) Fixing unmatched-goal oith KAS: SG-CREATE—HETHOD-FROH-SCRATCH—TO-ACHIEVE-GOAL * Create from SCRATCH (8) fame: find-crater-area body includes goals: (FIHD (OBJ (SPEC-OF VALUE)) (OF PI)) (FIHD (OBJ (SPEC-OF VALUE)) (from triangle-factor)) Got unmatched-goal error: (FIHD (OBJ (SPEC-OF VALUE)) (from triangle-factor)) 128 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. E Fix user mistake: * Edit method: find-crater-area (8) P Got unmatched-goal error: (FIHD (OBJ (SPEC-OF VALUE)) (from triangle-f actor)) E Fix user mistake: * Edit method: find-crater-area ********** Ho more Errors ********** Hotes: (1) Tool: L - Loom, E — Expect, K — ETH. P - Build PS Tree (2) inserted patch with initial changes (prepared off-line) (3) When the subject was doing this, this KAS didn’t come out because of a limitation in the isomorphism between PS Tress that I have corrected now. She had chosen “create a method from scratch, but because it showed how to reformulated the goal and suggested her to base the new method in the methods for the other subgoals, she could use similar guidance from ETH than I have now. (4) Substitutions were proposed by ETH (5) Base method was one of the few methods proposed by ETH (7) Did not suggest a specific base method. Subject choose from a complete list of methods. (8) **• Hade mistake: Parameter name FBOH instead of OF in goal (FIHD (OBJ (SPEC-OF VALUE)) (from triangle-factor)) Extension 4 The Notes shown to the right are described at the end of the table. Tool Hodification Actions E * Edit method: estimate-time-for-hasty-breach-with-region (2) Inserts a big nested IF expression (an exception?) that includes the goal: (FIHD (OBJ (SPEC-OF AREA-TO-CLEAR)) (FOR (region-of ?s))) P Got unmatched goal error: (FIHD (OBJ (SPEC-OF AREA-TO-CLEAR)) (FOR (inst-of geographical-region))) E * Edit method: estimate-time-for—hasty-breach-with-region (3) In added goal changes (region-of ?s) by (crater-of ?s) P Got IHVALID-ARGUHEHT errors: Hotes Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. * * * * * * * * Abandon modification due to time limitations ******1 (this error was not detected when done same scenario before) Votes: (1) Tool: L - Loom, E - Expect, K - ETH. P - Build PS Tree (2) * * * * Hade mistake: REGIOH-OF instead of CRATER-OF (3) * * * Hade mistake. Extension 5 The Notes shown to the right are described a t the end of the table. Tool Hodification Actions Hotes ( 1 ) * Add member "assemble-raft" and "move-asset-across-raft" to "real-wg-step" partitions (2) Defined concept "assemble-raft", "move-asset-across-raft" and their relations "assembly-num-bays" and "raft-of" (2) Got unmatched-goal error: (estimate (obj (spec-of time)) (for (inst-of workaround-step))) Fixing unmatched-goal with KAS: SG-CREATE-AHALOGOUS-TO-ORIGIHAL-HETHOD-TO-ACHIEVE- (3) HEW—HEHBER-OF-GOAL-REFORHULATIOH * Create analogous to estimate-time-to-move-asset-over-bridge (5) Fame: estimate-move-asset-across-raft Substitute: (((IHST-OF move-asset-across-bride) (IHST-OF move-asset-across-raft))) (4) * Edit: change body. Inserts goal: (FIHD (OBJ (SPEC-OF TIHE)) (FOR ?S) (WITH (WIDTH-OF-OBJECT (RIVER-OF (GAP-OF ?S))))) * Create analogous to estimate-time-to-emplace-bridge (5) Hame: estimate-assemble-raft Substitute: (((IHST-OF emplace-bridge) (IHST-OF assemble-raft))) (4) * Edit: change body. Inserts goal: Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. (ESTIHATE (OBJ (SPEC-OF ASSEHBLE-TIME)) (FOR ?S) (WITH (RAFT-OF ?S))) P Got unmatched-goal error: (find (obj (spec-of time)) (for (inst-of move-asset-across-raft)) (with (inst-of number))) K Fixing unmatched-goal with KAS: SG-CREATE-HETHOD-FROH-SCRATCH—TO-ACHIEVE-GOAL * Create analogous to get-obtain-control-time-with-unit-level (7) lame: find-time-to-move-asset-across-raft Substitute: (((IHST-OF military-unit-level) (IHST-OF move-asset-across-raft) ) (4) * Edit: Add extra capability parameter (oith (inst-of number) )and edit body. Ho new goals are inserted. P Got unmatched-goal error: (estimate (obj (spec-of assemble-time)) (for (inst-of assemble-raft)) (with (inst-of ribbon-bridge-or-m4t6) ) ) (8) K Fixing unmatched-goal with KAS: (9) SG-CREATE-HETHOD-FROH-SCRATCH—TO-ACHIEVE-GOAL * Create from SCRATCH Edit: changed capability parameter to (with (inst-of m4t6)) Ho new goals sure inserted. P Got unmatched-goal error: (estimate (obj (spec-of assemble-time)) (for (inst-of assemble-raft) ) (with (inst-of ribbon-bridge-or-m4t6))) (10) K Fixing unmatched-goal with KAS: SG-CREATE-HETHOD-FROH-SCRATCH-TO-ACKIEVE-GOAL (10) * Create analogous to estimate-m4t6-raft Same: estimate-for-ribbon Substitute: (((IHST-OF m4tS) (inst-of ribbon-bridge)) ((ribbon-bridge-or-m4t6) (ribbon-bridge)))) « Edit body. Ho new goals were inserted. *************** Abandoned due to a Tool malfunction »«**•**«***» Hotes: (1) Tool: L - Loom, E - Expect, K - ETH. P - Build PS Tree (2) inserted patch with initial changes (prepared off-line) (3) When the subject was doing this, this KAS didn’t come out because of a limitation in the isomorphism between PS Tress that I have corrected now. She had chosen "create a method from scratch, but because it showed how to reformulated the goal and suggested her Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. to base the new method in the methods for the other subgoals, she could use similar guidance from ETH than I have nos. (4) Substitutions sere proposed by ETH (5) Base method sas one of the feu methods proposed by ETH (6) The KA Scripts that folios up the deletion of a goal argument sere not implemented and hence sere not offered. (T) Did not suggest a specific base method. Subject choose from a complete list of methods. (8) ****** System error?. EXPECT is not working properly. Strange concept in goal (ribbon-bridge-or-m4t6) . Should have used FLOATIBG-BRIDGE, shich is equivalent. EXPECT is not able to reformulate this goal!!!! (9) **** If EXPECT would have worked properly, this KA Script sould have created methods for each member of the goal reformulation, based on the methods already used. (10) Should not have happened this if EXPECT sould have worked fine. Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.

Asset Metadata

Creator Tallis, Marcelo (author)

Core Title A script-based approach to modifying knowledge -based systems

Contributor Digitized by ProQuest (provenance)

School Graduate School

Degree Doctor of Philosophy

Degree Program Computer Science

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag Computer Science,OAI-PMH Harvest

Language English

Advisor Gil, Yolanda (committee chair), O'Leary, Daniel (committee member), Rosenbloom, Paul (committee member), Swartout, William (committee member)

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c16-74244

Unique identifier UC11337789

Identifier 3018037.pdf (filename),usctheses-c16-74244 (legacy record id)

Legacy Identifier 3018037.pdf

Dmrecord 74244

Document Type Dissertation

Rights Tallis, Marcelo

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the au...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus, Los Angeles, California 90089, USA

Linked assets

University of Southern California Dissertations and Theses