Annotation databases for distributed documents

PDF

Download

Open document

Flip pages

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content INFORMATION TO USERS This manuscript has been reproduced from the microfilm master. UM I films the text directly from the original or copy submitted. Thus, some thesis and dissertation copies are in typewriter face, while others may be from any type of computer printer. The quality of this reproduction is dependent upon the quality of the copy submitted. Broken or indistinct print, colored or poor quality illustrations and photographs, print bleedthrough, substandard margins, and improper alignment can adversely affect reproduction. In the unlikely event that the author did not send U M I a complete manuscript and there are missing pages, these will be noted. Also, if unauthorized copyright material had to be removed, a note will indicate the deletion. Oversize materials (e.g., maps, drawings, charts) are reproduced by sectioning the original, beginning at the upper left-hand comer and continuing from left to right in equal sections with small overlaps. ProQuest Information and Learning 300 North Zeeb Road, Ann Arbor, M l 48106-1346 USA 800-521-0600 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. NOTE TO USERS This reproduction is the best copy available. U M I" Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. ANNOTATION DATABASES FOR DISTRIBUTED DOCUMENTS by Ilia Ovsiannikov A Dissertation Presented to the FACULTY OF THE GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (COMPUTER SCIENCE) A ugust 2002 Copyright 2002 Ilia Ovsiannikov Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. UMI Number: 3073829 — ___ UMI UMI Microform 3073829 Copyright 2003 by ProQuest Information and Learning Company. All rights reserved. This microform edition is protected against unauthorized copying under Title 17, United States Code. ProQuest Information and Learning Company 300 North Zeeb Road P.O. Box 1346 Ann Arbor, Ml 48106-1346 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. UNIVERSITY OF SOUTHERN CALIFORNIA The Graduate School University Park LOS ANGELES, CALIFORNIA 90089-1695 This dissertation,, w ritte n b y I I ifij O v s i f i f l / h i i K O i r ________________ U nder th e d irectio n o f h. u, S. . D issertation C om m ittee, a n d approved b y a ll its m em bers, has been p resen ted to a n d accepted b y The G raduate School, in p a rtia l fu lfillm e n t o f requirem ents fo r th e degree o f DOCTOR O F PHILOSO PHY Dean o f Graduate Studies D ate D1SSER TA H O N CO M M ITTEE Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. A cknow ledgem ents This research was supported by USC Human Brain Project (NIMH grant MH 52194-03) and a grant from Fuji Xerox Palo Alto Laboratories (FXPAL). Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. C ontents Acknowledgements • • ii List of Tables .................................................................................................................... v List of F ig u re s .................................. . ....................... vi Abstract ............................................................................................................................. x 1 Introduction ....................... 1 2 Human Handwritten Annotation Theory ............................................................... 5 2.1 Introduction................................................................................................... 5 2.2 Handwritten annotation ap p earan ce............................................................... 8 2.2.1 S tru c tu re ................................................................................................ 8 2.2.2 A to m s....................................................................................................... 11 2.2.3 Annotations and n o tes........................................................................... 19 2.2.4 Links ........................................................................................................... 22 2.2.5 Markup procedure..................................................................... 25 2.3 Annotation functions.............................................................................................. 26 2.3.1 Reading goals, behaviors and styles.......................................................... 27 2.3.2 Reading s u p p o rt.........................................................................................29 2.4 Implications to system d esig n ............................................................................... 37 3 Functional Analysis And Architectural Design .......................................................... 40 3.1 Introduction..............................................................................................................40 3.2 System review...........................................................................................................43 3.2.1 Annotations in word processors................................................................ 43 3.2.2 Web-oriented annotation software .......................................................... 44 3.2.3 Annotations on electronic books and documents ..................................46 3.2.4 Pen-enabled applications.............................................................................47 3.2.5 Other annotation system s..................................................................... 48 3.3 Task-based functional analysis............................................................................... 49 3.4 Backend architecture o v erv iew ............................................................................ 55 3.5 Mark-up anchoring..................................................................................................58 3.5.1 Definitions .............................................................................................. 59 3.5.2 A document change probabilistic m o d e l................................................. 61 3.5.3 Anchoring algorithm ...................................................................................63 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 3.5.4 Statistical anchor construction analysis ............................................. 65 3.5.5 Performance ........................................................................ 68 3.6 Prototype softw are............................................................................................ 69 3.6.1 Features and user in terface................................................................... 69 3.6.2 Implementation overview ...................................................................... 71 3.6.3 Evaluation .............................................................................................. 73 4 Front-End Design ....................................................................................................... 77 4.1 Introduction............................................................................................ 77 4.2 Mark-up layout engine....................................................................................... 81 4.2.1 Functional specification and element style review ................................ 82 4.2.2 Markup visualization constructs..... ........................................................ 86 4.2.3 Layout generation as optimization p ro b le m ........................................ 86 4.2.4 Style table construction......................................................................... 92 4.2.5 Inline markup placement ..........................................................................96 4.2.6 Margin markup placement....................................... 97 4.2.7 Link m a rk u p ...................................... ... ................................................101 4.2.8 Concluding processing ................................................ 108 4.2.9 Implementation evaluation and performance..........................................109 4.3 Input in te rfa c e ..................................................................................................... 112 5 Conclusions ........................................................................................................ 115 References ........................................................................ 116 iv Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. List o f Tables 2.1 Common text atom selection techniques.......................................... 14 2.2 Annotation appearance versus annotation function...............................................38 4.1 Clump binding compatibility and resolution.......................................................104 v Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. List o f Figures 2.1 Annotated clump ........................................................................................ 9 2.2 Typical explicit character granularity selectors................................................. 12 (a) C ircling................................................................................................... . 12 (b) U nderscore...................................................................... 12 (c) H ighlight..................................................................................................... 12 (d) Box enclosure......................................................................... 12 (e) Brackets............... 12 (f) S trik e o u t................................................................ 12 (g) Emphasis using double underscore........................................................... 12 (h) Emphasis by means of c irc lin g ................................................................. 12 2.3 Typical explicit line granularity selectors.......................................................... 13 (a) Open range ................................................................... 13 (b) Right open range with em p h asis.............................................................. 13 (c) Closed range . ......................................................................................... 13 (d) Closed range without spanning line........................................................... 13 (e) Detailed range uses brackets to enhance g ran u larity ............................. 13 (f) Detailed range with no spanning line and non-directional end-points . 13 (g) C ross-out..................................................................................................... 13 (h) Cross-out combined with box en clo su re.................................................. 13 2.4 Typical explicit pictorial selectors......................... 18 (a) C ircling................................................................ 18 (b) Highlighting ............................................................................................... 18 2.5 Inscriptions implicitly create atoms.................................................................... 18 (a) Sentence or paragraph................................................................................ 18 (b) P ag e............................................................................................................... 18 (c) Document or section................................................................................... 18 (d) Pictorial........................................................................................................ 18 2.6 Link endpoints can create atoms......................................................................... 18 (a) Implicit text a to m ...................................................................................... 18 (b) Implicit pictorial ato m ................................................................................ 18 (c) Explicit p o sitio n ......................................................................................... 18 2.7 Typical explicit positional selectors.........................................................................18 (a) In-line............................................................................................................ 18 v i Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. (b) On the margin 18 2.8 More annotation examples................................................................................... (a) Annotation on the m argin............................................................ (b) Footer annotation with reference m ark............................................. (c) Inscription in a picture ........................................................................... 2.9 Semantic clarity of annotations......................................................................... (a) Conventional . ........................................................................................ (b) Idiom atic.................................................................................................... (c) Idiosyncratic................................................................................. ... (d) Idiomatic with emphasis ......................................................... 2.10 Link types............................................................................... .............................. (a) Link proper ............................................................ ....................... (b) Link by reference in lin e ........................................................................... (c) Implicit link in text via aligned o v e rla p .................. ............................ (d) Link by reference on the margin ......................................................... (e) Link chaining.............................................................................................. 2.11 Implicit link formation via aligned overlap........................................................ (a) Overlap can take precedence over proximity.. (b) Explicitly bound annotation does not create implicit l i n k .. (c) Annotations in opposite side text areas create independent implicit links 2.12 Navigation aids...................................................................................................... (a) H ighlight....................................................................................................... (b) Number marking ...................................................................................... (c) Text m arking.............................................................................................. .. (d) Symbolic m a rk in g ...................................................................................... 2.13 Understanding and memory aids........................................................................ (a) Clarification .............................................................................................. (b) R elations............................................................. ....................................... .. (c) Un-abbreviation............................... ........................................................... (d) Re-formulation............................................................................................ (e) Breakdow n.................................................................................................. (f) Associated concept.................................................................................... (g) G rouping..................................................................................................... (h) Contraction....................................................................................... 2.14 Summaries............................................................................ ................................ (a) Footnote sum m ary...................................................................................... (b) Summary on the front p a g e ....................................................................... 2.15 Reflection aids....................................................................................................... (a) Comparison.................................................................................................. (b) Reflections recorded as n o te s .................................................................... 20 20 20 20 21 21 21 21 21 23 23 23 23 23 23 24 24 24 24 31 31 31 31 31 32 32 32 32 32 32 32 32 32 35 35 35 35 35 35 vii Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. (c) Problem s o lv in g ......................................................................................... 35 2.16 Communication aids............................................................................................ 36 (a) Library information markings.................................................................. 36 (b) R e v ie w ........................................................................ 36 (c) Proof m a rk ...................... 36 (d) S ta m p ........................................... 36 3.1 Text anchor s tru c tu re ...................................................................................... 60 3.2 Anchoring algorithm............................................................................................64 (a) Individual anchor construction................................................................ 64 (b) Atom anchoring record construction . ..................................................... 64 3.3 Minimal unique string......................................................................... 67 (a) Typical MUS length histogram . ............................ 67 (b) MUS histogram with an 11-word fragment re p e a te d ............................. 67 (c) Histogram of ( b ) ............................ 67 (d) MUS length can substantially increase after text replication.....................67 3.4 Annotator 2.0 user interface..................................................................... 70 (a) Annotation index with to o lb a r................................................................ 70 (b) Two inline text atoms with annotations......................................................70 4.1 In-line selectors................................................................................... 84 (a) H ighlight........................................ 84 (b) U nderscore.................................................................................................. 84 (c) U nder-arch......................................................................................................84 (d) O v e rarch ..................................................................................................... 84 (e) Brackets........................................................................................................ 84 (f) B o x .............................................................................................................. 84 (g) Thick underscore ...................................................................................... 84 (h) Double underscore...................................................................................... 84 (i) S trik e o u t.............................................................................................. 84 (j) Wavy underscore......................................................................................... 84 4.2 Margin selectors ................................................................................................ 85 (a) O p e n ........................................... 85 (b) Closed........................................................................................................... 85 (c) Closed with no spanning elem ent................... 85 (d) Open detailed ................................................................................... 85 (e) Closed detailed.................................................... 85 (f) Open with double lin e ....................................................................................85 (g) Closed with thick l i n e ................................................................................ 85 (h) Open with wavy lin e ................................................................................... 85 4.3 Atom markup layers............................................................................................. 85 4.4 The minimal markup principle............................................................................... 87 viii Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. (a) Too few markup creates am biguity.......................................................... 87 (b) Too much markup ensures clarity, but creates clutter and requires human e ffo rt............................................................................................. 87 (c) an optimal solution ................................................................................. 87 4.5 Markup layout algorithm block-diagram............................................................ 92 4.6 Arch selectors help handling overlapping atoms ............................... 95 (a) Over-arch . ............................................................................................... 95 (b) U nder-arch.................................................................................................. 95 4.7 Vertical alignment of annotation with respect to the atom . ...................... 99 4.8 Side-text a r e a s .......................................................................................................100 4.9 Types of clump binding...........................................................................................102 (a) Hard binding .....................................................................................102 (b) Soft binding.................................................................................................... 102 (c) Two overlapping soft-bound clumps can lead to ambiguous associations 102 (d) A hardbound element does not associate with a soft-bound..................102 4.10 Sample results calculated by the experimental implementation..........................110 (a) Low density ........................................................................................... 110 (b) Two-source medium density with two overlapping atom s.........................110 (c) Higher density with up to three overlapping atoms and collapsed an notations 110 4.11 Explicit mark object state diagram.......................................................................114 ix Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. A bstract The appearance of portable pen-enabled tablet PCs has prompted a renewed interest to the task of system design of human electronic document annotation applications. We approach this problem in three steps. Annotation system design requires a solid understanding of the bases of handwritten annotation of paper documents. We present results of an extensive case-based study of handwritten annotation appearance and functionality situated within a theoretical framework. Document annotation is an activity integrally assisting essential reading tasks. Based on a review of existing annotation systems and taxonomies of reading goals and styles we undertake a functional systems analysis. An architectural and algorithmic system back-end framework that provides the required functionality is presented thereafter. In conclusion, we focus on front-end design of human electronic document annota tion applications and describe algorithms for automated high quality layout of annotation markup on pages. The algorithms build around principles underlying human document annotation on paper to assure high visual appeal and familiar appearance that parallels paper markup. Two proof-of-concept software systems have been developed and evaluated. Annotator 2.0 implements system back-end supporting distributed annotations on on-line documents. The experimental layout engine realizes the front-end and generates high quality markup. x Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. C hapter 1 In tro d u ctio n The appearance of portable pen-enabled tablet PCs has prompted a renewed interest to the task of advanced system design for Human Electronic Annotation of Documents (HEAD). With few exceptions, experimental and commercial annotation systems developed so far have not enjoyed substantial success and widespread use. They could not support active reading (Adler and van Doren, 1972) by combining reading with annotation in one effortless process (O’Hara and Sellen, 1997). The lack of adequate hardware platforms was one of the key factors leading to this situation. Numerous high-quality inexpensive pen-enabled PCs recently became available from the largest hardware OEM manufactures such as Sony, Phillips, Fujitsu, Samsung, IBM, Hitachi, Honeywell and Wacom. In November 2000 Microsoft announced its new strategic concept of Tablet PC (Microsoft, 2001c). A typical Tablet PC device will combine a mobile lightweight design with full Windows capabilities, pen input and wireless network access. Tablet PC is expected to address the previously unmet needs of ergonomic on-screen reading, note taking and annotation. Advanced HEAD system design requires a solid understanding of the bases of handwrit ten annotation of paper documents. We present results of an extensive case-based study of handwritten annotation appearance and functionality. These results are situated within 1 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. a theoretical framework called Human Handwritten Annotation Theory (HHAT). Hand written annotations consist of structural elements of three types. The annotation process involves the use of markings to single out document fragments and associate with them second-party statements. The fragments, statements and associations are called clumps of atoms, annotations and links respectively. We analyze clump form, size, granularity, positioning, markup sub-elements, emphasis and visibility. Annotations are classified with respect to form, positioning, emphasis, visibility and semantic clarity. Links are examined with respect to their form, visibility and associated semantics. Handwritten annotations are inherently ambiguous since many components and relations are contained in the mark ings implicitly. For each component we analyze details of their both explicit and implicit formation. A description of annotated clump construction in its entirety concludes the investigation of markup appearance. The second part of HHAT is devoted to annotation functionality. We review reading goals, behaviors, styles and examine how annotations critically enable some and support others. Reading involves many constituent activities. In particular, we analyze and illus trate how annotations aid navigation, understanding, memory, reflection, communication and other processes. Document annotation is an activity integrally assisting essential reading tasks. The mi gration of annotations to electronic media introduces new capabilities for communication, processing, dynamic visualization and archiving. Based on a review of existing annotation systems presented herein and taxonomies of reading goals and styles we undertake a func tional systems analysis. The analysis shows how individual paper document reading tasks can be assisted and new use cases of collaboration and knowledge lifecycle support arise. Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Subsequently, we present an architectural framework for electronic document annota tion system back-end design. The framework takes into account characteristic differences between legacy and electronic media and advances principles of markup insensitivity to document modification, format and location. We describe an improved computationally efficient algorithm for markup anchoring in text documents subjected to change. The algorithm and a subset of postulated functional specifications are implemented in a pi lot annotation system. Annotator 2.0 is a two-tier application featuring a lightweight client based on a seamless plug-in to Internet Explorer. It incorporates features such as distributed annotated clumps, shared annotation database, annotation database search, annotation index and others. We evaluate the system and report our experience as users. Next, we focus on front-end design for human electronic document annotation appli cations and describe algorithms for automated high quality layout of annotation markup on pages. The algorithms build around principles underlying human document annota tion on paper to assure high visual appeal and familiar appearance that parallels paper markup. The capability for automated layout allows retaining annotations after document modification, simultaneous display of dense overlapping markup from several sources and serves as output facility for applications that generate annotations. Markup generation is formulated as an optimization problem that minimizes layout clutter, ambiguity and style inconsistencies. We construct appropriate numerical estimators and define style guidelines based on the findings of HHAT. The layout engine begins by constructing style-rich expres sive inline and margin markup that provides requested visibility, emphasis and associated semantics. Subsequently, mark linking is performed to guarantee clarity of perceived as sociations. To that end we utilize both explicit and implicit links based on proximity and 3 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. aligned overlap cues. To validate the design we present results from our experimental implementation. The task of handwritten annotation markup input requires recognition and resolution of associated ambiguity. To this end we present an algorithm that recovers implicit markup elements. It relies on same concepts used by the layout engine for markup linking. Our research into bases of handwritten annotations and algorithmic foundations of HEAD back-end and front-end design provides sill ingredients necessary for advanced HEAD system development. A typical Tablet PC-based HEAD application will feature natural, paper-like annotation of electronic documents, provide advanced support for ac tive reading, collaboration, knowledge lifecycle and ensure tolerance to electronic document changes. We hope that this work will help the advent of a new generation of personal com puting applications. Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. C hapter 2 H um an H an d w ritte n A nnotation T heory 2.1 In tro d u ctio n With the appearance of new affordable pen-enabled hardware and software computing platforms the task of advanced system design for human electronic annotation of docu ments (HEAD) has gained considerable importance. In the past, with few exceptions, electronic annotations on documents did not enjoy much success and by far were unable to compete directly with annotations on paper. First and foremost, this situation has emerged due the hardware interface capabilities being fundamentally inadequate to enable active reading (Adler and van Doren, 1972). In other words, existing systems could not provide a seamless integration of reading and annotation (O’Hara and Sellen, 1997). The state of affairs in hum an document electronic annotation computing has been experiencing a substantial change. Numerous high-quality inexpensive pen-enabled PCs recently became available from the largest hardware OEM manufactures such as Sony, Phillips, Fujitsu, Samsung, IBM, Hitachi, Honeywell and Wacom. In November 2000 Mi crosoft announced its new strategic concept of Tablet PC (Microsoft, 2001c). A typical Tablet PC device will combine a mobile lightweight design with full Windows capabilities, 5 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. pea input and wireless network access. Tablet PC is expected to address the previously unmet needs of ergonomic on-screen reading, note taking and annotation. Microsoft mar kets this concept as the next generation of personal computing and plans to release a Windows XP Tablet PC edition in 2002. Another interesting potential hardware platform is electronic paper under development by Gyricon (Sheridon and Berkovitz, 1977) and elnk (Jacobson, 1997). It intends to augment traditional paper with the ability to change its displayed contents. The availability of this medium may not be so immediate, but it has the potential to become a model instrument in human electronic document annotation. The area of human document electronic annotation has enjoyed certain attention (Marshall, 1997; Marshall, 1998) but has not received a systematic and comprehensive treatment. Our goal is to fill in this gap by offering a methodical, in-depth theoretical framework corroborated by pilot software implementations. This chapter presents the re sults of this effort. We begin by focusing on understanding of issues associated with human handwritten annotations on paper documents (HHA). Chapter 3 continues to address top ics of functional and architectural back-end design. In conclusion, Chapter 4 presents the algorithmic bases of building a graphical system front-end. Human handwritten annotations on paper documents are HEAD close counterparts. Advanced HEAD system design can benefit substantially from understanding the bases underlying HHA. To obtain an insight in HHA we have conducted a case study. An ex tensive collection of annotation samples was accumulated from a variety of sources that included library books, journals, textbooks, manuscripts, papers, reviews, proofs and of fice documents. The source documents were annotated by university students, faculty Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. and staff. Document topics included mathematics, neuroscience, history, philosophy, re ligion, psychology, entertainment and business. Everything reasonably possible was done to ensure that the study covers as many kinds of annotation activity as possible. Annotation samples were scrutinized and categorized with respect to four main char acteristics. First of all, we observed and formalized the markup structure. We found that HHA markup typically consists of elements of three types and this arrangement is repeat- able and robust. Individually, each element type can be expressed and communicated in many ways. In addition, the elements can be combined together and arranged in different fashions. In this regards we infer a number of classifications of markup appearance. With a better understanding of markup exterior properties we can proceed to study the functional and behavioral aspects. Each annotation satisfies certain user needs by serving particular functions. The former are defined by goals of the activity the reader is involved in. A process distinguished by characteristic patterns of activity whereby such goals are achieved constitutes an annotation behavior. We review most common annotation goals and activities, construct functional and behavioral taxonomies and analyze typical markup appearance associated with each function or behavior. The case study of annotation samples lays foundation of the human handwritten an notation theory (HHAT). The first and foremost purpose of HHAT is to equip us with an in-depth understanding of HHA and thus prepare the grounds for research into the de sign of advanced HEAD systems. HHAT is intended to enable the development of formal computational algorithms for high-quality automated text document annotation, which distinguishes it from related work in (Marshall, 1997; Marshall, 1998). In particular, in this article we discuss important implications of HHAT to two key problems of HEAD Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. system design, which are automated markup layout generation and handwritten markup recognition. Human handwritten annotations are inherently ambiguous. Typically ambi guity arises from leaving certain annotation elements implicit. This is a direct result of the desire to convey information with as little markup as possible to both reduce the page clutter and spare the effort. We refer to this phenomenon as the minimal markup principle and discuss how it applies to HEAD system design. Successful markup recognition requires unraveling any uncertainties, while automated markup layout demands a re-introduction of implicitness in a proper fashion to increase the visually appeal of the output. To this end we have developed a number of algorithms, which are treated at length in chapter 4. The algorithms offer formal means of ambiguity detection and resolution with a strong emphasis on the markup layout task. 2.2 H an d w ritte n annotation appearance 2.2.1 Stru ctu re In this section we will address the structure of annotation markings and introduce key terminology. Consider Fig. 2.1, which shows an example of a human handwritten annota tion in a printed document. The markup consists of two selections, a link and a comment. Each selection demarcates a fragment of the source text. We refer to such selections indi vidually as atoms. The two atoms taken together make a clump of atoms. The comment contains a statement from a second party. The link establishes a relation between these components. Since the comment is attached to a clump of atoms we can identify it as annotation. If the comment does not relate directly to any document fragment, we call it 8 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. a note to distinguish from annotation. The markup in Fig. 2.1 taken as a whole forms an annotated clump. gamma function according to the relation r(m) r(n) r(m + r m, n) for i of beta or gamma functions. Two useful Figure 2.1: Annotated clump consists of a clump of atoms, a link and an annotation. Clump of atoms, link and annotation are three basic tools enabling one to attach statements to document fragments, which is the essence of the annotation process. We observe the three-element structure in all handwritten annotation samples collected so far. It appears to be robust and repeatable. In different cases the elements can assume different will address element expression varieties shortly after defining the key concepts in a more formal fashion. respect to the appropriate metric. According to the definition, the text atom on the left in Fig. 2.1 corresponds to ‘ m < 0'. Any proper substring of this fragment would not qualify as an atom due to the entirety requirement. Also, “ m < O ' and ‘ n < 0 ' in Fig. 2.1 cannot be a single atom since the unselected comma separates the two text portions. The reference to a metric being as sociated with each atom calls our attention to the fact that atoms can be created on document content conveyed in any acceptable format, such as text, graphics, tables and formulas. Typically, we assess the continuity and completeness of pictorial atoms using appearances and occasionally make their identification complicated and ambiguous. We Def. 2.2.1 An atom is an entire marked up fragment of document that is continuous with 9 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. two-dimensional Euclidian metrics. A practical metrics for text atoms is to use the direc tion of the local text flow and follow its sequence. Def. 2.2.2 A clump (or clump of atoms) is a set of associated atoms. In most cases an annotation refers to only one atom. However in certain cases, such as one shown in Fig. 2.1, it can associate to a collection of atoms. The collection membership is typically established schematically by means of a link or it can be implied in the way the markings are positioned and drawn. Note that when talking about clumps by default we assume the entire set of associated atoms. Def. 2.2.3 An annotation is a datum added to a document by a second party and associ ated with the entire document or one or more of its parts. The mentioning of entire document or its fragments highlights the fact that the frag ments can come in any size. An annotation can be associated with an atom as small as few characters, words or lines or as large a page, a chapter or the entire document itself. Def. 2.2.4 A link is a means for associating atoms into a clump and a clump to its annotation. Def. 2.2.5 Annotated clump is an entity consisting of a clump together with its annotation and link(s). Def. 2.2.6 Annotation markup is a set of markings added to a document by a second party. Def. 2.2.7 Annotator (or second party) is the person assuming the role of annotation markup creator. 10 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Def. 2.2.8 Viewer (or third party) is the person assuming the role of annotation markup reader. In the definitions above the concept of annotation markup specifically designates the visible physical markings created by the second party. The annotator and viewer role assignments depend on whether the annotated clump is created for private or public use. Private annotated clumps are created and viewed by the same person. In the other case the creator and viewer are typically not same. Note that in any case more than one person can be acting in both roles. Any document can be annotated and viewed by any number of people. 2.2.2 A tom s According to the definition, an atom is a fragment of a document. Let us examine how these fragments are formed. Consider for instance Fig. 2.2(a) that shows two words selected by circling. Circling is a precise in-line technique allowing the establishment of boundaries on a per-character basis. Other typical per-character in-line selection methods include the use of underscores, highlight, box enclosures, brackets and sometimes strikeouts as shown in Fig. 2.2. Ah of these techniques are practical for fragments of small size. The typical length varies from several characters to several lines. Fig. 2.3(a) shows a larger atom created by drawing a vertical line on the margin called range. Ranges are so-called margin selectors. They are convenient for fragments varying from one line up to one page in size. Imagine using an underscore to create the atom in Fig. 2.3(a) and you will see that in this case the range spares the markup effort and retains document legibility. 11 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. (a) Circling (b) Underscore (c) Highlight s in different Fonts-— arc in associative m em o ry , w h ich w ork than less frequently the [neural substrate! they nglefatteniional fix a tio n } F u r put mapping » faster and and pattern activation sul of phon em es can b e directly nges, the scope of the a t t» (d) Box enclosure (e) Brackets (f) Strikeout ere racing past the bam, or sufficient cue for suet art of w arden oath sentence handfprosodyjbhanges v evise tne interpretation of s, which does not dete (g) Emphasis using (h) Emphasis by means double underscore of circling Figure 2.2: Typical explicit character granularity selectors. Def. 2.2.9 Selector is the markup used to form an atom. The fragment intended for selection using a range may not always exactly correspond to the marked atom. The discrepancy takes place due to the fact that the boundaries of the intended fragment may not coincide with line breaks. This allows us to speak of selector granularity. For instance, the granularity of the ranges in Figs. 2.3(a), 2.3(b), 2.3(c) and 2.3(d) is one line. On the other hand, the granularity of in-line selectors shown in Fig. 2.2 is one character. Other typical levels of granularity include paragraph, page, chapter and entire document. Def. 2.2.10 Granularity of a selector is the precision with which its boundaries are marked. For selectors having implicit boundaries the granularity corresponds to the size of the atom. Certain types of selectors do not have their boundaries marked. For instance, we can select an entire journal article by attaching a post-it note to its front page. The post-it 12 with permission of the copyright owner. Further reproduction prohibited without permission. probably at Prosopa o f these pati never becor prosopagnos their skin (i sponses) wh the elephant? ects pictures, iratus, and so i farther away similar effects illy present, imagery also characteristic scenes. First, to generate i takes time tc window to tl _part. The in dude not 01 (a) Open range (b) Right open range with emphasis (c) Closed range and possib ("way. In the which the parently b moving th |Jield. The these patie (d) Closed range with out spanning line encoded. For fontsjthe av; selves to rect (in Chapter representatic howeverjone on the'Tasis (e) Detailed range uses brackets to enhance gran ularity parts, letters, thcm.j His ey inspecting an or family me unable to idc time he ident He did identi (f) Detailed range with no spanning line and non- directional end-points did not alway is tickled enoi This sin e mu: a given input (g) Cross-out (h) Cross-out combined with box enclosure Figure 2.3: Typical explicit line granularity selectors. Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Size-Granularity Character Line Paragraph Page Chapter Document Character CBHKUS n/a n/a n/a n/a n/a Word CBHKUS n/a n/a n/a n/a n/a Sentence CBHKUSD R n/a n/a n/a n/a Paragraph CBHKUSD R XP n/a n/a n/a Page MPI n/a n/a Chapter MPI n/a Document PI Table 2.1: Common text atom selection techniques. C-circling, B-box enclosure, H-highlight, K-brackets, U-underscore, S-strikeout, X-cross-out, D-detailed range, R-non-detailed range, M-bookmark, P-post-it note, I-inscription. note creates a large atom with implicit boundaries. According to the definition above, the granularity of this atom corresponds to the document level. In the course of his activity the annotator may need to create atom of different sizes and granularities. Granularity and size jointly determine the selection techniques available to the annotator for atom construction. Table 2.1 presents a typical correspondence between these two parameters and the applicable selector types. Ranges differ with respect to boundary demarcation. The range in Fig. 2.3(c) utilizes bracketed endpoints to emphasize that its boundary corresponds to the line to the right. This is an example of a closed range. In contrast, the markup in Fig. 2.3(a) does not have this decoration. We call such ranges open. Fig. 2.3(e) shows a range that combines per-line selection with the per-character granularity. The enhanced granularity is achieved by using a companion in-line bracket selector. We refer to such rtinges as detailed. Correspondingly, the range in Fig. 2.3(a) is called non-detailed. Open ranges are faster to mark and may be helpful during speed-reading. They are typically used in four situations. Firstly, the absence of a prominently marked endpoint allows for fuzzy selection. For example, an open range can indicate that some interesting 14 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. information is located in its proximity. Secondly, the use of an open range is appropriate if it continues onto the ad jo in in g page and their page and range boundaries coincide. An open range can serve as a margin landmark for one or more associated in-line selectors to make their location easy. Precision endpoints are also not necessary if the exact range boundary is obvious to the reader. For instance, open range is suitable to select a whole paragraph. On the other hand, the closed range is helpful when an atom ends at a sentence, but not paragraph boundary and that boundary is the only one in the line. In this case the atom implicitly has a per-character granularity, while the markup granularity is coarser and corresponds to one line. Fig. 2.3(c) presents an example of such situation. Ranges also differ with respect to the choice of the vertical mark shape. Some an notators prefer vertical lines or square brackets. Others favor curly brackets or even tall parentheses. Occasionally the vertical line in the closed range can be omitted and the endpoints appear as horizontal bars as shown in Fig. 2.3(d). The choice is often a matter of habit and has a number of implications. In general, we can speak of many text selectors as consisting of opening, closing end point marks and a span mark. Figs. 2.2(a), 2.2(b), 2.2(c), 2.2(d), 2.2(f), 2.2(g), and 2.2(h) show typical examples of in-line span marks. Fig. 2.13(g) shows a rare case where an in-line span mark is situated above the line. Figs. 2.3(a), 2.3(b), 2.3(c) and 2.3(e) demon strate this concept on margin selectors. When present, the span mark adds to the atom’ s visibility and is often used to convey a degree of emphasis. Figs. 2.2(g) and 2.3(b) show how using a double line span mark can increase the saliency and associated importance of an atom. On the other hand, a span mark absence trades visibility for space and effort savings, reduces page clutter and leaves more room for annotations. Consequently, such 15 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. selectors can be used to create large atoms crossing page boundaries. Figs. 2.2(e), 2.3(d) and 2.3(f) illustrate the case at point. Endpoints specify selector boundaries. We distinguish at least three types of these markings. Closed range and bracket inline selectors utilize directional endpoints. Figs. 2.2(e), 2.3(e) and 2.3(d) in particular illustrate how angular marks can indicate which way the selector extends. Non-directional endpoints such as one shown at the bottom mark of Fig. 2.3(f) do no more than demarcate the boundary. Endpoints can also be absent. For instance, the underscore in Fig. 2.2(b) and the open range in Fig. 2.3(a) contain only span markings and no endpoints. Composite selectors that combine in-line and margin mark ings have two sets of endpoints as shown in Figs. 2.3(e) and 2.3(f). Note that in Fig. 2.3(f) the type and granularity of upper and lower markings do not match. This indicates that occasionally annotators prefer to use different endpoints for the same selector. Selections larger than ones created by ranges are typically formed using bookmarks, post-it notes and inscriptions. An inscription shown in Fig. 2.5(b) selects the entire page. Notice that it is written at an angle with the document text to make it stand out. Placing the inscription at the top distances it from any particular text fragment and associates it with the whole sheet. The use of bookmarks to select pages and chapters is very traditional. Post-it notes can be flexible and versatile. They can be affixed in such a fashion as to mark atoms of smaller and bigger sizes as well. For instance, a post-it note can be placed on the margin to mark a paragraph and attached to the title page to select the whole document. So far we have addressed text atoms. Atom creation rules for formulary atoms typically adhere to text atom rules since formula display relies on text flow as well. Figs. 2.10(a), 2.10(e), 2.13(b) and 2.13(g) provide severed examples of this type of atom. Tabular atom 16 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. formation rules depend on the level of selection and cell contents. Regulations for selections inside cells correspond to the type of the contents of the cell. Cell-level selection normally complies with pictorial atom rules. Fig. 2.4 shows examples of pictorial atoms. Typical pictorial selection techniques in clude highlighting, circling and the use of inscriptions. Circling is acceptable if the object being selected is not already located inside a box or circle. In general, highlighting proves to be a more flexible pictorial selector. This is due to the fact that elements of certain irregular shapes are difficult to circle. For pictures containing words or sentences those elements can be selected using text selectors. Inscriptions are annotations positioned in such a fashion as to implicitly select a docu ment fragment and thus create and atom. The selection in pictures is usually achieved by the virtue of 2D proximity or the markup simply overlaying the part of interest. Inscrip tions are also adequate for text atom formation. For instance, they can be positioned next to a sentence of interest, at the top of a page, on the first page of a chapter or on the title page thus implicitly selecting the entire document. Fig. 2.5 illustrates these cases. We will address issues concerning explicit and implicit elements in the section dedicated to links. Def. 2.2.11 Inscription is an annotated clump with implicit link and clump elements. Link endpoints can create implicit atoms in a manner similar to inscriptions. Figs. 2.6(a) and 2.6(b) illustrate how a text and a pictorial atom are formed by link endpoints. Fig. 2.6(c) shows a link establishing a position within text. This is an instance of a positional selector. Fig. 2.7 demonstrates typical marks used to specify positions in-line or on the margin. 17 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. (a) Circling (b) Highlighting Figure 2.4: Typical explicit pictorial selectors. Fortunately, p speaker is being “i f0' act. Indeed, cruel The origin of language I one know* bow man t o t L i t " u * t ' i ' ‘ ‘ ■ ' * i.**«— The Updating o f the Rep in Parietal Cortez by Intt Je a n -R e n T O u h a m e l , C a jl o l L . (a) Sentence or para graph (b) Page (c) Document or sec tion (d) Pictorial Figure 2.5: Inscriptions implicitly create atoms. ion that will bear on an is called topdown — i to look up relevant te discussed, there is [ (a) Implicit text atom (b) Implicit pictorial atom is present. He 1 replaced them he context•oi' a ' i but also heard (c) Explicit position Figure 2.6: Link endpoints can create atoms. :m, which in turn produces a ive memor^This represen ns associated with words, (a) In-line /hich places strong_ d with other wordsN. associative memo- (b) On the margin Figure 2.7: Typical explicit positional selectors. 18 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Atom markings can perform a secondary function, which is conveying additional se mantics. Two most common semantics are emphasis and action instructions. Emphasis expression depends on atom selection type. We have mentioned that due to their visibility span marks can serve as a vehicle to convey a degree of importance associated with selec tors. For ranges and underscores the emphasis can appear as a thicker, double or triple line, see Figs. 2.2(g) and 2.3(b). Fig. 2.2(h) shows how circling can indicate additional focus. Circling may be preferred in certain cases over underscoring to reduce page clutter if the line spacing is too small. For highlights emphasis can be communicated through the choice of the color. Emphasis can also be achieved by means of a nested selection, such as an underscore located within a greater range. Action instructions are customarily communicated by using special atom selection shapes. For example, a crossed out block of text is most likely marked for deletion, see Fig. 2.3(g). Proof marks such as those defined by the British BS 5261 (BSI, 1976) and international ISO-5776 (ISO, 1983) copy correction standards go a long way to utilize action instructions. 2.2.3 A nnotation s and notes Handwritten annotations on paper documents can assume several forms: symbolic, text, formulary, tabular and pictorial. Figs. 2.9(b) and 2.12(d) show exclamation and question mark symbolic annotations. Figs. 2.8(a), 2.8(c) and 2.9(a) illustrate text annotations. A pictorial annotation can be seen in Fig. 2.8(b). Annotations can be located on margins, in-line, in footer, header and blank space. Figs. 2.8(a), 2.8(b), 2.5(b) and 2.8(c) show relevant examples. Annotation positioning is 19 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. SMA <Wco. C? ttcqucnccs (a) Annotation on the mar gin Hi* M fM al d*t* «* not m d**r K k .c.1 ; ti jtC U f 5 > '« u { « r (c) Inscription in a picture (b) Footer annotation with reference mark Figure 2.8: More annotation examples. determined by several factors. Its primary goals are to record a statement, establish correct associations and keep ambiguity, effort and page clutter to a minimum. If the clump is explicitly marked, the selector type has a direct influence on annotation placement. Annotations for ranges selectors usually go on the margins next to the clump. If the space on the margins is insufficient, the comment can be placed on the footer or, more seldom, in blank space or on the header. Notes typically go on the footer too. Short in-line text selectors may have their annotations located in the blank space nearby, if available. Otherwise margins are acceptable. Fig. 2.13(f) illustrates this situation. If the clump is not marked, the annotation is placed in such a way as to implicitly select it and at the same time achieve required visibility, see Fig. 2.5. In certain cases annotations can be recorded separately from the document. We refer to such annotations as conspectual. They usually appear as a list of notes containing references to clumps in the document. Annotations differ with respect to semantic clarity. Conventional annotations, such as one shown in Fig. 2.9(a), are carefully written out so that viewers other than the annotator 20 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. ing the troth and j , molt the existence of mce of phenomena/1 faty sion of his society for / j aa a mere instru-/ I i w *i"ff if f f a *r society. „ 0 .. f ‘ -----------------^ — —---------- — :n . Man ha* ceaeed| iw himself as poor, and (a) Conventional (b) Idiomatic contemporary see /■j. > church but with im on't you? They see t \ \ J J symptom of the fa he name—“docsn’ t\ | v , Those who try their power.” . • ' (c) Idiosyncratic (d) Idiomatic with empha sis Figure 2.9: Semantic clarity of annotations. himself can understand it with relative ease. Fig. 2.9(b) demonstrates an idiomatic anno tation. We can reasonably assume that the exclamation symbol indicates the annotator’ s interest. Idiomatic annotations are also characteristic of people of certain professions, e.g. proofreaders. Fig. 2.16(c) shows a proof mark example. In contrast, the meaning of the inscription in Fig. 2.9(c) is difficult to determine with certainty. The mark appears to be meaningful only to the person who jotted it down. This is an example of an idiosyncratic annotation. Emphatic annotations commonly occur in handwritten markup. They convey informa tion about annotated clump importance. Emphasis is typically achieved by writing down one or more symbolic stars or exclamation marks, see Fig. 2.9(d). Examples that we have seen in this section feature explicit annotation markings. In certain cases atom and even link markings can carry additional semantic that acts in place of an annotation. Figs. 2.2(f) and 2.3(g) demonstrated how action markings could imply deletion. We refer to such annotated clumps as having implicit annotations. 21 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 2.2.4 Links The primary function of links is to associate atoms and annotations to form annotated clumps. The associations can be created in three ways. A proper link connects annotated clump elements using a certain shape. Typical shapes include line, arrow or a short stroke pointing in the direction of the associated element. The latter method is acceptable if no ambiguity arises. Fig. 2.10(a) shows an example of a line shape. Fig. 2.13(b) illus trates an arrow link. Occasionally proper links can carry piggyback semantics such as causality, order or direction. Links can be also labeled and used creatively to convey an additional message, such as one in Fig. 2.13(b). There the equality sign can be considered an annotation, but on the other hand it can be recognized as a part of link. Proper links work well only for relatively short distances. If annotated clump elements are located far away from each other annotators favor links by reference, see Fig. 2.10(d). A unique symbol, color or a resource locator with document and page number is used as a placeholder for the annotation. The annotation itself is located elsewhere and is also marked with that symbol. Fig. 2.8(b) shows the annotation for the atom in Fig. 2.10(d) located in the footer. A proper link in this case would create too much clutter and therefore was replaced by a reference. The third method of establishing association utilizes implicit relations. It is often the first choice for the annotator since it does not create clutter or require markup effort. On the other hand, implicit links can create ambiguity. In that case a proper explicit link or link by reference are used. Fig. 2.10(c) shows an example where the atom and annotation are connected using aligned overlap. Aligned overlap establishes associations if the element projections on the axis perpendicular to the predominant text flow overlap. 22 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 0,(t At) = (t - 0,(t) + • ^ ^lnputj(t) + £ u^State* (t)j Out = f(o(t)) lo o i« x - C o J u u 'v o ' ,v ’ (a) Link proper of the written numbers is spatial acalculiar&nd. itself, which is referred (b) Link by reference inline and for fixed-CS conditioning, the whereas the most ty conditioning is sing ISI. In fixed* • conditioning for (d) Link by reference on the margin connections with noiljr lsc ftfeundi between tea- prry~u- crtrK Given its very Uije me (c) Implicit link in text via aligned overlap i — d)‘ + 4befgf), a n equal eigenvalues < j l - .5 * / m atrix, i.e., that. L-cy -^Jd, 3.2.9, kj(t) - 0 a s ■ a e > state at q - 0) is other hand, *,(r) (e) Link chaining Figure 2.10: Link types. Other methods of implicit linking include adjacency and direct overlap. V V e have discussed these techniques in a previous section. Aligned overlap is quite common in text documents. It appears to work with all types of text atom selectors. Let us look more into the rules governing its formation. We would like to demonstrate how aligned overlap can take precedence over other possible implicit associations, how explicit associations can preclude aligned overlap from forming a link and how simultaneous aligned overlap is resolved. Fig. 2.11(a) shows an ambiguous situation 23 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. to compensate for the imbalan ^ brain-damaged person is not a ® ln the S l y year* of the 19th century r L missing an ability O f two. R at in hia researches on Jugt_conductioji was ( p J tions will automatically adju trigonometric series which now bear his ^ in the new context. The “ CCOlOf (a) Overlap can take precedence over (b) Explicitly bound annotation does proximity not create implicit link to semantic features than concrete words. They found that all (A) types of damage near the input layer resulted in more errors for (6) abstract words than concrete ones, which would explain the (c) Annotations in opposite side text areas create independent implicit links Figure 2.11: Implicit link formation via aligned overlap. where two annotations establish implicit relations with one atom. For an atom it is very uncommon to have two annotations. Therefore in this situation the annotations compete for an association. Mark A align-overlaps the atom, while mark B is positioned closer to it. From the perceptual point of view it is possible to associate mark A with the atom and mark B with word ‘ early’ . That demonstrates that associations created aligned overlap can be strong enough to override associations by proximity. Fig. 2.11(b) shows an annotation align-overlapped with an atom while being explicitly linked to another atom. The explicit link weakens the implicit association and the aligned overlap does not form an implicit link. In this case we can say that the annotation is explicitly bound to its clump and therefore cannot form an implicit association. Fig. 2.11(c) shows two sets of atoms and annotations each residing on the opposite sides of the page and all align-overlapped with each other. Due to the latter all four components should be linked together. However, from the figure we can reasonably assume that mark A corresponds to the left atom and mark B corresponds to the right one. We can say that in this case the two annotated clumps belong to the opposite side-text areas thereby 24 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. influencing the rules of implicit association formation. Please see chapter 4 for a formal analysis of implicit links and detection and resolution of arising ambiguities. Relations of different types can be chained together to form composite links. Fig. 2.10(e) shows how aligned overlap is combined with a proper link to connect the annotation to its clump. 2.2.5 M arkup procedure Now that we have considered each constituent element individually, let us examine the process of annotation clump creation in its entirety. At an abstract level this task can be viewed as a constraint satisfaction and optimization problem. Given a particular page layout, a fragment of a certain selection size and granularity and an annotation we must generate appropriate markup and ensure required emphasis and visibility while minimizing clutter, effort and ambiguity. In practice this problem can be approached in several consecutive steps. In a typical scenario the annotator can proceed as follows. During document reading a fragment of interest is found and a decision is made to create an annotated clump. The annotator must choose a selector type. The latter depends on atom type, size and granularity. In addition, the annotator may have certain preferences influencing the selector choice. For example, having a highlighter pen forces the use of highlight selector. Having a regular pen, the annotator may prefer using underscores over ranges or vice versa. We can reason that the annotation process should not be distracting to permit easy active reading. In our collection of annotation markup we observe certain annotation styles. It appears that a person may have a strategy chosen in advance and then focus on reading. 25 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. The strategy includes preferences to selector types, annotation content, positioning and visibility. It is chosen according to the goals and mode of reading. For instance, during fast reading annotators may prefer to place annotations on the margins. Clump selectors are often coarse grained. Annotations are often trivial or implicit. On the other hand, annotations during slow and meticulous reading can appear anywhere in the page. The clumps are typically fine-grained. The number of implicit annotations is less. After marking the clump, a location for annotation must be found. Knowing the size of the comment is important at this step. The annotator should be able to estimate it mentally. Annotation positioning is conducted according to the principles set forth in the previous section. The order of the next two steps can be reversed. Now the annotator has to write down the annotation and choose and draw a link. Link selection is carried out using link formation rules that we discussed. Once again, the minimum markup principle is must be ensured to minimize clutter, effort and ambiguity. If the annotator can dispense with the link, we have an implicit link component. If marking the clump can be also avoided, we have an inscription. If the message can be conveyed through clump markings, we have a semantic clump mark (SCM). 2.3 A nnotation functions Understanding the aspects of markup formation and appearance is essential for construc tion of HEAD front-end components such as layout engine and input interface. At the more abstract level of application feature selection the software designer must also take into consideration the functional side of annotations. In this section we will look at the 26 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. use of annotations in different situations. We will reason as to what goals the annotator is trying to achieve through that use and how. Answers to these two questions will help the annotation software designer choose the tasks the application will support along with corresponding enabling techniques. 2.3.1 R eading goals, behaviors and styles Annotation is an activity intimately and inseparably associated with reading. Under standing why people read goes a long way towards understanding why people annotate. (O’Hara, 1996) presents a literature review summarizing goals of reading. The list includes items such as reading and re-reading to learn, reading to search and answer questions, reading for research, critical review, problem solving and decision making, reading to sum marize, to do, to write and revise documents, proofreading and reading for enjoyment. Each goal has characteristic patterns of associated reading and annotation activities that we call behaviors. Note that there are two kinds of goals and behaviors, those of the annotator and those of the viewer. These can be the same person acting in different roles by annotating a document and reading it again later, or it can be different people. Some behaviors do not utilize annotation. Others rely heavily and depend on it, such as proof reading and review. We can say that annotations enable certain behaviors and participate in others by providing new affordances. There are two taxonomies that are relevant and immediately important to our study. Let us briefly review each one here. (Adler and van Doren, 1972) discusses four levels of reading. The first level is called elementary. It corresponds to rudimentary reading by recognizing words and is learned in elementary school. Reading at the next, inspectional 27 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. level involves systematically skimming the material. Its goal is to obtain as much informa tion as possible within a relatively short period of time. The third level is called analytical reading. Its primary purpose is attaining in-depth understanding rather than just infor mation or entertainment. At the highest, synoptical, level the reader relates books to one another for comparison and constructive analysis. (O’Hara, 1996) presents a taxonomy of how text is read. Receptive reading involves uninterrupted reading in the manner of listening. In the reflective mode the reader can suspend the activity and think about the contents. Skimming is used to quickly obtain a general idea about the document. Scanning involves searching the document for particular information. Based on the two reading taxonomies we can make several conjectures on the use of annotation markup. Scanning and skimming involve locating key words and concepts. These fragments are marked as clumps to increasingly assist navigation within the docu ment. Analytical reading is synonymous to reading for understanding through a meticu lous examination of the material. The understanding grows as the reader interprets and later summarizes new concepts in his current knowledge framework. The interpretations and summaries appear as annotations attached to the concepts and document sections respectively. Reflective and synoptical reading can generate new ideas that relate to the document at hand in general, but not to any particular fragment. Such stand-alone reflec tions can be recorded as notes. In the following section we will provide more annotation examples to confirm the abovementioned suppositions. 28 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 2.3.2 R eading support Reading is a complex activity involving many processes that can occur at the cognitive level and include navigation, understanding, memorization, recollection, communication, summarization, reflection and others. Annotation is a record of the mental activity of the reader. It influences the thinking and actions of the reader and viewer at the cognitive level and functions alongside these processes to support active reading (Adler and van Doren, 1972). Typically, annotation markup conveys information such as location, importance, various thoughts and facts. This effectively helps selective information filtering, expedient navigation and assists reflection. Def. 2.3.1 Annotation markup function is the cognitive action associated with creating the markup as perceived by the annotator or viewing the annotation as perceived by the viewer. People exploit annotations as tools or instruments. For each supported reading process there is a set of commonly used annotation techniques with characteristic appearance and functions. Using the instrument analogy we call these empirical groups annotation instrumental classes (AIC). In the previous sections we examined appearance of elements individually. Below we will study element arrangements and observe a relation between annotation function and appearance. Def. 2.3.2 Annotation instrumental classes (AIC) are general empirical annotated clump classification groups observed in common use and characterized by certain appearance and functions. 29 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. In our review of annotation functions below we offer a generalized process-oriented framework. We refer to AICs supporting a reading process as aids. In this classification a given annotated clump may belong to several categories. Such joint membership would indicate that the annotated clump supports several processes at once. Another possible taxonomy could relate AICs to reading tasks and behaviors. 2.3.2.1 N avigation aids In Fig. 2.12(a) annotator explicitly marked up an important fragment, but did not indicate as to why it is important or include any remarks on the subject. Such markup functions to assist viewer’s navigation in the document by selectively increasing the fragment salience. We refer to such fragments as selection clumps and hypothesize that they can mark key points in the text. Def. 2.3.3 Clumps of atoms having with an implicit or absent annotation are called se lection clumps. Def. 2.3.4 Clumps of atoms marking document fragments that are especially important to the annotator and viewer are called key clumps. Two AICs are typically used to assist navigation. Highlights without margin marks appear quite salient. For better visibility, in-line or margin selectors other than highlight can combine with a margin mark, typically symbol and rarely annotation. Figs. 2.12(c), 2.12(b) and 2.12(d) illustrate this AIC. Margin annotations make good landmarks since they are easy to scan and find. Navaids do not convey complicated statements. We can say that they are semantically trivial. They can carry emphasis as shown in Figs. 2.9(b), 2.9(d) and 2.3(b). When placed 30 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. network aocceufally lesnu eere diet « te » ' a t time i , t&en the (a) Highlight before having been s / f ) b. o w g y r w ' truth or falsity of ai (b) Number marking were considered • faith or civiiiza* ^ universal bazaar. ^ ehren worked or * — 1 — y'. « - « —« ■ a d a alaB O U B L b y t nlurinn gning ' ‘ critchy in empty (c) Text marking Figure 2.12: Navigation aids. a She in | bunkacros I He wore n , sH e was an It had (d) Symbolic marking on the front page, navaids can carry a simple evaluation. For instance, writing ‘Good!’ indicates that the document is valuable. Navaids can appear as conventional annotations, see Figs. 2.9(a) and 2.12(c). However, writing qualifiers like ‘important’ next to every key clump may prove to be slow and tiresome. Idiomatic or idiosyncratic symbols can readily take up the function of words. Fig. 2.9(c) showed a personal idiosyncratic navaid. Fig. 2.12(d) illustrates the idiomatic case. Occasionally, annotators number clumps to establish a structure or navigation sequence as seen in Fig. 2.12(b). Other navaids indicate the reading progress. For instance, tick marks can identify chapters that have been read. Sometimes, a margin mark is appropriate to landmark another annotation, e.g. inscription. In this case an annotated clump may have more than one annotation element, each serving different functions. The viewer has to steer not only to document fragments, but to annotation markup too. On a sideline we should note that highlights can represent a trace of attention. In this case it helps annotator to focus. Such annotations are called attention aids. 31 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. illumination. Subae hood to come up wi ■nation may be quite M M g i i i l S jH IB H Im - 1 descriptors of edges the aspects of the v cesses, which can e bon. It is important t level vision to comp (a) Clarification v H e ftv c u fc iw t. ♦(f) = *iU)flt + \ i \ 3 4(f) - ^ f p - *i*i(r> 4i - *:*2(f)«i (b) Relations SECOI cT w a r r e n h (c) Un-abbreviation ion of and 43. if 9 * * i9 i + ■ * 2 * 2 ' * - ^ S * ,k a (d) Re-formulation “even number”). d. B^fuijcation.—The fac t that a number of wc (e) Breakdown (f) Associated concept tance and is always completed in one time step. ^ 0 c hange accordin&to the following equation: I) = + c[z(f) ~z(t - 1 )]y; (r - l)jrf(f - (3) ' 7 o-a:p±*on te constant determining the rate of learning. Hence w j f will •movement takes place (y.-fr - 1 ) > 0) and the asterisk is k (x;(t - 1 ) > 0), and it will change in the direction of z(t) - (g) Grouping n t t o t h e m a t r i x d i f f e r e n t i a l e q u a t i o n p - ? ; v os 4(f) = c y o u t * i 0 a. £11 JC(/)J i r n m JLjt(r ) J ” m m t h e m a t r i x m u l t i p l i c a t i o n , t h e t o p S i n e j u s t s a y s t h a t U . 1 1 (f) , **0 ~ m m (h) Contraction Figure 2.13: Understanding and memory aids. 32 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 2.3.2.2 Understanding and memory aids Understanding and memory aids (UMA) serve to interpret the narrative in the reader’ s own vocabulary to incrementally bridge gaps in comprehension and help retention. Reading for studying is a demanding and sophisticated activity. When someone starts studying a document at a minimum he is likely to encounter new concepts and jargon that have to be expressed in the current reader’s knowledge framework. This process reflects richly in handwritten annotations. The goal is achieved through various types of re-formulations, contractions, expansions and re-casting. Fig. 2.13(a) shows key clumps assigned with conventional text annotations. The key clumps represent new terms that have occurred at that particular place of the document. The annotations expand the clumps by providing detailed clarifications. Fig. 2.13(c) shows how a clump is expanded by means of un-abbreviation. Un-abbreviation lengthens clumps to return them back into their unabridged state. Fig. 2.8(c) shows un-abbreviation in picture. Figs. 2.5(d), 2.6(b) shows explanatory labeling. It is possible that under some circumstances the reader’s erudition may approximate or exceed the one of the material. In such cases we can observe clump contractions. Sometimes reducing the level of details is also necessary for simplification, see Fig. 2.13(h). Expansions as well as contractions change the level of detail. Equivalent reformula tion expresses the clump in different terms or form while retaining the amount of details. Fig. 2.13(d) shows how a formula is expressed in pictorial form. In text equivalent refor mulations can appear as synonyms. Understanding involves other activities, such as establishing relations, associations and correspondence between concepts, grouping, breakdown and otherwise structuring and 33 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. organizing material. The number mark on the margins in Fig. 2.12(b) is a navaid that also may act as a UMA. Examples of grouping, breakdown and relations are shown in Figs. 2.13(g), 2.13(e) and 2.13(b) respectively. Fig. 2.13(f) illustrates how a germane concept is associated with the clump. In summary, UMAs feature non-trivial annotations and rich in semantics and diverse in expression clumps. The diversity reflects the complexity of the understanding process. UMAs can be located on the margins and often in-line. They tend to be fine-grained and personal. 2.3.2.3 Summaries A summary conveys a substantial amount of information in a concise and organized fashion. Key notions are collected together making their location easy during re-reading. Writing them down can refine understanding. A summary is useful when placed on the document title page to sum up the whole document. This spares viewer from skimming the document when making a decision on whether to read it again. One way to create a summary is to simply replicate key clumps in handwriting as stand-alone notes instead of or in addition to selecting them in-place. Let us refer to such notes as clump copies. Fig. 2.14(a) shows such an example. Summaries can also contain re-worked content abstracted from the document. Summaries typically appear stand-alone and attached to an entity, which typically is a large coarse-grained fragment. In particular, summaries on the front page typically attach to the entire document as shown in Figs. 2.14(b) and 2.5(c). Summaries on the first page of a chapter or section sum up chapters and sections respectively. Occasionally a summary can cover and attach 34 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. y & \* i ?'«ao ckOc.ev* / ^ S ’V W t < v v ^ o t i L & l — - ^ c t I o r N euii b*^ V ^ o i^ < W ^ v .» a U i of “* ■ - ' L - '- < * - * '^ - 5 .t^ O X .O r t op - ^ U jt - j» - 3.1 A e t w O r i t m t t d ftw y t e (a) Footnote summary potation of new domains of knowledge. For exa V ^ g . A ftt> A * y system can provide a "growth node" for a com f c t y v x c b o v v ) : > ) o i f e o - d r i ' . c r t c x c fc iv ftf c .CVT y pert V u t-tfiS - drivO* l * L aw ds -J f P * .i» '-k ^ "Y (b) Summary on the front page Figure 2.14: Summaries. Models of Pavlovian Idrc-tc. - O uc^J ~ bdidtn*) - !(«({ ( ■ * - M f-fclt 1' ” ! • \ \ M ~ t t ' f t * T ' (a) Comparison corccal rspfocnu oon « u .u uu I'U.-iW b > rtyt*'i> > • J -3 ? ( 4 (b) Reflections recorded as n. '-s ^ I —j 7 4 ^ ' yH*) = • - 7 7 ^ this fact in the sense that it causes the evolution of activity in (c) Problem solving Figure 2.15: Reflection aids. to smaller fragments, such as page or several paragraphs using appropriate selectors, see Fig. 2.14(a). Footer and header space is typically used in this case. Conspectual summaries are recorded separately from the document. In any case summaries are non-trivial. They can appear as text and in other forms, such as pictorial in Fig. 2.14(b). 2.3.2.4 Reflection aids The purpose of aids in this category is to convey reflective statements that add new content to the document. This involves reflective reasoning, problem solving, opinions, 35 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. k t r a A / y i i i v l u w u i a i i B |/ i « a « i i M » i i w i i v i u i * aids the results of Expt. 2*and also ~ othesis. and the architecture of the the internal representation of the vv/*f i*l& 7 " V \**Ajxvt£ \c .o u rr"'A c^^’ 1 (a) Library information markings (b) Review of its content, form and A U G 0 2 Z 0 Q 1 TON sets of their functionality. (Date) (c) Proof mark (d) Stamp Figure 2.16: Communication aids. ideas, remarks, observations and comparisons. Fig. 2.15(c) shows an example of problem solving and Fig. 2.15(a) illustrates a comparison. Reflection aids associated with a particular clump appear as annotations. However, there are many cases when reflective thoughts go beyond the content or otherwise distance from the document and appear as notes as in Fig. 2.15(b). They can facilitate integration, synthesis, and interpretations in another context. As a rule, reflective aids are not trivial. 2.3.2.5 Communication aids Annotations can be classified into personal and public depending on the viewer they are intended for. Personal annotations are created and used by the annotator himself. Since convenience and speed is of importance, the annotator may not exercise the extra effort of making them readily comprehensible to other viewers. It is very characteristic of such 36 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. annotations to be concise and cryptic to keep the distraction from reading to a minimum. All types of aids addressed so far are primarily for personal use. Communication aids convey information to a third party other than the annotator himself. Typically they appear as clear, spelled out annotations or idiomatic symbols. Ex amples include approvals, stamps as in Fig. 2.16(d), resolutions, reviews as in Fig. 2.16(b), instructions, and information markings illustrated in Fig. 2.16(a). Idiosyncratic symbols can be used if the viewer is expected to understand them, e.g. for specialty tasks such as proofreading, see Fig. 2.16(c). Communication aids associated with the entire document typically appear on the front page as shown in Fig. 2.16(a). Critical reviewing and revising focus on document contents, while proofreading deals with its form. For this reason revi sions and reviews can appear as UMAs and reflection aids, but clearly spelled and easy to navigate. Proof marks often show as fine-grained clumps with associated margin markings to simplify its location. (BSI, 1976) and (ISO, 1983) provide relevant standards. 2.4 Im plications to system design In this chapter we have examined appearance and functionality of handwritten annotation markup. Table 2.2 shows typical mapping between the two. Incidentally, for a trained observer the type, location and amount of markup found in a document may reveal many of the things that we have discussed. The markup signature can illuminate on the mode in which the document was read, annotator goals and interests, the level of annotator expertise before and after reading and many other interesting issues. We chose the example cases to be typical and representative. Annotation markup can be very versatile and creative and it is quite difficult to record all of its manifestations. We 37 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Function-Feature Margin markup Front-page location Personal Fine-grained clumps Trivial annot. Notes Navaids + -F + + / - + - UMA -F - -F -F - - Summaries - -F + /- - - -F Reflection aids -F + / - -F -F - -F Comm, aids + -F - + / - + /- + / - Table 2.2: Annotation appearance versus annotation function. also recognize that many conclusions need verification. Some of them can be ascertained through experiments. Others can be proved by construction, which is designing a HEAD system based on those principles and demonstrating satisfactory performance. Consider the elements a HEAD system needs in order to provide high-quality annota tion capabilities. First of all, its functionality should be determined based on a choice of reading behaviors it will support. Implementation of annotation affordances should take into account, but not be limited to handwritten annotation appearance and functions. Electronic media and networked environment enable new kinds of reading activities and behaviors. We analyze these issues in chapter 3. HEAD system back-end must implement annotation storage and manipulation. In chapter 3 we present a back-end design built mound the three-element annotation clump structure we have observed. A database is used to enable storage and manipulation. Net working makes possible distributed annotated clumps. We also analyze issues of markup anchoring in text documents, describe and evaluate a pilot back-end system implementa tion. HEAD front-end deals with annotation input interface and graphical markup output. In chapter 4 we present a suit of annotation layout algorithms that recognize and utilize 38 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. inherent markup ambiguity and implicit components to produce high quality visually ap pealing output. The engine is built on principles of minimal markup. We also provide quantitative means to assess markup quality. An advanced HEAD front-end must also provide a rich selection of markup appearance and placement options, utilize emphasis and make available requested visibility. It should take into account human factors of reading on-screen. For generic issues in human factors of reading paper versus on-screen documents see comprehensive literature reviews in (Dillon, 1992) and (Haas, 1992). The front-end input interface should identify explicit annotation clump components and recon struct implicit ones. It should also recognize explicit and implicit relations and additional semantics. In chapters 3 and 4 we continue our investigation of HEAD system design and approach these problems to develop relevant algorithms. 39 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. C hapter 3 Functional A nalysis A nd A rchitectural Design 3.1 In tro d u ctio n Recent industry achievements in personal portable computing call for an increased at tention to the problem of human electronic annotation of documents (HEAD). A large number of manufacturers have introduced so-called tablet PCs featuring pen input on 14” or larger displays, wireless connectivity, enhanced lightweight design and computational capabilities matching those of typical notebook PCs. In parallel, Microsoft is releasing a Tablet PC version of Windows XP (Microsoft, 2001c), which will provide enhanced pen in put capabilities at the operating system level. Such situation prompts the development of practical HEAD solutions based on good understanding of principles of human document annotation. In the past, with few exceptions, electronic annotations did not enjoy much of a success and were unable to compete with their legacy counterparts by far. The available hard ware was fundamentally inadequate to enable active reading (Adler and van Doren, 1972), which requires a seamless integration of reading and annotation (O’Hara and Sellen, 1997). 40 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Though some issues may still remain, tablet PCs are better situated to fulfill this require ment. In this chapter we concentrate on the HEAD system functionality and architecture. We start off by reviewing existing electronic annotation systems. A good deal of research in this area focused on Web-enabled applications. With the appearance of e-books and pen-enabled portable systems application developers became directly involved and their attention is drawn to annotation during editing and reading. Reading is the most perva sive activity associated with documents. Based on this fact we proceed to analyze how HEAD system functionality can assist various traditional reading tasks, such as learning, research, question answering and others. Annotations implemented in electronic media bring in capabilities for communication, computation, dynamic visualization and archiv ing. We reason on how each of these capabilities can help each reading task in particular. Combination of legacy annotations with electronic media also gives rise to new use cases, which are collaboration and knowledge lifecycle support. Electronic documents have several special characteristics that set them apart from legacy ones. The former are easily copied, modified and, to a lesser degree, converted from one format to another. While legacy annotations rely on document natural fixity, their electronic counterparts cannot always do that. Legacy annotations become am inseparable part of the document, whereas electronic ones must exist independently when we annotate a document we do not own. These differences pose a fundamental challenge for annotation system design. We approach this problem in two ways. In this chapter we describe an architectural framework for text HEAD systems that feature a desired property of docu ment change, format and location insensitivity. Document change and format insensitivity 41 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. is ensured by special markup attachment algorithms. We review existing techniques and describe an improved version that relies on detection of unique substrings located in the vicinity of the annotated text fragment. Several enhancements to standard versions of this procedure ensure a practical and robust solution to this problem. The algorithm was im plemented and tested in a pilot system described thereafter. Annotator 2.0 was designed as a plug-in to Internet Explorer (IE). It allows marking up hypertext documents and uses a shared database for annotation storage and manipulation. It implements a subset of the back-end specific functionality cited in the specification analysis and characteristic to system back-ends. Annotator 2.0 includes some unique annotation features such as construction of distributed clumps. A companion system described in chapter 4 treats the problem of front end design. It presents algorithms for automated high quality generation of markup graphical layout accompanied with a pilot implementation. Our functional and architectural analysis rely on a number of concepts developed in the human handwritten annotation theory (HHAT) reported in chapter 2. Based on a case study of handwritten markup, HHAT observed and formalized annotation structure, appearance and hypothesized associated functionality. Let us briefly recapitulate some important terminology. Document annotation typically involves tagging one or more frag ments and attaching to them some datum. We refer to fragments as atoms. The datum is called annotation, the collection of fragments is a clump of atoms and the entire structure including annotation is called annotated clump. The set of applied markings is called markup. Atoms come in different sizes and of different granularities. Fine-grained atoms have their boundaries established precisely, such as using a marker pen to highlight a word. Coarser-grained atoms are created by tagging text by the line, page, chapter or even entire 42 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. document. The person applying markup is called annotator, whereas the person reading the annotated document is called viewer. Reading can be categorized with respect to goals and styles (Adler and van Doren, 1972; O’Hara, 1996). Predicated on these, behaviors are characteristic patterns of reading and annotation activities. Annotation activities are integral components of reading (Adler and van Doren, 1972). Annotation markup functions at the cognitive level to assist pro cesses that take place during reading. These include navigation, filtering, understanding, memorization, recollection, reflection and many others. 3.2 System review Historically, hardware platforms were incapable to deliver front-end capabilities sufficient to enable active reading. Electronic annotation research and development predominantly targeted system back-ends. In particular, due to the rise of the World Wide Web many applications explored the new capabilities in communication and collaboration. With the appearance of portable e-book and tablet PCs the focus of attention has moved to annotation while editing, reading and annotation using pen input. Since the number of annotation systems has grown very large, this review cannot possibly cover all of them. Instead, we will concentrate on the aspects of application functionality. 3.2.1 A n n otation s in word processors Microsoft Word ’97 represents comments as footnotes in a separate window linked to highlighted portions of text in the main document. In Microsoft Office XP (Microsoft, 2001a) comments can appear as balloons on the margins with a dashed line connecting 43 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. them to the in-text selection. To create a comment the user must select a contiguous portion of text, click on the new comment button and type in the text. Other features related to annotation focus on change control. Starting with version 4, Adobe Acrobat (Adobe, 2001) includes powerful markup tools, such as electronic notes, highlighting, underlining, strike-through, text circling and mark ing files as ‘Approved’ or ‘Confidential’ with clip-art stamps. The reader can automatically generate a summary of comments organized by author name and compare documents to view pre- and post-review versions. Earlier versions had similar capabilities realized using an third party plug-in called Rermark from Ambia. 3.2.2 W eb-oriented ann otation software MS Word and Adobe Acrobat store annotations within the body of the document. Al ternatively, ComMentor (Roscheisen and Winograd, 1995) separates the markup from the document and stores it in a stand-alone collection. One of the first Web-oriented sys tem, ComMentor allows creation of comments on HTML documents by highlighting a text fragment and attaching a small image that links it to a text note. The comments can be shared on-line to support collaboration in a group. ComMentor was implemented as a NCSA Mosaic browser with special extensions. Version 2.6 of NCSA Mosaic (NCSA, 1993) also provided webpage annotation capabilities. User comments can be conveniently archived, managed and accessed using an anno tation database (ADB) (Ovsiannikov et al., 1999; Ovsiannikov and Arbib, 2000). An notator 1.0 implemented searchable annotation collections with instant access to anno tated documents. Other systems such as W3C’ s Annotea (Annotea, 2001) and iMarkup 44 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. (iMarkup, 2000) also use ADB-like storage. iMarkup is an excellent present-day commer cial plug-in application for web page annotation in Internet Explorer. Users can create freeform drawings, highlight portions of text and place sticky notes. The markup is stored on the PC or a shared server. An organizer bar helps filing notes by category, conducting keyword search and navigating to annotated documents. Annotation sharing is one of the key features that distinguish collaborative systems from personal ones. An early version of Third Voice (ThirdVoice, 2000) allowed attaching notes to selected parts of hypertext pages in Internet Explorer and Netscape Navigator. If a note was marked ‘public’, it also showed up on the many screens of other users that come to this page. These users could answer that note to start off a discussion and thus share information. Truly public comments can be easily abused. YAWAS (Denoue, 1999) expressly addresses problems associated with untrusted annotation servers. Later versions of Third Voice abandoned public annotations, but introduced ‘active words’, which are pre-selected keywords displayed as a link that leads to additional information or instant search. Several other systems use technologies similar to active words. RichLink (RichLink, 2000) is a commercial patented database linking technology for automatic addition of in context information to be displayed on demand. The RichLink intelligent engine parses documents and matches terms against a collection of databases. The matching terms in text are augmented with pop-up annotations containing corresponding records. Still {mother system called Annotate.net (Annotate, 2001) offers annotations from trusted ex clusive sources called voices. 45 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Annotations on the Web can create a basis for on-line conversations. In addition to Third Voice, examples of relevant systems include chat-board systems like HyperNews (HyperNews, 1998) and applications for discussions embedded in a webpage like Page- Seeder (Pageseeder, 2000), Web4Groups (Web4Groups, 1999), CoNote (CoNote, 1999) and many others. This class of systems has enjoyed a substantial acceptance. In combi nation with a web server, Microsoft Office (Microsoft, 2001a) allows inserting discussion points into Word documents. To conduct conversations in real time Active Annotations Project (Hummes, 1997) provides a mechanism for immediate synchronous annotation updates. Another class of systems uses annotations primarily as links. The links can differ in granularity. Coarse links point to entire pages, while fine-grained ones address page fragments and positions. Pertinent titles in this category include Distributed Link Server (Carr et al., 1995), CritLink (CritLink, 1999), Webvise (Gronbaek et al., 1999), Inter media (Haan, 1992), HyperWave (Maurer, 1996), DHM/WWW (Gronbaek et al., 1997), Chimera (Anderson et al., 1994), Annotation Engine (AnnEngine, 2000), The Xanadu project (Nelson, 1987) as well as the already mentioned ComMentor. 3.2.3 A nn otations on electron ic book s and docum ents Evolution of handheld computer devices has led to the appearance of portable electronic book viewers. A large number of models and software is now available and nearly all of them include basic annotation capabilities. For example, Microsoft Reader (Microsoft, 2001b) allows highlighting text, attaching notes, free-form drawings and bookmarks. Once created, the annotations can be viewed, organized and navigated to in the annotation 46 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. index. Other relevant titles are GlassBook Reader, Rocket eBook and AportisDoc. Elec tronic book systems have been a focus of many standardization efforts by various groups and consortiums like Association of Electronic Publishers, E-books in Print, the Electronic Literature Organization, Electronically Published Internet Connection, Japanese El-book Consortium, National Writers Union and Teleread. A number of older document-viewing systems support annotations as well. DynaText was an example of such a system. Its implementation relied on XML technologies. Other applications target manly multimedia documents, see for example DIANE (Benz et al., 1997). Also, (Phelps and Wilensky, 1997) presents multivalent annotations that can add layers of content and behaviors to a document. 3.2.4 P en-enabled applications A large class of applications uses pen and mouse input for teleconferencing purposes. For example, Microsoft NetMeeting (Microsoft, 1999) offers freeform drawing on a whiteboard combined with image exchange, file transfer and other features. Other systems take steps toward handling handwritten annotations on documents. A successor to Dynomite (Wilcox et al., 1997), XLibris (Price et al., 1998; Schilit et al., 1998) addressed the idea of active reading (Adler and van Doren, 1972) which is the combination of reading with thinking and adding annotations. It is based on a thin high-resolution tablet display with pen input. The user has colored pens, highlighters and an eraser at his command to mark up, draw and write anywhere on the page. Annotation clippings can be reviewed, searched and sorted in the Reader’s Notebook. Terms used in search are extracted from selected document text. XLibris supports automatic and manual links. Automatic links are pointers to relevant 47 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. documents dynamically generated from searches. Manual links are multi-way pointers created by circling document text. Gnotate (Informal, 2000) is another pen-enabled annotation application. It enables free-form annotations on Microsoft Word and other applications by writing on a Palm Pilot personal digital assistant (PDA) . Handwritten annotation on electronic documents can become much easier and more natural with the recent appearance of pen tablets. 3.2.5 O th er ann otation system s A substantial number of applications serve specific annotation-related tasks. XEmacs en ables source code annotation using bitmaps, strings and actions. Applications like Desk- topNotes, Sticklt Notes and QuickNotes Plus realize desktop post-it notes. Standard features for this category include selectable color, programmable alarms, and networking. Note organizers assist the creation and filing of simple notes, which are organized either by category or in the form of a prioritized list. Relevant titles include Microsoft Cardfiler, NextNote, NoteBook, NoteCenter and others. Bibliographic databases like EndNote (ISI, 1998) maintain publication references and automate the tasks of citation and bibliography compilation by interfacing to popular word processor applications. The databases can be conveniently searched and organized. Paperless office applications specialize in input, filing and retrieval of business-related documents. PaperPort by ScanSoft (ScanSoft, 2000) can enables volume document scan ning, text recognition, image editing and Web pages capture. The data can be organized using color-coded folders and searched by keywords. Documents can also be annotated 48 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. using sticky notes, highlights and freeform drawings. See also the Paperless Office by Computing, Inc. (Computhink, 2000) for another example. C Technologies’, formerly known as QuickLink, CPen (C-Technologies, 2000) sells portable OCR-enabled scanners in the form of a pen. The pen can scan and upload to PC or PDA various notes, text excerpts, business cards, tables and charts. Images can be scanned in for faxing. Integrated vocabularies allow instant word lookup. For more relevant information please see Annotate! (Ginsburg, 1998), Notable (Baldonado et al., 1999) and Fluid Documents (Chang et al., 2000). Annotate! uses annotations in con text of information retrieval. Notable reports a PDA-based document annotation system and presents a comparative system review. Fluid documents address document appear ance volatility when using electronic annotations. Also, e-Quill was an annotation system similar to iMarkup and was acquired by Microsoft. 3.3 Task-based functional analysis Traditional goals of reading (O’ Hara, 1996) include reading and re-reading to learn, search and answer questions, research, summarize, proof reading, reading for composition, re vision, critical review, to apply, problem solving and enjoyment. Let us briefly recount each goal’s description. Reading to learn necessitates efforts for understanding and mem orization. Before engaging in learning an unfamiliar material the reader may want to obtain a general idea about its contents by speedy pre-reading using skimming. To read to search and answer questions one would scan or otherwise examine the document for requisite information. Reading for research includes searching for document sources and within documents to collect, organize and analyze information. To create a summary the 49 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. reader needs to extract information of interest and restate it in a concise and non-repetitive fashion. Proofreading requires an orderly examination for grammatical, syntactical and associated errors. Reading for composition entails organization and thoughtful integration of information from various sources. When reading text for revision one must evaluate and correct its content, style and presentation. A critical review demands a methodical evaluation of soundness and consistency of statements in many respects. Reading to apply deals with executing orders or instructions. Lastly, problem solving related reading may involve location and assessment of information followed by decision making. For our purposes the reading activities associated with these tasks can be conveniently viewed as consisting of three interrelated courses of action, which are access to information, information analysis and recording of results, if any. Also, in very broad terms and based on the system review above we can state that electronic media introduces new capabilities in four areas that include communications, computational processing, dynamic visualization and archiving. Let us examine how these capabilities can assist the three reading courses of action. The reader may need to access information selectively or in a progressive fashion. For instance, skimming text during pre-reading and scanning the document to search and answer questions requires discrimination and speediness. Conversely, receptive reading resembles listening to a contiguous narration. The selective access can be enhanced through information filtering and expedited navigation. One technique to achieve filtering employs selectable annotation sets (SAS), which are collection of annotated clumps that can be displayed on demand. SAS can be either prepared in advance in anticipation of the readers’ needs and maintained statically or constructed dynamically in real time. Static SAS may 50 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. be helpful for first-time readers. For instance, a document could contain a special static annotation set that optimizes initial pre-reading by highlighting key notions. Similarly, other individual static sets could alter the document’s appearance to make it more suitable for studying or summarization. On the other hand, re-reading to answer questions and reading for research may benefit from dynamic SAS. For example, off-the-shelf software applications can search documents for keywords by highlighting matching occurrences, which essentially corresponds to a dynamic annotation set. In general there are four possible sources for SAS construction. These are the author, the reader, other readers and the document contents. The examples of static SAS above assume the author origin. The dynamic SAS were constructed based on the reader’s input. Annotations created by other readers could correlate with the viewer’ s interests and thus serve as a basis for SAS. Alternatively, task-specific SAS could conceivably be constructed by an algorithmic routine that examines the document text, similar to automatic text summarization procedures. Overall, information filtering using SAS can assist reading tasks like pre-reading, searching and question answering, learning, research, summarization and problem solving. Expedited navigation during reading can be facilitated through hypertext transitions, dynamic visualization and distributed clumps. The first technique is self-explanatory and has received a wide use. Dynamic visualization refers to the ability to adjust the doc ument’s appearance as desired. Using highlighting to show an SAS is one example of this method. Lastly, distributed clumps (Ovsiannikov et al., 1999) can have their compo nents situated in different documents thus transcending the implicit locality of traditional annotated clump. This allows bringing together, integrating and accessing information 51 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. from numerous sources. On the whole, improved navigation is helpful for most of the reading tasks, specifically pre-reading, learning, search and question answering, research, summarization, composition, review and problem solving. Note that navigation as well as other design considerations is subject to specific ergonomic requirements (O’ Hara and SeUen, 1997). Information analysis during reading involves activities such as organization, summa rization, reflection and many others. Annotation index can assist these by providing a structured annotation-centered view of document markup with instant navigation to the source. Annotator 2.0, which we describe later, also adds unique features for distributed clump linking and browsing. Annotation index content could possibly be used as a con spectus recording analysis results. Thus with respect to reading tasks, annotation index can practically assist summarization, learning, research and composition. Analytic activities can also be aided by enhanced visualization and integrated reference lookup. Enhanced visualization can make the workspace more manageable and convenient to use. Expandable annotations, floating hints, iconic representation of larger objects have been commonly used in interactive applications. When appropriate, annotation systems could also utilize post-it notes, separate window display for notes and document source and showing or hiding certain annotations. Enhanced visualization can influence annotation- heavy reading tasks like learning, research, revision, review, proofreading and problem solving. Lastly, integrated reference lookup associated with markup can offer a tighter integration with support materials used during active reading, see (Price et al., 1998). Research, learning and question answering tasks are likely to benefit from this capability. 52 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Besides improving the traditional functionality, electronic media introduces certain all-new capabilities. Firstly, annotation databases (Ovsiannikov et al., 1999; Ovsiannikov and Arbib, 2000; Bischoff-Grethe et al., 1998) have allowed practical archiving, manipula tion, versioning and access to annotation collections and annotated documents. Annotator 2.0 combines ADB with distributed clumps and linking to offer new features in this cate gory. ADB access can be conducted through keyword search, topically and chronologically. Search results can be used to generate dynamic SAS used for research, summarization and question answering. Versioning can be helpful for composition, review, revision and proof reading. The capacity for communication also allows annotation sharing and on-line conver sations. Annotation sharing applies to ADB, which can be used personally or mutually within a group. On the other hand, annotation systems for on-line conversations focus on messaging and real time response. They enable a new kind of task, which is reading for communication and collaboration. It is similar to the traditional task of reading for corre spondence, but is more dynamic due to real-time interaction, remote access and dynamic visualization. ADB’s practical archiving and access to large amounts of data collected over long periods of time creates the bases for another new task of knowledge lifecycle support. Learning is the first step in a cycle where new knowledge can be used to some purpose, forgotten, refreshed and used again, forgotten once more and so on. A system having access to the source document, distributed clumps and notes associated with its studying can be in a good position to assist its recollection. This is subtly different from just re reading an individual text for refreshment. As information keeps accumulating over time, 53 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. such a system could possibly become the primary lookup source to recollect a concept and gain immediate access to associated material. In other words, ADB could instrumentally assist reliable long-term maintenance of large cognitive maps. In this respect the term of personal information archive (PIA) would be more appropriate for use. Similarly, a shared ADB in this context would constitute a group information archive. Annotator 2.0 offers much of the functionality we have cited. Its annotation index provides annotation-centered view for linkable distributed clumps with instant navigation to fine-grained atoms with dynamic visualization capabilities. A shared and searchable annotation database can carry out PIA tasks, support group collaboration and realize static selectable annotation sets. To achieve this functionality Annotator 2.0 employs architectural and algorithmic solutions that we will now address. Incidentally, electronic media can assist many reading tasks without using annotations. Interactive reading assistant is one such tool that warrants its mentioning. Certain needs for information filtering and access can be fulfilled by communicating with an intelligent agent rather than inspecting the document. For instance, instead of pre-reading one could ask the assistant what the book is about. Based on the agent’s reply the prospective reader may then decide to ask more detailed questions about the book, to find and display certain information, request suggestions about reading on related topics and so on. On the technical side, the agents’ reactions could utilize pre-programmed responses to commonly asked questions or make use of custom question answering algorithmic procedures. 54 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 3.4 B ackend architecture overview An application backend is typically concerned with data storage, manipulation and access. This section presents a systems view on implementing annotation processing and archiving with a focus on text documents. Communication-related functionality can be realized using industry-standard solutions, examples of which can be found in the report on Annotator 2.0 implementation. For a detailed analysis of visualization aspects please see chapter 4 that describes a system user interface and associated algorithms. Several requirements will constrain and guide our construction of the application ar chitecture. First of all, we can reasonably assume that a substantial number of documents that the reader would like to annotate will be open for online access, but not be in the reader’s ownership. This supposition rules out any document manipulation, modification and copying and dictates that annotation markup must be stored and maintained as a separate and independent collection. If the document content, representation or location changes the system must be able to detect this event and correctly reattach markup where possible with graceful performance degradation. Let us call this requirement the document format, location and change insensitivity. HHAT in chapter 2 has showed that annotation markup possesses a robust structure having elements of three types, which are clumps of atoms, annotations and links. Individ ual markup elements have associated properties like emphasis, visibility, style, granularity and semantics. Data structures of this type can be readily handled by relational structures and call for the use of off-the-shelf relational database management system (RDBMS). The position of atoms must also be recorded. HHAT reported that text atoms vary with re spect to size and precision of boundary demarcation. Therefore we must support both 55 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. fine-grained and coarse-grained atoms of any length. To support distributed clumps the system must be able to associate clump elements with particular documents and their location. Besides recording element position the system should maintain content and in dexed content data, timestamp, ownership and any custom fields. The indexed content is necessary for text search of atoms in documents not in our ownership. To honor copyright regulations we cannot store verbatim text copies, but still can compile them into search structures just like the global on-line search engines do. The addressing of arbitrary fragments and positions within a document represents a separate problem by itself. To be insensitive to content and format change the addressing scheme has to rely on the local constancy of text. One possible solution (Roscheisen and Winograd, 1995; Ovsiannikov et al., 1999) is to record a short unique substring at the position of interest and expect that after all subsequent document changes it will remain intact and unique. The substring is called text anchor string. To recover the position later on, the application would have to find the anchor string. We will analyze this method at length in the following section. Document identification for annotation is another special problem. We must be able to recognize documents regardless of their location, format and slight alterations of content. Hypothetically the identification would work if each electronic document were assigned a globally unique identifier and information about its version. However, we cannot impose this requirement in any practical way. Another more realistic solution is to construct a document signature by splitting it in blocks and calculate hash values. The blocks could be paragraphs or sentences with formatting information discarded. To verify the identity and relationship of two documents one would define a signature metric and compute a distance 56 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. using a sequence matching algorithm. The distance tells us how different documents are, which parts have been changed, repositioned, duplicated or deleted. A user performing annotated clump keyword search can encounter yet another problem. Compared to the length of a document, the length of clumps and annotations is very small. This translates into a lack of terms for indexing. (Bernheim-Brush et al., 2000) reports that annotators tend to remember keywords, proper names and quotations contained in the atom, while the surrounding context receives less attention. However, viewers search ing other people’ s annotations may still consider it helpful. We call this clump context indexing. Also, the reader may have a need to find the entire document. To this end the document itself should be indexed as well. In fact, annotated clump content could pos sibly assist the document search and vice versa. The former is called annotation-assisted document search. Other kinds of search besides keyword include the search by document, location within the document, creation time, owner, category and style. The annotation database itself can be personal or shared. If shared, the developer should consider how the participating members will benefit from the common access. Most likely the membership will be based on collaboration. ADB makes possible a wide range of experiments with data mining algorithms. Examples include tracking user interests, member interest matching, improving search and generating topic summaries. After careful editing the latter can be collected and shared to the benefit of all participating users. We call such valuable collections summary databases. 57 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 3.5 M ark-up anchoring Existing annotation systems have used several methods of attaching fine-grained markup to text documents. (CoNote, 1999) inserted special tags and used offsets into the file. The first solution interferes with document contents and requires write access. The sec ond one is not robust with respect to document modifications. Any changes can result in markup misplacement upon reattachment. (Roscheisen and Winograd, 1995; Ovsiannikov et al., 1999) relied on local unique text substrings for anchoring to work around both problems. This solution is acceptable for relatively small document changes that do not affect the anchors. More extensive changes will create so called orphaned markup. XML’s XPointer and XPath provide for external links that can be anchored to text substrings and tree elements. See (Annotea, 2001) for example of implementation. These specifications define multidirectional and multi-destination links and inline insertion or replacement of the content of one document with the content of another one. These techniques are flexible and powerful, but applicable only to XML documents. (Phelps and Wilensky, 2000) de fines robust positions as descriptors using multiple location methods. The methods depend on document type and include unique node identifiers, tree walks and contexts, which are similar to local substrings. Webvise (Gronbaek et al., 1999) uses a similar ideology. (Bernheim-Brush et al., 2000) reported that given a substantially changed anchor users would prefer the system to orphan its annotation rather than speculate about its posi tions. Also, annotators considered atom’ s content as more important than its surrounding context. 58 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. In this section we describe an improved algorithm for text atom anchor construction (Ovsiannikov et al., 1999). It relies on the notion of minimal string (Roscheisen and Wino grad, 1995), which contributes to robust performance. We investigate the practical value of minimal string length and emphasize complications with text repetitions. Based on (Bernheim-Brush et al., 2000) we conclude that the algorithm should be kept uncompli cated, since recovery of substantially modified anchors is unwanted. For the same reason the procedure should not be designed to work on extensively edited documents. 3.5.1 D efinitions The anchor construction procedure relies on a basic assumption of document change lo cality, which states that changes in documents occur locally and their extent is limited. In other words, in document D the probability of a portion X to change is a function of its length L \ . When D is changed, longer portions have more chances to be altered and short text sequences are less likely to change. We will utilize this desirable property and use short fragments to construct reliable text anchors. Def. 3.5.1 Text anchor A is a triplet A = (T ,0 , R}, where T is anchor text, O is offset and R is repetition count. Suppose that we need to record position of a text atom. At first we will attempt finding a unique substring T in the immediate vicinity of the atom, see Fig. 3.1. To find the atom’s position later on in the anchor recovery phase, we will search for this or similar substring in the document. Offset O specifies the distance in characters between the beginning of the anchor string and the actual start of the mark-up. In the case when a unique string of reasonably short length cannot be found, we record the number of its repetitions in R. 59 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Text anchor Offset ■ * The quick brown fox Opposite direction Preferred direction Figure 3.1: Text anchor structure The uniqueness requirement contradicts the need for brevity. Short strings are less likely to be affected by change, but also are less likely to be unique. Moreover, short anchors can be erroneously recovered if D has changed and the change contains A as substring by accident. A problem arises when a large portion of text in a document is duplicated several times. A local unique string in this case can be too long to be used in a practical way. This forces us to use the repetition count. Using R degrades the anchor’s immunity to change, but keeps the anchor string length within practical limits. The length of the atom being anchored is also important. Short atoms require only one anchor. Let us call such atoms single-anchored atoms (SAA). Longer atoms are more likely to change. If they do, the mark-up could still be recovered if we use two anchors. In a double-anchored atom (DAA) one anchor marks its beginning and one designates its end. To this purpose, a minimal SAA anchoring record includes anchor A and the atom’ s length LAtom, (A, LAtom)• A minimal DAA anchoring record contains two anchors, (Ax, A<i), where Ax precedes A2. For verification purposes we can also record the atom’s text in both cases. When copyright regulations do not permit verbatim recording of anchor and atom text we can instead use their hash values instead. Yet another issue relates to anchor text positioning. To decrease the chances of a DCL document change corrupting the atom or its anchor, the anchor text should be located completely within the mark-up. The combined length of the atom and anchor 60 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. strings exposed to change, combined exposed length (CEL), L e = L ai om ~ h L Anchor Overlap(Atom, Anchor), should be minimized. Often such overlap is not possible. If the atom is too short, its content may not be unique and the anchor must be longer. Alternatively, the atom may be long, but the anchor text of reasonable length within the atom may still not be unique. To help the construction of a unique anchor in these cases, we can use text immediately adjacent to the atom and utilize the offset to specify the distance between anchor and atom starting points. Actual implementations of anchor string processing must decide on text case and punc tuation sensitivity. The ability to recover mark-up even when case and punctuation has changed is quite attractive and seems to be the method of choice. However, this can slightly worsen the algorithm performance. 3.5.2 A docum ent change probabilistic m odel Algorithm performance can be quantified by the probability of successful mark-up recovery P r e c after document change. The reasons for recovery failure include anchor corruption, Pc o r r and erroneous anchor recovery, P e r r • An anchor is corrupted when change C to document D overlaps with anchor text A. An anchor is recovered erroneously if the updated document D' has a new substring equal to A. Consider a simple probabilistic document change model constructed to estimate the algorithm performance. For small single continuous length-preserving modifications to document D, the probability that change C will overlap with anchor A is P(Overlap\LA,L c ) = L a + 2 L c ~ 2 = (3.1) Ld = P c o r r , {0 < L a , L c L d } (3.2) 61 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Let P{Lc) designate the probability of a change to have length Lc- The probability of anchor corruption given a change in D can therefore be approximated as L c = L d P(Overlap\L,A) = 5Z P{Overlap\La, Lc) ■ P{Lc) = (3.3) £.c=t _ (3.4, = fi(Lp) • La + /2(Ld) (3-5) We can see that the P(Overlap\LA) = Pc o r r is a linear function of anchor length. We would like to make A short to minimize this value. For example, if we assume a linear distribution of L c , P{LC) = fc - (1 “ x f ) (3-6) where k is a normalization factor (3.7) Ld — 1 the overlap probability evaluates to k = (3.8) P(Overlap\LA,P (L c )) = (3.9) - ‘£ £4±£ 1=! - e M 1- £ ) - = ^ + / ( L d) (3-H) Lc 62 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Given document change probability Pd and using conditional probability calculus we can also find that P con « = P(O verlaP) = P(- 0ve£ ’’lL* ) (3.12) An estimate for P e r r requires knowing the probability of a copy of A being introduced as a result of a change. It can appear either spuriously or by intentional replication of a text fragment. In the first case we will always choose the anchor length to ensure that the anchor text is unique in D and D' with small local changes. If we have enough evidence to believe that the anchor text was replicated elsewhere in D' on purpose, the algorithm can respond in a reasonable way by replicating annotations as well. Therefore we can approximate P r e c ~ (1 - Pc o r r )- 3.5.3 A nchoring algorithm The algorithm described in this section generates anchoring records for any given text atom. A version of this procedure adapted to Internet Explorer is used by Annotator 2.0. The algorithm contains two major parts. The first subroutine computes anchor A given document D , position P in D and preferred direction Dir. The anchor construction subroutine performs as follows, see Fig. 3.2(a). At first, the anchor is initialized by collapsing and positioning at one of the two atom boundaries. The boundary is decided based on preferred direction, so that initial expansion will be directed into the atom. The anchor is then expanded in the preferred direction. This process is accompanied by checking if the resulting string is unique in the document. If it is, the function returns the anchor. When the expansion is not possible due to encountering the end of the document, the direction changes and the growth continues on the other side 63 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Start No Y e s A ':» Expand(A',Oir) R := CourtteadReps(A') Fail No Yes N O Yes End Len(Range)>k2 Yes Type := aaDouMe Type:= aaSngle A ,:= CreateAnchor A , := CreateAnchor (Doc.Ranoe.LeftddRightJ (Doc.Ranoe.Left.ddRiohtl I i Aj := CreateAnchor (Doc,Aange.Right.dcf left) L := Len(Range.Text) C End (a) Individual anchor construction (b) Atom anchoring record construction Figure 3.2: Anchoring algorithm. of the anchor. This logic ensures the maximum possible overlap between the atom and its anchor. If the anchor length reaches threshold A q, we assume that the anchor is likely to be situated in a repeating text portion. At this point we terminate the expansion and compute the number of anchor string repetitions preceding and including the anchor. The algorithm returns with A. If expansion in the opposite direction fails, this means that anchor string is equal to the document and is therefore unique. Expansion units for this subroutine can be characters, whole words or any other met ric allowing for position comparison. Annotator 2.0 experience shows that working with words is fast and convenient. The Annotator 2.0 version of this algorithm behaves slightly differently. It tries to expand in the opposite direction and see if it helps making the an chor string unique. The expansion step is defined statically and is greater than one word to accelerate the process. Upon finding a unique anchor, it may be useful to additionally extend it a small amount to decrease the spurious component of P e ru - 64 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. The main algorithm function computes an anchoring record and calls on the anchor creation subroutine, see Fig. 3.2(b). The anchor length is compared to £2- Two anchors are created for long atoms, one for the starting point and one for the ending. Only one anchor is created for short atoms and the atom’ s length is recorded. The choice of ki and fa is based on document statistics and will be discussed below. During the recovery phase, anchor text T is searched R times and offset is applied to find the atom boundary position. If the substring is not found, we can try to find a similar one. In case of failure, the atom is declared orphan. Double-anchor atoms become orphans in the case when at least one of the two anchors are not found or both anchors are found but their positions are reversed. 3.5.4 S tatistical anchor construction analysis Def. 3.5.2 A minimal unique string MUS(D, P, Dir) in text document D is a string starting at position P and continuing in direction Dir; the string is unique in D and any prefix of the string is not unique in D. Let us determine the threshold ki at which the anchoring algorithm detects possible text replication and falls back to repetition counting. Consider Fig. 3.3(a) that shows an M U S distribution in a typical document calculated for all possible positions P. For the purposes of Annotator 2.0 evaluation, a typical document represents a hypertext document several pages long and written in English. The length in the figure is specified in words. The calculation is case insensitive and punctuation was retained. For ease of perception the y axis has logarithmic scale and shows M US counts rather than probability. The 65 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. document contains no purposely-repeated portions of text. In the histogram the mode equals 2, average is 2.4 and the maximum M U S, M M U S s t d > is 7. Fig. 3.3(b) shows the previous document with an 11-word sentence repeated. Notice that the skew and M M U S become larger. Fig. 3.3(d) illustrates how text replication causes the maximum M U S length to grow. Consider a set of M US starting within the repeating portions. Assume downward direction. For each downward M U S there exists am identical upward MU S. Therefore we cam assume one direction amd not be concerned about directions anymore. M US in repeated portions must continue to the end of that replicated portion plus a few extra words that will m adce it unique. The repetition length in Fig. 3.3(b) is L r e p = 11 and M M U S is 12 indicating that just one extra word is enough to make it unique, L m m u s = L r e p + 1. Also, for each position within the repeated text there will be N r e p M U S of identical length, where N r e p is the number of repetitions including the original. This is evident in the Fig. 3.3(b) distribution’s tail showing severed M U S of length 2. The taul effect becomes even more apparent if we plot a histogram of Fig. 3.3(b). Fig. 3.3(c) shows a second-order histogram of M U S distribution. The peaik in bin 2 counts the M U S lengths positioned in the N r e p — 2 repeating portions. We cam set ki to the meotimum no-repetitions M U S length from Fig. 3.3(a). M U S exceeding this vadue would indicate that the string in question is likely to be explicitly replicated by the author elsewhere in the document. At this point the adgorithm cam justifiably fall back on repetition counting. Repetition counting keeps anchor string length within practical limits. This is importamt to ensure fast searches during the recovery phase. However, this workaround is more prone to failure since the exposed length grows, L'e = L e + (N r e p — 1 ) - L a s* N r e p - L a • That increases P c o r r ~ Pc o r r * N r e p . 66 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 1 2 3 4 5 6 7 7* MEljngffinWbrtt (a) Typical MUS length histogram 1000 _ 1 0 0 3 e o 1 0 1 1 2 3 * 5 5 7 8 9 10 11 12 12* MUS isngfft in Words (b) MUS histogram with an 11-word frag ment repeated 1 2 3 4 5 0 7 8 9 9 - MJS LangVi Count Longest M U S #1 Longest M U S #1 --------- ‘l --------- & A b e.. Xocrj... A b e.. yyyxi— TextT C O py of textT (c) Histogram of (b) (d) MUS length can substantially crease after text replication Figure 3.3: Minimal unique string. 67 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Another important related question is the one of determining & 2 , which is the atom length threshold distinguishing SSA from DAA. According to DCL, change probability depends on the atom length. With two anchors, anchor corruption probability depends only on the sum of anchor lengths, P c o r r = a • C ^ ai + £ .4 2 ) -F b. Note that we allow the atom to change and update L e definition for DAA L e ,d a a — L a i + L a 2 - To a first order approximation, in this case the atom can grow to large sizes without significantly co m p ro m isin g the ability to recover it. This justifies our desire to use double-anchored atoms as often as possible. On the other hand, the smallest practical atom length to choose DAA has a bound. In a short atom, the two anchors could overlap or even use identical M SU text, which is undesirable. Therefore, we can set to a minimal value that makes the expected overlap near or equal zero, £2 ~ 2 • M M U S std • Note that, there is also a consideration that having one search versus two is preferable with respect to the system performance. 3.5.5 P erform ance The algorithm has been implemented in Annotator 2.0. In most cases its performance is quite robust and meets design expectations. The algorithm complexity for each of the two phases is equal. In the construction phase, since the number of expansions is bounded the anchor creation complexity is equal to Oc o n str — Osearch{Ld) * N ai where N a is the number of anchors to be constructed and Osearch is the complexity of the implemented search. The recovery phase also relies on search and its complexity is also Orec = Ose ar c h(Ld) ■ N a - Annotator 2.0 works on entire words, ignores text case and locates anchor words sequentially using search interfaces provided by IE. 68 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Annotator 2.0 experience indicates that there is a certain class of documents where the algorithm fails persistently. These documents are certain fast-changing, fine-grained online HTML documents, such as news site front pages. Typically such pages contain a list of short headlines that change dynamically on a daily or even hourly basis. Headline mark-up disappears when the original text is replaced with new one. Even when the marked-up portion of the original remains, that mark-up can become irrecoverable if the atom is short and its anchor text overlaps an adjacent headline. This shortcoming can sometimes be worked around by truncating the anchor string to match the atom’s length and using it for search. 3.6 P ro to ty p e softw are Annotator 2.0 was developed based on the results of the functional analysis and employs the proposed backend architecture. It allows us to test and evaluate concepts of annotation database, distributed clumps, annotation index, ADB search and others. Our front-end experimental software is described in chapter 4. The design of an earlier version of An notator was reported in (Ovsiannikov et al., 1999). Although both systems target on-line annotations, their implementation and capabilities differ substantially. Annotator 1.0 was designed using HTTP proxy architecture. Annotator 2.0 is a seamless extension to Internet Explorer with a superior user interface and new features. 3.6.1 Features and user interface As an IE plug-in, Annotator 2.0 adds several components to the standard browser interface. These components are in-document markup elements, a sidebar and a toolbar, see Fig. 3.4. 69 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. ( ■ '.I * .|S |£ | H lillllfi Annotator /. X 3 - £ 3 http://www.cnn.com/200 S ^ <Cats & Dogs> ® My text note 4] http:/Aoolbaf.netscape.cc fSEAR (a) Annotation index with toolbar a f a m i household pets feuding o\ ■ H W . debuted as movie with a weekend gr million, acco: M y ,ext note I s - Sunday. Jul 14,20011 (b) Two inline text atoms with anno tations Figure 3.4: Annotator 2.0 user interface. To mark up a fine-grained text fragment the user must select a document portion and click on the annotation toolbar or in the context pop-up menu. The state of controls on the toolbar determines markup style and color. Available styles include underline, highlight, strikeout and bold and italic fonts. To add an annotation the user must click on an icon that accompanies the markup of interest and type in text. It will be displayed as a floating tooltip. The sidebar control represents annotation index. It is split into two windows. The first window shows a tree of currently open document frames and associated clumps. If a clump is annotated, its entry will display the text of the annotation. Otherwise it will show the text of the atom in angled brackets. Annotator’s database can be shared, which allows seeing markup by other group members. The second window shows a tree of search results and a text input control to enter search keywords. Clump tree nodes can be expanded to show more linked atoms and annotations. The user can add clump elements via dragging. Nodes can also be edited, deleted and unlinked. The URL of any node can be copied to the 70 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. clipboard. Lastly, we can instantly navigate from any node to the markup in the source document and back by clicking on the node and markup icon respectively. 3.6.2 Im plem entation overview Annotator 2.0 was developed using Borland Delphi 5. It employs a two-tiered architecture where the client is seamlessly integrated into the IE host along with a lightweight database wrapper. The shareable annotation database comprises the second tier. Annotator makes intensive use of industry-standard technologies. EE integration is achieved by developing a custom browser helper object (BHO). The BHO uses document object model (DOM) interfaces to create inline markup and insert special ActiveX icon objects representing annotations. The sidebar and toolbar were implemented as Explorer band objects (EBO). A script helper object (SHO) was introduced to enable EE context menus by providing script access to the BHO. Active data objects (ADO) were chosen to implement database connectivity. The default database type is Microsoft Jet 4.0. Let us examine the key objects in more detail. The browser helper object is central to the Annotator 2.0 architecture. It coordinates the logic of retrieval, creation, insertion or deletion of mark-ups on every page. Upon startup, each IE main window runs registered BHOs using apartment-threading model. That allows direct access to the main IWeb- Browser2 interface, the frame and document models and event interception. Host browser events inform Annotator BHO about key page load and change events. Explorer bar objects (EBO) load independently from the BHO. Unlike BHO these are GUI-enabled visible elements that E ire specially registered to be properly recognized and started. Upon loEiding the objects locate each other and connect to the BHO event sink 71 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. that sends markup change alerts. EBOs call on the BHO interfaces to manipulate clumps or invoke search. Due to IE specifics and demands for speed and precision the in-page annotation markup code has required a high level of sophistication. A separate library handles all mark-up insertions, deletions, anchor creation and location. All mark-up in every frame of the current document is kept synchronized to the set of database records. The anchoring algorithm was implemented as reported. It ensures a very good performance and a graceful degradation on pages with large amounts of repeating text. Annotator stores anchor, mark-up, link and all other information in a database that is accessed through a plug-in ADO-enabled wrapper. In a 3-tiered architecture the wrapper would be running in a COM+ environment and comprise the middle tier. However, for personal and limited group use the additional maintenance may be too complicated and thus was deemed unnecessary. Other components in the package include plug-in dialogs, which implement custom preferences forms. New dialogs can be added by registration. Typical operation cases for Annotator 2.0 include page loading, creating a new note and deleting a note. In the first scenario, the BHO detects when an IE open operation has completed for each and every form on the page. The database is queried for records for the current frame set and the frame set mark-up is synchronized with the record set. The BHO performs mark-up insertion and deletion in a single mark-up synchronization operation. The synchronization subroutine locates all existing mark-up in the page, if any, and performs creation, removal or update. Once the synchronization is complete, the BHO fires a ready event that is used by the Explorer band objects to update their state. The sidebar draws an ordered tree reflecting the current set of frames and notes. 72 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Objects like EBO and SHO can call BHO to create or delete mark-up. To create a mark-up the BHO locates highlighted text using dynamic HTML text range interfaces and calculates anchors. On success the BHO adds a new record to the current record set and calls on the database wrapper to save the data. Finally, the BHO performs the synchronization operation that will add the mark-up and the accompanying ActiveX icon to the document. Annotation deletion is performed in a similar way. Clumps can be linked and unlinked on the sidebar EBO using a drag-and-drop interface. The linking bypasses the BHO and calls on the database wrapper object directly. 3.6.3 E valuation The evaluation addressed Annotator 2.0 features, behaviors, design and performance de tails. It is based on experience of several users carrying out a set of tasks. The first task consisted of researching information on-line on a given topic using personal annotations. After that we used annotation database search to locate and show the best findings. The remaining tasks were similar except that the findings had to be communicated using public distributed clumps and the annotation search was performed on unfamiliar public records on a different topic. In our choice of tasks we looked for use cases typical for Annotator. Online research and collaborative sharing seem to be especially suited for our evaluation. As evidenced by the systems review, both tasks can involve annotation and are quite popular among the online community due to uncomplicated access to large numbers of documents and instant connectivity to other users. Online research included activities of searching for document 73 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. sources, scannin g and skim m ing documents and annotations to collect, organize and an alyze information. The need for communication required summarization and elements of composition. 3.6.3.1 Features An important conclusion concerns the markup granularity. Fine-grained atoms were help ful only when the research topic was very specific. Otherwise the users will prefer book- marking entire pages or websites. For this reason annotation systems should provide support for atoms of both fine and coarse granularity. In our evaluation coarse-grained book-marking was not available. To work around this restriction entire documents were tagged by marking up their headings. The index panel displays annotations attached to the current page. This helped navi gation in large documents. It was also instrumental for clump construction using linking. Public annotations require atoms to be well organized to ensure good communicability. Distributed clumps are frequently used in this task. They required careful assembly and were useful for fast search and navigation. On the other hand, personal annotations did not require much organization and clump linking occured less frequently. Atoms in a clump should appear in a fixed order for consistency. The user may want to order atoms by relevance, chronologically, topically or any other suitable way. Otherwise the distributed clump implementation was considered adequate. During research of a new topic the quality and importance of individual intermediate findings may not be recognized at once. In such case the reader can create markup without a solid understanding of its true relevance and quality. Such markup tends to clutter the 74 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. database. To decrease the number of garbage records, explicit annotation and especially linking can be delayed until a good understanding of the topic is achieved. We could also use temporary annotations that will be deleted automatically unless explicitly validated. The latter can be done by using the annotation index to display the recent annotation creation history. Having a history of recent document access may also be helpful. A clump consisting of a single atom can contain too few terms. Finding such records using ADB search can be very difficult unless clump context indexing is used. A reader searching for his own annotated clumps has a better chance of success since he may be able to recollect the right terms. Searching for small clumps created by someone else is much more challenging. Distributed clumps have more components and are less subject to the lack of terms problem. In this case search results can be satisfactory. Inserting additional potential search keywords into notes can improve the retrieval even more. Also, the quality of annotation search tends to get better with practice. Search for fine-grained elements was more effective when search topics were specific. The content of the ADB was too special and particular for queries on general subjects. On-line search engines were much more appropriate for that kind of task. In this respect ADB acted to complement global search engines. We noticed that annotation organization by topic is a matter of personal preference. For collaboration to be efficient, public ADBs should have predefined annotation classi fication. This may appear too restrictive, but it does provide a uniform structure for team use. If necessary, individual users can work around this limitation and use their own taxonomies locally by making clump copies. 75 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 3.6.3.2 Design and performance Several observations concerned the technical side of Annotator. Maintaining and accessing a shared database was quite convenient. It proves to be a solid and reliable architectural design. The speed of database access and search was acceptable. The evaluation was conducted using a Pentium II 366MHz computer with 192 Mb of memory running Internet Explorer 5.5 and Microsoft Access 2000. There was no perceptible slowdown displaying text pages up to 50 Kb long. Retrieval properties of the search engine were not evaluated since it is provided by the database engine and is not application-specific. Application interface was found to be user friendly and well-suited for on-line anno tation tasks. However, Annotator 2.0 does not provide sufficient support for overlapping atom markup. This is especially true when two or more people annotate same page frag ment. As expected, annotation of documents having dynamic URLs caused problems. Such documents do not have a single unique URL, but rather the URL changes each time the page is accessed leading to annotations disappearing. This problem underscores the need for document location insensitivity and can be resolved by implementing associated routines. 76 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Chapter 4 F ront-E nd D esign 4.1 In tro d u c tio n Recent developments in portable personal computing have resulted in the emergence of new affordable portable personal computing systems featuring pen-input, large full-format displays, wireless connectivity, lightweight design and computational power equivalent to a typical notebook PC. Such systems can enable applications that make use of handwritten input including handwriting recognition and freeform drawing. These services will be provided at the operating system level with the release of MS Windows XP Tablet PC edition (Microsoft, 2001c). In this situation the task of advanced system design for human electronic annotation of documents (HEAD) has acquired substantial significance. With few exceptions, experimental and commercial annotation systems developed so far have not enjoyed substantial success and widespread use. They could not support active reading (Adler and van Doren, 1972) by combining reading with annotation in one effortless process (O’Hara and Sellen, 1997). The lack of adequate hardware platforms was one of the key factors leading to this situation. 77 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. We continue the topic of HEAD system design started in chapter 3 and focus on the system front end. Front-end design deals with user interface issues for information input and output. We will describe practical approaches and algorithms that enable electronic document annotation with a high quality visually appealing dynamic markup display and natural markup input with implicit element detection. In general, existing systems de veloped so far have not been concerned with generating markup comparable to what individuals can produce when annotating paper documents. Many of them are limited to having a fixed markup layout also known as surface inking. Examples of such sys tems range from the popular, commercially available Adobe Acrobat (Adobe, 2001) to the experimental XLibris (Price et al., 1998; Schilit et al., 1998) and Dynomite proto types (Wilcox et al., 1997). A fixed layout is attractive for its simplicity, but has many inherent disadvantages. Once annotated, the document cannot be changed without com promising the markup. Pen scribbling may have a low visual appeal, especially when the markup is to be demonstrated to another person. Few systems (Phelps and Wilen- sky, 1996; Phelps and Wilensky, 1997) treat markup other than as a graphical layer. It is a rarity that markup elements are recognized as structural entities to assist intelligent pro cessing (Price et al., 1998; Schilit et al., 1998; Wilcox et al., 1997). Other systems support flexible layout, which is generated by a web browser. IMarkup (iMarkup, 2000), Annotator (Ovsiannikov et al., 1999; Ovsiannikov and Arbib, 2000) and ComMentor (Roscheisen and Winograd, 1995) are just a few examples of this class of systems. Our approach specifically concentrates on generating visually appealing markup closely resembling that of handwritten annotation of paper documents. The markup is maintained independently of the document format and contents. The layout supports very dense 78 with permission of the copyright owner. Further reproduction prohibited without permission. overlapping markup including cases where annotations generated by several people are superimposed without causing clutter and ambiguity. Free-form input can be translated into standard marks to make it readable for public use. The algorithms are aware of annotation element structure and enable intelligent markup processing. Finally, the front- end can interface to automated document annotation applications to provide quality visual output capabilities. The algorithms presented in this chapter are motivated by the case study presented in chapter 2. In that study we analyzed the appearance and functionality of human hand written annotations on paper documents. Let us briefly review relevant key concepts. Annotation markup consists of elements of three types. In a typical scenario the anno tator selects fragments of the document and attaches remarks. Selected fragments are called atoms and the remarks are called annotations. The associations between atoms and annotations are called links. In general, one annotation can relate to one or more atoms. The set of annotated atoms is called a clump of atoms or simply a clump. A clump of atoms together with its annotation forms an annotated clump. Text atoms are created using selectors, which can be inline or marginal. Other atom types include pictorial, formulary and tabular. Atoms are distinguished by their granular ity and size. Granularity refers to selection precision. For instance, margin selectors tag document text by the line, while inline selectors use per-character accuracy, see Figs. 4.1 and 4.2. A selector consists of a body element and two endpoints. The type of body influences visibility. Margin selectors are also called ranges and can be open, closed and detailed. Closed ranges have angled endpoints, while open ranges do not. Detailed ranges 79 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. are open or closed ranges paired with inline bracket selectors to achieve the character granularity of selection. Unlike annotations, notes do not associate with a particular atom. The three annotated clump elements can be marked explicitly or exist implicitly. For instance, an annotation can have an implicit atom by being located close to a paragraph. Such an annotation is called an inscription. Conversely, an atom having no explicit annotation is called a selection. Explicit linking is achieved by means of drawing a line, using a reference symbol or a certain color. Implicit links are commonly created by proximity and aligned overlap cues. The latter typically designates the atom and annotation being aligned so that their vertical projections overlap. Each of the three element types can carry associated emphasis and semantics. In general, to create an annotation the person must tag document fragments, record a statement and establish correct associations while keeping ambiguity, effort and page clutter to a minimum. These three requirements are known as the minimal markup prin ciple. The requirements are contradictory, since less effort will produce fewer markup and associated clutter, but at the expense of clarity. For example, to reduce the effort annota tors frequently utilize implicit components and relations, which makes annotation markup inherently ambiguous. HEAD front-end design must dead with user interface input-output tasks. The output is realized as a markup layout engine. In our investigation of layout principles we cover theoretical bases and practical design details. We have constructed a pilot implementation of the algorithm, sample outputs of which are presented herein to validate the approach. 80 with permission of the copyright owner. Further reproduction prohibited without permission We also address the construction of the input interface in our analysis of issues concerning the recognition of implicit markup elements in user input. 4.2 M ark-up layout engine Annotation markup presentation is a special problem for electronic annotations. The presentation module is part of the system’s front-end and the quality of its output has a great influence on the application’ s appeal to the user. In our design we regard visual appeal as the most important goal. To achieve a high quality we model markup styles and layout techniques after human handwritten annotations on paper documents. In addition we pay special attention to accommodate dense overlapping markup. Unlike annotations on paper, electronic annotation applications may need to display annotations from many people or applications simultaneously. Dense overlapping markup capabilities come in handy in such situations. The markup layout engine accepts annotated clump records as input from a system back-end, see chapter 3. The back-end contains an annotation database (ADB) that main tains records with annotation contents, atom location and associations between the two. The atom location information is encoded to be insensitive to small document changes. Given a text document and a set of annotated clump records a back-end anchoring pro cedure recovers exact atom boundaries. Based on annotated clump records and location, the layout engine determines selector and annotation styles, their actual positions in the layout and how they are linked into clumps. 81 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. The layout algorithm description starts by defining functional requirements, examin ing available markup element styles and constructing a representation framework. Subse quently we formulate the layout procedure as an optimization task and construct necessary estimators. The optimization function is strongly influenced by style considerations, which are examined in depth and accompanied by examples. A computational solution and an experimental implementation are presented after that. Due to a number of practical con siderations the optimization procedure is subdivided into several tasks that are executed in sequence. In conclusion we demonstrate sample results to validate the approach and showcase its capabilities. 4.2.1 Functional specification and elem ent style review Annotation-enabled systems must provide certain basic means for presenting annotated clump elements. To enable clumps of atoms we must provide means to select document fragments of any desired size and granularity. In addition, we should have adequate func tionality to convey selection visibility, emphasis and semantics. The system must accom modate annotations, notes and capabilities to link elements into annotated clumps. Ac cording to the minimal markup principle, we should also provide for implicit components and relations. Chapter 2 reported on techniques and individual marking appearance to fulfill the above requirements. We can use the practical set of selectors shown in Fig. 4.1. Highlight, brackets, underscores, arches and box drawing can realize fine-grain inline atoms markings. Underscores and arches have a medium visibility while the visibility of the highlight is higher and that of the brackets is lower. The box, the double and thick underscores 82 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. can convey emphasis. The strikeout and wave underscore can carry additional common sense semantics of deletion and uncertainty. Various types of ranges can serve as margin selectors of coarser granularity. Fig. 4.2 shows a number of open, closed and detailed ranges in different configurations. Detailed ranges combine large size selection capabilities with fine granularity. Closed ranges have bracket-like endpoints and their application differs from open ones in a subtle way. Although each type can select text, open ranges can also appear as an alerting symbol for the presence of important information. To that purpose it can accompany inline selectors to make them more visible. Closed ranges emphasize their boundaries and act strictly as selectors. Figs. 4.2(d) and 4.2(e) show an example of proper detailed range construction. In Fig. 4.2(d) the open range height matches that of the bracket selector. In contrast, the height of the closed range in Fig. 4.2(e) should be adjusted for better appearance. Similarly to inline selectors, ranges can carry semantics, emphasis and differ in visibility. Thick and double ranges imply stress. The closed range with no body element has a lower visibility. The wavy vertical line may suggest uncertainty. The choice of selector style is primarily, but not entirely, dictated by atom properties such as length, required visibility, emphasis and semantics. Atoms can assume sizes and granularities larger than of ranges such as page, chapter and entire document. However in this particular case we will ignore large atoms and focus on page layout only. For annotations, margins, footer and header can provide the necessary space. Lastly, implicit links can be constructed using aligned overlap and proximity cues. 83 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. quick H K H fox quick brown fox quick brown, fox (a) Highlight (b) Underscore (c) Under-arch quick Brown fox quickjbrownjfox quick|brown|fox (d) Overarch (e) Brackets (f) Box quick brown fox quick brown fox quick brown fox (g) Thick underscore (h) Double underscore (i) Strikeout quick brown fox (j) Wavy underscore Figure 4.1: In-line selectors: a) highlight has higher visibility; b) underscore, c) under-arch and d) overarch have average visibility; e) brackets have lower visibility; f) box, g) thick underscore and h) double underscore convey emphasis; i) strikeout and j) wavy underscore carry additional semantics. 84 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. The quick brown fox jumped over a lazy dog. (a) Open ’ The quick brown fox jumped over a Jazy dog. (b) Closed r The quick brown fox jumped over a Jazy dog. (c) Closed with no spanning element Thefquick brown fox jumped over a lazy dog.} (d) Open detailed The(quick brown [fox jumped over a . i lazy dog.} (e) Closed detailed The quick brown fox jumped over a lazy dog. (f) Open with double line The quick brown fox jumped over a ,lazy dog. (g) Closed with thick line The quick brown fox jumped over a lazy dog. (h) Open with wavy line Figure 4.2: Margin selectors also known as ranges: a) open, b) closed, c) closed with no spanning element and lower visibility; d) open detailed, e) closed detailed with adjusted height; f) double and g) thick lines express emphasis; h) wavy line can imply uncertainty or emphasis. / Y ‘ - Y , - Y . \ Y 1 Figure 4.3: Atom markup layers. 85 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 4.2.2 M arkup visu alization constructs Atom selector markup can be conveniently expressed and manipulated using so-called atom markup layers. The layers stack up vertically and horizontally, see Fig. 4.3. Horizontal layers are numbered Y fc, k e I. These layers accommodate inline selectors. Zero index corresponds to the original text line itself. With the text flowing top-to-bottom and left- to-right, negative indices correspond to upper layers and positive to lower. Vertical layers are numbered Xk, k E l, and accommodate margin selectors. X q is an imaginary layer that corresponds to the text body. X-layers with negative indices are located on the left margin, while those with positive belong to the right. The set of all markup layers in use is called a layer stack. Its configuration strongly influences many layout aspects such as appearance, selector style use and the maximum achievable density and overlap of markup. 4.2.3 Layout generation as optim ization problem The minimal markup principle that we encountered in the human handwritten annotation theory (HHAT) strives to reduce ambiguity, clutter and effort, see chapter 2 and Fig. 4.4. In the case of computer-generated markup layout the human effort is irrelevant unless we are talking about the application developer. On the other hand we must be concerned about ensuring a certain layout style. A layout style consists of a number of rules governing choices of element style and location to produce visually consistent and sensible output. To assess the overall quality of a layout and compare different solutions we can define a global function Q that takes into account these three factors. In this function A, S and C 86 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. reading the individ reading the individ reading the individ K ej triangle symbols in K ej '^fianglJ^ymbols in Heu triangle symbols in Pouut was in the occipita po7»A was in the occipita poiwt was in the occipita right side. This fine right side. This fine right side. This fine (a) Too few markup (b) Too much markup (c) an optimal solution creates ambiguity ensures clarity, but cre ates clutter and re quires human effort Figure 4.4: The minimal markup principle. are numeric estimators of perceived layout ambiguity, style and clutter respectively, k is a factor that allows trading clutter and style for ambiguity and vice versa. Q = (l-fc)-A + fc-(C —5);0< fc< 1 (4.1) In order to evaluate Q we must provide detailed models of ambiguity, style and clutter (ASC). ASC values are determined based on the appearance of individual elements and relations between them. For simplicity, we will limit evaluation of relations to binary pairs. Let us temporarily assume that ASC estimators possess the additive property so that the perceived ASC-value of a layout approximately equals the sum of perceived values of individual factors. In the formulas below N el is the total number of markup elements in the layout. When computing Q we assume that the layout is complete in the sense that all input annotated clumps are represented. A = £ £ A ij (4.2) 1 < i< N E C . l < i.j< N E L ,i< j s = £ s < + £ (4 -3 ) 1 < i< iV g£ 1 < i J < l ^ E L 4 < j C = £ c t + £ Cij (4.4) 1 <i<N 'EL 1 < i j < N e l 87 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Consider the individual ASC components. Clutter C % of the individual element i is de termined by this element’s size and style. The length of the body element and type of endpoints are two important factors for selectors. Clutter C ij can arise when two marks of certain styles overlap or abut each other. The layout style 5 is an artificial construct that conveys preferences such as the favored margin, inline and margin selector style. 5 biases Q so that layouts complying with the style preferences have a better Q score. S is zero for markup consistent with the chosen style and is negative for all others. Just like clutter, layout style is calculated for each element and pairwise relation. Finally, ambiguity stems from dubious markings and dubious relations between elements, such as overlapping selectors and atoms visually associating with more than one annotation. Besides evaluating layout quality, Q can also act as a fitness function in a layout optimization task. The layout construction algorithm can achieve its goal by minimizing Q. One way to obtain an optimal solution is through search. The solver would look for an optimal or near optimal solution in the state space. The state space is the closure of variables determining individual layout element properties such as mark style and location for each element. Once an optimal state is found, the engine will output the corresponding layout. Several issues may complicate the direct use of the layout quality function Q. Primarily, it forces us to juxtapose clutter and style estimators of three disparate kinds - inline, markup selectors and links. This seriously complicates the choice of values. Suppose that among other element types we would like to use underscores and margin annotations. Comparing their contribution to clutter and style is not straightforward. We cannot say with certainty if their contributions are equal, greater or less. If the choice of the number 88 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. is erroneous, we will see unwanted inline clutter optimization at the expense of margin annotation clutter or vice versa. Our experience with the pilot implementation indicates that we should group markup by location and assign comparative values only within those groups. These groups are inline, margin and link markup. Comparing clutter and style contribution within these groups becomes uncomplicated. For instance, we can argue that a single underscore should contribute less clutter than a double underscore of the same length and shape simply because it has one line instead of two. This makes clutter estimates much more robust and suitable for practical use. The markup can be grouped due to several reasons. Firstly, we will observe that the choices of inline and margin markup location are independent of each other since they are primarily influenced by atom positioning. Secondly, we will witness that margin markup placement style rules already provide for low link clutter. Based on that we can split the Q function into the three components to minimize them independently and sequentially. We also split the joint state space accordingly. Q — (1 — fc) * (A[ + A m + A i) - 1 - (4.5) +k • ((C/ + Cm + CL) - {St + S M + SL)) = (4.6) = Qr 4- Qm + Ql (4.7) Here Qi, Q m and Q l are three functions for inline, margin and link markup. We ignore footer and header clutters since although it must follow style conventions, it is not subject to the minimization principle. In the pilot implementation we optimize each component of Q separately. This allows relaxing the assumption of additivity. At first we find a 89 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. near-optimal solution for inline markup only. In this phase the solver ignores annotations, margin marks and links thereby making the state space much smaller. Subsequently we find an approximate solution for the margin layout. At this stage annotation and range positioning is chosen. We ignore the inline selector solution but apply style rules that will help linking. In the third phase we solve element linking based on the inline and margin markup solutions. In conclusion, based on all previous solutions we can beautify the layout by performing the tasks of range merging. Our experience with the pilot implementation strongly indicates that the ambiguity must be minimized not only in the global sense. In addition, local ambiguity must be limited. During the linking phase we will check each markup element and relation for acceptability in addition to the Ql function. Before we address the layout algorithm in detail let us define some criteria to assess input complexity. Element density D is the number of input record elements related to fragment size. Our evaluation demonstrates examples for various values of D. The density can be computed either locally, for a page or entire document. It is convenient to specify it as average per page and peak per line. In the formula below N l is the number of lines and NAn is the number of annotations. Note that layout quality for a given D depends on the choice of style. D * - % t ( 4 ' 8 ) ^ <**> D = D \t + E > A n (4.10) 90 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Element density does not account for overlapping and abutting elements. Such elements can be difficult to display when several of them overlay same location. Overlap level indicates how many atoms overlap an atom at a given position. Maximum overlap level is the maximum value calculated over a fragment. Overlap level is helpful to designate the peak capabilities of a given style. Nov.el is the number of elements overlapping a text fragment. O = Nov.ei — 1 (4.11) Fig. 4.5 shows a high-level simplified diagram of the atom markup algorithm. The pro cedure requires a set of input parameters consisting of a list of annotated clump records, a markup style table and page size parameters. Annotated clump records contain data entries for each atom, annotation and link. Atom data contain starting and ending posi tions for each atom, associated emphasis, visibility, semantics and some other properties described later. At the output, each atom is assigned a selector style. If the chosen style is a range, the side and endpoint style are defined as well. Annotation data contain text of the comment. Annotations are positioned and all elements are linked. A solver routine conducts search in three state spaces and evaluates fitness functions. We will address the construction of each state space below. If all markups can be accom modated without violating ambiguity, style and other constraints, the procedure reports success and outputs the highest-ranking state and associated quality estimator values. The resulting state corresponds to a near-optimal solution. The algorithm can also run in a faster incremental mode as discussed later. The layout engine presented herein has some simplifications with ensuing limitations. We do not support in-line annotations, position markings, atoms of page size and larger, 91 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Design style tables and evaluation functions Link markup placement Inline selector placement Pre-layout constraint checks and state space construction Range merging Margin markup placement Figure 4.5: Markup layout algorithm block-diagram. inscriptions and arrow links. For our purposes there are no breaks between pages. Markup simply continues on from one page to the next. All these limitations can be easily removed in a full-fledged implementation. 4.2.4 S tyle table con stru ction To construct layouts we will need a markup style table. Its purpose is to map atom properties to particular selector styles. Atoms can have two associated selectors, one marginal and one inline. The table has two parts, one for each type. Atom properties used in style selection include its length, emphasis, preference, source, semantics and visibility. Short atoms are assigned inline selectors, while those longer that one line’s length should use ranges. Emphasis is specified as none, strong and very strong. Values of the semantics argument can include ‘none’, ‘delete’ and ‘uncertain’. Visibility can be set to low, normal and high. The layout engine can simultaneously display annotations created by several people or applications. The source parameter indicates which party the atom belongs to. 92 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Preference is an index of style preference, starting with the most desirable. For instance, if the preferred style of index zero cannot be used in the layout for some reason, the application can request the next-to-best one by setting ‘preference’ to one. The style table should have an accompanying evaluation function that translates selector properties into the ASC value triplet. This function can take into account selector style, body length and endpoint type. The inline style compatibility evaluation function returns an ACS value triplet for each pair of styles. Its task is to detect conflicting styles that create clutter and ambiguity when used by overlapping selectors. For example, overlapping underscores create ambiguity of the atom boundary location. We should use this function to check abutting endpoints too. Under such circumstances two adjacent underscores may look like one continuous one, for instance. A higher A-value returned by the compatibility function in this case will direct the solver to use a different inline style for one of the two atoms. With respect to the two input styles the function is not reflexive or transitive, but can be symmetric. The layer stack framework described above is helpful in detecting markup conflicts. After defining which styles use which layers we can detect if selector endpoints and body elements overlap, invade inner layers or abut. For example, two overlapping selectors with body elements of conflicting styles occupying the same layer will be in conflict. Many popular inline selector styles, such as underscores and highlights, are not overlap- compatible. This is acceptable when the markup is sparse. However, it will create a problem for dense overlapping inline selectors. In such situations we can use arched under and over-scores, see Fig. 4.6. Arches can also be used to display overlapping ranges, which helps creating large coarse-grained overlapping atoms. The restrictions on overlap for this 93 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. selection type are less limiting. Firstly, we would like to prevent any two arches from almost coinciding. Secondly, to avoid clutter, the number of overlapping arches at any position in a particular Y layer should not exceed two. Thirdly, for aesthetic purposes the arch’ s width-to-height ratio should be limited to preclude flat lines. Here is a simple empirical formula that conforms to the requirements and can be used to compute overlap. O N is normalized to range from 0 to 1. OverlapH designates the horizontal overlap length. The style compatibility value is be a function of ON. r>\rt\r i f \ 2 • OverlapH (Aft, M2) ( A, * 2) Length(Mi) + Length(Mo) ' 0 < O N (M u M2) < 1 (4.13) By using over- and under-arches simultaneously we can achieve the overlap level of three. Using highlight and brackets increases it to five. Having arches on both margins can contribute four more overlapping selectors. Even larger numbers can be obtained by using more inline markup layers and additional bracket selectors. However, the latter measures can substantially degrade the visual appeal. Any overlapping bracket selectors must be explicitly indexed on both ends to guarantee clarity. All in all, we can see that the available means allow us successful handling of overlapping selectors. Style tables must be designed to accommodate a known number of sources. We must decide which source uses which markup style. We must also decide on what constitutes the base level of inline and margin visibility. To achieve a higher visibility we can use highlight-based markup. The downside of this style is that it does not handle overlaps 94 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. (a) Over-arch (b) Under-arch Figure 4.6: Arch selectors help handling overlapping atoms. well. We can utilize underscores and brackets for a lower base inline visibility. Several choices are available with respect to margin selector tactics. For low margin visibility we should not use ranges together with inline selectors, especially highlight. For medium margin visibility we can allow the use of ranges for large atoms only. Higher visibility can be achieved by using ranges with each inline selector. It is advisable to never mix different styles. A chosen style should continue throughout the entire document. The choice of a particular markup style set for a particular source is based on document type, selector density, overlap level and personal annotator pref erences. The HHAT markup function survey reports how annotators choose a particular style depending on the task at hand. Here are two style table examples for a single source. • Use inline highlight and no margin marks. This is achieved by assigning yellow highlight to all table entries and clearing all entries for margin selectors. Change color to red for emphatic selectors. This style is good for simple, visible markup of low to middle density with no overlaps. • Mark large atoms using ranges and possibly inline brackets. Small atoms should use underscores. Overarches are acceptable for overlapping inline selectors. Express emphasis using double underscores. For extra visibility use underscores for larger 95 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. atoms too. This style provides a moderate margin and inline visibility and is good for dense overlapping markup, see Fig. 4.10(c). Typically, annotator markup created by one person exhibits little or no overlap within in-line selector and margin selector groups. Two persons annotating the same document may mark same key clumps. To display markup from several sources simultaneously we should ensure a visibly clear differentiation. That can be achieved by reserving certain colors, texture and markup stack layers for each source. Layers for different sources should not be mixed or shared. Earlier markup should go on the inner layers. Markup on the outer layers should not intervene into inner ones. E.g. a box selection on the outer layers will cross inner V-layers and is therefore undesirable. All these rules must be incorporated in the style compatibility function. Annotation markup from several sources can share same margins and differ in font style. Annotations on the footer should be grouped by the source. Here are some two- source style table examples. • Source A uses V o for highlighting. Source B uses Y± 1 and X_i. No inline overlaps are allowed for A, see Fig. 4.10(b). • Source A uses V±i and X±i- Source B is assigned Y±2 and X±2 - Both sources can use denser, overlapping markup. Two-side margins accommodate more annotations. 4 .2.5 In lin e m arkup placem ent The state describing inline markup consists of styles chosen for each inline selector. In our implementation it is a concatenation of the preference variables associated with each atom. A straightforward search procedure would iteratively generate a next state and evaluate its 96 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. fitness until it covers the entire state space. A next inline markup state can be obtained by recursively incrementing the preference variable of each atom. More sophisticated search routines could benefit from starting from a near-optimal state obtained by assigning to each atom its preferred inline style from the style table. The evaluation function in each state checks every inline selector for individual style and clutter. The 5-value can be modeled as being proportional to the atom’ s inline markup preference variable. Pairwise relations between selectors must be checked for compatibility by applying the inline style compatibility function. Overall, we must make sure that ambiguity is low or zero for each inline selector and selector pair. If the ambiguity is too high that state should be discarded. For instance, suppose that default inline markup styles for short selectors are ‘under score, under-arch, over-arch, brackets’ in that order of preference. Suppose also that two short inline atoms overlap. Initially we assign underscore to all selectors. The evaluation function checks individual elements and pairwise relations and detects the overlap. For two overlapping underscores the inline style compatibility function assigns a high ambigu ity value thereby rendering the state invalid. The search continues to other states to try alternative style assignments until a highest-ranking one having ‘underscore, under-arch’ is encountered, see Fig. 4.10(c). 4.2.6 M argin m arkup placem ent In the second phase we work with margins to place ranges and annotations. The evaluation function seeks solutions that avoid conflicts, reduces clutter and increases implicit element 97 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. association. The latter is necessary to use proximity and aligned overlap cues to reduce expected link clutter produced later on in the third step. Annotations can be placed on left, right or both margins and on the footer. These lo cations constitute legitimate annotation markup layers. Unlike inline markup, annotation layers cannot overlap. Such markup can make the page too hard to read. In our study in chapter 2, annotators practically never used align-overlapping annotations. Each annotation markup layer can be characterized as either proximal or distal with respect to any given atom. Suppose that we have an annotated text atom somewhere on the page. Placing the note on any margin and aligning it horizontally with the atom can possibly create an unambiguous implicit correspondence. In contrast, barring a placement immediately adjacent to text, putting the note in the footer or header cannot lead to an implicit correspondence. In this case we call the margins proximal and the footer and header distal. The practical importance of this concept is as follows. Since notes have no associated clumps, they can and should be placed in a distal layer to avoid a likely, unwanted and ambiguous implicit association. Annotations should be located in proximal layers to utilize implicit correspondence that can obviate explicit linking. If for some reason an annotation must be placed in a distal layer, it must be explicitly linked to its clump. Annotations and atoms are assigned a vertical alignment. In combination, these two settings define the preferred style of annotation positioning with respect to its clump. The set of possible values consists of upper, lower and middle. If an annotation has a clump consisting of exactly one atom, their alignment points should ideally match, see Fig. 4.7. For clumps containing more than one element we need to construct an artificial representative atom that will be used for alignment. Depending on style settings, the 98 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. N ote X \ xxxxx f*. X X X L quick brown fox over a lazy dog Figure 4.7: Vertical alignment of annotation with respect to the atom. The tick marks indicate upper, middle and lower alignment points. representative atom can correspond to the first, last, middle or largest atom in the clump. It can also correspond to a group of merged atoms in case they are located close together. Typically, setting both alignm ents to upper and using the first atom in the text flow produces good results. Layouts having annotations misaligned with their clumps incur a style penalty. The order of annotations in the layout should not be ignored. In chapter 2 we have observed that the markup, which consists of selectors and annotations, is created incre mentally along the direction of the text flow. Therefore annotation ordering should follow the clump ordering. This rule should be enforced both on the margins and inline. The former can be checked by pairwise comparison of annotation locations and the locations of their representative atoms. For a clump further down the text flow the reader will expect to find its annotation further down on the margins. A style penalty is incurred when this rule does not hold. The inline ordering is enforced through so-called side-text areas (STA). In chapter 2 we defined STA as text located next to a margin, see Fig. 4.8. STAs allow the use of proximity cues to create candidate implicit correspondences inline for atoms fitting entirely on a single line. Annotators tend to put annotations on the margin closest to the short inline selector. We could also say that annotations gravitate towards their atoms. In our implementation if an inline selector fits entirely in the left or right third of a line and its annotation is 99 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Note A | Atom a | _ _ _ _ _ _ _ _ _ _ _ I Atom B | Note B I . Left STA. | . | Right STA. | Figure 4.8: Side-text areas work inline and combine proximity and aligned overlap cues to create candidate implicit correspondences. located on the opposite margin, that state will incur a style penalty. Annotation number one in Fig. 4.10(a) is an example of such STA gravity rule. When placing a range we have to decide on its side and style. Just like for inline selectors, the preferred range style for each atom is obtained from the style table. Several principles apply to range side selection. All margin selectors must be located on the same margin with its annotation. This rule must be strictly enforced. We should also check for style conflicts in the same X-layer. Two overlapping ranges of different styles will cause clutter and ambiguity. Notice how in Fig. 4.10(c) the double-line range is located on the right margin to avoid a conflict with the single-line open range on the left. Range height and style affect the margin clutter. We can make a practical simplification for clutter calculation of this kind. Since range height will remain same regardless of on which margin it is placed, its contribution to the clutter in each possible state will be constant. Therefore for state comparison purposes its calculation can be excluded. This is a good example of how clutter calculation can be simplified. In contrast, annotation height is a state variable and contributes a clutter proportional to the number of occupied lines. To each margin annotation we also assign one more state variable that indicates if it is collapsed. A collapsed annotation is put away into the footer leaving on the margin only a reference symbol, see annotations number one and two in Fig. 4.10(c). This also costs some style penalty, but effectively limits the height of margin annotations. 100 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. In summary, margin layout consists of margin selector and annotation placement. We can use one, both margins and none. Notes are put away on the footer. The state consists of annotation and range side, annotation shift, collapsing and range style variables. The evaluation function assesses each atom and annotation individually and for pairwise inter action using the following criteria. Annotations and ranges are subject to style penalties for the wrong side, collapsing and shifting. The latter is a function of distance plus a fixed threshold. To ensure candidate implicit correspondences, annotations must overlap their representative atoms and their ranges. Annotations and representative atom ranges must be on the same side. Annotation height contributes to clutter. Pairwise interactions contribute to style. Ranges of incompatible styles located on the same margin should not overlap. Regular annotations on the same margins must not overlap either. In contrast, the reference symbols of collapsed annotations can do that. Same-margin ordering rules check margin selectors and their annotations starting positions. Annotations are encouraged to gravitate to the STA where its representative atom is located. 4.2 .7 Link m arkup Part of the input data set, link elements specify information about atom and annota tion membership in clumps. These recorded correspondences between clump members are called actual. When the data records are translated into a layout the viewer can see markup elements forming perceived associations. Perceived associations result from inter actions of layout graphical elements. The interactions can appear as explicit or implicit. Explicit associations are created using linking. Linking typically indicates clump mem bership in a clear fashion leaving little chance for ambiguity, see Fig. 4.9(a). In chapter 2 101 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. N o te A 1 quick brown1 N o te a j quick brown (a) Hard binding (b) Soft binding N o te A quick brown fox N o te B (c) Two overlapping soft-bound clumps can lead to ambiguous associations N o te A quick1 brown fox (d) A hardbound element does not associate with a soft-bound Figure 4.9: Types of clump binding. we showed that implicit associations could arise from aligned overlap and proximity cues. Expressing actual correspondences through implicit associations avoids clutter but may create ambiguous situations, see Fig. 4.9(c). If a perceived association has a matching actual correspondence it is called true or correct Otherwise the association is called in correct or false. In certain cases the viewer cannot decide with certainty that particular two markup elements correspond. The goal of the linking phase is to guarantee correct and unambiguous perceived relations between markup elements while minimizing page clut ter and upholding style guidelines. In a correct and unambiguous markup all perceived pairwise element relations are correct, unambiguous and form a bijective mapping to the entire data membership set. Generally speaking, the ambiguity can. be completely eliminated and correct relations ensured by explicitly marking each annotated clump element with an index mark that is unique for that clump. However, a large amount of index marks can clutter the page. Let 102 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. us define a figure of merit c*, that conveniently indicates the quality of link placement. cl represents the ratio of the link clutter to the total the number of data elements. Our implementation utilizes numeric references for linking. If each link symbol contributes a unit clutter, cl is the proportion of data elements assigned a reference mark. C L NAt+N An ( 4 ' 1 4 ) To approach the link placement solution we need to define the concept of annotated clump binding. It determines the method of clump element association in a layout. There are two types of binding, hard and soft, that correspond to explicit and implicit association. A clump using hard binding has all its elements connected by some kind of link markings. The association must be unequivocal and may produce some clutter, see Fig. 4.9(a). In a soft- bound clump, all elements associate by means of location cues, see Fig. 4.9(b). Singleton selections, which are clumps of atoms consisting of only one atom without annotation, are called unbound. Such clumps cannot have correspondences in data records. Likewise, notes are also unbound. An unbound clump cannot have a link and in relation to other clumps it acts as soft-bound. Fig. 4.9(c) shows how align-overlapped markup can create ambiguities and false cor respondences. In the figure we cannot infer with a reasonable confidence which of the two annotations the atom refers to. The main implication of element binding is that it offers formal means of detecting and resolving ambiguities created by implicit associations. Suppose that we would like to check a markup layout for ambiguity and false associations. We need to examine the perceived relations between each annotation and atom mark. If both elements in the pair belong to the same clump, we consider the binding. The hard 103 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Atom binding Hard Soft Ann. binding Hard OK OK Soft OK Check Rij Table 4.1: Clump binding compatibility and resolution binding will guarantee a true correspondence. The soft binding does not do that and we need to estimate the degree of implicit relation Rij before making a decision. Soft binding is acceptable if Rij is sufficiently high on the absolute scale and is the highest among all other possible correspondences that each of the two elements can form with other soft- bound clumps. Otherwise at least one of the two elements will be predominantly and falsely associated with another clump unless we change its binding to hard. If the atom and annotation in the pair belong to different clumps we should check the binding of both according to Table 4.1. There will be no perceived association if any of the two elements is hard-bound. Fig. 4.9(d) illustrates this case. Note A does not appear to correspond to word ‘quick’ because the latter has a link symbol and the former does not. However, if both elements in the examined pair are soft-bound, the Rij check must be performed and its results applied as described above. Table 4.1 specifies element relations between atoms and annotations only, with no regard to atom-to-atom or annotation-to-annotation associations. The reason is as follows. Firstly, we assume that annotations cannot form strong perceived implicit associations with each other in layout and do not have correspondences in the data records. Our observations in chapter 2 indicate that this is a rule with few exceptions. Secondly, atom- to-atom associations can be ignored since atom membership is already defined by its relation to its annotations. In the special case when a selection may have more than one 104 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. atom and no associated annotation, the layout engine can work around it in three ways. The HHAT study indicates that annotators rarely make an attempt to indicate clearly whether a number of adjacent selections belong to the same clump or not. Therefore we can treat the member atoms as singleton selectors. If clarity is important, we can lock the binding of such clumps to hard. At last, we can convert selectors into ordinary annotated clumps by creating an artificial symbolic annotation. Rij estimates the perceived relation between an annotation on the margin and an atom. At least two factors contribute to the strength of the association, fey estimates how well the elements overlaps vertically with respect to each other. It is maximal for exactly overlapping markup of matching sizes, kff accounts for proximity. In the formula below LC is the average of line center positions for the atom. L is the line length. LC/L is the normalized position of the atom’ s horizontal first moment, or figuratively speaking its center of gravity on the line, kff approaches unity in situations such as shown in Fig. 4.8. Rij = kff • fey; 0 < Rij < 1 (4-15) , 2 • OverlapV(Arii, Atj) V ~ Height{Am) + Height(Atj) ; 0 ~ v ~ ( ' ) kH = l - \Side(Ani) - < kH < 1 (4.17) L t Side{M) = 0,1 (4.18) 105 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission The formula above applies to all explicit inline selectors. It also applies to explicit margin selectors if they are located on the same margin with the annotation; otherwise the relation will not exist. Implicit margin and inline selectors require a special treatment. Note that Rij is zero for elements that do not align-overlap and no check is necessary in this case. This observation allows increasing the algorithm speed by breaking the data record set into groups. The grouping is such that elements from any two different groups do not align-overlap. This allows running the layout separately for each group to reduce the size of search space. In the pilot implementation linking is conducted as search. Each clump is assigned a binding type variable and their concatenation forms the state space. Binding of certain clumps if assigned and locked before executing search. Any clumps that have atoms with ambiguous inline selector markup, such as overlapping brackets, axe hard-bound. Collapsed annotations and clumps that do not overlap its annotation are also assigned hard binding. Singleton selections use soft binding. Non-singleton selections should be handled according to the three methods addressed above. Link markings in the form of reference symbols can be attached to annotations and explicit inline selectors. Note that we do not link ranges due to a low visual appeal. The global linking goal function Ql is composed as follows. Unlike previous phases, here we do not have any applicable style regulations and Sl is set to zero. The ambiguity is a function of Rc,min and Rt,max> which are the lowest correct association in the layout 106 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. and the highest incorrect one. Since Rc,min and R[,max range from zero to one we can conveniently use the relative link clutter c l - Ql — — • (1 — fc) • (— Rc,min — (1 — R[,max)) + & • C£, = (4.19) = i - (k - 1) - (Rc,min + (1 - Rr,max)) + k-CL (4.20) During state evaluation we examine each atom and annotation individually and pairwise. Individual hard-bound elements receive a unity clutter since they have a symbolic link marking. Collapsed annotations already have a margin link symbol and do not contribute to Cl- If at least one of the elements in pairwise examination is hard-bound, Rc,min and Rl,max for that pair are set to unity and zero respectively. In addition to minimizing the Q l we guarantee a limit on local ambiguity. For each atom and annotation we enforce that it associates predominantly with its clump’ s annotation or atom respectively. In the formula below Rrp designates a constant threshold. If the inequality does not hold that state is deemed invalid. Rr ,max — Rc.min Rt (4.21) Styles containing implicit selectors require special compatibility checks and handling to eliminate ambiguity and problems that can arise in the linking phase. We can construct implicit styles by defining artificial transparent selector types to be used in the style table. Atoms having an explicit range and an implicit inline selector are subject to the following restrictions. If the range is a vertical line mark it may not overlap ranges of the same style on the same margin. Also the range cannot be merged with any other range. We will address range merging shortly. The clump should not align-overlap other annotations that 107 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. also have their binding locked to soft. Otherwise the linking procedure cannot guarantee an unambiguous solution. The atom boundaries must correspond to sentence boundaries. The latter must not share their lines with boundaries of any other sentence. Chapter 2 showed that it is natural for the reader to assume that implicit selections are likely to end at a sentence boundary. This rule can be tested in the pre-layout constraint-checking phase. In inscriptions annotations act as margin selectors, see chapter 2. Both margin and inline atom selectors are set to transparent. This constrains the layout even more. The annotation height must be less or equal than the atom’s and the latter must completely cover the annotation. Otherwise the annotation will be selecting the wrong text. The atom size should be limited to one paragraph. Larger atoms may have an ambiguity prob lem. Atom boundaries are constrained identically to those of range-only atoms. Likewise, inscription binding must be locked to soft and the clump must not align-overlap with other clumps with locked soft binding. The annotation on the margin must not overlap with any explicit margin selector. 4.2.8 C oncluding processing Margin selector merging concludes layout construction. The merging procedure can help reducing margin clutter Cm by bridging abutting range markings to eliminate endpoints. A pair of ranges can be merged without compromising ambiguity if they have the same style and have corresponding explicit inline selectors. For atoms having both margin and inline selectors explicit the range effectively acts as an additional highlight mark to increase 108 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. visibility. Therefore such ranges can be merged while retaining clarity. We can also slightly increase ambiguity by bridging short gaps with the height of one line or so. If the margin layout solution still contains overlapping ranges of incompatible style, we have to combine the selectors. Typically, a more salient style will override the less salient one. A warning should be issued during this action since it creates a significant ambiguity. To avoid this situation we can enable two-sided annotation or otherwise increase the number of margin markup layers available for range placement. 4.2.9 Im plem entation evaluation and perform ance To validate the approach and demonstrate its capabilities we have constructed a pilot implementation of the algorithm. The implementation realized all features described above except for range merging, implicit selectors and has a limited choice of range styles. The output contains markup positions, styles and linking. After obtaining a calculated layout the markup is applied to the original text for visual evaluation. In our evaluation we pursued two goals. Firstly, we wanted to produce visually appeal ing layout comparable to that of HHAT study standards in chapter 2. Secondly, we took existing annotated text fragments and tried to reproduce same or similar layout. Both goals were successfully achieved. Fig. 4.10 demonstrates sample results. The first example shows a low-density single-source markup. All three clumps in the layout use soft bind ing. One of the clumps is emphasized. Fig. 4.10(b) shows a denser layout coming from two sources and having overlapping atoms. One of the four clumps uses hard binding. Fig. 4.10(c) shows an even denser layout having up to three simultaneously overlapping atoms and two collapsed annotations. 109 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. list of notes. Annotations vary with respect to . Ann. Conventional annotations are carefully written out so # I that viewers other than the annotator himself can understand it with ease. Idiomatic annotations are Ann. characteristic of people of certain professions. In #2 (a) Low density Ann.1 an example of an idiosyncratic annotation* # / Ann. #2 E x am p les t h a t we h a v e s e e n i n t h i s s e c t i o n f e a t u r e (b) Two-source medium density with two overlapping atoms Ann. ' Ranges also differ with respect to the choice of the #/ vertical shape. Some annotators grefer^vertical2lines | (1,2) or square brackets. Others favor curly brackets or even tall parentheses. Occasionally the (yertical line in the closed range can be qo^^gfi^and the end- | | 3Ann. points appear as horizontal bar5) as shown in Fig. 3d. #2 (c) Higher density with up to three overlapping atoms and collapsed annotations Figure 4.10: Sample results calculated by the experimental implementation. 110 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission Overall, the results clearly demonstrate the algorithm’s capability to elegantly handle dense overlapping and multi-source markup. This functionality is not available in other applications. The sensitivity and stability of output quality to evaluation function param eters was quite good. We believe this was mostly due to splitting of evaluation function Q into three independent components. Consider the layout engine performance. A straightforward search-based implementa tion of the algorithm is too slow for practical use. The number states is on the order of 0(aN) where N = N m + N,\n is the total number of atoms and annotations and a is a constant. In addition, the state evaluation routine performs as 0(N2) to check all pairwise element combinations. Optimizing Q in several steps, using locked constraints and care ful design of the state spaces as described above, can increase the performance. Caching search results can also help. However, to achieve a qualitative improvement in speed we could utilize incremental annotation. Incremental annotation is the method used to create handwritten markup. Annotators do not plan their layout in entirety before applying it to the document but rather add clumps as they read on. In our case, incremental annotation utilizes the same layout procedure. Clumps are added one at a time and their appearance is frozen. The layout engine runs once for each clump. This reduces the overall complexity to 0(iV2). There is one practical issue concerning how the algorithm is used. The application must ensure that a layout is reproducible for the same document. Viewing the same document having a different look each time the user opens it can be confusing. Therefore implementations should not rely on random choices. I l l Reproduced with permission of the copyright owner. Further reproduction prohibited without permission A full-fledged implementation will need to extend the algorithm in several ways. Page and document-level granularity clumps should be provided. Inscriptions, annotations in line, on tables and pictures should be supported. The latter may require substantially extending the annotation framework. In our pilot implementation the ASC coefficients were chosen based on ‘common sense’. The coefficients can also be learned using various stochastic and neural network algorithms by inputting a large number of selected examples for a given annotation style. 4.3 In p u t interface A system supporting handwritten electronic annotations will require a user interface for markup input. HHAT observations in chapter 2 suggest that user input is governed by the ‘minimal effort’ rule. The user will create only a minimal amount of markup necessary to fulfill his objective at hand. This strategy saves time and reduces clutter. We have ignored this principle in the previous section since layout was computer-generated. How ever markup input interface design must take it into consideration. A minimal effort in markup input is typically achieved by making use of implicit annotated clump elements. Therefore the interface design involves two tasks: support of explicit markup entry and detection of implicit components. Consider the task of entering explicit markup. Fig. 4.11 shows an example state di agram for markup objects. Markup objects encapsulate visible markup and are created by writing or drawing in blank space or over text. Writing in blank space creates an an notation. Drawing a vertical line on the margins creates an atom with a range selector. A horizontal underscore in text creates an atom with an inline selector. Newly created 112 with permission of the copyright owner. Further reproduction prohibited without permission objects start from state SL. SL differs from the regular selected state S by allowing for accelerated linking of elements into clumps. In the diagram state transitions are marked with choices of input and associated responses. Types of input include tapping, double tapping, executing a pen gesture and writing or drawing. Such input is preferred over using menus since pen gestures facilitate active reading (Adler and van Doren, 1972) by avoiding visual distractions. Pen gestures should provide commands such as delete, unlink, show menu and show properties. The character input and drawings can be captured using operating system resources. Whenever there is a mark on the page residing in state SL, tapping another mark will add it to the clump if it is not already there or unlink it other wise. After timeout t£ , the accelerated linking is discontinued by transitioning to state S. In both of the selected states the user can also carry out boundary resizing and editing. Edit commands select, delete and input text in annotations using gesture sequences stan dard to word processors with handwritten input. Double tapping on a mark invokes its default action, such as displaying object properties dialog. Tapping, writing or drawing in a blank space or selecting another mark puts the current element back in the unselected state. Finally, executing a delete pen gesture over a mark erases it from the page. This description of a markup input interface gives us enough understanding to approach the second task of implicit component detection, which involves discovering implicit links and atoms. Let us see how the perceived relation strength estimation technique from the layout engine can be reused to this end. The implicit component detection procedure starts by examining the list of markup elements. Elements with hard binding are ignored since their clumps are explicitly con structed. The binding of all atoms is reset to the default value of unbound. Subsequently, 113 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Dd.gesture over ett Tap * ---------------- Tap blank space or select dlff. elt Dbl-tap/exec. # Write c -Vnone; In Man W-tap/exec. Tap dlff. elt. /link-unlinl^ Dbl-tap/exec; Gesture, edit resize Figure 4.11: Explicit mark object state diagram. Rij are computed for each atom with respect to each margin annotation. A soft link is added if Rij exceeds a certain threshold and is maximal for that atom. The procedure continues by examining all annotations that do not have any attached atoms. An implicit atom is created in each case with the vertical size matching that of the annotation and the atom boundaries located at sentence boundaries. Thus upon the procedure completion we have recovered a set of links and atoms that are likely to represent the implicit handwritten annotation components. 114 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. C hapter 5 Conclusions Our approach to annotation system design is novel in that we focus on markup that parallels human handwritten annotation. Our study of human handwritten annotations on paper in chapter 2 has created a foundation for algorithmic design of markup layout and recognition. The layout engine described in chapter 4 is capable of generating high quality, visually appealing output, which substantially exceeds that of existing systems. Annotation systems must be built around the needs or reading. Electronic media can support traditional reading tasks and enable all-new ones. However, the benefits come in tandem with unsteadiness of document content, format and location. Our architecture described in chapter 3 aims at solving these problems. By combining our back-end architecture with our front-end an application developer can realize a complete, high-quality advanced system for human electronic annotation of documents on Tablet PCs. We hope that this dissertation will stimulate and constructively influence the advent of applications of this kind. 115 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. References Adler, M. and van Doren, C. (1972). How to read a book, Simon and Schuster, New York, NY. Adobe (2001). Adobe Acrobat website, http://w w w .adobe.com /products/acrobat/. Anderson, K., Taylor, R. and Whitehead, E. (1994). Chimera: hypertext for heterogeneous software environments, Proc. European Conference on Hypermedia Technologies ’ 94, Vienna, pp. 94-107. AnnEngine (2000). Annotation Engine website, h ttp ://cy b er.law .h arv ard .ed u / pro jects/annotate.htm l. Annotate (2001). Annotate Technologies, http://w w w .axm otate.net/. Annotea (2001). Annotea website. http://www.w3.org/2001/Axmotea/. Baldonado, M., Cousins, S., Lee, B. and Paepcke, A. (1999). Notable: an annotation system for networked handheld devices, Proc. CHI’ 99, ACM Press, Pittsburgh, PA, pp. 210- 211. Benz, H., Fischer, S. and Mecklenburg, R. (1997). DIANE: hypermedia documents in a distributed annotation environment, Proc. HIM’ 97, Dortmund, Germany. Bemheim-Brush, A., Bargeron, D., Gupta, A. and Cadiz, J. (2000). Robust annotation positioning in digital documents, MSR-TR 2000-95, Microsoft Research. Bischoff-Grethe, A., Ovsiannikov, I. and McNeill, T. (1998). An on-line database for neu roscience information retrieval and computational modeling, Proc. Society for Neu roscience, 28th Annual Meeting, Society for Neuroscience, Los Angeles, CA. BSI (1976). Copy preparation and proof correction. (BS 5261:part 2:1976) 5QP, British Standards Institution, London. C-Technologies (2000). C-Pen website, http://www.cpen.com/. Carr, L., Roure, D. and Hall, W. (1995). The distributed link service: a tool for publishers, authors and readers, Proc. WWW4, Boston, MA. Chang, B., Mackinlay, J. and Zellweger, P. (2000). Fluidly revealing information in fluid documents, Proc. A A A I Smart Graphics Spring Symposium, ACM Press, Pittsburgh, PA. Computhink (2000). Computhink website, http://www.computhink.com/. 116 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. CoNote (1999). CoNote website. h ttp ://d ri.co rn ell.ed u /p u b /d av is/A n n o tatio n / an n o tatio n .html. CritLink (1999). CritLink website, h ttp ://c r it.o r g /c r itlin k .h tm l. Denoue, L. (1999). Adding metadata to improve retrieval: Yet another web annotation system, TR 1999-01, University of Savoie, France, h ttp ://ju lie .u n iv - s a v o ie .f r / labos/syscom/Laurent.Denoue/. Dillon, A. (1992). Reading from paper versus screens: a critical review of the empirical literature., Ergonomics 35(10): 1297-1326. Ginsburg, M. (1998). Annotate! a tool for collaborative information retrieval, Proc. WET ICE ’ 98. Gronbaek, K., Bouvin, N. and Sloth, L. (1997). Designing Dexter-based hypermedia ser vices for the World Wide Web, Proc. Hypertext 97, Southampton, UK. Gronbaek, K., Sloth, L. and 0rbaek, P. (1999). Webvise: browser and proxy sup port for open hypermedia structuring mechanisms on the WWW, Proc. Fifth In ternational World Web Web Conference, Toronto, h ttp ://w w 8 .org/w8-papers/ 3a- search-query/w ebvise/w ebvise. html. Haan, B. (1992). IRIS hypermedia services, Communications of the ACM 35(1): 36-51. Haas, C. (1992). Writing technology: studies on the materiality of literacy, Lawrence Erlbaum Associates, Mahwah, N.J. Hummes, J. (1997). Active annotations of Web pages, Proc. First European Conference on Voting, Rating, Annotation, Vienna. HyperNews (1998). HyperNews website. http://ww .hypem ew s.org/H yperN ew s/get/ hypemevs .html. iMarkup (2000). M arkup website, http://w w .im arkup.com /. Informal (2000). Informed Software website. http://ww.inform aLl.com /. ISI (1998). ISI Research website, h ttp ://w w .isire se a rc h so ft.c o m /. ISO (1983). ISO 5776:1983. Graphic technology - symbols for text correction, International Organization for Standartization. Jacobson, J. (1997). The last book, IBM Systems Journal 36(3). Marshall, C. (1997). Annotation: from paper books to the digital library, Proc. ACM Digital Libraries ’ 97, ACM Press, Philadelphia, PA, pp. 335-342. Marshall, C. (1998). Toward an ecology of hypertext annotation, Proc. Hypertext ’ 98, ACM Press, Pittsburgh, PA, pp. 335-342. Maurer, H. (1996). Hyperwave - the next generation Web solution, Addison Wesley, Har low, UK. 117 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Microsoft (1999). Microsoft NetMeeting website. http://www.microsoft.com/ netmeeting/. Microsoft (2001a). Microsoft Office website, http://office.microsoft.com/. Microsoft (2001b). Microsoft Reader website, http://w w w .m icrosoft.com /reader/. Microsoft (2001c). Tablet PC website, http://www.microsoft.com/tabletpc/. NCSA (1993). NCSA Mosaic website. http://www.ncsa.uiuc.edu/SDG/Software/ Mosaic/Docs/mosaic-docs.html. Nelson, T. (1987). Literary machines, Mindscape Press. First ed. published in 1981. O’Hara, K. (1996). Towards a typology of reading goals, EPC 1996-107, Rank Xerox Research Centre. O’Hara, K. and Sellen, A. (1997). A comparison of reading paper and on-line documents, CHI97, ACM Press, Atlanta, GA, pp. 335-342. Ovsiannikov, I. and Arbib, M. (2000). Annotator: annotation technology for the WWW, in M. Arbib and J. Grethe (eds), Computing the brain, a guide to neuroinformatics, Academic press, pp. 255-263. Ovsiannikov, I., Arbib, M. and McNeill, T. (1999). Annotation technology, Int. J. Human- Computer Studies 50(4): 329-362. Pageseeder (2000). Pageseeder website, http://www.pageseeder.com/. Phelps, T. and Wilensky, R. (1996). Toward active, extensible, networked documents: multivalent architecture and applications, Proc. Digital Libraries ’ 96, ACM Press, Qethesda, MD. Phelps, T. and Wilensky, R. (1997). Multivalent annotations, Proc. First European Con ference on Research and Advanced Technology for Digital Libraries, Pisa, Italy. Phelps, T. and Wilensky, R. (2000). Robust intra-document locations, Proc. 9th World Wide Web Conference, Amsterdam. Price, M., Schilit, B. and Golovchinsky, G. (1998). XLibris: the active reading machine, Proc. CHI’ 98, ACM Press, Los Angeles, CA. RichLink (2000). RichLink website, http://www.richlink.com/. Roscheisen, M. and Winograd, T. (1995). Generalized annotations for shared commenting, content rating, and other collaborative usages, Proc. W3C Workshop on W W W and Collaboration, Cambridge, MA. ScanSoft (2000). ScanSoft website, http://www.scansoft.com/. Schilit, B., Golovchinsky, G. and Price, M. (1998). Beyond paper: supporting active reading with free form digital ink annotations, Proc. CHI’ 98, ACM Press, Los Angeles, CA, pp. 249-256. 118 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Sheridon, N. and Berkovitz, M. (1977). The Gyricon - a twisting ball display, Proc. of the Society for Information Display, Boston, MA, p. 289. ThirdVoice (2000). ThirdVoice website, http://w w w .thirdvoice.com /. Web4Groups (1999). Web4Groups website. http://w w w .dsv.su.se/jpalm e/w 4g/ web4groups-summary.html. Wilcox, L., Schilit, B. and Sawhney, N. (1997). Dynomite: a dynamically organized ink and audio notebook, Proc. CHI’ 97, ACM Press, Atlanta, GA. Reproduced with permission of the copyright owner. Further reproduction prohibited without permission

Asset Metadata

Creator Ovsiannikov, Ilia (author)

Core Title Annotation databases for distributed documents

Contributor Digitized by ProQuest (provenance)

Degree Doctor of Philosophy

Degree Program Computer Science

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag Computer Science,OAI-PMH Harvest

Language English

Advisor [illegible] (committee chair), [illegible] (committee member), Waltman, Michael (committee member)

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c16-227272

Unique identifier UC11339064

Identifier 3073829.pdf (filename),usctheses-c16-227272 (legacy record id)

Legacy Identifier 3073829.pdf

Dmrecord 227272

Document Type Dissertation

Rights Ovsiannikov, Ilia

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the au...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus, Los Angeles, California 90089, USA

Linked assets

University of Southern California Dissertations and Theses