00001.tif

PDF

Download

Open document

Transcript (if available)

Content PER C E PTIO N OF 3-D SHAPE FROM 2-D IM AGE O F CONTOURS by Fatih Ulupm ar A D issertation Presented to the FACULTY OF THE GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In P artial Fulfillment of the Requirem ents for the Degree D O C TO R O F PH ILO SO PH Y (Com puter Engineering) December 1991 Copyright 1991 Fatih U lupm ar UMI Number: D P22839 All rights reserved INFORMATION TO ALL USERS T he quality of this reproduction is d ep en d en t upon the quality of the copy subm itted. In the unlikely event that the author did not sen d a com plete m anuscript and th ere are missing pag es, th e se will be noted. Also, if m aterial had to be rem oved, a note will indicate the deletion. Dissertation Publishing UMI DP22839 Published by P roQ uest LLC (2014). Copyright in th e D issertation held by the Author. Microform Edition © P roQ uest LLC. All rights reserved. This work is protected against unauthorized copying under Title 17, United S ta tes C ode ProQ uest LLC. 789 E ast Eisenhow er Parkway P.O. Box 1346 Ann Arbor, Ml 4 8 1 0 6 -1 3 4 6 UNIVERSITY OF SOUTHERN CALIFORNIA THE GRADUATE SCHOOL UNIVERSITY PARK LOS ANGELES, CALIFORNIA 90089-4015 This dissertation, written by Fatih Ulupinar under the direction of /u s Dissertation Committee, and approved by all its members, has been presented to and accepted by The Graduate School, in partial fulfillment of re quirements for the degree of P L D . CpS " 9 ) 0 * 7 Hs7 B /'^° DOCTOR OF PHILOSOPHY Dean of Graduate Studies Date A ugust 16* 1991 DISSERTATION COMMITTEE Chairperson Dedication This thesis is dedicated to my wife who supported me throughout my Ph.D . and to my son Doga Can Ulupinar. ii Acknowledgments I would like to thank Prof. Ram akant Nevatia, my thesis advisor, who was always there when I needed him and m ade this thesis possible. He formed an environm ent th a t supported creative thinking and hard but com fortable working w ithout any pressure. I would like to thank Dr. G erard Medioni who continuously provided me w ith very valuable feedbacks. I would like to thank Dr. K eith Price and Andreas H uertas for providing and m aintaining the com putational environm ent for this thesis. I would like to thank my com m ittee m em bers, Prof. Ram akant Nevatia, Dr. G erard Medioni and Dr. Francis Bonahon, for their valuable participation in my thesis. I would like to thank all IRIS faculty, staff and students for providing a friendly environm ent to work in. Finally, I would like to thank my wife who supported me, throughout my Ph.D . studies, emotionally, psychologically, and even financially and w ithout whom my studies here would not be as meaningful. iii Contents D ed ica tio n ii A ck n o w led g m en ts iii L ist O f F igu res v iii A b stra ct xii 1 In tro d u ctio n 1 1.1 Sum m ary of the Previous W o r k ................................................................ 4 1.2 Sum m ary of the Proposed M ethod ........................................................... 5 1.2.1 Original C o n trib u tio n s .................................................................... 7 1.3 O rganization of the T h e s is .......................................................................... 9 2 P r ev io u s W ork 10 2.1 G radient S p a c e ............................................................................................... 11 2.2 E xtrem al M ethods ........................................................................................ 12 2.2.1 Smoothness M easures...................................................................... 12 2.2.1.1 Barrow and T e n e n b a u m .................... 12 2.2.1.2 Weiss ................................................................................. 13 2.2.2 Com pactness Measures (Brady and Y u ille ) ............................ 13 2.2.3 O ther E xtrem al M e a s u r e s ............................................................ 14 2.3 Constraint A p p ro a c h e s .................. 15 2.3.1 K anade ............................................................................................... 15 2.3.2 Using Lines of C u r v a tu r e .............................................................. 17 2.3.2.1 S t e v e n s .............................................................................. 17 2.3.2.2 Xu and T s u j i ................................................................... 18 2.3.3 Horaud and B r a d y .......................................................................... 19 2.3.4 O t h e r s ................................................................................................... 20 3 S y m m e tr ie s, Q u a lita tiv e In feren ces and C o n stra in ts 21 3.1 S y m m e trie s ...................................................................................................... 21 iv 3.1.1 Parallel S y m m e tr y ........................................................................... 22 3.1.2 Line-Convergent S y m m e tr y ......................................................... 23 3.1.3 Skew S y m m e try .................................................................................. 24 3.2 Q ualitative Shape In feren ces.................................................................. . 25 3.2.1 Case I ................................................................................................... 26 3.2.2 Case I I ................................................................................................... 28 3.2.3 C a s e U I ................................................................................................ 28 3.3 C o n s tr a in ts ................................................................................................. 30 3.3.1 Curved Shared Boundary C onstraint (CSBC) ...................... 31 3.3.2 Inner Surface C onstraint (ISC) .................................................. 32 3.3.3 O rthogonality C onstraint ( O C ) .................................................. 35 4 Z e ro G a u ssia n C u r v a tu r e S u rfa c e s 37 4.1 Symmetries and ZGC S u rfaces............................................................... 38 4.2 Q uantitative Shape Recovery of ZGCs w ith Parallel Sym m etry . . 43 4.2.1 Recovering R u lin g s ........................................................................... 44 4.2.2 Application of CSBC and I S C ...................................................... 44 4.2.3 Combining Three C o n s tr a in ts ......................... 45 4.2.3.1 Cylindrical S u r f a c e s .................. 46 4.2.3.2 Circular C o n e s ................................................................ 48 4.2.3.3 General ZGC S u rfa c e s .................................................. 48 4.2.4 E stim ating (pc, qc) 49 4.2.4.1 Com puting (pc?9c) given a line I : ............................. 50 4.2.5 Com putational R e s u lts .................................................................... 54 4.2.5.1 Synthetic I m a g e s ............................................................. 56 4.2.5.2 A Real Image E x a m p le .................................................. 56 4.3 Q uantitative Shape Recovery of ZGCs Cut by Non Parallel Planes 59 4.3.1 Shape R e c o v e ry .................................................................................. 60 4.3.2 R e s u lts ................................................................................................... 62 5 M u ltip le Z G C S u rfa c e s 65 5.1 Representing S urfaces................................................................................ 65 5.1.1 Planar S u rfa c e s .................................................................................. 66 5.1.2 ZGC S u rfa c e s ..................................................................................... 66 5.2 Combined Shape R e c o v e ry ...................................................................... 69 5.2.1 Internal Constraints ................................................................ 70 5.2.2 External C o n s tra in ts ....................................................................... 72 5.2.3 Solving Constraint E q u a tio n s ...................................................... 74 5.3 R e s u lts ............................................................................................................ 74 6 S tra ig h t H o m o g e n e o u s G e n e ra liz e d C y lin d e rs 78 6.1 Surfaces and Their Limb E d g e s ........................................................... 79 v 6.2 Properties of S H G C s ................................... ? ............................................. 82 6.2.1 Recovering the Cross S e c tio n s .................................................... 84 6.2.2 Observing S H G C s............................................................................. 86 6.3 Q uantitative Shape Recovery of S H G C s.................................................. 86 6.3.1 C S B C ................................................................................................. 87 6.3.2 I S C ........................................................................................................ 87 6.3.3 P lanarity of M e rid ia n s ................................................................... 88 6.3.4 O rth o g o n a lity ................................................................................... 89 6.3.5 R e su lts.............................................................................................. . 90 7 P la n a r R ig h t C on stan t C ross S ectio n G en era lized C y lin d ers 92 7.1 Properties of C G C s ........................................................................................ 93 7.1.1 Observing P R C G C s........................................................................ 96 7.2 Q uantitative Shape Recovery of PRCG Cs ........................................... 96 7.2.1 C S B C ................................................................................................. 96 7.2.2 I S C ........................................................................................................ 97 7.2.3 O rth o g o n a lity ................................................................................... 97 7.3 Recovering Cross Section Curves ............................................................ 98 7.3.1 Com puting (pa>qa) ............................................................................ 101 7.3.2 R e su lts.....................................................................................................103 8 C on clu sion 104 8.1 Future R esea rc h ................................................................................................... 105 R eferen ce L ist 106 A p p en d ix A P l'O o fs .............................................................................................................................. I l l A .l Proofs of Theorems for ZGC S u rfa c e s....................................................... I l l A .1.1 Proof of Theorem 5 112 A .1.2 Curves of M axim um Curvature for ZGC S u r f a c e s .................113 A. 1.3 Proof of the Inner Surface Constraint for Z G C s .................... 114 A p p en d ix B Experim ents on Perception of qc by H u m a n s...................................................... 115 A p p en d ix C Prespective P r o je c tio n ...............................................................................................123 C .l C onstraints Under Perspective P ro je c tio n ...................................................125 C.1.1 Choosing a Representation for Surface O rien ta tio n ................. 126 C .l.2 Shared Boundary C o n s tra in t...........................................................126 C .l.3 Parallelity T h e o re m ............................................................................ 127 vi C .l.4 O rthogonality C o n stra in t.................................................................. 128 C .l.5 Convergent S y m m e tr y ......................................................................130 C .l.5.1 Com puting O rientation Using Convergent Sym m etry .................................................................................... 132 C.2 Usage of the Constraints for Polyhedral O b j e c t s ..................................136 C.3 Extensions to Curved S u rfaces..................................................................... 138 C.3.1 Parallel S y m m e tr y ............................................................................ 138 C.3.2 Analysis of a Conic Surface ........................................................... 140 C.4 C onclusion.......................... 142 vii List Of Figures 1.1 Some objects th a t we can readily perceive a 3-D shape from the contours alone.................................................................................................... 2 1.2 Shape perception; (a) A smooth curve (b) two sym m etric curves (c) two non sym m etric c u r v e s ................................................................... 5 1.3 The shaded images of the of the com puted 3-D shapes for the contours given in figure 1 .1 ......................................................................... 8 2.1 Each point on the ellipse on th e image plane may be the projection of any point on the line of sight.................................................................. 11 2.2 (a) A typical rectangle and its skew symmetry, (b) corresponding constraint hyperbola in p — q space........................................................... 16 2.3 Two cylinders resting at different slant angles......................................... 18 3.1 Examples to (a) and (c) parallel sym m etry w ith curved contours, (b) parallel sym m etry w ith straight contours. T he dotted curves are axes of sym m etry and the dashed lines are lines of symm etry. 22 3.2 Two line-convergent sym m etric curves....................................................... 23 3.3 Examples to (a) skew sym m etry w ith curved contours, (b) and skew sym m etry w ith straight contours. The dotted curves are axis of sym m etry and the dashed lines are lines of sym m etry................... 24 3.4 Some exam ples of case I.............................................. 27 3.5 (a) A figure w ith two skew sym m etries, (b) addition of an extra curve clarifies the perceived s h a p e ............................................................ 29 3.6 (a) Face of a cylinder w ith a clipped corner, (b) the parallel sym m etry cover only part of the surface, (c) the top curve is extended for the parallel sym m etry to cover the whole face, (d) the top of th e surface is also in c lu d e d .......................................................................... 30 3.7 Two curved surfaces meeting along a curve T ............................................ 32 3.8 Two cylinders (a) is cut along the curves of maximal curvature, and (b) is cut in an arbitrary direction while preserving parallel symmetry, now we have the perception of an elliptical cylinder................................... 35 3.9 Orthogonality c o n stra in t.................................................................................. 36 viii 4.1 Exam ples to ZGC surfaces.............................................................................. 38 4.2 A ZGC surface cut along th e “ruling”........................................................ 41 4.3 Form ation of the line-convergent sym m etry w ith a ZGC surface an two non parallel planes................................................................................... 42 4.4 O bjects w ith cross sections having (a) only one skew sym m etry, (b) two skew s y m m e trie s............................................................................. 43 4.5 The three degrees of freedom present, pc,qc,d, in a ZGC surface after applying the constraints ISC and CSBC........................................................ 45 4.6 (a) A cylindrical surface w ith axis of sym m etry and th e rulings m arked, (b) the constraints ISC, CSBC and the orthogonality for the cylindrical surface ................................................................................. 47 4.7 (a) A cylindrical object and th e ellipse fitted to the cross section, (b) th e orientation (pe, < 7 e ) th a t would make th e ellipse a circle and its projection on the q axis gives qe, first approxim ation to qc. . . 51 4.8 O bjects and ellipses fit for their cross sections. The cross sections of the objects are segmented based on their concavities (or inflec tion points) and the whole cross section has the same slan t............. 53 4.9 Sample contours, the needle images com puted and their images after shading the object w ith the com puted orientation at every point on the surface......................................................................................... 57 4.10 The processing of a real image; the cone image, edges, com puted surface norm als and the shaded image w ith the com puted surface norm als................................................................................................................. 58 4.11 A conic surface w ith line-convergent sym m etry....................................... 60 4.12 Three ZGCs cut by non parallel planes...................................................... 61 4.13 Constraints on the orientation of the cutting planes of a ZGC surface. 63 4.14 Recovering the surface shape of a ZGC cut by non parallel planes. 64 5.1 Some objects consisting of m ultiple planar and curved surfaces. . 66 5.2 The param eters of a ZGC surface and the constraints in the gra dient (p, q) of the surface along the ruling r. . . . . ......................... 68 5.3 The segmented surfaces, and the sym m etries com puted for each surface. T he skew sym m etry of planar surfaces are shown by crosses, the long line is the axis of sym m etry and the short one is th e direction of the lines of sym m etries. Parallel and line- convergent sym m etries are shown by their axis..................................... 75 5.4 T he needle and the shaded images obtained from the com puted surface norm als for the objects in 5 . 1 ..................................................... 77 6.1 An SHGC along the z coordinate axis w ith both m eridians and cross sections m arked...................................................................................... 79 6.2 Sample Straight Homogeneous Generalized Cylinders.......................... 80 IX 6.3 Tangent line, L v, of a surface S at point P in direction V ................ 80 6.4 Tangent plane, Tp, of a surface, S, containing all th e tangent lines at point P ......................................................................................................... 81 6.5 (a) An SHGC, and its tangent lines, in the direction of the axis em itting from a single cross section, intersecting at a single point on the axis, (b) The tangent lines, X }, of limb edges are not the sam e as the tangents lines, Tm, of th e m eridians in 3-D..................... 83 6.6 Image of an SHGC cut along its cross sections. Image of the top cross section curve is the bottom one is Cy and the limb bound aries are on the left Ci and on the right Cr............................................. 85 6.7 Images of th e cross sections and axes, the dashed lines, recovered for the SHGCs in figure 6.2 ...................................................................... 86 6.8 T he needle images and the shaded images generated w ith the com puted gradients at each point of the SHGCs in figure 6 . 2 .............. 91 7.1 Sam ple PR C G C s............................................................................................... 93 7.2 A PR C G C w ith both meridians and cross sections m arked.............. 94 7.3 A PRC G C (half of a torus) (a) from a general view and (b) semi transparent top view with th e limb edges of the previous view and th e m eridians passing from the points Pi and P 2 m arked along w ith their tangent lines................................................................................... 95 7.4 A PR C G C w ith a non-rotationaliy sym m etric cross section............. 100 7.5 A PR C G C w ith, (a) none, (b) one, and (c) both end cross sections available............................................................................................................... 100 7.6 The recovered cross sections for th e PRCG Cs in figure 7.1. . . . 101 7.7 T he recovered orientations shown by b o th needle image and by shading the objects for th e PRCG Cs in figure 7.1................................ 102 B .l The test figures used in th e experim ent................................................... 116 B.2 The reference cylinders w ith cross section plane slants, qc, ranging from 20% th e top left one, to 75% the bottom right one, w ith 5° increm ents........................................................................................................... 117 B.3 Com parison of hum an responses and th e algorithm discussed in section 4.2.4. In the graph the x axis is th e object num ber and th e y axis is the slant angle in degrees. For each object the shaded bars show the interval of uncertainty ( range of the slant angles containing the 90% of th e hum an responses), w hite circles shows the m ean of the slant angles given by hum an subjects, and the crosses show the slant angle com puted by the algorithm ................... 121 X B.4 The cum ulative distribution of the errors. T he x axis indicates the error in degrees (the difference between th e response of the algorithm and the m ean of the hum ans), and th e y axis indicate the num ber of objects. T he graph shows the num ber of objects having error which is greater than or equal to a given error, for each error value..................................................................................................... 122 C .l (a) An arrow like planar object w ith its axis of sym m etry, solid vertical line, and lines of sym m etry, dashed horizontal lines, (b) Projection of th e arrow like object and its convergent sym m etry lines; dashed lines are th e lines of sym m etry m eeting at the point (wc, vc), Li is one of the lines of sym m etry m eeting the boundary at points E and F. T he vertical solid line is th e axis of sym m etry, L s, having N IG P of A = (a, 6, c)..................................................................... 131 C.2 A cube under perspective projection (a), and com puted orienta tions for the faces shown as points on th e p — q space w ith the shared boundary constraints overlayed, dashed lines, (b )......................... 137 C.3 Contours of a conic surface under perspective projection, w ith the point of convergence for the rulings P, and th e line L($) ......................139 xi Abstract In to d ay ’s technology, the capabilities of machine vision system s are far be hind those of the hum an vision system . To be able to design and m anufacture vision system s th a t rival th e hum ans, we have to study and understand th e prin ciples and tools th a t the hum an visual system utilizes. One of th e m ost im portant features of these systems is the variety of sources of inform ation used. Hum ans can perceive definite shapes for a large class of objects from their sim plest ab straction, nam ely their contours. The most common exam ple is our ability to “see” and understand cartoons. Shape com putation from a single 2-D image of object contours is an in trin sically am biguous problem , i.e., given a contour in the image plane there are infinitely m any objects th a t can produce th e given contour. Various preference criteria, in addition to th e constraints available, m ust be used to resolve the am biguity in the perception process. Previous work on shape from contour con centrated on objects w ith planar faces. In this thesis the problem of shape from contour for curved surfaces is addressed. By studying the geom etric and differential properties of various types of sur faces, the constraints im posed on the object surfaces by images of its contours are form ulated. It is shown th a t three classes of sym m etries, form ed by object contours, convey im portant shape inform ation. These sym m etries, called par allel sym m etry line-convergent sym m etry and skew sym m etry, have enabled us to analyze a wide class of objects, such as objects w ith zero Gaussian curvature surfaces, classes of generalized cylinders w ith straight axis (SHGC) and with non-straight axis (CGC). xii Image contours containing the above type of sym m etries impose two essential constraints on th e shape of the surface; th e “Curved Shared Boundary Con strain t” and th e Inner Surface C onstraint. A pplication of these constraints to surface shape leaves three or four degrees of freedom depending on th e type of surface. These degrees of freedom are fixed by using assum ptions of regularity inspired from hum an visual system , like preference of orthogonality. Various al gorithm s are devised to reconstruct th e surfaces from their contours for th e object classes m entioned above. xiii Chapter 1 Introduction One of th e basic goals of com puter vision is to be able to “perceive” the environ m ent from 2-D pictures of it w ithout having specific models of the com ponents of th e environm ent. Recovering 3-D shape of the objects constituting the environ m ent from their 2-D images is an essential com ponent of perception. 3-D shape is needed to achieve m ost of the higher level goals of com puter vision. For object recognition, recovering 3-D shape enables us to obtain a richer description of the scene, thus, enhancing the ability to discrim inate between wider range of model objects. And for m any other applications th at requires the “understanding” of th e environm ent, 3-D shape recovery is essential. Hence, one of the essential goals of the mid-level vision is to recover the local orientations of the surfaces of the objects in a scene. T he basic difficulty, of course, is th a t an im age is a 2-D projection of th e 3-D scene, therefore, the 3-D structure can not be recovered w ithout some assum ptions. In spite of the inherent am biguities in a single view, hum ans are able to perceive 3-D surfaces in single images. Much effort has been devoted in the past few years to understanding ability to perceive 3-D shape and has led to development of techniques such as shape from shading, shape from tex tu re and shape from contour (sometimes also known as shape from shape). 1 Figure 1.1: Some objects th a t we can readily perceive a 3-D shape from the contours alone. We believe th a t of all th e m onocular cues, shape of the 2-D contour itself is the m ost im portant cue for th e shape of the 3-D surfaces. Strictly speaking, not only is such interpretation infinitely ambiguous but the contours can only give shape inform ation near the contours; shape of the surface in between can vary sm oothly w ithout producing other contours. Nonetheless, hum ans when presented with contours of various, not necessarily fam iliar, objects perceive com plete surfaces (and even solids). Some exam ples for such contours th a t we readily perceive a shape are given in figure 1.1. Barrow and Tenenbaum [BT81] show, by some exam ples, th a t in case of conflict betw een contour and shading inform ation hum ans use th e contour in form ation. Biederm an [Bie87] claims th a t in the experim ents w ith hum ans the 2 recognition of a full colored image of an object is not faster th an the recogni tion of th e line draw ing of the object. He also shows th a t we do not necessarily need to have any fam iliarity w ith the object in order to perceive a shape from its boundary. We conjecture th a t th e reason for preferring shape from contour over other cues, such as shading, m ay be th a t even though shape from contour m eth ods need to m ake some assum ptions, other m ethods need to m ake even stronger assum ptions. Shape from shading m ethods, for exam ple, need to assum e th a t the reflectance properties of the surface are known, th a t th e albedo is constant, and th a t the light sources are known precisely. On the other hand, contour is m ostly invariant under large variations of the lighting and surface reflectance. These observations are not to argue th a t only shape from contour is needed, b u t th a t it is an essential elem ent in m onocular perception th a t can not be ig nored. We believe th a t such an ability will also be essential for m achine vision system s, if they are to work w ith m onocular images in the absence of highly specific models. Even stereo vision systems ignoring all the m onocular cues and relying solely on stereo inform ation, as they are done today, handicaps these sys tem s severely. For such systems to rival or exceed th e hum an visual system , they should use the m onocular shape cues available in th e images too. In this thesis, we study the perception of object shape from 2-D images of the contours of the surfaces surrounding the object. Specifically, we recover the local surface norm als of th e object surfaces. W ithout knowing th e com plexity of the processes involved, hum ans can easily perceive a 3-D shape from 2-D contours for a large variety of objects. Cartoons and engineering drawings or any drawing in millions of the books are some exam ples. However this does not m ean th at hum ans can perceive a 3-D shape for any given random line drawing. Only the line drawings w ith certain properties, carry inform ation for 3-D shape. In this thesis we a ttem p t to identify some of the properties of the line drawings required for 3-D shape recovery. Zero Gaussian curvature surfaces, straight homogeneous generalized cylinders, and constant cross section generalized cylinders cut by pla nar planes are studied in this thesis, because, such surfaces produce contours th a t 3 contain the properties required to recover 3-D shape. A lthough we do not claim th a t all th e classes of contour th a t hum ans can perceive a 3-D shape are identi fied and studied, we believe th a t these classes of surfaces constitute an im portant subclass and a line drawing carries a little (if not none) shape inform ation, if it does not fit into one of th e categories, studied in this thesis, or some derivation of these categories. 1.1 Summary of the Previous Work The early work on inferring 3-D structure from a 2-D shape was focused on anal ysis of line drawings of polyhedra [Huf71], [Clo71], [Mac73], [Kan81], [KK83]. In polyhedral scenes, the problem is th a t of segm entation and estim ating orienta tions of faces. In early 70’s [Huf71], [Clo71], [Mac73] th e work on contours has concentrated on line drawing interpretation and edge labelling. K anade [Kan81] has pioneered th e work on shape recovery from contours for polyhedral scenes us ing skew symm etry. Techniques for non-polyhedral scenes have been proposed in [BT81], [Wei88], [BY84], [Ste81], [XT87], [HB88]. Barrow and Tenenbaum [BT81] and Stevens [Ste81] pioneered the work on shape recovery for curved surfaces from 2-D contours. Barrow and Tenenbaum proposed a m ethod th a t minimizes the smoothness of the curve in 3-D and Stevens has proposed a m ethod th a t is only applicable to cylindrical surfaces. Xu and Tsuji .[XT87] extended Stevens work to other surfaces than cylindrical surfaces. Weiss [Wei88] has proposed another smoothness based m ethod w ith a different definition of smoothness. B rady and Yuille [BY84] proposed a com pactness based m ethod for planar surfaces th a t are not necessarily enclosed by straight lines. Later Horaud and Brady extended this work to apply to conic surfaces. We discuss these techniques in more detail in section 2 and contrast w ith our work later. It is interesting to note th at, m ost of these techniques exam ine a single surface in the scene whereas our perception of a surface can be strongly influenced by our perception of the entire object. 4 (a) (b) (c) Figure 1.2: Shape perception; (a) A sm ooth curve (b) two sym m etric curves (c) two non sym m etric curves Recently some papers (e.g. [PCM89, Nal89]), have been published th a t study th e projection and geom etry of certain classes of curved surfaces, however the analysis is not sufficient to recover the 3-D surfaces. 1.2 Summary of the Proposed Method In this thesis we present m ethods for recovering the 3-D shape of th e surfaces from th eir contours. We follow a constraint based approach, where constraints on shape are separated from th e assum ptions of regularity. T he two essential constraints on shape im posed by the contours are Curved Shared Boundary C onstraint (CSBC) and the Inner Surface C onstraint (ISC). CSBC restricts th e orientation of the neighboring surfaces. This constraint was first used in the study of th e planar surfaces and is extended to apply to curved surfaces here. ISC restricts the orientation of th e inner points w ithin a curved surface along certain selected curves. T he perception of the shape of a contour is sharply influenced by its rela tionship w ith other contours. One of the m ost im portant relationships is th at of sym m etry. For exam ple in figure 1.2(a) the single curve does not give any strong 3-D shape im pression, but in figure 1.2(b) another curve which is per fectly sym m etric to th e first one is added and a definite shape is perceived. In figure 1.2(c) two non sym m etric curve are displayed and again there is no definite shape perception. 5 Sym m etries play an im portant role in our analysis in two ways. F irst, sym m e tries carry im portant inform ation about the surface type: by ju st looking at the sym m etries of the contours of a surface, in m any cases, we can identify th e type of the surface w ithout even perform ing any q uantitative analysis, and in some cases we show th a t th e surface m ust be of the identified type under the assum ption of general viewpoint. Second, sym m etries provides vital inform ation needed for quantitative analysis. For exam ple, one of th e two essential constraints, th e inner surface constraint described in section 3.3.2 is based m ostly on a sym m etry th a t we call parallel sym m etry, and orthogonality assum ption is based on sym m etry properties of the contours. We also conjecture th a t little shape inform ation is available in figures lacking sym m etries for hum ans as well. Our work m ay be viewed as being based on gener alizations of concepts th a t have been used previously such as by K anade [KanSl] for polyhedral scenes and by Stevens [Ste81] and Xu and Tsuji [XT87] for curved surfaces. We believe th a t our m ethod is of wide applicability. In particular, we provide a detailed analysis for the case of Zero-Gaussian C urvature (ZGC) surfaces, Straight Homogeneous Generalized Cylinders (SHGCs), P lanar Right C onstant cross section Generalized Cylinders (PRCG C s) and objects form ed by com bination of planar and ZGC surfaces. T he types of surfaces, analyzed in this thesis, are commonly found in ev eryday life, especially in m an-m ade objects. For each class of surface, the con straints available are identified. Com bination of constraints reduces, but does not uniquely determ ine the degrees of freedom of surface orientations. For ZGCs, after the CSBC and ISC are applied; only three degrees of freedom rem ain to recover the local surface gradient at each point on the surface. For SHGCs and CGCs four degrees of freedom are left. T he assum ption driven constraints are sym m etry based and they are applied to obtain a unique or a few m ost likely interpretations. The m ain assum ption used is the 3-D orthogonality of 2-D sym m etries. W hat exactly is m ade orthogonal depends on the type of surface studied. For planar surfaces the skew sym m etry is m ade orthogonal in 3-D. For curved 6 surfaces th e intersection of the lines and curves of parallel sym m etry are m ade orthogonal. In all of th e surfaces, we have studied, application of all the con straints, including th e one based on orthogonality assum ption, consistently left one degree of freedom undeterm ined. The shape of th e cross section curve is used to fix this last degree of freedom. For objects form ed by m any surfaces the constraints and assum ptions for each surface are identified. T he constraints are essentially th e curved shared boundary constraints between surfaces. In com puting a consistent global shape for the object, these constraints are satisfied exactly while the error of not satisfying the orthogonality constraints are minimized. T he set of param eters of th e surfaces th a t minimizes the constraint m inim ization define the object shape. In figure 1.3 th e synthetically shaded images of the objects, given in figure 1.1, are shown after the local surface orientations are recovered using the m ethods described in this thesis. These are typical objects th a t we are able to handle by the m ethods described here. O ur m ethod assumes th a t clean, closed boundaries are given (or can be ex tracted from th e real image). We do not address th e issue of separating object boundaries from surface m arkings, or other perceptual grouping operations here. We believe th a t th e specific conditions needed for an object surface to be recon structed by our m ethod will provide further constraints for the perceptual group ing operations when surface m arkings and other noisy boundaries are present. 1.2.1 O rigin al C o n trib u tio n s T he following item s are individual im portant contributions introduced in this thesis. • Definition of sym m etries, especially parallel and line-convergent symm etry, found in various classes of curved surfaces. •! -N W 'W ''5 Figure 1.3: T he shaded images of the of th e com puted 3-D shapes for th e contours given in figure 1.1 8 • Explicit analysis of contour based constraints on surface shape for curved surfaces. • Introduction of Inner Surface C onstraint for curved surfaces. • A detailed analysis of th e following classes of surfaces and objects; — Zero Gaussian curvature surfaces, — Straight homogeneous generalized cylinders, — P lan ar Right Constant cross section generalized cylinders. • Com bining constraints from intersecting surfaces to obtain a consistent re construction for an object consisting of planar and ZGC surfaces. • Extension of all the constraints and the analysis of ZGC surfaces to P er spective Projection. 1.3 Organization of the Thesis In chapter 2, we discuss previous related m ethods. In chapter 3, we define two kinds of sym m etries and discuss the constraints we obtain from contours and the sym m etries. In chapter 4 we describe our technique for qualitative and quan titativ e shape recovery of Zero Gaussian C urvature (ZGC) surfaces. Straight Homogeneous G eneralized Cylinders (SHGCs) are studied in chapter 6 and Con stan t Cross section generalized Cylinders (CGCs) are studied in chapter 7. In chapter 5 we discuss objects composed of planar and ZGC surfaces. Finally we state our conclusion in chapter 8. 9 Chapter 2 Previous Work Here we present an overview of the im portant classes of previously m ethods. We also give our view of their strengths and weaknesses. As our m ethod builds on some of th e previous work, this section will also help provide some of th e relevant background for describing our work. 3-D in terp retatio n of a 2-D line drawing is, of course, inherently ambiguous. Given a curve in th e image, we can find an infinite num ber of 3-D curves th a t the 2-D curve could be a projection of. T he construction is simple. Consider figure 2.1. Given a point in the image, we can associate a ray in 3-D w ith it on which th e corresponding point of th e 3-D curve m ust lie. Any choice of points on these rays will project in the same 2-D curve. Note th at inferring the 3-D curve gives th e surface orientation only if the curve is planar (and closed). Otherwise, another process of estim ating surface orientations from the bounding curve m ust be applied. Two classes of m ethods have been used to reduce this am biguity. One is to choose a shape th a t satisfies some preference criteria over the others. The second approach is to define some constraints on the 3-D interpretation- some of the constraints come from purely geom etric considerations, others require assum ption of some regularity in the 3-D shape which m ay be signaled by some regularity in the 2-D shape. In the following, we survey th e two classes of m ethods. F irst we 10 Image plane Figure 2.1: Each point on the ellipse on the im age plane m ay be the projection of any point on the line of sight. give a definition of th e gradient space th at has been used by m any researchers before and is extensively used in this thesis. 2.1 Gradient Space W e assume orthographic projection throughout the paper unless specifically m en tioned otherwise. In appendix C, we have shown how th e constraints for ortho graphic projection can be transform ed to the case of perspective projection. In this paper we will use Gradient Space to represent the orientation of surfaces (given by their norm als). To review, th e normal, N , of a plane ax + by + cz + d — 0 is given by the vector N = (a,6, c). This can be rew ritten as (p, q, 1), where p = a/c and q = b/c. N ote th a t this excludes cases where c = 0, however, such planes are parallel to the line of sight and are not im aged under orthographic projection from a general view-point anyway, (p, q) can be thought of as defining a two dim ensional space, called the gradient space, such th a t every point in this space corresponds to the norm al of a plane in 3-D. 11 2.2 Extremal Methods In this class of m ethods, each interpretation (of a curve or figure) has, a m easure of some desirable property, such as sm oothness or com pactness, associated with it. In th e following we briefly describe some of th e m ore influential m ethods. 2 .2 .1 S m o o th n e ss M ea su res Sm oothness is the first regularity m easure tried by researchers like [BT81, Wei88]. In th e following these m ethods are described briefly. 2 .2 .1 .1 B arrow and T en en b au m Barrow and Tenenbaum [BT81] proposed to use the sm oothness of th e curve as th e preference criterion. Their m ethods is not restricted to planar curves. The sm oothness function they propose is an integral function of curvature and torsion of th e 3-D curve. Over all possible 3-D curves th a t can generate a given 2-D image curve, they choose the one th a t minimizes th e integral functional where k is the curvature, t is the torsion and s is the arc length of th e 3-D curve. They im plem ented an iterative optim ization procedure to com pute the m inim um of the integral in equation 2.1. B ut they noted th at the convergence of th e algorithm was slow. If planar surfaces are assum ed, only the first term in th e integrand above need be considered. Given an ellipse in the image, this m ethod will find a 3-D circle as th e curve th a t m inim izes th e above function; an inference generally in agreem ent w ith hum an perception. (In fact, for a circle the m easure in equation 2.1 is identical to zero everywhere.) This m ethod has th e advantage of having a well-defined com putational model though th e com putation m ay require a search over an n dim ensional space where (2.1) 1 2 n is th e num ber of points on the curve. It is also applicable to non-planar curves. However, the answers are not always in agreem ent w ith th e hum an perception. This m ethod is likely to be highly sensitive to noise as it uses the derivatives of the curvature which are th ird order derivative operations. As an exam ple consider a circle w ith a small notch. T he contribution of th e notch to th e smoothness integral m ay be m uch larger th an the contribution of all other parts of the circle, in this case, th e resulting 3-D curve m ay not resem ble a circle. Strictly speaking, this m ethod is applicable to sm ooth image curves only and not to polygons, for exam ple. 2 .2 .1 .2 W eiss Weiss [Wei88] has proposed a modified m easure th a t uses curvature rather th an its derivative and also handles polygons in a cleaner way. For planar curves he proposes to m inim ize the integral of the square of the curvature (norm alized by the length of the curve). This m ethod is slightly less sensitive, compared to th e previous m ethod, to small changes on a curve (like a notch on a circle) but it is still quite sensitive. Although, this m ethod interprets an ellipse on the im age plane as a circle in 3-D, partial contours result in a consistent bias due to norm alization, therefore a section of a ellipse will not be interpreted as a section of a circle. He also proposes using square of th e angles of corners for polygonal scenes as a m inim ization criteria. And he shows th a t a parallelogram is interpreted as a slanted rectangle which may be in agreem ent w ith hum an visual system . However, if a corner of th e same parallelogram is clipped th e m ethod interprets it as a pentagon rath er than a clipped rectangle. 2 .2 .2 C o m p a c tn e ss M easu res (B ra d y and Y u ille) B rady and Yuille in [BY84] used the “com pactness” of a figure as th eir preference criterion. M easure of com pactness is chosen to be (area) / (perimeter)2. (This m easure im plicitly assumes th at the curve is planar.) This m easure is com puted 13 over all 3-D orientations and th e m axim um chosen. This requires a search though th e search can be speeded up for an approxim ate answer. T he area can be com puted using th e Stoke’s theorem , which is: tion efficient. This m ethod should be insensitive to sm all noise as the area of a surface is alm ost unaffected by noise on th e boundary, and the perim eter is at th e orientation. Also, this m ethod processes sm ooth curves and polygons in a unified way. A lthough, this m ethod has m any nice features from a com putational point of view, the real question is: how often does the m ethod give the desired answers. The m ethod works perfectly when th e the input is an ellipse and when the input is a projection of a square. However, it fails on m any very sim ple shapes. For exam ple it interprets a rectangle in th e picture as a slanted square; not in conform ity w ith th e hum an perception. Also when the boundary is not com plete th e behav ior of the m ethod is different th an w ith a com plete boundary which is not totally desirable. In a recent paper H oraud and Brady [HB88] presents a m ethod for interpreting generalized cylinders w ith the help of (area)/(perimeter)2 m ethod. T heir m ethod is discussed in the next subsection. 2 .2 .3 O th er E x tr e m a l M ea su res We have studied other preference m easures ourselves, for exam ple th e regularity of a figure, which for a polygonal figure can be defined to be the equality of th e angles of th e polygon. Again, this m easure works well on simple exam ples, including a rectangle not handled by the Brady-Yuille m ethod. However, this (a re a )n where (area) is a scalar, n is a vector norm al to th e boundary curve and r is a vector coordinate system in th e plane of the figure. This allows both area and perim eter to be com puted by an integral on the boundary m aking the com puta- m ost affected by a linear factor and this does not change rapidly when we change 14 m easure is difficult to apply to continuous curves, is sensitive to sm all changes in th e boundary and also does not always produce results th a t agree w ith our perception. For exam ple, a rectangle with a clipped corner is interpreted as a slightly distorted pentagon. It is our conclusion th a t the m ethods based on m axim izing some simple prop erty of a 3-D figure viewed in isolation are not very effective even for planar figures and generally do not apply to curved surfaces at all. However, we believe th a t the m ajor deficiency of these m ethods rem ains, nam ely th a t they use only the inform ation of a single curve and ignore all other context. 2.3 Constraint Approaches In this class of m ethods, constraints on 3-D surface orientations are obtained by a variety of observations, w ith the expectation of getting unique (or a few) solutions when the various constraints are combined. Som etim es, the constraints are based on an assumption th at an observed regularity in th e image corresponds to a regularity in the 3-D scene. Almost all of these techniques rely on making some lines (or norm als) orthogonal in 3-D. If it is possible to find two lines (or vectors) in an image, th a t we have reasons to believe are orthogonal in 3-D, we can derive some constraints on the plane containing th e two lines (vectors). K anade [Kan81] used the observation of skew symmetry to signal such orthogonality, others [Ste81] [XT87] have used the assum ptions th a t some of th e curves in image are lines of curvature. We briefly survey these im portant techniques below. 2.3.1 K a n a d e A sym m etric figure in 3-D, w ith a straight axis and lines of sym m etry th a t are orthogonal to it, projects in a figure which is skew symmetric (under orthographic projection), i.e. th e lines of sym m etry are no longer orthogonal to the axis but 15 (b) (a) Figure 2.2: (a) A typical rectangle and its skew sym m etry, (b) corresponding constraint hyperbola in p — q space. are at a constant angle to it. K anade [Kan81] showed th a t if we assum e th a t the inverse also holds, i.e. th a t an observed skew sym m etry in the image is in fact due to orthogonal sym m etry in 3-D, some useful constraints can be obtained. Consider Figure 2.2(a) showing a parallelogram w ith its axis and skewed line of sym m etry m aking angles a and fi w ith the horizontal. Imposing the requirem ent th a t these lines are orthogonal in 3-D constraints the possible orientations of the figure in 3-D. Expressing the orientation of th e surface in gradient space (p,q ), the constraint on th e surface orientation is given by a hyperbola. Figure 2.2(b) shows th e constraint hyperbola for th e parallelogram in figure 2.2(a). Note th at the skew sym m etry observation by itself does not give a unique orientation. K anade suggests th a t in the absence of any other constraint one m ay choose th e orientation th a t gives m inim um slant; this is the point on the hyperbola th a t is nearest to th e origin (the two possible answers are equivalent to a “Necker reversal”). However, this simple selection m ethod does not always give th e desired answers. For polyhedral scenes, K anade combined the skew sym m etry constraints with w hat we have called the shared boundary constraints to obtain unique (or a small set of consistent orientations). T he answers given by this m ethod appear to be consistent w ith hum an interpretation. Of course, this m ethod applies only when skew sym m etric objects are present. 16 2 .3 .2 U sin g L in es o f C u rvatu re Some researchers [Ste81, XT87] have proposed using the lines of curvature to recover th e shape assum ing th e object is cut along th e lines of curvature. 2 .3 .2 .1 S tev en s Stevens [Ste81] studied cylindrical surfaces using orthogonality property. A cylin drical surface is one where one of the principal curvatures is zero and th e lines of zero curvature (the rulings of the surface) are parallel to each other. For such a surface th e lines of m axim um curvature are planar and parallel to each other. Stevens assumes th a t th e lines of m axim um curvature are given. T he rulings can be obtained from these by connecting points w ith the same tangent. T he surface is thus covered by a grid of curves, w ith the property th a t on the actual surface, th e curves are orthogonal at th e points of intersection. Thus, constraints sim ilar to those of th e skew sym m etry analysis can be applied. Stevens chooses to use the slant1 and tilt representation instead of (p, q) representation. As before, th e constraint is not enough to give unique orientations. However, Stevens observes th a t slant and tilt param eters can be bounded and th a t the bound depends on the angle between the two intersecting curves, w ith error in tilt approaching zero as fi approaches 7 r. This happens near th e occlusion boundaries of a cylindrical surface. S tarting from these points where tilt can be fixed accurately, Stevens gives a m ethod of propagating th e estim ates along the lines of m axim um curvature by th e following form ula (given w ithout proof here): tan Ti ta n /A = ta n r 2 tan /?2 (2-3) where Ti are th e tilt angles (3{ are the angles between th e lines of m axim um and m inim um curvature, at two points along a line of m axim um curvature. 1Orientation of a surface, having gradient (p,q), can be alternately described in terms of its tilt and slant which can be viewed as the polar coordinates of a point in gradient space, specifically tilt = arctan slant = arctan \/p 2 + < 7 2 17 (b) (a) Figure 2.3: Two cylinders resting at different slant angles. This m ethod, however, does not always give correct results even when a cir cular cylinder is given to it, as can be shown by a sim ple exam ple. Consider the two cylinders in figure 2.3. T he points on the limb edges for both cases have /3j equal to it which give unam biguous values of zero tilt. Using Stevens m ethod to extrapolate, along th e cross-section, we will get the same orientations for the m id-points of th e two cylinders where /?2 is ?r/2, in clear contradiction to our per ception (which indicates th a t the top surface of (b) appears much m ore slanted to us th an th a t of (a)). 2 .3 .2 .2 X u and T su ji Xu and Tsuji [XT87] have described an extension of this m ethod to apply to m ore general, curved surfaces. Their m ethod does not require th a t all lines of curvature be given but th a t th e surface is cut along these lines of curvature. Given a figure w ith four sides, such th at two of the opposite sides are lines of m axim um curvature and th e other two are lines of m inim um curvature, they show how a net (or a grid) over th e figure can be constructed w ith th e expected property th a t th e corresponding net on the 3-D surface is orthogonal (this construction is strictly valid for a restricted class of surfaces only). Surface orientations are first com puted at special points on th e net where error in tilt is sm all (as in Stevens’ m ethod) and then propagated to other points on the net. 18 This m ethod seems to work well in some cases but has several draw backs. The propagation scheme for non-cylindrical surfaces is only approxim ate and errors can add up. It will always find an answer for a four-sided figure, even when hum ans perceive no specific 3-D shape in it. Like Stevens’ m ethod, this m ethod also would not differentiate betw een the two cylinders in the figure 2.3. 2 .3 .3 H orau d an d B ra d y H oraud and B rady [HB88] present a m ethod for interpreting linear straight ho mogeneous generalized cylinders (LSHGCs). Their m ethod makes the following assum ptions: a) th e axis of the LSHGC projects as th e axis of th e ribbon form ed by th e two limb contours in th e im age plane, b) th e cross section of th e LSHGC is planar, and c) th e cross section is orthogonal to the axis in 3-D. Satisfying assum ption (b) above gives a constraint th a t the orientation of th e cross section m ust be along a certain curve in th e orientation space (the curve is shaped like th e character “s” and hence called “s-curve”). They also require th a t th e back projected cross section satisfy the Brady-Yuille com pactness m easure. If an ori entation satisfies both constraints then th at orientation is chosen. They do not specify w hat should be done if this is not the case, though a n atu ral extension would be to take the m ost com pact shape constrained by the s-curve. Finally they suggest sweeping the reconstructed cross section along the 3-D axis to reconstruct the surface of the LSHGC. This m ethod has the attractiv e property th a t it attem p ts to com bine the con straints from two surfaces. However it has several deficiencies. T he com pactness m easure can only be applied to com plete cross sections. M ore seriously, the as sum ption (a) above is incorrect. Given the image of an LSHGC, we can choose any axis th a t passes through th e apex of the LSHGC in th e im age plane and reconstruct an orthogonal LSHGC in 3-D, including th e ends, th a t have th e same projection in th e image. To see this, take any backprojection of the chosen axis, backproject th e two cross sections (the top and the bottom ) on any two planes 19 orthogonal to th e backprojected axis; th e orthogonal LSHGC can be com pleted by joining th e points on the cross sections, such th a t th e lines joining these points pass through th e backprojected apex. Finally even if an axis is chosen in the im age plane, th e point through which the 3-D axis pierces the reconstructed 3-D cross section m ust be chosen in order to reconstruct the LSHGC. This point is not addressed in th e H oraud and B rady paper. 2 .3 .4 O th ers In other work, Nalwa [Nal87] has derived a sym m etry condition th a t m ust be satisfied by the lim b boundaries of a solid of revolution (sufficiency of these conditions is also claim ed under certain general viewing conditions). However, this paper does not show how to actually reconstruct the surface. Ponce et al. [PCM89] have derived properties th a t m ust be satisfied by a broader class of surfaces known as straight homogeneous generalized cylinders (SHGCs). Again, these properties by them selves are not sufficient to reconstruct the 3-D surface of SHGCs. Recently Gross and Boult [GB90] have provided a m ethod for recovering SHGCs from contour and shading data. Their analysis of the contour is quite sim ilar to ours, however they use shading inform ation to further constrain the possible orientations. They claim th at, using shading inform ation elim inates the need to m ake assum ptions about the shape of the object, as needed for shape from contour m ethods. However, using shading inform ation itself requires drastic assum ptions about the surface reflectance properties of th e objects. 20 Chapter 3 Symmetries, Qualitative Inferences and Constraints O ur proposed technique is based on observations of sym m etries in figures. We believe th a t sym m etries have an im portant role in shape perception, this also has been noted and used by m any researchers [NB77b, Nal87, Kas88, Kan81, Ste81, MN89]. We define three types of sym m etries, th a t we call parallel sym m e try, line-convergent sym m etry and skew sym m etry. Parallel and line-convergent sym m etries are m ainly found in curved surfaces, skew sym m etry is usually an indicator of planar surfaces. O ther types of sym m etry relations betw een curves specific to various classes of surfaces will also be discussed as the surfaces gen erating such curves are discussed. Later, we discuss qualitative shape inference from th e sym m etries. Finally, constraints th a t constitute th e basis of surface reconstruction for all types of surfaces, studied in this thesis, are presented. 3.1 Symmetries We define two new sym m etries called parallel sym m etry line-convergent sym m etry and redefine skew sym m etry, previously defined and used by K anade [Kan81]. For curves to be sym m etric (parallel or skew) certain point-wise correspondences betw een two curves m ust exist. We will call the lines joining th e corresponding points on th e curves as the lines of symmetry, the locus of th e m id points of these 21 '\ \ \ \ \ (a) (b) (c) Figure 3.1: Exam ples to (a) and (c) parallel sym m etry w ith curved contours, (b) parallel sym m etry w ith straight contours. T he dotted curves are axes of sym m etry and th e dashed lines are lines of sym m etry. lines as the axis of symmetry, and the curves form ing th e sym m etry as the curves of symmetry. 3.1.1 P a ra llel S y m m e tr y Consider two curves Xi(s) = (®,-(s), yi(s)), for i = 1,2, param eterized by arc length s. Let T)(s) = (:r'(s), t/8 '(s)) be th e unit tangent of th e curves. Then, X i(s) and A T 2(.s) are said to be parallel sym m etric if there exists a correspondence function / ( s ) between them such th a t, T .M = r 2( /( s ) ) (3.1) for all values of s for which X \ and X 2 are defined and f(s) is a continuous m onotonic function. Note th a t com puting sym m etry between two curves using this definition requires estim ating the function f(s) as well. A useful special case is when f(s) is restricted to be a linear function. In th a t case, the sym m etry condition becomes: Ti(s) = T2(as + b) (3.2) where a and b are constant (a m ay be thought of as a scale param eter). Some exam ples of parallel sym m etry are given in figure 3.1; th e correspondence function 22 Figure 3.2: Two line-convergent sym m etric curves. f(s) is linear for (a) and (b) but not for (c). Note th a t th e above definition of parallel sym m etry also holds for curves consisting of straight lines and corners as in figure 3.1 (b). Presence of parallel sym m etry w ith non straight curves of sym m etry, is a very strong indication of curved surfaces. We can extend the definition of parallel sym m etry to 3-D curves. T he parallel sym m etry in 3-D is still defined by equation 3.1 where Ti is the unit tangent in 3-D. 3.1.2 L in e-C o n v erg en t S y m m e tr y Two image curves C\ and Ci are line-convergent sym m etric if the tangents of C\ and C > 2 j &t the corresponding points, intersect along a line, say I, on th e image plane. This is shown in figure 3.2. This sym m etry is found in curves obtained by cutting ZGC surfaces w ith two non parallel planes. Also this sym m etry is present between th e projection of limb edges of straight homogeneous generalized cylinders (SHGCs) described in chapter 6. 23 2 (a) Figure 3.3: Exam ples to (a) skew sym m etry w ith curved contours, (b) and skew sym m etry w ith straight contours. T he dotted curves are axis of sym m etry and th e dashed lines are lines of symm etry. 3 .1 .3 S k ew S y m m e tr y In this sym m etry, the point-wise correspondence should be such th a t the axis of the sym m etry is straight, and th e lines of sym m etry are at a constant angle (not necessarily orthogonal) to the axis of sym m etry. Skew sym m etry was first proposed by K anade [Kan81] and used in the analysis of scenes of polyhedral objects. Ponce [Pon88] has given point-w ise conditions for two curves to be skew sym m etric. We state these here w ithout proof. For the case when th e lines of sym m etry are orthogonal to the axis of sym m etry th e criterion for the two curves, X l(s) and 2f2(s), to be skew sym m etric is: /c i(s ) = — /c2 (s + b) (3 -3 ) where k ( s ) is th e curvature, and b is the offset. In general, let a,-(s) be the angle betw een the line of sym m etry and th e tangent to th e curve i at the corresponding points, then the necessary condition for the two curves to be skew sym m etric is: Ki(s) sin3(o:2(s)) = — k 2( s + b) sin3(o:2(s + b)) (3.4) 24 An exam ple is given in figure 3.3 (a). The above conditions are only valid for curves, not for lines, th a t is curvature should be non zero. For lines the first definition of skew sym m etry can simply be applied as: two lines are skew sym m etric if another set of two lines th a t joins th e end points of th e given lines are parallel to each other. In this case the new two lines are th e lines of sym m etry and the lines joining the end points of these lines is the axis of sym m etry. An exam ple to skew sym m etry for straight lines is given in figure 3.3 (b). 3.2 Qualitative Shape Inferences We now describe some qualitative inferences about th e shape of surfaces from their sym m etries. We prove some of the inferences th a t we m ake w ith th e assum ption of general viewpoint defined as: D e fin itio n 1 G e n e ra l V ie w p o in t : A scene is said to be imaged from a gen eral viewpoint} if perceptual properties of the image are preserved under slight variations of the viewing direction. Specifically, the properties we are interested in are: straightness and paral- lelity of lines and sym m etry of curves Here, we discuss th e interpretation of individual surfaces independently. In an object, of course, several surfaces may be visible and th eir interpretations m ust be m utually consistent. This can provide a m echanism for either reinforcing individ ual surface interpretations or choosing among possible m ultiple interpretations of individual surfaces. It will be useful to consider figures as belonging to one of th e following three classes: 25 3.2.1 C a se I Here, one skew sym m etry covers the entire boundary of the surface. We allow m ore th an one alternative descriptions for a figure (figure 3.4 shows two exam ples). For exam ple, th e ellipse in figure 3.4 (a) can be described as being skew sym m etric about any axis th a t passes through its center, and th a t all sym m etries include all the points on th e ellipse boundary. Surfaces belonging to case I are generally perceived to be planar. We prove th a t if a contour belongs to case I bounded by non-lim b edges, then the contour has to be planar under the assum ption of general viewpoint and if the correspon dence is static w ith respect to chcnging viewpoint. Limb edges (or limbs) of a surface are generated by points on the surface whose norm al is orthogonal to the viewing direction. Such edges changes its position on the surface as the viewpoint changes. Non-limb edges, on the other hand, do not change their position on the surface as th e view point moves, they include creases and wireframes. L e m m a 1 A 3-D skew symmetric figure projects as a skew symmetric figure under orthographic projection. P r o o f It is a direct result of the property of the orthographic projection th at parallel lines project as parallel lines and th a t m id points of lines project as mid points of th e projected lines. Therefore, the 3-D lines of sym m etry project as the lines of sym m etry on the image plane and the projection of th e 3-D axis is the line joining the m id points of the lines of sym m etry on th e image plane. T h e o re m 1 If a 3-D contour, formed by non-limb edges, produces a skew sym metric line drawing in the image plane such that the 3-D correspondence is in variant under small perturbations of the viewpoint then the 3-D contour must be planar (under the assumption of general viewpoint). P ro o f: Since the 3-D correspondence is invariant w ith respect to variations of the view point (th at is the projection of the same set of 3-D points correspond 26 (a) < b ) Figure 3.4: Some exam ples of case I. from different view points) then, the assum ption of general viewpoint implies th a t parallel lines in th e image plane m ust be th e projection of parallel 3-D lines, otherw ise they would not project parallel from nearby view points. Therefore we conclude th a t th e 3-D lines, say U, th a t project as the lines of skew sym m etry on the im age plane, m ust be parallel to each other in 3-D, because lines of skew sym m etry are parallel to each other in the im age plane. T he axis of sym m etry in 3-D, which can be obtained by joining the m idpoints of the 3-D lines /j, m ust be straight because its projection on the image plane, which is the axis of skew sym m etry is straight. Therefore, the lines have to lie on a plane, because they are parallel to each other and a single line, th e 3-D axis of sym m etry, intersects them . Hence the 3-D contour, which encloses the lines k is planar. □ Lem m a 1 shows th a t planar skew sym m etric figures projects as skew sym m et ric figures on th e im age plane. Theorem 1 shows th e reverse is also tru e under the stated conditions. We conjecture th a t invariance of th e correspondence is not necessary, b u t have not proved it. Note th a t, if the 3-D contour form ing th e skew sym m etry on the image plane is limb edge, th en th e 3-D contour could be non-planar. For exam ple, lim b edges on surfaces of revolution produce an orthogonal skew sym m etry, [Nal89]. Generally, such surfaces also produce a parallel sym m etry together w ith skew sym m etry and they belong to case II, defined below. 27 3 .2 .2 C a se II Here, th e boundary of th e figure is covered by exactly two sym m etries, identified a th e first and the second sym m etry. The first sym m etry is required to be a parallel sym m etry (or could also be a line-convergent sym m etry in the case of ZGC surfaces). We will argue th a t case II figures are the ones th a t give us the m ost inform ation about the surface shape and th a t such cases are com m on in scenes of everyday experience. T he first four objects in figure 1.1 shows some exam ples of this case. In this thesis we concentrate on such surfaces. Depending on the second sym m etry the type of the perceived surface changes. In this thesis we identify and study three types of surfaces: • If th e second sym m etry is a skew sym m etry w ith straight curves of sym m e try then the surface is hypothesized to be a zero Gaussian curvature (ZGC) surface. Also if th e first sym m etry is an line-convergent sym m etry term i nated by line segments, the surface is hypothesized to be a ZGC surface. In chapter 4, ZGC surfaces are studied under th e assum ption of general view point. • If the second sym m etry is a line convergent sym m etry th a t satisfies the limb boundary condition given in section 6.2, then the surface is hypothesized to belong to Straight Homogeneous Generalized Cylinders (SHGCs). Such surfaces are studied in chapter 6. • If the second sym m etry satisfies the term inator condition given in section 7.3 then th e surface is hypothesized to belong to planar right constant cross section generalized cylinders (PR C G C s). This class of objects are studied in chapter 7 in detail. 3 .2 .3 C ase III This class includes all rem aining cases. Two interesting sub-classes occur here. 28 (b) (a) Figure 3.5: (a) A figure w ith two skew sym m etries, (b) addition of an ex tra curve clarifies th e perceived shape a) We hypothesize th e presence of some boundaries not present in the im age to convert the figure into case I or II. For exam ple consider figure 3.5 (a) w ith two skew sym m etries. Note th a t we have no strong feel for the 3-D shape of this surface. However, if we assum e th a t there is one missing boundary th a t would introduce a parallel sym m etry (and an additional skew sym m etry) as shown in figure 3.5 (b), th e surface shape becomes very distinct. Of course, m ore th an one such construction m ay be possible, each giving an alternative interpretation; some constructions m ay be preferable according to some heuristic criteria. A nother interesting case is where two sym m etries cover m ost of the bound ary b u t not all of it; an exam ple is given in figure 3.6 (a) Here, two choices are available. E ither we can inscribe a sm aller figure inside the larger one, or ex trapolate some of the boundaries to m eet the requirem ents of case II. T he two choices are shown in figure 3.6 (b) and (c). N ote th a t the ex trapo latio n is preferred if the “to p ” surface is also shown as in figure 3.6 (d). b) All other cases. We have not studied such surfaces, b u t we conjecture th a t it is difficult to perceive a specific shape in such cases in general. 29 ~ r A - " (a) (c) Figure 3.6: (a) Face of a cylinder w ith a clipped corner, (b) the parallel sym m etry cover only p art of the surface, (c) the top curve is extended for th e parallel sym m etry to cover the whole face, (d) the top of th e surface is also included 3.3 Constraints We now give th e constraints th a t derive from observations of th e sym m etries and other boundaries in the image. T he constraints are: • Curved shared boundary constraint (CSBC) : relates the orientation of two intersecting surfaces along the curve of intersection. • Inner surface constraint (ISC) : restricts th e orientation of the neighboring points on a surface. • Orthogonality constraint (OC) : uses the assum ption th a t some observed regularity is a result of some 3-D orthogonality. Usage of these constraints will be discussed when individual types of surfaces are studied. Here we give th e definitions and proofs, when needed, of these constraints. 30 3.3.1 C u rv ed S h ared B o u n d a ry C o n stra in t (C S B C ) T his constraint relates the orientations of th e two surfaces on opposite sides of an edge. T he planar version has been used since early days in polyhedral scene analysis [Mac73]. Shafer et al. [SKK83] extended it to th e case of intersection of curved surfaces. Consider two surfaces X i(u ,u ) and X 2(u,v) m eeting along a curve T (s) = (a;(s), y(s ), 2 (5 )) as in figure 3.7. Let N i(u , u) and N 2(u, v ) be th e norm als of X i and X 2 respectively. Along the curve T(s) we can represent the norm als Ni and N 2 as N{(s) = Ni(ui(s),Vi(s)). Since T'(.s) is on the tangent planes of bo th X i and X 2, T '(s) is orthogonal to both N i(s ) and N 2(s) in 3-D. T h a t is Ni(s) ■ T'{s) = 0 N 2( s ) ■ r ( s ) = 0 (3.5) We can rew rite it as r 7 (s) • (N 2(s) — iVi(s)) = 0. Say the norm als Ni(s) are represented in p —q space as Ni(s) = (Pt(s), 1). S ubstituting these in the above equation gives: (x \s),y '(s):z \s )) ■ ((p2(s),q2(s),l) - (p1(s),q1(s),l)) = 0 ^'(s)(p2(s) - p d s )) + y/(s)(92(3) - qi(s)) = 0 (3.6) This is the Curved Shared Boundary C onstraint (CSBC) which states th a t along th e curve T(s) th e orientation of th e surfaces X i and X 2 are constrained by th e tangent, (x'(s),y'(s)) of th e im age of th e curve T(s) under orthographic projection. A stronger constraint can be obtained if we can assum e th a t the intersection curve, T, is planar. Say, T lies in a plane w ith orientation (pc,qc). W ith the assum ption of p lanarity th e constraint equation becomes: - p(s)) + y(s)(qc - q(s)) = 0 (3.7) For ZGC, SHGC and CGC surfaces, we will assum e th a t th e parallel sym m et ric curves are planar either due to some regularity in the figure or by definition of th e individual surface model. 31 Figure 3.7: Two curved surfaces meeting along a curve F 3 .3 .2 In n er S u rface C o n stra in t (IS C ) T he inner surface constraint restricts th e relative orientations of th e neighboring points, w ithin a surface. Consider a curve C{t) = y(t ), z{t )) on a C 2 surface S. For each point P £ C associate a vector R £ Tp such th a t gn ■ dNp = 0 (3.8) w here Tp is th e tangent plane of th e surface S at th e point P and (INp is the derivative of the norm al N of th e surface S in the direction R. T h e o r e m 2 Inner Surface Constraint: Under orthographic projection, if an im age curve Ci is the projection of the curve C on the surface S and R i = (rx,ry) is the projection of the vector R satisfying equation 3.8, then the change of the orientation, (p,q), of the surface S, along the curve C, in the p — q space is restricted by the image vector Rj, as: d(p,q)c > ■ R i = 0 (3.9) P r o o f Let X (n ,u ) be the local param eterization of the surface S around the point P £ C(t) such th a t for P = X (n 0,uo), th e curve X(u,t>o) is th e curve C and th e curve X { u q , v ) is in th e direction R. T h at is, u param eter curve is along th e curve C and v param eter curve is in th e direction R at th e point P. Here we have to show th a t ■ R j = 0 where (p, q) is the norm al of th e surface in 32 the gradient space, du is in the direction of C', R j is th e im age, (xv,yv), of th e vector R = X v = (x v, yv,z v) under orthographic projection. N orm al, N , of this surface at any point is given by: N = i y ' v y \ (310) \ S \ U X u \ y J T hen, th e functions dC}dt and dNn are: dC d X v 1A T d N A T —- = — — = X v, dNR = -j,— = N u (3-11) dt ov ou By equation 3.8 we have X v ■ N u = 0. Let the norm al N of th e surface around point P is represented in th e (p,q) space as N — c(p,q, 1). W here c is the scale coefficient and equal to (p 2 T q2 + l ) -1/ 2. D ifferentiation of N w ith respect to the p aram eter u gives: Nu = cu(p,q, 1) + c(pw ,? u,0) = — iV + c(pu,? tt,0) (3.12) c If we set X v ■ N u = 0 where X v — (x v,yv, zv) and N u is given in equation 3.12 we get: X v • N u = — X v • N + c(xv, yv, zv) ■ (pu, qu, 0) = 0 (3.13) c We also have N • X v = 0 from 3.10. Therefore x vpu + yvqu = ■ R i = 0 (3-14) □ To apply this constraint, we need to identify a curve C in the im age plane for which th e orientation R can be determ ined. In appendix A. 1.3 we show th a t for zero G aussian curvature surfaces any curve on th e surface can be the C curve if th e direction R is chosen to be th e direction of th e rulings of the surface. Following theorem shows how we can use parallel sym m etric curves for this purpose for other classes of surfaces. T h e o r e m 3 Let the family of curves,{Ci}, be on a surface S such that the curves, Ci, are parallel symmetric in 3-D. If the curves Ci are used as the C curves of 33 equation 3.8 then, the tangent of the curves obtained by joining the symmetric points of the curves Ci gives the direction R of the ISC. Conversely, if the curves obtained by joining the parallel symmetric points of curves, Ci, are used as C curves of equation 3.8 then the tangents of the curves Ci give the direction R. P r o o f Consider th e param etric representation S(u, u) of th e surface S' such th a t th e u param eter curves are parallel sym m etric to each other (th e {Cf} family of curves) and v param eter curves join the parallel sym m etric points of th e u param eter curves. For th e first p art of the theorem we have to show th a t equation 3.8 holds or w ith th e current param eterization Su -N v = 0 (3.15) is tru e, w here N = is the unit norm al of the surface. N ote th a t N ■ Su — P u X | N • Sv = 0 by definition. We can substitute — Suv ■ N for Su ■ N v since: 0 = — Sg v N ^ = S UV- N + SU- N V ^ S u - N v = - S uv - N (3.16) S u is th e tangent of th e u param eter curves, and since th e v param eter curves join th e parallel sym m etric points of u param eter curves th e direction of S u(u, v) is independent of the v param eter, th a t is 5 u(u, v) = c(v)Su(u) where c is a scalar function. And; S U v = = c » s » (3.17) By substituting this in equation 3.16 we get N v ■ Su = - N • Suv = —c'(v)(N • Su(u)) = 0 (3.18) For the second p art of the theorem we have to show th a t Sv • N u — 0. Using equation 3.18 we get: 0 = N v ■ Su = - N ■ Suv = - N ■ Svu = Sv ■ N u (3.19) □ 34 « Figure 3.8: Two cylinders (a) is cut along the curves of maximal curvature, and (b) is cut in an arbitrary direction while preserving parallel symmetry, now we have the perception of an elliptical cylinder. 3 .3 .3 O rth o g o n a lity C o n stra in t (O C ) C ertain properties or sym m etries invoke the assum ption of orthogonality in 3-D. T he assum ption of orthogonality was first studied for skew sym m etric contours by K anade [Kan81]. We will assum e orthogonality between th e axis of parallel sym m etry and the lines of parallel sym m etry. For a ZGC surface, this is equivalent to slicing th e surface along rulings to obtain th in skew sym m etric planar strips and assum ing th a t these strips are orthogonally sym m etric in 3-D. This preference is illu strated in 3.8 where in (a) we see a circular cylinder, b u t in (b) we prefer to see an orthogonal elliptic cylinder rath er th an a slanted cylinder. For other surfaces, SHGCs and CGCs, the curves of parallel sym m etry will be assum ed to be orthogonal to th e curves obtained by joining points of correspondence. On th e surface analyzed, say th e tangent of the parallel sym m etry curve makes an angle a w ith th e horizontal and the tangent of the other curve (the curve th a t parallel sym m etry curve will be m ade orthogonal w ith) makes an angle at some point on the surface, as in figure 3.9. Let the norm al of the surface be N = (p, q, 1) at th a t point. Since th e 3-D tangent vectors A and B lie on th e tangent plane of the surface they can be represented as: A = (cos(a), sin (a ),p c o s(a ) + q sin(a)) B — (cos(/?),sin(/3),pcos(/3) + qsin(P)) (3.20) 35 Figure 3.9: Orthogonality constraint and from th e orthogonality of th e 3-D vectors A and B we get: A ■ B — 0, this is the equation of a hyperbola as: (3.21) cos(a — /?) + (p cos a -f q sin a)(p cos j3 -f q sin /3) = 0 This is th e equation of a hyperbola in the p — q space as shown in figure 2.2, constraining the possible orientations for th e surface norm al N. Chapter 4 Zero Gaussian Curvature Surfaces A Zero Gaussian C urvature (ZGC) surface is one where th e th e G aussian cur vature (the product of th e m axim um and m inim um principal curvatures) of the surface is zero everywhere. Cylinders and cones are exam ples of a ZGC surface. Some exam ples are given in figure 4.1. These surfaces are also called developable surfaces since they can be generated from a piece of paper by rolling an d /o r bending w ithout cutting. We feel th a t ZGC surfaces com prise a large and useful class and th a t they represent a n atu ral step up in com plexity from the study of planar surfaces th a t have dom inated previous work in th e field. Lines of m inim um curvature for a ZGC surface, also called rulings, are straight, i.e. it is possible to em bed straight lines on a ZGC surface along these rulings. In section 3.2.2 we argue th a t a surface bounded by parallel sym m etric curves term in ated by straight lines are perceived as Zero G aussian C urvature (ZGC) surfaces. In this chapter we actually prove th a t such a surface m ust be ZGC, along its contours, w ith th e assum ption of general viewpoint. L ater we apply the constraints discussed in section 3.3 to recover the shape of th e ZGC surfaces. At th e end of th e chapter we extend th e reconstruction algorithm s to ZGC surfaces th a t are cut by non parallel planes. 37 Figure 4.1: Exam ples to ZGC surfaces. 4.1 Symmetries and ZGC Surfaces In this section th e types of sym m etries found in ZGC surfaces cut by parallel planes are discussed. T he following theorem asserts th a t case II figures satisfying specific properties m ust have Zero G aussian curvature along its skew sym m etry contours. T h eo r e m 4 If a surface patch generates one parallel sym m etry and one skew sym m etry, with straight curves of skew sym m etry on the image plane, and the straight curves of skew sym m etry are also the lines of sym m etry for the parallel sym m etry, then the Gaussian curvature of the surface must he zero along the curves of skew symm etry. P ro o f: There are two sub-cases, depending on w hether th e curves of skew sym m etry are limb edges or not. a) T h e straight curves of skew sym m etry are produced by lim b edges: In this case ju st th e straightness of th e limb is sufficient for the surface to have zero 38 G aussian curvature along the limb. This can be inferred as a special case of a theorem given by K oenderink [Koe84]. We give an alternative proof here th a t does not need the additional assum ptions used in K oenderink’s proof. Let th e surface X (u ,v ) be param eterized such th a t a u param eter curve is along the limb boundary for v = va. Since th e curve X { u ,v 0) is along th e lim b boundary and the projection of this curve is straight the surface norm al JV * along this curve is constant, th a t is Afu(u,v0) = 0. This condition is a sufficient condition for th e G aussian curvature of th e surface along the X ( u ,v 0) to be zero. T he G aussian curvature, k , of a surface is given by; L N - M 2 * E G - F 2 ^ ^ where L , M, N are the coefficients of th e second fundam ental form of the surface and E , F, G are th e coefficients of the first fundam ental form. The equations of these coefficients are given in equation A .3. Particularly, the coefficients L and M can be w ritten as; L = - X u -Mu M = - X v ■ Afu (4.2) Since N (u ,v 0) = 0, th e G aussian curvature, /c, m ust be zero along th e limb. T he above proof does not require the assum ption of general view point, hence, it only shows th a t along the curve of th e limb boundary th e surface has zei'o G aussian curvature. W ith the assum ption of general view point, we conclude th a t an open region surrounding the limb boundary also has zero G aussian curvature. b) T he straight curves of skew sym m etry are cut edges. Consider figure 4.2, w here a pair of parallel sym m etric curves on a ZGC surface cut along a ruling is shown. Since, in the im age plane, th e tangents t\ and t2 of the top and bottom curves are parallel, by the assum ption of general viewpoint they m ust be parallel in 3-D. Also, since the skew sym m etry curves (one of which is th e ruling in figure 4.2) are straight on th e im age plane, th e 3-D 39 corresponding curves m ust also be straight. T h a t is, the surface em beds straight lines. Therefore th e surface can locally be represented as a ruled surface having equation: X(v,v) = f ( v ) + ug(v) (4.3) w here f(v) and g(v) are arbitrary vector functions of th e param eter v only. T he vector function g(v ) indicates th e direction of th e ruling which are also th e u param eter curves. T he norm al of this surface is: M (u< v ) = ( / ' ( » ) + x g ' W ) X g(v) |(/'( v ) + ug’(v)) X g(v) I For figure 4.2, let the dotted line (ruling) be th e cut boundary for v = v0, and J\fi(ui,v0) and Af2(u2, v 0) be the norm als of th e surface at points where th e ruling intersects th e parallel sym m etry curves. Since the tangents t\ and t2 are the sam e and of course the tangent of th e ruling is constant along it, the surface norm als and A/2 , which are th e cross products of C and t2 w ith th e tangent of th e ruling, m ust be th e same. T h a t is: M (u i,v 0) = J\f(u2,v0) ( f ( vo) + uxg'(v0)) x g{va) = ( f ( v 0) + u2g'(v0)) x g(v0) (4.5) w here = indicates parallelism of th e vectors. T hen, we have th a t either = u2 or th e three vectors f , g \ g are dependent. Clearly «i u2, so / ' , 1 q\ g are dependent and hence the surface norm al J\f is independent of the the u p aram eter curve, th a t is J\f(u,v0) = 0. As in th e case (a) above, this is a sufficient condition th a t the Gaussian curvature of the surface along th e X (u,v0) curve is zero. □ G e n e r a liz a tio n If we assum e th a t the type of th e surface does not change w ithout producing a visible edge, th en we conclude th a t the whole surface m ust be a ZGC surface if it satisfies th e property given in theorem 4. This generalization 40 ,ruling Figure 4.2: A ZGC surface cut along th e “ruling”. m ay appear to be a rath er sweeping one. However, it is no m ore so th an the com m on assum ption th a t a polygonal line drawing corresponds to polyhedral objects. It follows th a t if the parallel sym m etry has a linear correspondence function then th e surface is conic, and if th e correspondence function is an identity then th e surface is cylindrical. We now show how we can infer th e rulings and th e cross sections of the ZGC surface. Rulings are the lines along which th e curvature of th e surface is zero. Cross sections are th e transverse (not necessarily orthogonal) curves; specifically th e curves th a t project into parallel sym m etric curves. We first give two theorem s th a t is key in inferring properties of cross sections and rulings. T h eo r e m 5 Curves obtained by intersecting a Z G C surface with two parallel planes are parallel sym m etric such that the lines of sym m etry are the rulings of the surface. T he proof of this theorem is given in appendix A. 1.1. T h e o r e m 6 Curves, C\ and C%, obtained by cutting a Z G C surface, S, by two non parallel planes, IR and H2 project as line-convergent sym m etric curves such that the lines joining the corresponding points of the image curves are the projec tions of the rulings of S and the line I form ed on the image plane by joining the intersection points of the tangent lines of the line-convergent sym m etric curves is the projection of the 3-D intersection line of the planes and II2 . 41 Figure 4.3: Form ation of the line-convergent sym m etry w ith a ZGC surface an two non parallel planes. P r o o f T he above theorem is visualized in figure 4.3. T he key to the proof of this theorem is th a t th e tangent plane, plane T , in figure 4.3 of th e ZGC surface S is sam e along th e rulings of S. Therefore, both the tangent lines, t\ and £2 , of th e curves C\ and C2 from points P\ and P2 are on plane T. Also the tangent line tx is on plane III and 12 is on n 2. Therefore intersection of G and t 2 is necessarily at th e intersection point of the three planes III, II2 and T. For other rulings the same things repeat for a different T plane, and all the tangent line intersections tak e place along th e line I, th e intersection line for planes Ifi and 1I2. Hence, on th e im age plane too th e intersection of th e tangents takes place on the projection of the line I. N ote th a t th e reverse of these theorem s, th a t parallel sym m etry or line- convergent sym m etry curves must come from parallel or non parallel planar cuts, is not valid; In appendix A. 1.2 we show th a t lines of m axim um curvature which are not necessarily planar, also project as parallel sym m etric curves. However, we believe th a t it is reasonable to infer th a t parallel sym m etry (or line-convergent) curves are planar, unless we have evidence to th e contrary. Lines of curvature 42 (a) Figure 4.4: O bjects w ith cross sections having (a) only one skew sym m etry, (b) two skew sym m etries of a ZGC, in general, can be very complex and it is unlikely th a t an observed surface would be cut in this way. If the curves are neither planar nor along the curves of m axim um curvature, it is quite difficult to obtain parallel symm etry. For exam ple, in order to obtain parallel sym m etry for a conic surface, a sub case of ZGC surfaces, by cutting w ith non-planar cross sections the cuts m ust be tran slated along th e axis of the cone and scaled exactly w ith th e scaling function of the cone. O ur interpretation does allow for piecewise planar cross sections as indicated by m ultiple skew sym m etries. Figure 4.4 shows an exam ple, th e cross section of th e object in figure 4.4 (a) has a single skew sym m etry and is perceived planar, whereas the cross section of th e object in figure 4.4 (b) has two skew sym m etries and the perception is th a t th e cross section has two planar parts. T h at is if the cross section has m ultiple skew sym m etries then it will be piecewise planar such th a t each planar section has one skew symmetry. 4.2 Quantitative Shape Recovery of ZGCs with Parallel Symmetry We now describe our technique of quantitative shape recovery for ZGC surfaces. R em em ber th a t presence of ZGC surfaces is indicated by observing the properties given in theorem 4. T he constraints described in section 3.3 will be applied to 43 ZGC surfaces to recover the local surface orientations. In order to apply these constraints we need to recover the rulings of ZGC surfaces. 4.2.1 R ec o v erin g R u lin g s We can infer the rulings of the surface by joining the corresponding points on the two curves forming the parallel sym m etry by straight lines, as shown in figure 3.1 (c) (the corresponding points on th e two curves have the same tangent). Note th a t the orientation of a ZGC surface does not change along a ruling (this is also proved as a byproduct of th e above proofs in th e appendix). Therefore, if we find th e orientation of th e surface at a single point on a ruling we can extend it along th e ruling. 4 .2 .2 A p p lic a tio n o f C S B C an d ISC T he curved shared boundary constraint (CSBC) given in section 3.3.1 is applica ble along the parallel sym m etry curve which is generated by th e intersection of the ZGC surface and the cross section plane (the top plane in figure 4.5). In dis crete dom ain we need to quantize (p(s),q(s)) as (pt -, <p) and estim ate (:r'(,s),y'(s)) from the image of T(s), which is (x (s),y (s)) under orthographic projection. If the ZGC surface is to be described at n points then there are 2n -f 2 unknowns, 2n for the surface orientations (pt -,<p) and 2 for the cross section plane (pc,qc). This constraint provides us w ith n constraint equations. T he inner surface constraint (ISC) is proved in section 3.3.2 to be applicable along the curves of parallel sym m etry. In appendix A. 1.3 we prove th a t ISC is applicable to ZGC along any arb itrary curve C if th e R direction of equation 3.8 is chosen to be along the rulings of the surface. Inner surface constraint applied along the curve of parallel sym m etry provides n — 1 constraint equations. By using the curved shared boundary constraint (CSBC) in conjunction w ith the inner surface constraint (ISC), we get 2n — 1 equations. This leaves us w ith 3 degrees of freedom for describing a ZGC surface totally. 44 ISC V \ Figure 4.5: The three degrees of freedom present, pc,Qc,d, in a ZGC surface after applying the constraints ISC and CSBC. T he two constraints are shown graphically in figure 4.5. A ZGC surface (a frustrum ) is shown in (a) w ith rulings and the axis of the sym m etry m arked on the surface. T he inner surface constraint (ISC) curve is shown on th e p —q plane. Here th e section of the ISC curve from th e point (pi,qi) to (p;+1,g;+i) is orthogonal to th e ruling r{. T he straight lines on th e p — q plane are the curved shared boundary constraints (CSBC) such th a t at each point i the tangent of th e axis of sym m etry (the dotted curve on th e surface) is orthogonal to the corresponding CSBC line on the p — q plane. T hree param eters required to fix all the orientations (pi, g;) are: the orientation of the plane containing the intersection curve, (pc, qc), and th e quan tity shown as d in figure 4.5 which we call angle parameter. T he angle p aram eter can be described as distance of th e ISC curve from th e point (pC 5?c), which corresponds to an angle in 3-D. Specifying th e length of one of th e CSBC lines is enough to fix th e angle param eter, d. 4 .2 .3 C o m b in in g T h r e e C o n stra in ts T he orthogonality constraint is applied between the direction of the rulings and th e axis of parallel sym m etry. This is equivalent to slicing th e surface along rulings producing th in skew sym m etric planar strips and assum ing th a t these strips are orthogonal sym m etric in 3-D. O rthogonality constraint applied at n points along th e axis of parallel sym m etry provides n constraint equations. The CSBC and ISC constraints provides 45 2n — 1 equations. T he three different constraints of th e previous sections provide 3n — 1 constraint equations, for n points w ith 2n + 2 unknowns (including (pc, qc))• This suggests th a t the system of equations is over constrained (for n > 3). Thus in general, it m ay not be possible to find an in terp retatio n for th e contours such th a t th e surface obeys all the given constraints exactly. This is directly related to th e conflict introduced in section 3.3.3, th a t strict orthogonality forces parallel sym m etry curves to be curves of m axim um curvature which are not planar. How ever for special b u t im p o rtan t cases, these set of constraints are dependent and m ay give a unique answer or even leave one degree of freedom unconstrained. In the following, special cases of cylindrical surfaces and circular cone are analyzed th en a general algorithm is given for all types of ZGC surfaces. 4 .2 .3 .1 C y lin d rica l S u rfaces A cylindrical surface is a ZGC surface for which rulings are parallel to each other, in 3-D. An exam ple is given in figure 4.6(a). Let this surface be param eterized by W (it,u) = (x (u ,u ), j/(u, u),;z(u,d)) such th a t u is along th e axis of sym m etry and v is along th e rulings. As we move along th e axis of sym m etry let th e angle betw een th e tangent of the axis of sym m etry and th e horizontal be a(u) note th a t a is a function of u only, and let the angle betw een th e ruling and th e horizontal be /? as in figure 4.6. Note th a t 0 is constant since all rulings are parallel. We can always ro tate the coordinate system to m ake /3 equal to II/2 , as in the figure. W ith these angles we have X u = (xu, yu, zu) = (cos or, sin a, zu) X v — (xv,yv,z v) — (cos/?, sin /?, zv) = (0, l , z v) (4.6) O ur purpose is to com pute th e surface orientation (p(u),q(u)) along th e axis of sym m etry. A pplying inner surface constraint in equation 3.8 gives pux v + quyv — 0 = $ ■ q(u) = q0(constant) (4.7) 46 CSBC ISC ( a ) Figure 4.6: (a) A cylindrical surface w ith axis of sym m etry and th e rulings m arked, (b) th e constraints ISC, CSBC and the orthogonality for th e cylindrical surface T h a t is th e ISC curve is a horizontal line on the p — q plane as shown by d o tted line in figure 4.6(b). Say th e orientation of th e cross section plane, is ( <7c)• Then the curved shared boundary constraint gives: Xu(pc - p(u)) + yu(qc - qo) = 0 co s(a(u ))(p c — p(u)) + sin(o:(n))(^c — qo) = 0 - » > - + » . < « ) cos(a(tt)) T h a t is, if we fix pc,qc and qo then the surface orientation (p(u), qo) for all values of u is fixed. T he last constraint is given by orthogonality as given in equation 3.21. Since ( 3 = 7r/2 we have: sin (a(n )) + qop(u) cos(a(u)) + q$ sin (a(u )) = 0 (4-9) su b stitu tin g p(u) by 4.8 in above equation gives sin (a (u ))(l + q0qc) + Pc cos(« (« )) = 0 (4.10) Since th e above equation is equal to zero for all values of u then we both have pc = 0 and l + ?o9c = 0 =4 q0 = - l f q c (4-11) 47 W ith th e orthogonality constraint we have pc = 0 and qo = — 1 / qc leaving qc as a variable. T h at is th e three constraints CSBC, ISC and OC, are satisfied for a cylindrical surface and still one degree of freedom , nam ely qc, rem ains. In section 4.2.4 we describe a m ethod to estim ate qc. T he m ethod uses th e shape of th e parallel sym m etry curves. 4 .2 .3 .2 C ircu lar C o n es A circular cone is a Linear Straight Homogeneous G eneralized Cylinder LSHGC whose cross section is a circle. The im portance of circular cones is th a t these are the only ZGC surface th a t have a unique (2 corresponding to Neckers reversal) solution to th e three constraints (ISC, CSBC and OC) given before. In [UN88] we have analyzed the im age of a cone under these constraints and a unique solution is found which is also in agreem ent w ith th e assum ption th a t the ellipse of the cross section in the im age plane is th e projection of a circle in 3-D. The details are o m itted from th e thesis. 4 .2 .3 .3 G en era l Z G C S u rfaces For surfaces other th an cylindrical surfaces and th e circular cone, the three con straints can not be satisfied exactly. We believe th a t in m ost cases the planarity assum ption is stronger th a n th e orthogonality assum ption. Therefore, the fol lowing process tries to m axim ize the orthogonality while keeping the constraints ISC and CSBC satisfied exactly. As discussed in section 4.2.2 there are 3 degrees of freedom left for recon structing a ZGC surface. T he free variables are (pc,? c) and d. We choose the values for these free variables th a t m inim izes th e orthogonality error: E = £ c o s 0 t - (4-12) »=l 48 W here 0, is the angle betw een the two 3-D vectors (A and B in figure 3.9 whose projection on th e im age plane m ake angles a ; and w ith th e horizontal, cos 0t is given by (cos(aj — 1 3 j) -f (p i cos c tj + q j sin a,•)(/?,• cos ( 3 j -f q % sin A ))2 (1 -f (p i cos a i + q i s in a i)2)(l + (p i cos # + q i sin # )2) H ere (p.;, q, ) are dependent on (pc, qc) and d as given by constraints ISC and CSBC. We w ant to m axim ize th e orthogonality by m inim izing the above function E for (pc, qc) and d. We can convert this problem into a 2-D m inim ization problem by associating a d value to each choice of (pc, qc) th a t m inim izes E. U nfortunately, for a general ZGC surface the global m inim um for E occurs when (p c, qc) = (0, 0) and d = oo; this is an infeasible in terpretation. However, the function E, in term s of (po^c) has a “valley” of local m inim a (passing through the origin of the p — q space) and the valley is typically a straight line. Any choice of ( p c , # c ) along this valley is essentially equally acceptable, i.e., we have one degree of freedom to fix. In section 4.2.4 we discuss how to choose a specific value of (pc,qc) on this line using th e shape of th e cross section. 4 .2 .4 E stim a tin g (pc, qc) As discussed in section 4.2.3 the previous three constraints (ISC, CSBC, OC) leave one degree of freedom, such th a t constraining the orientation of th e cross section plane, ( p c , # c ) , is constrained to be along the m inim um line of th e function E. It is expensive to com pute this m inim um line. Instead we use the following gradient descent algorithm to com pute (pc,g c). 1. Choosing a starting line, l0, passing through th e origin, in th e p — q space, in th e direction of th e skew sym m etry axis. Set the current line I = Iq- 2. C om pute th e (pc,q c) for the line I using the m ethod described below. 3. C om pute th e value of E for (pc,gc), check if (pc,qc) is along the m inim um line of E by repeating the above process for lines ±60 degrees off the line and by com paring th e E values for these lines. 49 4. If (pc,qc) is along the m inim um line of E stop. O therw ise choose another line by rotating the line I by 66 degrees in the direction of descending E, and go to step 2. For th e above algorithm we use the im age axis, the line joining the m id points of th e lines joining end points of parallel sym m etry curves, as the startin g line, lo- 4 .2 .4 .1 C o m p u tin g (pcQc) g iv en a lin e /: We ro ta te the coordinate system such th a t th e line I is aligned w ith th e q axis of th e p — q plane then we have pc = 0 and qc is the unknow n quantity. To fix qc, we use the shape of th e cross section. If the cross section is skew sym m etric, th e lines and the axis of sym m etry are m ade orthogonal in 3-D. Ap plication of this orthogonality constraint, given in equation 3.21, restricts (pc,9c) to be on a hyperbola in the p — q space as given in figure 2.2 (b). T he intersection of th e line I and this hyperbola uniquely determ ines (pc, qc). If the cross section is not skew sym m etric, we propose another m ethod based on perceptual properties rath er th an on m athem atical constraints. Specifically, our inform al studies of hum an perception indicate th a t we prefer com pact shapes (as also observed in [BY84]) and th a t we prefer interpretations w here the slant of th e surface is not very high or very low. O ur m ethod is based on th e following observations of hum an perception th a t • We prefer com pact shapes, • We prefer m edium slant to very high or very low slant and • We have a large range of uncertainty for the perceived slants. Based on these observations we propose a two stage m ethod for determ ining qc. F irst we estim ate a value for qc then we u p d ate it w ith a bias tow ards 45°. 50 Fiist approximation to q « q (a) (b) Figure 4.7: (a) A cylindrical object and th e ellipse fitted to th e cross section, (b) th e orientation (pe,qe) th a t would m ake th e ellipse a circle and its projection on th e q axis gives qe, first approxim ation to qc. For th e first estim ation, an ellipse is fit to th e cross section and back projected to an orientation th a t makes it a circle (apart from being m uch faster, this has the advantage over (area)/ (perimeter)2 m easure used by B rady and Yuille [BY84] in th a t it does not require th a t closed contours be given ). T he two steps are described in detail below. F irst E stim a tio n o f qc: An ellipse fitting process is utilized as a first approx im ation for qc. An ellipse is fit to th e cross section contour, then the orientation of the circle (pe,qe), th a t would project as th e fitted ellipse is projected on the q axis, on th e p — q plane to obtain th e first approxim ation of qc, call it qe. Figure 4.7 shows an exam ple. N ote th a t there are two values of (pe,qe) th a t m ake a circle project as the ellipse in the im age plane. These correspond to a N ecker’s reversal and we choose th e one th a t gives a solid shape in terp retatio n to the one th a t gives th e in terp retatio n of a hole. 51 T he behavior of the m ethod is dependent on th e choice of th e ellipse fitting algorithm used. We have experim ented w ith two different ellipse fitting algo rithm s. F irst one is based on the scattering of the boundary points. Covariance m atrix of the equally spaced contour points is com puted by: c = i n * * - * ) 2 l T . { x i - x ) { y i - y ) ^ £ 0 * - - x)(y{ - y) ^ J 2 ( y i - y ) 2 W here (Xi,yi) are th e equally spaced boundary points and (x,y) is th e mean. T he scattering of these contour points is given by th e eigenvalues ei and e2 of th e covariance m atrix C. Say th e un it vectors v\ and v2 are eigenvectors of C corresponding to ei and e2 respectively, then we can approxim ate the cluster of points w ith an ellipse whose m ajor and m inor axes are in the direc tions v\ and v2 w ith m agnitudes yjei/2 and yJe2/2. This m ethod is quite robust when the contour is closed, however for open contours, th e m ethod consistently underestim ates th e eccentricity of the ellipse. The second m ethod is a regular least squares fit of the param eters of the q uadratic representation of the ellipse to th e boundary points. This m ethod is robust when th e contour is sim ilar to an ellipse w hether it is closed or not, but m ay give a bad fit if the contour is not sim ilar to an ellipse. We apply both m ethods to a contour and choose the one having the sm aller fit error (the e in equation 4.15). If th e cross section has repetitive parts, as in figure 4.8, then th e slant percep tion is governed by th e shape of individual parts rath er th an th e overall figure. For closed cross sections we segment them by finding the concavities on the two sides and then m atching them . For open cross sections (as in figure 4.8 (e)) we sim ply segm ent it at inflection points. An ellipse is fit to each p art and corre sponding qe s are com puted for each part. T he qe for the whole cross section is given by weighted average of th e qe values, where the weight is given by the length of th e curve to which the ellipse was fit. Note th a t this is different than th e segm entation of th e cross section described in section 3.2.2 which is based on 52 (a) Figure 4.8: O bjects and ellipses fit for th eir cross sections. T he cross sections of the objects are segm ented based on th eir concavities (or inflection points) and the whole cross section has th e sam e slant. broken skew sym m etry axis and results in a different slant com putation for each segm ented p art as in figure 4.4. Figure 4.8 shows ellipses fit to th e cross sections of various objects. The objects in (b) and (c) are segm ented by th e above m ethod and an ellipse is fit to each part. U p d a tin g qc: T he purpose of this updating process is to sim ulate th e behavior th a t hum ans have in preferring m edium slant to very high and very low slant. We u p d ate qe to obtain th e final qc as follows (after converting qc into degrees): qc = 45° + X(qe - 45°) (4.14) W here A is a confidence factor in th e range [0,1] and is a function of how well th e ellipse approxim ates th e cross section curve. Intuition suggests th a t the b e tte r the approxim ation of th e ellipse the higher the value of A should be and th e closer the qc is to the 45° th e less th e correction should be. T he A we are using : A(£) = ( l - £ 2) (4.15) W here e is the ellipse fit error given by d /yfab, where d is the average distance of the contour points from th e fitted ellipse and a and b are th e half-lengths of 53 th e m ajor and m inor axes of the ellipse. We use an approxim ation (not described here) to com pute d. Note th a t e is in the range [0,1]. We believe th a t th e exact form of the function is not critical. Small changes in qc do not radically affect th e perceived surface shape and hum ans too estim ate qc rath er imprecisely. V a lid a tio n We have conducted a psychological experim ent w ith hum an sub jects on th e perception of qc for cylindrical and conic objects. Results of the experim ent show th at; th e standard deviation of th e perceived angle for th e top plane is quite high, w ith an average standard deviation, a, of 8°. T he interval of uncertainty for the slant of each object, which is th e angle interval th a t contains 90% of th e responses given for th a t object, is 24°. T he com parison shows th a t th e algorithm perform s quite well for a variety of shapes. T he average of th e dif ferences betw een th e m ean of the hum an response and com puted slants is only 6° (sm aller th an th e average standard deviation of hum an responses). In appendix B we provide th e details and results of this analysis. 4 .2 .5 C o m p u ta tio n a l R e s u lts For the results shown in this section the following im plem entation is used. The input to the program are the segm ented curves th a t define th e contour of each object. These segm ented curves are grouped into closed regions using continuity. Each closed region is taken to correspond to an object surface. N ext, we find sym m etries am ong segments of a surface. Every segm ent in a surface is checked for parallel sym m etry against every other segment in the surface. Two segm ents are considered to be parallel sym m etric, if they retu rn a low parallel sym m etry error defined as; j^ J o \ / ( 4 0 ) - x i(p a ))2 + (y'2(s) - y[(ps))2ds (4.16) 54 w here th e segm ents Ci(.s) = (ari(s), y-i(s)) and C 2{s) = ( ^ ( s ) , V2{s)) are para m etrized in term s of their arclength s, p = I1/I 2 is a scaling p aram eter where h and l2 are th e lengths of the segm ents C\ and C2. T he above error m easure is effective only if the entire lengths of two segments are parallel sym m etric to each other. Also, this m easure is lim ited to linear parallel sym m etry (found in cylindrical and conic surfaces). Segm ents are also checked for having the same curvature sign at th e corre sponding points. This m easure is especially useful when th e segm ents are alm ost straight, in which case th e error m easure given in equation 4.16 m ay be low even if th e segm ents are not parallel sym m etric. T he surfaces containing parallel sym m etric segment pair are tre ated as curved and others are treated as planar. For curved surfaces th e curves joining parallel sym m etric curves are checked if they are straight to confirm th a t the surface is a ZGC. T he curved surfaces are associated w ith their planar neighbors, which will be treated as th e cross section having th e norm al (pc> < 7 o 1)- For each object th e orientation of the planar cross section, ( p c ><7c), is com pu ted using the m ethod describe in section 4.2.4. Then th e angle param eter d is com puted by m inim izing th e orthogonality error E given in equation 4.12. T he surface orientation, [pll < ? ,- ) at each point th en is com puted by using constraints ISC and CSBC as illustrated in figure 4.5. In th e following, we show results on some synthetic exam ples as well as a real image. Evaluation of shape from contour results is difficult as there is no real “ground tru th ” . Even when contours are derived from a real object, or from projection of a synthetic object, th e sam e boundaries could have been derived from a projection of infinitely m any other real or synthetic objects. Thus, in a sense, the only good m easure of the perform ance of our algorithm s is a com parison w ith hum an perform ance. We use two graphical m ethods to display th e com puted orientations. T he first one shows the surface norm als as oriented needles along one cross section. T he orientations along other cross sections are th e same, as the orientations are tran slated along rulings. T he orientation and length of a needle 55 is th e projection of a unit surface norm al at th a t point (for us to perceive 3-D orientation from this requires solving a shape from contour problem in itself). The second m ethod is to display th e surface orientations by constructing a synthetic shaded im age from th e reconstructed surface, by assum ing L am bertian reflection and a point source of light (for hum ans to perceive this requires solving a shape from shading problem ). We believe the needle diagram s to be m ore effective th an shaded images for this purpose. We present graphical results for a reader to make h is/h e r own judgm ent. U nfortunately, this can only give qualitative rath er than q u an titativ e evaluation. For th e real im age, we also give a com parison w ith the real object. 4 .2 .5 .1 S y n th e tic Im a g es Figure 4.9 shows com puted surface orientations using our m ethod from contours of objects in figure 4.1. T he input to the algorithm are th e curves (given as a list of points) defining the contour for each object. In our judgm ent, the reconstruction is consistent w ith hum an perception of the given figures. It is w orth noting th a t for all th e objects the com puted orientation at th e lim b boundaries of th e objects is orthogonal to the boundary, even though this is not an explicit constraint in our m ethod. T he cross section of the object in the bottom last row is segm ented into two planar sections based on the observation of th e skew sym m etry of th e cross section. Each section is processed individually b u t th e inner surface constraint is required to apply betw een th e two sections of the object. 4 .2 .5 .2 A R ea l Im a g e E x a m p le To apply our m ethod to real images, we need to first find th e boundaries of the objects and then the sym m etries, if any, contained in them . In general, we can expect object boundaries to be fragm ented, and several intensity boundaries th a t correspond to surface m arkings, shadows and noise to be present. To separate the 56 Figure 4.9: Sam ple contours, th e needle images com puted and th eir images after shading th e object w ith the com puted orientation at every point on th e surface. 57 (a) (b) (c) (d) Figure 4.10: T he processing of a real image; th e cone im age, edges, com puted surface norm als and th e shaded image w ith the com puted surface norm als. object boundaries from these other boundaries, and to fill in th e gaps in object boundaries as appropriate, is a difficult problem in m onocular im age analysis and this thesis is not about such analysis. P erceptual grouping has been suggested as one solution to such problem s and in our group we have developed such tech niques th a t we believe are p art of the solution to these problem s [MN89]. Also, th e fact th a t we are seeking certain specific relations betw een curves should help in th e process of perceptual organization. Here, we only show an exam ple where boundaries can be detected cleanly and no surface m arkings, shadows or high lights are present. Nonetheless, we still deal with the noise in the location of the detected edges and the effects of this noise on com puting tangents. In th e exam ple to be shown, we first detect edges in the im age using a Canny edge detector [Can86]. Edges are then linked into curves. T he curves are seg m ented into sm aller curves by detecting corners using a m ultiscale version of the curvature based corner finder described in [MY87]. T hen th e segm ented curves are given to th e reconstruction system as before. T he tangent of th e axis of the parallel sym m etric curves (necessary for th e curved shared boundary constraint) is com puted by convolving the larger of th e sym m etric curves w ith w ith a first derivative of G aussian kernel having a large standard deviation (a = 10.0), to sm ooth out th e noise. In figure 4.10 we show the results on a real image. T he im age (245 x 300 x 8) is th a t of a circular cone. Figure 4.10 (b) shows edges, figure 4.10 (c) shows the 58 recovered surface norm als as needles, and figure 4.10 (d) shows a reconstructed im age assum ing L am bertian reflection and point source of light. We believe th a t th e results agree well w ith hum an perception of th e original image. T h e average error of th e surface norm al from the actual cone is about 5° (the average error in tangent estim ates of the im age curves is also about 5°). M ost of this error is concentrated near the lim b boundaries where th e im age tangent estim ates also have higher error. N ote th a t th e im age was obtained by perspective projection b u t we have processed it as if it were obtained by an orthographic projection. For this exam ple, th e difference in the two projections apparently does not create a large error. This exam ple is intended to dem onstrate th a t our reconstruction algorithm is robust enough to work w ith real images at least those obtained under controlled conditions. We do not, however, claim to have solved other problem s of m onocular im age analysis. 4.3 Quantitative Shape Recovery of ZGCs Cut by Non Parallel Planes If a ZGC surface is cut by two non-parallel planes, we get a line-convergent sym m etry instead of a parallel sym m etry. In order to com pute line-convergent sym m etry betw een two curves on a general ZGC surface we m ust try all possi ble m onotonic point correspondences betw een th e curves. This is a very costly search. However, for th e case of cylindrical and conic surfaces, th e com putation of line-convergent sym m etry is m uch simpler. M ost of the ZGC surfaces th a t we encounter in our environm ent are in fact cylindrical or conic surfaces. M ore over, we can always segment a general ZGC surface into cylindrical and conic sections (m ostly at the inflection points) and process each section in d iv id u al^ w ith appropriate constraints applied along the lines of segm entation. 59 A p e x Figure 4.11: A conic surface w ith line-convergent sym m etry. For conic surfaces correspondence of line-convergent sym m etry is restricted to be along th e lines th a t pass through a com m on point, th e apex of th e cone. In figure 4.11 line-convergent sym m etry is shown for a cone. T he com putation of line-convergent sym m etry for conic surfaces is, therefore, restricted to checking for the correspondences betw een the curves such th a t the rulings, when extended, intersect at a single point on th e image plane. T he process is further simplified when th e end points of the curves are available. In th a t case the apex point can easily be com puted on th e im age plane and it is only needed to check the correspondence for th a t apex point. For cylindrical surfaces th e apex point is at infinity, therefore, the direction of the apex is used rath er th a n th e location of it. 4 .3 .1 S h a p e R e c o v e r y Process of recovering local surface norm als for ZGCs cut by non parallel planes is sim ilar to th a t of recovering ZGCs cut by parallel planes. F irst we need to decide which cross section curve is to be m ade orthogonal to th e rulings. Figure 4.12 shows three possibilities. In (a) the general preference is to m ake the top cross section curve orthogonal, in (b) the b o tto m one is preferred and in (c) we prefer th e m iddle curve, which is the axis of line-convergent sym m etry. In 60 I,III' " " 1 l,‘llllllllfllllllllll"',l"“ (a ) (b ) (c) Figure 4.12: T hree ZGCs cut by non parallel planes. our im plem entation the curve th a t “looks” orthogonal to th e im age axis, which is the line joining the m id points of th e lines joining th e end points of the line- convergent sym m etry curves, is chosen as th e curve th a t will be m ade orthogonal. How orthogonal a curve looks is determ ined by th e angle betw een th e im age axis and th e line joining two ends of th e curve. For a ZGC surface cut by non parallel planes there are two additional un knowns which are th e gradient param eters of th e second cutting plane. Also, there are two additional constraints on th e orientation of th e planes cutting the ZGC surface. T he first one is a shared boundary constraint; for a ZGC surface S, w ith line-convergent sym m etry, let (pt,qt) be the gradient of th e top plane and let (pb, qb) be th e gradient of th e bottom plane, and let th e intersection line have direction (lXJly) on th e im age plane. Since the top and th e bottom planes actually intersect each other along th e line I in 3-D we have the shared boundary constraint as; (P t Pbt qt ~ qb) ' (C> ly ) — 0 (4-17) 61 Consider figure 4.13; let (pr,qr) be th e local surface gradient of th e surface S along ruling r. This constraint enforces th a t th e 3-D lines tx and t2 be on the sam e tangent plane having gradient (pr,qr )• T he constraint is: t\ x r = t2 x r (4.18) in long form; (a?! - x, y x - y, ~ ( p b( x i - x) + qb{ y x - y))) x (;x 2 - 2/2 — 2/1, ~ ( P r ( x 2 - x x) + qr (y2 - t /i) ) ) = ( x 2 - x, 2/2 - y, ~ ( P t ( x 2 - x ) + qt(y 2 - y ) ) ) x ( x 2 - Xi, V2-yi, ~ { p r { x 2 - Xi) + qr (y 2 - y i ))) (4-19) W ith these two constraints we can com pute th e orientation of th e plane con taining th e curve th a t is not chosen to be m ade orthogonal. Once the orientation of the cross section th a t is chosen to be m ade orthogonal is com puted, com puta tion of the local surface norm als is exactly th e sam e as for ZGCs cu t w ith parallel planes. T he curved shared boundary constraint is applied betw een th e orientation of th e chosen cross section and th e ZGC surface (in the case of parallel sym m etry this constraint is applied betw een any cross section and th e ZGC surface). Inner surface constraint is applied along th e sam e cross section. 4 .3 .2 R e s u lts A ctual com putation of local surface norm als for ZGCs cut by non parallel planes is perform ed by the algorithm for m ultiple ZGC surfaces described in chapter 5. T he m ultiple surface algorithm finds a best fit solution th a t m inim izes the error of not satisfying each constraint available for these surfaces. In a single non parallel cut ZGC surface case, the algorithm ignores the constraints obtained from intersection of m ultiple surfaces (except for the intersection of th e ZGC surface and th e cross section plane). 62 (x,y) (x2,y2) Figure 4.13: C onstraints on th e orientation of the cutting planes of a ZGC surface. Figure 4.14 shows th e results for ZGC surfaces cut by non parallel planes. T he figure shows th e local surface norm als as oriented needles and by a shaded im age obtained by using a Lam bertian shading m odel. For planar surfaces we use a little coordinate fram e, w ith two direction lines orthogonal to th e surface norm al are joined by a line, instead of a single line parallel to th e norm al of the surface, to enhance th e perception of the com puted surface orientation. 63 v S\l/'///M " " Figure 4.14: Recovering th e surface shape of a ZGC cut by non parallel planes. 64 Chapter 5 Multiple ZGC Surfaces M any objects of interest consists of several curved surfaces. Here th e recovered 3-D individual surfaces m ust be in agreem ent w ith th e neighboring surfaces, i.e., surfaces sharing a common boundary. We describe a technique for such integrated m ultiple surface recovery for objects consisting of planar and ZGC surfaces. Fig ure 5.1 shows some sam ple objects. T he problem of finding a consistent shape for all the surfaces is form ulated as a constraint optim ization problem , where the curved shared boundary constraints and inner surface constraints are satis fied exactly and m inim ization is perform ed on assum ption driven constraints of orthogonality of surfaces. 5.1 Representing Surfaces In order to be able to include the contributions of the constraints from each sur face and inter-surface constraints into pool of constraints, appropriate param eter representation for each surface is very im p o rtan t. We use the following param e terization for th e two surface types. 65 Figure 5.1: Some objects consisting of m ultiple planar and curved surfaces. 5.1.1 P lan ar Surfaces For planar surfaces th e gradient space representation (p , q) of th e surface norm al of the plane is used. This is the n atu ral and m ost versatile (for our purposes) representation for planar surfaces. 5.1.2 Z G C Surfaces Due to difficulties of com puting sym m etries for a general ZGC surface, and since a ZGC surface can be decom posed into cylindrical and conic surfaces, in this section we study cylindrical and conic surfaces only. If a ZGC surface is cut by parallel planes, (i.e., it has parallel sym m etry), then it has three degrees of freedom as discussed in section 4.2.2. In chapter 4 these are represented as (pcQc) and d, where (pc, qc) is th e gradient of th e cross section planes and d is the angle param eter as stated in section 4.2.2. In th e m ore general case of ZGC surfaces cut by non parallel planes, studied in section 4.3, two additional param eters are involved, th e gradient of the second cross section plane. In to tal a ZGC surface cut by non parallel planes have five degrees of freedom . Since parallel cut ZGCs are special cases of non-parallel cut ZGCs, we will always use th e non-parallel cut ZGC case. Of th e five degrees 66 of freedom , th e four param eters are the gradients (pt,qt ) and (pb, qb) of the top and th e b o tto m planes cutting th e ZGC surface. T he fifth p aram eter is related to th e angle param eter d introduced in section 4.2.3. However, here we will use a different param eter th an the angle param eter. If th e angle param eter is used, in order to com pute a local surface norm al at a given location, the surface norm als for the whole surface needs to be com puted. This posed no problem for th e single surface m inim ization problem described in section 4.2.3.3, because for th a t m inim ization all surface norm als were needed at each step of th e iteration. However, in m ultiple surfaces case the local surface orientations need to be com puted at random locations (especially at the ends of the surface) only. Therefore we use a different param eter in place of th e angle param eter, th a t enables us to com pute local surface norm als at any point on th e surface. We can m odel a conic surface by using any'3-D axis th a t goes through th e apex of th e cone. We use th e 3-D axis th a t projects as the 2-D axis of th e straight edges of th e cone. If the im age direction of the axis is {a,x,ay) then the 3-D direction of th e axis in gradient space representation is (uax, uay) w here u is a free variable, and it is th e fifth param eter of th e ZGC representation. Given (pt,qt)i (pf>,< 7& ), and u, surface gradient can be com puted at any point on th e surface. Consider figure 5.2, let (pt, qt) be the gradient of th e cross section plane th a t is chosen to be m ade orthogonal to th e surface. T he gradient (p, q) of th e surface along the ruling r is given by com bination of two linear constraints given below. T he first one is th e curved shared boundary constraint, given in section 3.3.1, betw een gradients (p, q) and (pt,qt) using th e tangent (x',y ') of th e intersection curve at th e point th e curve touches the ruling r. T he equation of the constraint is; (p - p t , q - qt) ■ (x',y') = 0 (5.1) This constraint is shown in the gradient space by the line labelled T (x(, y') in figure 5.2. T he second linear constraint is inherently equivalent to th e inner surface con strain t given in section 3.3.2. T he constraint is th a t th e 3-D gradient (pr,qr, 1) of 67 Figure 5.2: T he param eters of a ZGC surface and the constraints in th e gradient (Pi < ? ) o;f th e surface along the ruling r. th e ruling r m ust m ust be orthogonal to the 3-D gradient, (p, q, 1), of the surface along ruling r, th a t is; (p,q, 1) • (Pr,tfr, 1) = 0 (5.2) In figure 5.2 this constraint is shown by the line labelled _ L (pr,qr), which is th e orthogonal line of th e gradient (pr,q T ), i.e., th e gradient of th e set of th e directions th a t are orthogonal to (pr,qr) in 3-D. Note th a t this line is also orthogonal to th e 2-D direction of th e im age of th e ruling. T he gradient (pr, q r ) of th e ruling r is obtained by reconstructing the axis line ( a x , a y ), and th e line betw een points (x , y ) and (x p , y p ) in 3-D (i.e., com puting th e z coordinates of these image points). Since the gradient of the axis line is (u a x , u a y ), fixing zp — 0 , the z coordinate of the point (Xtip , y u p ), Ztip , is given by; Ztip 'Etip 3 C p (5 .3 ) T he z coordinate of th e point ( x , y ) is com puted using the gradient ( p t , q t ) as; z — p t ( x - x p ) + q t ( y - y p ) (5.4) T hen th e gradient (p r , q r ) of th e ruling is given by; 1 ( P r , q r ) = Ztip — Z (x t ip x , y tip y ) (5.5) 68 5.2 Combined Shape Recovery The shape of all the surfaces is recovered sim ultaneously by finding appropriate values for th e param eters of each surface. T he values of the surface param eters are com puted by solving th e following constraint m inim ization problem; m in Ei subject toEx = 0 (5-6) w here Ei stands for error term s resulting from internal constraints of each surface and E x are th e external term s, th a t is, th e constraints obtained by intersection of surfaces. B oth internal and external constraints are based on three simple essential constraint functions. These constraints have been defined before, however, here we redefine th em such th a t the error given by these constraints is uniform and norm alized, i.e., in th e range [0,1.0]. T he gradient space is not uniform , i.e., a constant shift at th e center of the gradient space corresponds to a larger vector difference in 3-D th an th e same shift somewhere farther away from the center. Therefore, th e uniform ity of the error function im plies th a t th e error returned by th e function depends on the 3-D vector differences rath er th an th e differences in gradient space. T he draw back of this norm alization is th a t linear error functions are no longer linear. T he first constraint function is the redefined shared boundary constraint given in equation 3.6. This constraint is applied betw een two gradients. (pi,qi) and (P2 j< ? 2 ), and a 2-D vector (x,y). The constraint is; o z ^ v 'k ((P2 ~ Pi)x + (q2 ~ qi)yY ft. ^ S B C (p 1,qi,p2,q2,x ,y ) = —---------- — — ---------- • (5.7) {{P2 - Pi )2 + {q* ~ qi )2 + l)(z 2 + r ) T he second constraint function is equality of two gradients (pi, qi) and (p2, q2). Since the gradient space is-not uniform , using Euclidean distance betw een vectors (pi,<?i) and (p2,q 2) is not a norm alized and uniform error m easure. Therefore, 69 square of th e sin of th e 3-D un it vectors corresponding to the gradients (pi,Qi) and (P2 ,< }2) is used; Fn(n a „ „ 'l - 1 _ ( f a ^ l ^ 1) ‘ ( f f 2 , 9 2 , l ) ) 2 E q\P l , 9 l , P2 , 92 ) 1 1/ iM 2U -i \ 1 2 V J |(P l,9 l, 1)11(^2,92, 1) | 2 where |v| is th e length of the vector v. T he last one is th e 3-D orthogonality of two 3-D vectors whose projections are (^i? 2/i), (^ 2 , 2/2) and lie on a plane w ith gradient (p , 9 ). This constraint is given in non-norm alized form in equation 3.21. The norm alized orthogonality constraint is; _ \ i(x i,yi,p x i + qyi) • (x2,V2,px2 + qy?))2 , K nX 0(p, q ,x u yu x2, y2) = ------------ r ------------ (5.9) IU h ,9 i,l)n (P 2 ,9 2 ,l)|2 5.2.1 In tern al C on strain ts T he internal constraints are th e constraints obtained from th e regularity assum p tions of each surface. In general they have the following form: Ei — u>pEp + Y2 w°Eo + Y2 wcEc (5.10) W here each ws are weight and Ep is th e error term for th e orthogonality constraint of th e planes, E 0 is th e error term for the orthogonality constraint of th e ZGC surfaces and E c is the error term for th e im plicit constraint of th e param eters of ZGC surfaces. These error term s are described in m ore detail in the following. E p is th e error term for the orthogonality constraint of planes. If a planar surface has a skew sym m etry th en this is the orthogonality function of th e lines and axis of skew sym m etry as given in 5.9, where and (^ 2 , 2/2) are the im age directions of the lines of sym m etry and the axis of sym m etry and (p, q) is th e gradient of the plane. wp is th e weight of E p and is proportional to the total length of th e contour enclosing th e surface. T he form ula used for wv is wp — y/Tc w here lc is th e to tal length of th e curve enclosing th e surface. If the surface does not have skew sym m etry E p is zero. 70 E 0 is th e error term for th e orthogonality of ZGC surfaces. This error m easure is an approxim ation of the one given by £ in equation 4.12. However com puting £ requires com putation of local norm als at each point on th e surface at each iteratio n of m inim ization, which is very tim e consuming. Therefore, we use an approxim ation of the tru e orthogonality error as follows (this error term is exact for cylindrical surfaces and is an approxim ation for conic surfaces): E 0 = E a x + E t (5.11) w here E a x = cos2 (a ), and a is the angle betw een th e gradient (pt, qt) of th e plane containing th e parallel (or line-convergent) sym m etry curve th a t is decided to be m ade orthogonal and the direction of the im age axis (ax,ay). E ax forces the 3-D orthogonal cone axis to be as close as possible to th e image axis. E t = (u in t — u )2 w here u is th e u-param eter of the ZGC surface and Uint is set at the initialization by m inim izing orthogonality error £ given in equation 4.12. wQ is th e weight of the orthogonality term and is proportional to th e to tal length of the perim eter of th e surface, w0 = y/ Tc where lc is th e total length of th e contour enclosing the surface. E c is th e error term for im plicit constraints of th e param eters of ZGC surfaces. Let (pt,qt) and (pb,qb) be th e gradients of the planes containing the two parallel (or line-convergent) sym m etry curves of the ZGC surface. If the ZGC surface has a parallel sym m etry then (pt,qt) should be equal to (Pb,qb), therefore, E c = Eq(Pt,qt,Pb,qb), where E q () is given in equation 5.8. If th e ZGC surface has an line-convergent sym m etry then E c is th e addition of th e constraints given in equations 4.17 and 4.19. wc is the weight and is inversely proportional to the eccentricity of the parallel (or line-convergent) sym m etry curves. If th e parallel (or line-convergent) sym m etry curves are highly eccentric, i.e., th ey are alm ost straight, then th e weight of this constraint is low. T he form ula for wc = 1/ecc, were ecc is the eccentricity of the to tal cross section curve (the eccentricity of a 71 curve is given by y e i / e 2 where e\ and e2 are the first and second eigenvalues of the covariance m atrix of th e curve given in equation 4.13). N ote th a t although ISC is an internal constraint of ZGC surfaces, it is not included in in Ei, because th a t constraint is inherently included in th e param eterization of the ZGC surfaces and in effect it is always satisfied exactly, not m inim ized. 5.2.2 E x tern a l C on strain ts E xternal constraints are the inter-surface restrictions im posed by each surface on neighboring surfaces. E xtrem al constraints have th e following form: Ex = w E v p + H wEpz + w E z z (5.12) w here w is th e weight of each constraint and is equal to y/ Tc where lc is th e length of the curve produced by intersection of the surfaces. Epp, E pz and E zz are the error term s for shared boundary constraint between planes, betw een planes and ZGC surfaces, and betw een ZGC surfaces respectively. In detail, the individual error term s are: E pp is th e error of th e shared boundary constraint betw een th e gradients of th e two intersecting planes as given in equation 5.7. E pz is th e error term for shared boundary constraint betw een a plane and a ZGC surface. T here are two possibilities; th e intersection is along a ruling of the ZGC surface or the intersection is along a parallel (or line-convergent) sym m etry of the ZGC surface. If the intersection is along th e ruling of the ZGC surface then Epz is the error from the shared boundary constraint betw een th e plane and th e local surface norm al of the ZGC surface at th e ruling of intersection. If th e intersection is along one of th e parallel (or line-convergent) sym m etry curves then; Epz = Eq(p,q,pt,qt) (5.13) 72 w here (pt,Qt) is the param eters of th e ZGC surfaces which is the gradient of the plane containing th e intersection curve and (p, q) is th e gradient of th e planar surface. E zz is th e error term for th e shared boundary constraint betw een two ZGC surfaces. T here are various ways two ZGC surfaces m ay intersect each other. Here we only handle the intersections th a t produce a planar intersection curve. There are two types of such intersections: along th e rulings of th e ZGC surfaces or along th e parallel (or line-convergent) sym m etry of the ZGC surfaces. If th e intersection is along th e rulings of the ZGC surfaces th en Shared Boundary C onstraint given in equation 5.7 is applied betw een the local surface norm als of ZGC surfaces at the ruling of intersection. If the intersection is along th e parallel (or line- convergent) sym m etry curves, then, let ( p i, 9 i) and (p2,q2) be th e gradients of th e planes containing the intersection curve in th e representations of the first and th e second intersecting ZGC surfaces. T he error term is: E zz = Eq(p1,q1,p2,q2) (5-14) W hen two ZGC surfaces intersect each other along their parallel sym m etry (or line-convergent sym m etry) curves, how orthogonal b o th surfaces can be m ade depends on how parallel their im age axes are. Therefore we form a new orthog onality error term E on for the intersecting ZGC surfaces to replace their original orthogonality error term s (E0's). Let a be the angle betw een the im age axis of these surfaces, let E 01 and E Q2 be th e error term s for the orthogonality of the intersecting ZGC surfaces. Then the new com bined orthogonality error term is; E on = cos2 ( a ) ( £ ol + E o2) + sm2(a)(EolE o2) (5.15) E on em phasizes th e orthogonality of b o th of th e ZGC surfaces when th e image axis are alm ost parallel to each other, and it em phasizes th e orthogonality of either of the ZGC surfaces when th e im age axes are alm ost orthogonal to each other. 73 5.2.3 S olvin g C on strain t E qu ation s T he to tal error function E is solved using a constraint m inim ization technique, where E x consists of “m ust-satisfy” constraints and Ei consists of assum ption driven error term s as defined earlier. To solve this constraint m inim ization (where th e constraints are non-linear), the problem is converted into a m inim ization form as follows: E = Ei + X E x (5.16) E is m inim ized for successively larger values of A, thus, em phasizing E x m ore at each m inim ization cycle. At the end E x constraints are satisfied alm ost exactly and ECs are m inim ized to th e extent possible. T he set of param eters of the surfaces m inim izing E is taken as th e solution set and used to reconstruct the local surface gradients. 5.3 Results Input to our im plem entation are clean segm ented curves. The surfaces and sym m etries w ithin each surface are com puted by th e m ethod described in section 4.2.5. Some surfaces are com bination of various curved surfaces and there is no distinctive boundary between them . This is the case for the curved surfaces of the left object in figure 5.1. Such surfaces contain m ore th an one parallel (or line- convergent) sym m etry and they are segm ented into sm aller surfaces containing only one parallel (or line-convergent) sym m etry. Figure 5.3 shows the segmented surfaces and the sym m etries (skew, parallel or line-convergent) for each surface. The constraints for each surface and inter-surface constraints including the newly form ed intersections are extracted form ing the error function E. Initial values of th e param eters of E, i.e., th e param eters of all th e surfaces involved in E, are com puted by an initializer. The initializer starts w ith an arb itrary ZGC surface, and sets its param eters as if it is an isolated surface. Then, the initializer 74 Figure 5.3: T he segm ented surfaces, and th e sym m etries com puted for each sur face. T he skew sym m etry of planar surfaces are shown by crosses, th e long line is th e axis of sym m etry and the short one is th e direction of the lines of sym m etries. Parallel and line-convergent sym m etries are shown by their axis. sets the param eters of the neighboring surfaces by keeping th em consistent with th e first surface, and, th e neighbors of these surfaces are processed progressively until th e param eters of all th e surfaces are initialized. T hen E is m inim ized by constraint m inim ization discussed in section 5.2.3. Figure 5.4 shows th e result of m inim ization for the objects in figure 5.1. The com puted surface norm als are shown by needle diagram , as needles sticking to th e surface in the direction of th e local surface norm als. For planar surfaces, a sm all coordinate fram e is used to b ette r show th e com puted surface norm al. We also provide th e shaded images of the objects shaded w ith com puted surface norm als using a Lam bertian shading model. It is interesting to note th a t, for the object on the bottom , th e m iddle sur face, w ith a parallel sym m etry, is initially classified as a curved surface based on th e parallel sym m etry of the surface. However, th e only consistent in terp reta tion for th e whole object is th a t, th e m iddle surface is planar (like a m ountain road), because the top surface is planar. A fter the m inim ization the com puted 75 orientation for th e m iddle surface is, in fact, planar (w ithin error bounds of the m inim ization). We believe th a t th e com putational results are in agreem ent w ith hum an per ception. 76 Figure 5.4: T he needle and th e shaded images obtained from th e com puted sur face norm als for the objects in 5.1 77 Chapter 6 Straight Homogeneous Generalized Cylinders G eneralized cylinders (GCs) are obtained by sliding a cross section (m ostly pla nar) along an arb itrary axis while scaling the cross section by an arb itrary scaling function. GCs are used as a powerful m odeling tool by th e com puter vision com m unity since early 1970s. T he ability of GCs to m odel a wide variety of objects a ttra cted m any researchers. GCs were first introduced by Binford [Bin71] and popularized by Agin and Binford [AB73] and N evatia and Binford [NB77a] then by M arr [Mar82]. Shafer [SK83] classified generalized cylinders according to the properties of th e cross section, axis and th e scaling function of th e GCs. He identified an im p o rtan t subclass of generalized cylinders th a t he called S traight Homogeneous Generalized Cylinders (SHGCs). This is the class of GCs where th e axis is straight and th e planar cross section is scaled but kept homogeneous. SHGCs have becom e the focus of m any researchers topics [Kas8 8 , PCM 89, GB90] because of their ability to m odel a wide class of objects while keeping th e geom ery relatively sim ple to deal with. Straight Homogeneous Generalized Cylinders (SHGCs) are obtained by sliding a cross section, say C , along a straight axis, say A. T he cross section is also scaled as it is swept along th e axis by a scaling function, say r. We can param eterize th e surface, S, of an SHGC, given th e planar cross section C{u) = (x(w), y(u), 0 ), and th e scaling function r(t), as : S(u,t) = (r(t)x(u),r(t)y(u),i) (6 .1 ) 78 Cross sections Figure 6.1: An SHGC along the 2 coordinate axis w ith both m eridians and cross sections m arked. T he axis of th e SHGC in this case is the z axis of th e coordinate system . An exam ple is shown in figure 6.1. N ote th a t th e cross section curves are generated by fixing t and varying u. We will call th e curves generated by fixing u and varying t as th e meridians of th e surface. N ote th a t the cross sections of an SHGC are p lanar because th e cross section function C'(w) is planar, and th e m eridians of an SHGC are planar since th e SHGC has no tw ist in its sweep. Let m eridian edges of an SHGC be edges th a t are along th e m eridians of the SHGC. Usually images of SHGCs do not contain m eridian edges, however, such edges m ay be present if th e cross section has a tangent discontinuity, a corner, (see b o tto m left SHGC in figure 6.2). Figure 6.2 shows some sam ple SHGCs. N ote th a t, even though th e bottom rightm ost exam ple does not seem to have a straight axis, it can be described by an SHGC w ith a vertical axis located outside th e object. 6.1 Surfaces and Their Limb Edges Limb boundaries are very im portant for describing surfaces they enclose. Here we prove a theorem for the projection of limb boundaries th a t enables us to prove im p o rtan t properties of the limb boundaries of SHGC and CGC surfaces. T he definitions and theorem s in this section are valid for both orthogonal and perspective projection geometries. 79 Figure 6.2: Sample Straight Homogeneous Generalized Cylinders. D e fin itio n 2 Tangent line, L v, of a surface, S, at point, P, in a given direction, V , is the line from the point P in the direction of the tangent of the curve, C , obtained by cutting the surface by a plane, n , that passes through P, and contains the normal, N , of the surface at P and the direction given by the vector V. Figure 6.3 shows an exam ple. It is a well known property in differential geom etry [DC76] th a t the tangent lines, L v i, of a surface, S , at point, P , in all Figure 6.3: Tangent line, L v, of a surface S at point P in direction V. 80 Figure 6.4: Tangent plane, Tp, of a surface, S, containing all the tangent lines at point P possible directions, Vi € R 3, are on a plane, Tp, called the tangent plane of the surface at P . M oreover th e plane Tp is orthogonal to th e norm al, T V , of th e surface at P. This property is shown graphically in figure 6.4. N ext, we define lim b edges and their projections for sm ooth surfaces. D e fin itio n 3 The limb edge of a surface is a viewpoint dependent curve on the surface such that at each point on the curve the surface normal is orthogonal to the viewing direction. T he limb edges project on the im age plane as the bounding curve of th e sur face. At these edges the surface sm oothly curves around to occlude itself. This definition of limb edges holds both for orthographic and perspective projection. Limb edges (also called “occluding contours”) can give some very im p o rtan t in form ation about th e 3-D surface they come from; K oenderink [Koe84] has given a nice analysis in previous work. We will show how th e lim b edges help us recover 3-D surface shape later in this paper. T h e o r e m 7 All the tangent lines of a surface at a point, P , which is on a limb edge of the surface for a given projection geometry, project as the same line on the image plane. P r o o f The proof involves a sim ple com bination of th e definition of lim b edges and th e property of tangent planes. Since th e norm al of the tangent plane at 81 P (which is also th e norm al of the surface at P ) is orthogonal to the viewing direction, th e tangent plane projects as a line on th e im age plane. Therefore all the tangent lines at P , which are included in the tangent plane also project to th e one line th a t the plane projects into. □ This theorem , though sim ple and rath er obvious, tu rn s out to be highly useful in proving other im portant properties of limb boundaries. 6.2 Properties of SHGCs Here we present and prove th e properties of the limb boundaries and the cross sections of SHGCs. These properties enable us to identify SHGCs from images of their contours. T h e o r e m 8 For an SHGC, the tangent lines of the surface in the direction of the axis from the points of any given cross section intersect at a common point on the axis of the SHGC. A proof of this theorem m ay be found in [SK83]. Figure 6.5 (a) graphically illustrates th e property. C o ro lla ry The tangents of all m eridian edges at th e points they intersect a single cross section intersect the axis of the SHGC at a single point. Therefore th e tangents of the images of the m eridian edges, at th e point they intersect a single cross section, intersect the im age of th e axis in a single point, under orthographic or perspective projection. It has been shown by Shafer[Sha83] th a t th e lim b edges on an SHGC are not necessarily planar. Therefore the lim b edges of an SHGC are not necessarily along its m eridians, and the tangents of th e lim b boundaries at th e point they intersect th e sam e cross section do not intersect the axis in 3-D. Figure 6.5 (b) shows the lim b edge and its tangent for an SHGC after rotating it, to show th a t in 3-D the tangent of th e limb edge does not intersect the axis of the SHGC. However, it has 82 Limb Meridians Figure 6.5: (a) An SHGC, and its tangent lines, in the direction of th e axis em itting from a single cross section, intersecting at a single point on th e axis, (b) T he tangent lines, Ti, of lim b edges are not th e sam e as th e tangents lines, Tm, of th e m eridians in 3-D. been shown by Ponce [PCM89] th a t under orthographic projection the tangents of th e lim b edges, at the point they intersect the same cross section, intersect th e im age of th e axis at a single point. Here we give a sim pler proof which is independent of th e projection geometry. T h e o r e m 9 The tangents of the projections of the limb edges at the points they intersect the same cross section, when extended, intersect the image of the axis of the SHGC at the same point. P r o o f Say th e lim b edge intersects a given cross section at point P (see figure 6.5). Since th e tangent line Tm from point P in the direction of th e axis of the SHGC (the tangent line of the m eridian passing through th e point P ) intersect th e axis of the SHGC, by theorem 7, the im age of the tangent line Ti from point P in th e direction of th e tangent of th e lim b edge project as th e sam e line as the tangent line Tm and thus im age of th e line T\ intersect th e im age of th e axis at th e sam e point as th e im age of the line Tm intersects. □ Since theorem 7 holds both under perspective and orthographic projection, th e above theorem and th e proof hold for b o th of th e projection geom etries. 83 In the following we show th a t th e cross sections of an SHGC are parallel sym m etric in 3-D w ith th e m eridian curves joining the parallel sym m etric points of th e cross sections. T h e o re m 1 0 The cross sections of an SHGC are parallel symmetric in 3-D with each other such that the meridian curves join the parallel symmetric points of the cross sections. P r o o f Given th e param eterization in equation 6 . 1 for an SHGC, we have to show th a t th e direction of the tangent of th e cross sections is independent of the t param eter curve. The tangent of the cross sections (u p aram eter curves) is given by: = (r(t)x'(u), r{t)y'(u), 0 ) = r(t)(x'(u ), y'(u), 0 ) (6 .2 ) Clearly the direction of Su is independent of th e t param eter. □ C o ro lla ry T he projection of th e cross section curves of an SHGC are also parallel sym m etric in the image plane. And the correspondence function is linear because cross sections are obtained by scaling a reference cross section curve w ithout deform ing it. 6.2.1 R ecoverin g th e C ross S ection s We next show how to find th e projections of cross sections in the im age of an SHGC, given th e images of its external contours. O ur m ethod does not require com plete cross sections, b u t only th e part th a t lies on th e visible face of the SHGC. However, we require th a t the SHGC be cut along its cross sections, otherw ise we would not have a parallel sym m etry between th e im age curves of th e two extrem e cross-sections (C* and Cb in figure 6 .6 ). We conjecture th a t hum ans too do not do well if this condition is not satisfied. The following algorithm recovers th e im age curves C % th a t correspond to the projections of th e cross sections of the SHGC. For each point Pi € Ci do: 84 Figure 6 .6 : Im age of an SHGC cut along its cross sections. Im age of th e top cross section curve is Ct, the bottom one is Cb and the lim b boundaries are on the left Ci and on th e right Cr. 1 . Find the point Pci E Ct such th a t C[{Pi) ~ C^(Pci).1 2 . T ranslate th e cross section curve Ct such th a t th e point Pci € Ct coincides w ith th e point Pi, obtaining th e curve Cit. 3. F ind the point Pcr € Ctt th a t m inim izes th e function f ( P cr) = + d^fdx which is the am ount of scaling required to be applied on the curve Ctt to bring the point Pcr to th e point Pr. T he quantities d\ and are th e length of th e line segm ents from Pi to Pcr and from Pcr to Pr. It can be shown th at local m inim a of the function /(• ) above gives th e correct point Pcr € Ctt such th a t the limb boundary condition C'tt(Pcr) = C'T{Pr) is m et. 4. Scale th e curve Ctt by f( P cr) so th a t th e point Pcr m eets w ith th e point Pr , obtaining th e curve Ci. T he curve Ci obtained by this algorithm is precisely the im age of th e cross section curve between the points Pi and Pr of th e SHGC. Once th e correspondence of th e points Pi and Pr betw een the limb edges Ci and Cr is obtained, we can recover th e im age of the axis of th e SHGC by using theorem 9. Figure 6.7 shows JT he = operator is used for parallelity o f vectors, that is, if Vi = V 2 then V i = AV2 for som e nonzero scalar A. 85 Figure 6.7: Images of the cross sections and axes, th e dashed lines, recovered for th e SHGCs in figure 6.2 the com puted images of th e cross section curves and the axes for SHGCs in figure 6.2. If th e parallel sym m etric points of th e cross section curves are joined, by theorem 1 0 , we obtain the m eridian curves. 6.2.2 O b servin g S H G C s If there are two parallel sym m etric curves w ith a linear correspondence function such th a t they are bound by curves th a t has a straight axis when th e axis is com puted by the above algorithm , then we can hypothesize th a t th e line drawing results from an SHGC. 6.3 Quantitative Shape Recovery of SHGCs To com pute the shape of an SHGC along each recovered cross section curve we can apply th e constraints discussed in section 3.3 as they are applied to a ZGC surface in section 4. For th e following; say th a t th ere are m cross section curves and we would like to com pute the orientation of th e surface at n points along 86 a cross section. T hen we have 2nm unknowns, initially, corresponding to the gradient (p,q) of the surface at nm points. Let (pc,qc) be th e orientation of the cross section of the SHGC, which is th e sam e for all cross sections. 6.3.1 C S B C T he curved shared boundary constraint applies betw een th e orientation, (pc,qc), of the cross section curves Cj and the orientation, (pi,Qi) of each of th e point on th e surface along a cross section. Note th a t (pc,qc) is th e sam e for all cross section curves. T he curved shared boundary states th a t th e line in th e p — q space from th e gradient (/y, qt ) of a point P,, 6 Cj to th e gradient (pc, qc) of the cross section plane is orthogonal to th e tangent, Cj(Pi), of th e cross section Cj at point Pi. Then the constraint equation is: (Pc - P i , q c - q.) ■ C '(P ) = 0 VP; € Cj (6.3) This provides n constraints along each cross section curve. 6.3.2 ISC Inner surface constraint is applied along a cross section using th e tangents of the m eridians at each point. The theorem 3 indicates th a t ISC is applicable along th e cross section curves because cross section curves are parallel sym m etric by theorem 1 0 w ith th e m eridian curves joining the parallel points of th e cross section curves. Inner surface constraint states th a t change of th e orientation (pi+i — pi, <?,;+1 — q% ) of the surface along a cross section curve Cj betw een two consecutive points Pi, P{+i € Cj m ust be orthogonal to th e tangent M 1 - +1^2(Pj+1/ 2) of the m eridian th a t passes through the point Pi+x/2 € Cj which is in th e m iddle m iddle of the points Pi and Pi+1. Then th e constraint equation is: (iPt+i — Pi}<k+ 1 — qi) • ■^'-^+1/2(■^+1/2) = 0 VPiPi+i/^Pi+i E Cj (6 .4) A pplication of ISC provides n — 1 equations for each cross section curve. 87 T here are 2n unknowns for each cross section curve, and two m ore unknowns for th e whole SHGC, the (pc,Qc), by combining the two constraints, we have 2n — 1 constraints for each cross section. Then for each cross section th ere are three degrees of freedom as in the case of a ZGC surface discussed in section 4. 6.3.3 P lan arity o f M eridian s T he m eridians of an SHGC are planar as discussed in section 6 . T hen th e shared boundary constraint can be applied along a m eridian curve as if the curve is obtained by cutting the surface of th e SHGC w ith a plane along th e m eridian. The shared boundary constraint is applied along a m eridian, M , betw een the gradient, (pm,<lm), of tb e plane th a t the m eridian M rests on and th e gradient (Pj, q3) of the points Pj £ M , using the tangent, M'{P3) of th e m eridian curve at each point Pj € M . The constraint equation is : (Pm - Pj,qm - Qj) • M'{Pj) = 0 VP, e M (6.5) Enforcing one m eridian curve to be planar autom atically m akes th e others to be planar too. Therefore, th e planarity is applied only to one of th e m eridians, giving m constraint equations w ith th e expense of two additional unknowns. In to tal, there are now 2nm + 4 unknowns, 2nm for the (p,q) of n m points on the surface, two for (pc,qc), two m ore for (p,n , qm), and there are 2 n m constraint equations, nm from th e CSBC betw een th e cross sections and th e face of the surface, m (n — 1 ) from the ISC, and to from the CSBC of a m eridian curve. T h at is there are four degrees of freedom for recovering the orientation of all the points on an SHGC. These four degrees of freedom corresponds to th e orientation, (Pc qc) , of the cross sections and the orientation, (pm, qm), of the plane containing the chosen m eridian. W ithout any^ assum ptions, we could arb itrarily set these four variables and get a valid reconstruction of the SHGC th a t w ould project like the figure in the image plane. However not all of these reconstructions look natural to hum ans when they observe the image of the contours of an SHGC. 88 H um ans prefer some interpretations over th e others. In th e following section we propose orthogonality as th e preference criteria. 6.3.4 O rth ogon ality For SHGCs we use the orthogonality of th e 3-D tangents of the cross sections and the m eridian curves, m aking each little patch, form ed by dividing the surface along m eridians and th e cross sections, orthogonal. We can apply th e orthogo nality constraint using the equation given in equation 3.21. This constraint is not always exactly satisfied, except for surfaces of revolution. Therefore we perform a m inim ization of the orthogonality constraint as: - = GG'cosfT) = (6 6) where (Ci(Pij))'3 and (A p(P q ) ) 3 are the 3-D tangents of the cross section and m eridian curves at point P^-. These 3-D tangents are dependent on th eir 2-D tangents on the im age and on the orientation (pq, Qij) of th e surface at point PtJ as given by th e equation 3.21. T he gradients (pij,q%j) at each point is dependent on the four variables, (pc, < ? c) an<f (Pm>#m)j discussed in th e previous section. We would like to m inim ize the function E for (pc, qc) and (pm, qm). However from our experim ents we observe th a t m inim ization of E chooses values th a t are always consistent w ith the assum ption th a t the 3-D axis of the SHGC is orthogonal to its cross section. If we enforce th e cross sections to be orthogonal to the axis of the SHGC, the orientation (pc?5 c) of the cross section lies along a line in the p — q space th a t passes through the origin and is in the direction of the im age of the axis of the SHGC. This constraint also, in effect, enforces th e gradient (pm,qm) of th e plane of the m eridians to be orthogonal to the gradient (pc,<?c) of the cross sections. T h at is : (.Pm5 q -m -, 1) ■ (pc5 1 ) = 0 (6*7) 89 For simplicity, say the coordinate system is ro tated such th a t th e im age of the axis of the SHGC is aligned w ith th e y axis of the coordinate system . Then, we have; pc = 0 from the orthogonality of the axis to th e cross section and qm = —1 / qc from equation 6.7. T he param eters pm and qc are th e free variables to be fixed by m inim izing the function E. However, the m inim um of th e function E does not fix the variable qc (except for surfaces of revolution). E ither the function forms a valley along qc m aking any choice as good as any other or fixes qc to be zero which is not a realistic solution. We use th e same m ethod for estim ating qc as described for ZGC surfaces in section 4.2.4. 6 .3 .5 R e su lts We have im plem ented th e SHGC surface recovery algorithm described in this section. The input to th e algorithm are cross section curves (the top and the bottom curves) and th e limb boundaries (the right and left curves). T he inner cross section and th e limb boundary correspondences are com puted using the algorithm given in section 6.2.1. Then, the image of th e axis is com puted using theorem 9. For an SHGC whose axis is aligned w ith the y axis of the coordinate system the m ethod is as follows; First the ellipse fit algorithm described in section 4.2.4.1 is applied to com pute qc, and the function E is m inim ized to com pute pm. Then, the surface is constructed using the constraints discussed in section 6.3 by com puting the surface orientation at each point. Figure 6.8 shows th e needle images and th e shaded images, w ith the com puted surface orientations, of the SHGCs in figure 6.2. T he needle m ap shows local surface orientations as oriented needles. The shaded images display the object by reconstructing a synthetic shaded im age from the reconstructed local surface norm als using a Lam bertian reflection m odel and point source of light. For a discussion of these local surface norm al display techniques please refer to section 4.2.5. As for ZGCs, there is no real “ground tru th ” can be available, only com parision w ith hum an perception is possible. 90 Fsvy/nuv^ ^ ^ 5i»,„vSP£r f / Z / I H " ’ 0 ' Figure 6 .8 : The needle images and th e shaded images generated w ith th e com pu ted gradients at each point of th e SHGCs in figure 6.2 91 Chapter 7 Planar Right Constant Cross Section Generalized Cylinders C onstant cross section G eneralized Cylinders (CGCs) are the class of generalized cylinders th a t have a constant cross section b u t th e axis m ay be an arb itrary 3-D curve. Here, we will focus on CGCs th a t have planar axis and th a t are “rig h t” , i.e., th e cross sections are orthogonal to th e axis; we call such objects P lan ar R ight C onstant cross section G eneralized Cylinders (PR C G C s). Figure 7.1 shows some exam ples. PR C G C s provide an im portant m odeling tool for m ostly “snake like” ob jects th a t can not be m odeled by SHGCs, however, th eir m athm atical properties received very little atten tio n in the past. Some researchers have studied some subclasses of PRCG Cs: Saint M arc et al. [SMM90] have studied a torus, which can be m odelled as a PRC G C w ith circular cross section and circular axis. They used th e property th a t limb edges of a torus produce parallel sym m etry w ithout proving this property. They also estim ated the pose of a torus from a single im age of it, using th e axis of th e parallel sym m etry of its limb edges. Rao [Kas88 ] has studied PR C G C s im aged from a direction orthogonal to th e plane of the axis of th e PR C G C . He has shown th a t, from this particular view point the 3-D lim b edges of PR C G C s are planar. He does not provide any anlysis of PR C G C s from a general view point. In this chapter, we provide a shape from contour m ethod for PR C G C s having arb itrary cross section and arb itrary axis. 92 Figure 7.1: Sam ple PRC G Cs. 7.1 Properties of CGCs In the following, we show th a t limb boundaries of a PR C G C project as parallel sym m etric curves under orthographic projection. Let us choose a coordinate system such th a t th e axis of th e PR C G C lies in the x — z plane and one of the cross-sections, say C(u) = (cx(u ), cy(u), 0), is aligned w ith th e x — y plane. Let A (t) = (ax(t), 0, az(t)) be th e axis param eterized in term s of its arc length, th a t is, \A\ = ax -j-a% = 1 for all t. Also, let A(0) = (0,0, 0) and since th e cross section is orthogonal to th e axis A'(0) = ( 0 ,0 ,1 ). Then the surface of the PR C G C , S {u ,t) is given by: S («, t) = R (A '(0), A '(f)) • C(u) + A{t) (7.1) where R (Vi, V2 ) is the rotation m atrix th a t transform s the direction vector V\ into vector V2 . For A'(0) = (0 ,0 ,1 ) and A '(t) — (a'x{t), 0, a ' (f)) the rotation m atrix R becomes: < (* ) 0 < ( 0 R ~ 0 1 0 (7.2) -a 'x(t) 0 a 'z(t) _ Note th a t the curves generated by fixing t and varying u are th e cross sections of the surface S (u ,t). We will call th e curves generated by fixing u and varying t as th e meridians of the surface. T he m eridians are also th e loci of points on 93 z Meridians A(t) .Cross sections Figure 7.2: A PR C G C w ith b o th m eridians and cross sections m arked. th e cross section as th e cross section is swept along the axis. Figure 7.2 shows an exam ple. L e m m a 2 The meridians of a P R C G C are parallel symmetric and the curves joining the parallel symmetric points of the meridians form the cross sections of the surface. P r o o f We need to show th a t th e direction of th e tangents of th e surface in the direction of th e m eridians, is independent of th e param eter u. d S ( u ,t ) dt dR dt C{u) + A'{t) a"(i) 0 a'j(t) 1 d * i <-(*) 0 0 0 Cy{u) + 0 - a " ( i ) 0 a"(t) 0 ......1 $ t < (* ) < (* ) 0 + cx{u) 0 (7.3) a'z{t) - a " ( f ) = A'{t) + cx(u)(A"(t))x w here (A " ^ ))1- is a vector which is orthogonal to th e vector A"{t) and is in th e x — z plane. Also note th a t, A"{t) ■ A '(t) = 0 since 0 = d(l) = d{A'{t) • A'{t)) = 2A'(t) ■ A"(t) (7.4) 94 meridians crossections (a) (b) Figure 7.3: A PR C G C (half of a torus) (a) from a general view and (b) semi tran sp aren t top view w ith th e limb edges of the previous view and th e m eridians passing from th e points Pi and P2 m arked along w ith th eir tangent lines. Therefore, th e vector (A "(t))x is parallel to th e vector A '(t), since d /( t) lA " ( t) , th e direction of it is independent of the u param eter. □ A lthough the m eridian curves on a PR C G C are parallel sym m etric it is easy to see th a t th e lim b edges of a PRC G C are not necessarily parallel sym m etric in 3-D (see Figure 7.3). However, the following theorem proves th a t the projections of th e lim b edges of a PR C G C are parallel sym m etric under orthographic projection. T h e o re m 11 The limb edges of a P R C G C project as parallel symmetric curves onto the image plane, such that the corresponding points on the limb boundaries belong to the same cross section. A//(t)jL(A,/(t))-L and all three vectors are on a plane (the x — z plane). Then, we can rew rite ^ as: (7.5) It is obvious th a t while the length of the vector ^ depends on the u param eter, 95 P r o o f Here we use th e property given in theorem 7 and in lem m a 2. Consider th e points Pi and P2 in figure 7.3 such th a t both points are on the same cross section. As can be seen in figure 7.3 (b) the tangent lines l\ and I2 from points P i and P2 in th e direction of the lim b edges are not parallel sym m etric in 3-D. However, the tangent lines m i and m2 from points Pi and P2 in th e direction of th e m eridians are parallel sym m etric by lem m a 2. Since th e tangent line l\ project th e same line as th e tangent line m i, and tangent line I2 project the same line as th e tangent line m 2, by theorem 7, th e projection of th e lim b boundaries of a PR C G C are parallel sym m etric. □ 7.1.1 O b serv in g P R C G C s If in th e im age plane there are parallel sym m etric curves th a t are term inated by two curves (possibly closed and having m irror sym m etry which enhances planarity of th e cross section) then we hypothesize th a t it is a PR C G C . T he real test for the line drawing to belong to a PR C G C m ay be perform ed after th e cross sections are recovered as described in section 7.3. For now we assum e th a t the line drawing belongs to a PR C G C and concentrate on th e geom etric constraints available in th e line drawing. 7.2 Quantitative Shape Recovery of PRCGCs Here we discuss the application of the three constraints discussed in section 3.3 along a cross section curve of a PR C G C , to recover th e surface orientation of the PR C G C s. 7 .2 .1 C S B C T he shared boundary constraint can be applied along th e im age of a cross sec tion curve betw een th e pane containing the cross section and th e surface of the 96 PR C G C . Let (pc, qc) be th e gradient of the plane th a t contains th e cross sec tion curve, C (u), whose im age is the im age curve Ci(u) = (c ^ it), cy(u)). Let (p(u),q(u)) be th e orientation of th e points along th e cross section curve C(u). T hen th e shared boundary constraint is: (pc - p{u),qc - q{u)) • (c'x(u),c'y(u)) = 0 (7.6) 7 .2 .2 IS C T heorem 3 indicates th a t ISC is applicable along the cross sections of a PR C G C because cross sections of a PR C G C join the parallel sym m etric points of the m eridian curves which are parallel sym m etric as given by lem m a 2. Since the m eridians of a PR C G C are parallel sym m etric w ith cross section curves forming th e correspondence, the tangent vectors of the m eridians along a cross section is a constant vector which is also parallel to th e axis of the PR C G C as given by equation 7.5. Let the tangent direction of the m eridians along th e cross section C(u) be A! and its image be A\ = {ar x^a'y), note th a t A[ is independent of th e u param eter. For th e sake of sim plicity let us assum e th a t the coordinate system is ro tated such th a t A\ is along the y axis of th e coordinate system , then a'x — 0. The inner surface constraint is : q(u)) • « , a 'y) = 0 =» q(u)'a'y = 0 q(u) = q0 (7.7) By com bining this constraint w ith the CSBC given in equation 7.6 we get: , \ C (u)(qc - q 0) p{u) = — — + ( 7 ' 8 ) 7 .2 .3 O rth o g o n a lity T he last constraint is the orthogonality of th e m eridians to th e cross section curves. T he reader can easily verify th a t the u and t param eter curves in equation 7.1 are orthogonal to each other for all points on th e surface S of th e PR C G C . T hen, we use th e orthogonality by enforcing th e tangent of the m eridians A', 97 whose im age is A[ = (0 , a^) to be orthogonal to th e tangent of th e cross section curve (7, whose im age is Ci(u) = (cx(u), cy(u)), at a point on the surface whose gradient is (p(u), qo)• (0,a’ y,qoay) * (c '.(u ),c'y(u)yp(u)c'x(u) + qo< !y{u)) = 0 (7.9) By substituting p{u) given in equation 7.8 in the above equation we get: a y 4 ( u ) ( 1 + 9 ogc) + a'vpcc fx(u) = 0 (7.10) Since the above equation is zero for all values of u we get both pc = 0 and l + q0qc = 0 =» q0 = - l / q c (7-11) Fixing qc fixes th e orientation of the surface along th e cross section C together w ith the gradient (pc,qc) (which is (0 , qc) in th e ro tated coordinate system ) of th e plane containing <7. However our constraint equations do not constrain qc. 7.3 Recovering Cross Section Curves In th e previous section we have discussed how to recover th e surface orientation at each point on a cross section curve given the im age of the cross section curve. However it is not directly possible to replicate the im ages of th e cross section curves of a PR C G C , such as the ones in figure 7.1, except for th e ends of the PR C G C , where we assum e th e cross section curve is given. T h a t is, we assum e th a t th e surface is cut along its cross sections. Here, we discuss a m ethod for recovering th e cross sections w hen one or both ends of the PR C G C are available, th e m ethod also enables us to reconstruct th e 3-D PR C G C from th e im age of it. At one end of th e PR C G C let the image of the end cross section curve be Ci(u) = (cx(u), cy(u)) and th e im age of th e axis be Aiit) — (ax(t),a y(t)) as in th e previous section. T he im age of th e axis at the point it intersect the cross section C is A;(0) = (ax(0), a^(0)). Say the coordinate system is ro tated such th a t a'x(0 ) = 0 , and th e orientation (pc,qc) of the plane containing th e cross 98 section curve C is com puted using the constraints discussed in section 7.2 w ith pc — 0. T he orientation of th e points along th e cross section curve C is (pu,qo) w here q0 = — 1 f qc and piu) is given by equation 7.8. Since the m eridian curves are parallel sym m etric to the axis of the PR C G C we can use the gradient (p(u),q0) to recover th e tangent of the 3-D axis at t = 0 as: / l ' ( 0 ) = « ( 0 ) , < ( 0 ) , - ( p ( i , X ( 0 ) + 4 o < ( 0 ) ) ) ( 7 . 1 2 ) = (0,<(0),2iM) = (0,fc l) T h a t is A^O) is parallel to norm al, (0,gc, 1), of the plane containing th e cross section C , or the plane IIa containing A'(0) is orthogonal to the plane of C. Also since th e axis, A, of th e PR C G C is planar the plane IIa contains th e whole axis curve A. In the following we give an algorithm for recovering the 3-D cross sections from th e im age of a PR C G C given the gradient (pa,qa) of the plane IIa containing the axis. T hen in th e next subsection we give a m ethod for com puting (pa,<?a) from th e image. T he gradient (pc,qc) of th e plane of the cross section C can be com puted if th e gradient (pa,g a) of IIa is given. T he gradient (pc,?c) m ust lie on a line th a t passes through th e origin and in the direction of A i(0 ), in our case pc = 0 , and (pc,qc, 1 ) is orthogonal to (pa,< 7a,l) then: (0 ,?c, 1) • (Pa,qa, 1) = 0 =* qc = — - (7-13) W e can com pute th e 3-D cross section C from th e im age Ci of it by backpro- jecting Ci to a plane having gradient (pc,qc)■ If th e cross section is rotationally sym m etric1 th e algorithm for recovering cross sections is m uch sim pler. In the following we give an algorithm th a t applies to general, not necessarily rotationally sym m etric case. 1A planar cross section is rotationally sym m etric iff the lines passing through the center of the cross section intersects both sides of the cross section at equal distances. 99 C J u ) A (t) Figure 7.4: A PR C G C w ith a non-rotationally sym m etric cross section. Figure 7.5: A PR C G C w ith, (a) none, (b) one, and (c) both end cross sections available. It can be shown th a t the im age of the axis of th e PR C G C , is not always th e same as th e axis, of th e parallel sym m etry of th e im age of th e limb edges, where th e axis of th e PR C G C is th e trace of a single point on the cross section as the cross section is swept. This is shown in figure 7.4. However the im age curves A i(t) and Bi(t) are always parallel sym m etric to each other such th a t th e corresponding points are on the same cross section. By using lem m a 2 and theorem 1 1 we conclude th a t the images of th e lim b edges are parallel sym m etric to each other (and of course to its axis) as well as to the images of th e m eridians of th e surface, and m eridians of th e surface are parallel sym m etric to the axis of th e PR C G C by equation 7.5, so are th eir images. Therefore the axis of the image of th e limb edges, the Bi(t) curve, is parallel sym m etric to the im age of th e axis of th e PR C G C , the A ,(t) curve. If we take th e axis A of the PR C G C as the trace of the point th a t is the backprojection of I?;(0) to th e cross section plane C. T hen Aj(0) = B{(0). Given th e orientation (pa,qa) of the plane n o containing th e axis A, to recover the 3-D 100 Figure 7.6: T he recovered cross sections for th e PRC G Cs in figure 7.1. cross section say at point Pj on th e im age axis B{\ T he backprojected C of Cj is ro tated by the rotation m atrix P (P '(0 ), B'(P )) to obtain th e 3-D cross section curve Cpiu ) at point P , where B '( 0) and B '{ P ) are obtained by backprojecting P j(0) and Bl(pi) onto the plane IIa. T hen th e points P\ and P 2 th a t produces the lim b edge on th e cross section Cp(u) is identified by equating th e image tangents of C p(u) to the image tangent of limb boundaries P 1 and P2. T he position of the cross section Cp in 3-D such th a t Cp(Px) and Cp(P 2) project as the points P i and P 2 on the im age and the point Pp on Cp th a t corresponds to th e point A(0) on C is on th e plane fla, gives the relative position of the cross section Cp w ith respect to end cross section C in 3-D. 7.3.1 C o m p u tin g (pa,qa) T he gradient (pa>3 a) of the plane IIa containing the axis is com puted by per form ing a search in th e gradient plane. T he objective of the search is to com pute (paiQa) th a t gives a valid reconstruction. A valid construction is one th a t makes the projection of th e cross section points Cp(Px) and Cp(P 2) exactly th e sam e as th e points Px and P 2 on th e im age plane (see figure 7.4). We form an objective function which is th e average distance, on th e im age plane, of th e reconstructed and projected point Cp(P 2) to the point P 2 w hen Cp(Px) and Px is aligned exactly. T hen this objective function is m inim ized for (pa,qa). T he search is facilitated by finding a good initial point for (pa> < Z a ) using the shapes of th e end cross sections. T he analysis in section 7.2 show th a t th e gradient 101 ,\'' 1 / Figure 7.7: T he recovered orientations shown by b o th needle im age and by shad ing th e objects for th e PR C G C s in figure 7.1. 102 (.Pc, Q c) of th e cross section at one end is constraint to be on a line in th e gradient space. A particular value on th a t line m ay be chosen by using th e ellipse fit discussed in section 4. Sim ilar analysis applies to the other end of th e PRC G C (if available). Say th e orientation of th e plane containing th e other end cross section Cn is (pn, 9n)- Then the plane of Cn is orthogonal to th e plane IIa. If (pn, qn) is not equal to (0, qc) we can com pute an initial norm al N a = (pa, qa, 1) of n a as N a = (pn, gn,l) x ( 0 ,^ c,l). If the other end cross section Cn is not available th en th e gradient (pa,qa) is constrained to be on a line by its orthogonality to (0,gc). T he equation of the line containing (pa,qa) is (0,qc, l ) • (pa,qa, l ) = 0. Any particular value of (pa,qa) m ay be chosen on this line as th e initial (pa,qo)- Figure 7.5 shows th a t perception is m ore definite when both ends are available, which confirms th e above observation th a t two ends are m ore inform ative than only one. 7 .3 .2 R e su lts We have im plem ented th e cross section recovery m ethod described in section 7.3. In the im plem entation first the orientations (pc,?c) and (pn,q n) of the end cross sections are com puted. T hen the norm al N a of IIa is found by searching around the gradient given by (p c, qc, 1 ) x (pn, qn, 1 ) th a t gives a valid reconstruction. The 3-D position of each cross section is then found by tran slatin g the end cross section rotating and aligning it w ith the limb boundaries and the plane of the axis na. Figure 7.6 shows the recovered cross sections and figure 7.7 shows the recovered orientations by both needle and shaded images for th e PR C G C s given in figure 7.1. As m entioned in sections 4.2.5 and 6.3.5 there is no real ground tru th to com pare the com puted surface norm als. Since m any objects could produce th e shown contours and th e specific m odel used to generate th e im age contours has no particular im portance, the real com parison is w ith hum an perception. In our opinion the results are in agreem ent w ith hum an perception. We provide graphical display for th e reader to judge th e results on their own. 103 Chapter 8 Conclusion We have presented a theory of how to infer 3-D shape from the contour of curved surfaces if certain sym m etry properties are present. We have given a detailed analysis for ZGC surfaces, objects composed of planar and ZGC surfaces, straight homogeneous generalized cylinders and constant cross section generalized cylin ders. O ur theory does m ake certain assum ptions, as m ust all shape from contour m ethods, to extract shape from contour, b u t we believe th a t our assum ptions are m inim al and th a t th e results agree w ith th e hum an perception. We have ai’gued th a t, in a certain sense, this is the only evaluation th a t can be m ade for shape from contour m ethods, as m any shapes can produce the same contour. We have presented results on several synthetic exam ples and real images. To apply our m ethod to complex real images, where surface m arkings, shadows and highlights m ay be present, will require solution of other m onocular im age analysis problem s. However, we believe th a t our approach can help provide a m odel for such analysis. We have shown th a t contours carry useful, im portant and in m any cases sufficient inform ation for 3-D surface perception. This is not to say th a t all the 3-D inform ation is in the contours and no study of other sources of inform ation, like shading, is needed. R ather, we suggest th a t contours should not be om itted in shape perception and recovery. 104 In this thesis we have provided theorem s and m ethods for effective usage of contours for shape recovery. We believe th a t for the surfaces and objects studied here, the m ethods proposed perform ed very well. 8.1 Future Research Shape from contour m ethod described here has certain lim itations. The algo rithm s described here work on perfect line drawings. However, the edges ob tained by state of th e a rt edge detectors are far from perfect. Also n atural effects like texture, m arkings on th e objects, lighting conditions, and shadows lim it the ability of local edge detectors to ex tract real edges of th e objects. We believe th a t the class of algorithm s known as perceptual grouping are quite promising for inferring real edges of objects. T he fact th a t our algorithm s require certain sym m etry relations also provides constraints for such algorithm s. A lthough a wide class of object surfaces is studied in this thesis, there still rem ain other classes of surfaces for which hum ans can perceive a 3-D shape but are not handled by our algorithm s. M any of these cases are some extensions of the classes studied here, like a ZGC surface having planar cut on one side b u t non-planar cut on th e other side. Such objects do not produce th e types of sym m etries used in our analysis. However, 3-D shape of such surfaces can be recovered w ith the m ethods studied here if we know which side is the planar cut. For a ZGC surface w ith non planar cuts on either side, we conjecture th a t, either one of th e sides is perceived planar or a specific shape is not perceived. However, surfaces w ith non planar cuts needs to be studied in detail. Contours contain the m ost im portant shape inform ation in m onocular im ages, however other shape recovery techniques contain im p o rtan t inform ation too. Com bination of other shape-from techniques, like shading, texture, even stereo, has a p otential to produce b e tte r results. Shading inform ation m ay pro vide useful qualitative shape inform ation which m ay supplem ent the quantitative techniques detailed in this thesis. Also, in this thesis th e effects of gravity and 105 support are not discussed; our algorithm s, as they are stated , are invariant to rotatio n in th e im age plane. Including a gravitational bias m ay enhance the perform ance of th e algorithm s in some cases. 106 Reference List [AB73] G. J. Agin and T. 0 . Binford. C om puter description of curved ob jects. In Proceedings of International Joint Conference of Ariificial Intelligence, pages 629-640, 1973. [Bar83] S. T. B arnard. Interpreting perspective images. Artificial Intelligence, 21:435-462, 1983. [Bar85] S. T. B arnard. Choosing a basis for perceptual space. Computer Vi sion, Graphics and Image Processing, 29:87-99, 1985. [BieS7] I. Biederm an. Recognition by com ponents: A theory of hum an image understanding. Psychological Review, 94(2): 115-147, 1987. [Bin71] T.O . Binford. Visual perception by com puter. IE E E Conference on Systems and Controls, December 1971. Miami. [BT81] H.G. Barrow and J.M . Tenenbaum . Interpreting line drawings as three dim ensional surfaces. Artificial Intelligence, 17:75-116, 1981. [BY84] M. Brady and A. Yuille. An extrem um principle for shape from con tour. IE E E Transactions on Pattern Analysis and Machine Intelli gence., 6:288-301, 1984. [Can8 6 ] J.F . Canny. A com putational approach to edge detection. IE E E Trans actions on Pattern Recognition and Machine Intelligence, 8(6):679-698, November 1986. [Clo7l] M.B. Clowes. On seeing things. Artificial Intelligence, 2( 1):79— 116, 1971. [DC76] M. P. Do Carmo. Differential Geometry of Curves and Surfaces. P ren tice Hall, 1976. [Dra81] W. S. D raper. T he use of gradient and dual space in line drawing interpretation. Artificial Intelligence, 17:461-508, 1981. 107 [GB90] [HB8 8 ] [Hul'71] [KanSl] [Kas8 8 ] [KK83] [Koe84] [Lip69] [Mac73] [Mar82] [MN89] [MY87] [Nal87] A. Gross and T. Boult. Recovery of generalized cylinders from a single intensity view. In Proceedings of the Image Understanding Workshop, pages 557-564, 1990. Pennsylvania. R. Horaud and M. Brady. On the geom etric in terp retatio n of image contours. Artificial Intelligence, 37:333-353, 1988. D.A. Huffman. Impossible objects as nonsense sentences. Machine Intelligence, 6:295-323, 1971. T. K anade. Recovery of the three-dim ensional shape of an object from a single view. Artificial Intelligence, 17:409-460, 1981. Rao K ashipati. Shape Description from Sparse and Imperfect Data. PhD thesis, University of Southern California, 1988. T. K anade and J. R. Render. M apping image properties into shape constraints: Skew sym m etry, affine transform able p attern s, and the shape from tex tu re paradigm . In J. Beck, B. Hope, and A. Rosenfeld, editors, Human and Machine Vision, pages 237-257. Academ ic Press, 1983. J. J. Koenderink. W hat does the occluding contour tell us about solid shape. Perception, 13:321-330, 1984. M. Lipschutz. Differential Geometry. M cGraw-Hill, 1969. A.K. M ackworth. Interpreting pictures of polyhedral scenes. Artificial Intelligence, 4:121-137, 1973. D. M arr. Vision. W. H. Freem an and Company, 1982. R. M ohan and R. Nevatia. Segm entation and description based on per ceptual organization. In Proceedings of Computer Vision and Pattern Recognition Conference, pages 333-341, 1989. San Diego. G. M edioni and Y. Ysum oto. Corner detection and curve represen tation using cubic b-splines. Computer Vision, Graphics and Image Processing, 39:267-278, 1987. V. Nalwa. Line drawing interpretation: B ilateral sym m etry. In Pro ceedings of the Image Understanding Workshop, pages 956-967, 1987. Los Angeles. 108 [Nal89] [NB77a] [NB77b] [PCM89] [Pon8 8 ] [Sha83] [SK83] [SKK83] [SMM90] [Ste81] [Sug8 6 ] [UN8 8 ] V. Nalwa. Line-drawing interpretation: B ilateral sym m etry. IEE E Transactions on Pattern Analysis and Machine Intelligence, 11:1117— 1120, 1989. R. N evatia and T. O. Binford. D escription and recognition of complex curved objects. Artificial Intelligence, 8(l):77-98, 1977. R. N evatia and T.O . Binford. D escription and recognition of complex- curved objects. Artificial Intelligence, 8:77-98, 1977. J. Ponce, D. Chelberg, and W . B. M ann. Invariant properties of straight homogeneous generalized cylinders and their contours. IEE E Transactions on Pattern Analysis and Machine Intelligence, 11(9):951— 966, 1989. J. Ponce. Ribbons, sym m etries and skew sym m etries. In Proceedings of the Image Understanding Workshop, pages 1074-1079, 1988. M as sachusetts. S.A. Shafer. Shadow geom etry and occluding contours of generalized cylinders. Technical R eport CS-83-131, Carnegie-M ellon University, May 1983. S.A. Shafer and T. Kanade. T he theory of straight homogeneous gen eralized cylinders. Technical R eport CS-083-105, Carnegie-M ellon U ni versity, 1983. S. A. Shafer, T. K anade, and J.R . Kender. G radient space under or thography and perspective. Computer Vision, Graphics and Image Processing, 24:182-199, 1983. P. Saint-M arc and G. Medioni. B-spline contour representation and sym m etry detection. In First European Conference on Computer Vi sion, pages 604-606, A ntibes, France, A pril 1990. K. A. Stevens. The visual interpretations of surface contours. Artificial Intelligence, 17:47-73, 1981. K Sugihara. Machine Interpretation of Line Drawings. M IT Press, 1986. F. U lupm ar and R. Nevatia. Using sym m etries for analysis of shape from contour. In Proceedings of the 2nd IC C V , pages 414-426, 1988. Florida. 109 [Wei88] [XT87] I. Weiss. 3-d shape representation by contours. Computer Vision, Graphics and Image Processing, 41:80-100, 1988. G. Xu and S. Tsuji. Inferring surfaces from boundaries. In Proceedings of the 1st IC C V , pages 716-720, 1987. London. 1 1 0 Appendix A Proofs A .l Proofs of Theorems for ZGC Surfaces In this section we give three proofs; two are related to the existence of parallel sym m etries on Zero Gaussian C urvature surfaces (theorem s 12 and 5) and the other proves the Inner Surface C onstraint. All of the proofs uses th e following surface representation. Let X ( u ,v ) = {x(u, v), y(u, v), z{u, u)) be a (u,v) param etric representation of the class C 2 Zero Gaussian C urvature surface X . L et’s assum e th a t the v pa ram eter curves are along the lines of m inim um curvature (rulings) of the surface. N orm al, Af, of this surface at any point is given by: ( a - 1 } where x is the vector product operator, and |V | is the length of th e vector V . Note here th a t \Af\ — 1 . F irst, I, and second, I I , fundam ental forms of such a surface are given by: I ( X udu + X vdv) = E d u 2 + 2Fdudv + Gdv2 (A. 2) 11 (X udu + X vdv) = Ldu2 2M dudv + N d v 2 where E = X u - X u F = X u • X v G = X v • X v L = X uu • A f M = X u v -A f N — X vv ■ Af (A.3) 111 Since th e param eter v is along the ruling (a line) the norm al curvature of the surface in the direction X v, given by I I ( X V ), should be zero, then we have: II( X V) — N = 0 (A .4) G aussian curvature, k, of such a surface is given by [Lip69] L N - M 2 K = E G - F 2 (A.5) Since th e G aussian curvature of the surface is zero setting k = 0 , w ith substi tu tin g 0 for N by equation A.4 gives: M = 0 (A .6 ) A .1.1 P r o o f o f T h e o r e m 5 Consider th e surface X , as given above. Also assum e th a t the u param eter curves on the surface X are planar and parallel to each other. We have to show th a t the tangent of the u param eter curves, is constant w ith respect to v (i.e. jjj^j is a function of u only). Let the planes, th a t th e u param eter curves lie on, have the norm al V (P is constant). T hen we have: • V = 0 = ► 0 = d<yX: ' V ) = X uv-V + X u -Vv = X uv-V (A .7) ov T h a t is X UX V and X uv±.V for all u and v. Also X UJ-Af by equation A .l and X U V ^ M since M = 0, therefore, unless Af / /V, we have X uv = ciAf x 'P and X u = C 2AA x 'P (A .8 ) for some constants c\ and c2. T h at is, X u/ / X uv, and the derivative of | ^ | w ith respect to v is: d -\r \ r | V I "V Xu 'Xuv / Xu X A UV\AU\ A u d v { \ x u\’ \Xu\2 1 j 112 Since X uj / X uv we can su b stitu te X uv by ^ j X u in the above equation: d ( X u N v f jJW„j (X u • X U)\X U V\ N _ n /a in \ ”(X p P C I3 ) _ ( ’ Therefore the tangent of the u param eter curves are parallel to each other at th e points they m eet a particular ruling, resulting in u param eter curves pro jecting as parallel sym m etric w ith the lines of sym m etry corresponding to the rulings. □ A . 1.2 C u rv es o f M a x im u m C u rvatu re for Z G C S u rfaces In this section we show th a t curves of m axim um curvature also projects as parallel sym m etric curves such th a t th e rulings corresponds to th e lines of sym m etry. T h e o re m 12 Curves o f maximum curvature on a Zero Gaussian Curvature sur face project into parallel symmetric curves in the image plane. Furthermore the lines of parallel sym m etry are the projections of the rulings. For this section the term cross section is used to refer to th e curves of m ax im um curvature on the surface. We have to show th a t 3-D tangent of every cross-section at the point th a t they intersect w ith a particular ruling is equal to each other. Consider the surface X given in section A.I. Also assum e th a t th e param eter u is along the cross sections of the surface then we have: X u ■ X v = 0 for all u and v. By equation A.l, we have A f± X v, then: o = di XF : A,r) = X ,„ • AT + X v • M v = N + x v • Afv ov T h at is X v±Afv, also X uJ lX"v since: o = d^Xu • * 0 = X uv • M + X u • J\fv = M + X u • Mv ov 113 (ATI) = X V-Afv (A.12) = X U-Afv (A. 13) Now we have X u±.Af, X v±.Af, X v±.Afv and X u A-Afv and X U±.XV. Therefore Afv should be parallel to N, but by definition dAf-LAf. Since any first order derivative , dAf = Afudu + Afvdv: of A f has to be orthogonal to Af\ observe: 0 = d l = d\Af\ = d(Af ■ Af) = 2dAf ■ Af (A .14) Hence, Afv has to be equal to zero. T h at is th e surface norm al A f is only a function of th e param eter u. Since A f — jv"x , X u • X v — 0, and the tangent j A il X A v | vector of v param eter curves y^y is a function of u on curves are straight lines) then p^yy is a function of the param eter u only, is the tangent of the cross-section and it is constant for all values of v. Consequently at th e points of intersections of cross-sections w ith a particular ruling cross-sections have the same 3-D tangents. Therefore cross-sections forms parallel sym m etry in th e im age plane under orthographic projection (i.e. a point (x,y,z) projects to th e point □ A .1.3 P r o o f o f th e In n er S u rface C o n stra in t for Z G C s Here we will prove th e inner surface constraint asserted by the equation 3.8. Consider a surface as given in appendix A.l, a ZGC surface w ith v param eter curves are along the rulings and u param eter curves are arbitrary. We need to show th a t X u ■ Afv = 0 which is a direct result of the property of ZGC surfaces as follows: 0 = ^ — X uv ■ Af + X u ■ Afv — M + X u ■ Afv = X u ■ Afv (A . 15) □ ly (because v param eter 114 Appendix B Experiments on Perception of qc by Humans We set up an experim ent to evaluate hum an perception of th e slant of the cross section qc. O ur study was lim ited to cylindrical and conic objects shown in figure B .l. We found th a t hum ans are unable to give the slant of th e top surface in useful absolute term s. We get very broad ranges, (such as betw een 30° and 70° for exam ple). Hence, we decided to ask our subjects to com pare the slant of a given object w ith a set of reference objects. For reference objects we choose 12 cylinders w ith slant ranging from 20° to 75° in 5° increm ents as shown in figure B.2. O ur choice of circular cylinders was based on th e belief th a t hum ans, in fact, perceive th e correct slant for tops of these cylinders (even though they are not actually able to give the precise num bers). T his belief is based on two observations. One is th a t th e tops of the reference objects are ellipses and if they are perceived as circles (as they seem to be) the correct slant is obtained. The second is th a t when subjects are shown a cube of unknow n slant and asked to choose the cylinder whose slant was closest, they typically found the right answer (w ithin ±5°). O ur test m ethodology was as follows: T he test figures shown in figure B .l were shown to the subjects one by one, and the subjects were asked to choose the cylinder, am ong the reference cylinders, whose cross section plane slant is closest to th a t of the test figures. They were also allowed to choose two reference 115 Figure B .l: The test figures used in the experim ent 116 20° 25° 30° 35° 40° 45° 50° 55° 60° 65° 70° 75° Figure B.2: T he reference cylinders w ith cross section plane slants, qc, ranging from 20°, the top left one, to 75°, the bottom right one, w ith 5° increm ents. cylinders if they were not certain of a single answer. T hirty three subjects1 participated in this study. T he results are sum m arized in table B .l. In this table each row sum m arizes th e responses given for a test figure by all th e subjects. For each row; the first entry shows th e test num ber, th e next two entries give the m ean and th e standard deviation of th e angles given for th a t test figure. R est of th e columns show twice th e frequency of a slant angle used for the test figure. If a subject has m arked only one slant angle for th e test figure then it is counted as two votes, if he/she has m arked two slant angles then it is counted as one vote for each of the m arked slant angles. R esults of th e experim ent show that: • T he standard deviation of th e slant angles is low only for the first three figures, which are circular cylinders and used for testing the consistency of the subjects. : 21 of these subjects work in the computer vision group, other 12 have different professions. There were no noticeable difference between the responses of the two different groups of people, and hence, we only give data for the combined set. 117 num me am + s t d e v 20 25 30 35 40 45 50 55 60 65 70 1 55 + 4 5 13 25 23 2 29 + 3 26 26 14 3 68 + 3 2 11 52 4 63 + 7 2 5 1 19 15 21 5 52 + 9 1 5 5 4 17 13 14 5 2 6 69 + 4 1 2 1 8 44 7 54 + 9 4 2 2 14 15 19 6 4 8 64 + 4 1 7 7 32 19 9 34 + 10 4 9 16 13 9 5 2 2 10 50 + 10 2 2 8 14 7 15 12 3 3 11 54 + 10 2 2 8 4 9 7 17 14 3 12 62 + 6 2 2 6 16 32 8 13 58 + 8 2 2 3 2 8 32 13 4 14 46 + 10 5 10 8 21 8 4 2 4 4 15 43 + 8 2 5 16 7 13 10 11 2 16 61 + 6 8 8 14 22 14 17 44 + 10 2 2 4 6 12 15 10 8 5 2 18 59 + 9 2 4 2 14 13 12 15 19 60 + 8 2 4 9 21 24 6 20 44 + 11 3 1 3 12 8 14 5 8 6 4 Table B .l: Sum m ary of the perceived slant angles by 33 hum an subjects. The columns are test figure num ber, m ean, standard deviation, and the histogram of th e responses. • S tandard deviation of the perceived angle for the top plane is quite high, w ith an average standard deviation, <r, of 8°. The interval of uncertainty for th e slant of each object, which is th e angle interval th a t contains the 90% of the responses given for th a t object is 24°. • The standard deviation is slightly lower for very high and very low slant objects (objects num bered 6 8 and 12). In these cases th e range of angles is lim ited by the reference cylinders. Table B.2 compares the average of th e slant angles given by hum an subjects and by the algorithm for each test object. F irst two columns of th e table lists th e test figure num ber and the average of the responses by hum an subjects, taken 118 from table B .l. R est of the columns list; the slant angle, qe, th a t would m ake the fitted ellipse a circle, th e corrected slant angle, qc (obtained from qe by equation 4.14) and the error of th e ellipse fit which is the average distance of the contour points to th e fitted ellipse. Figure B.3 shows th e com parison of the hum an responses w ith our algorithm . In this figure the solid bars indicate th e interval of uncertainty (in which 90% of hum an responses lie) and the m ean of th e hum an responses. T he result of our algorithm is shown by an x . For 18 out of 20 objects, th e algorithm com putes a slant value w ithin the shaded bars. Figure B.4 shows the cum ulative distribution of th e errors. The x axis indi cates th e error in degrees (the difference betw een the response of the algorithm and the m ean of the hum ans), and the y axis indicate th e num ber of objects. T he graph shows th e num ber of objects having error which is greater than or equal to a given error, for each error value. T he com parisons show th a t our algorithm perform s well for a wide variety of figures. T he average of the differences betw een th e m ean of the hum an response and com puted slants is only 6 ° (sm aller th a n the average standard deviation of hum an responses). Only for the “U ” shaped test figures, test num bers 14 through 17, there is a consistent high bias. We believe th a t this bias arises from almost perfect circular shape of the “U ” shape which suggests fitting a half ellipse rather th an a full one to obtain the slant angle. This shows th a t m any different sources affect th e perception of the slant angle for given shapes. 119 number mean e l l i p s e a n g le (d eg rees) e l l i p s e c o r r e c te d an gle e l l i p s e erro r 1 55 55 55 o o I 2 29 29 29 o o 3 68 71 70 0 .0 4 63 76 69 .47 5 52 72 67 .44 6 69 81 71 .51 7 54 53 53 . 16 8 64 66 65 . 15 9 34 39 40 . 16 10 50 52 51 . 17 11 54 67 65 o o 12 62 69 66 o o 13 58 63 61 0 .0 14 46 64 58 .56 15 43 63 58 .51 16 61 74 63 .62 17 44 67 60 .57 18 59 63 63 .15 19 60 63 63 .16 20 44 51 51 . 14 Table B.2: Comparison of the perceived slant angles by the human subjects, the second column, and the slant angles computed by the ellipse fitting algorithm. Third column is the first estim ate of qc (shown as qe in figure 4.7), fourth column is the corrected gc, and the last column is the ellipse fit error. 1 2 0 Slant Angle (degrees) i T X | T l 8 9 10 11 12 13 14 15 16 17 18 19 20 Object Number -► Figure B.3: Com parison of hum an responses and th e algorithm discussed in sec tion 4.2.4. In the graph the x axis is th e object num ber and the y axis is the slant angle in degrees. For each object th e shaded bars show the interval of un certainty ( range of the slant angles containing the 90% of the hum an responses), w hite circles shows the m ean of the slant angles given by hum an subjects, and th e crosses show th e slant angle com puted by the algorithm . 121 Number of Objects 20 Error (degrees) -► Figure B.4: T he cum ulative distribution of the errors. T he x axis indicates the error in degrees (the difference between the response of th e algorithm and the m ean of th e hum ans), and the y axis indicate the num ber of objects. T he graph shows th e num ber of objects having error which is greater th an or equal to a given error, for each error value. 122 Appendix C Prespective Projection In all th e previous sections orthographic projection m odel is used. In this section, extension of th e the m ethods and constraints to the perspective projection is discussed for planar and ZGC surfaces. Some researchers have investigated perspective projection before. D raper [Dra81] gave a constraint th at derives from boundary between two faces. Sugi- h ara [Sug8 6 ] gives a linear program m ing m ethod to determ ine the realizability of a line drawing under orthographic projection. His form ulation can also be carried out under perspective projection. However, this m ethod leaves m any degrees of freedom undeterm ined for surface orientations and does not provide a way of incorporating other geom etric constraints like sym m etries. Sugihara does show how to use additional constraints such as shape from shading in an optim ization scheme. In this section, we provide a set of techniques for perspective projection th a t parallel m any of the traditional techniques for orthographic projection and hence can be applied w here th e latter techniques apply. Some of the constraints we describe have been previously presented by Shafer, K anade and R ender [SKK83]; we will m ake specific references to their work in the appropriate places. B arnard [Bar83, Bar85] also studied perspective projection and used th e parallelism and orthogonality of the lines in a different and interesting way. 123 In section, C, we define some term s for perspective projection th at are used to develop the constraints for perspective projection. Some of these constraints are generalizations of the constraints for orthographic projection, however, some are new and apply to perspective projection only. One of our contributions is th e definition of a new kind of sym m etry in th e im age th a t we call convergent symmetry th at can provide unique orientations for such figures directly w ithout using any other constraints! In section C.2 , we show how to apply the m athe m atical constraints we have derived for an example. In section C.3, we give an analysis for curved surfaces. Perspective projection is th e exact projection m odel for th e pinhole cam era and a very good approxim ation for the lens systems used on cam eras for objects in focus. Tn perspective projection there is a focal point, let it be the origin of th e coordinate system . Any point, (x,y,z), in 3-D forms a ray passing through the point and the focal point (the origin). We can represent this ray as (u,v, 1) where u = x / z and v — yjz. The intersection of this ray w ith th e z — 1 plane forms th e image of this point, then the intersection has coordinates (u, v ) on th at plane. Note th a t any point (u ,u ) on th e u — v plane is also a point or a vector, (u,v, 1), in the x — y — z coordinate system . This duality of the points will be used throughout th e paper. In th e paper th e capital italic letters are used to denote th e vectors and points in 3-D, and the z — I plane is called the image plane, the projective plane or the u — v plane. Consider a line, L — R t -f P , in 3-D, parameterized in t, where R = (r#, ry, rz) is the orientation of the line and P = {p^iPyiPz) is any point on the line. From just the image of L on the projective plane we can not recover the parameters R or P . But we can extract some other useful parameters. The image, Li = R tt -f Pt, of the line L , can be considered as a line in 3-D which lies on the u — v plane. Say Li has equation au -f bv + c — 0 on the u — v plane, then R i = ( - 6 ,a ,0 ) P i = {pu,Pv: 1) (C .l) 124 where P,- is any point on th e line Li and P 2 is th e 3-D orientation of the line L{. N ote th a t and Pi are th e only observable quantities of th e line L from th e im age of it, and they are not unique but scalable. We define another useful observable. If the line L does not pass through th e origin 1 then we can define a plane by the line L and the origin. This plane has the property th a t the image, L{, of the line L is the intersection of this plane and the im age plane. This plane will be called th e image generating plane or IG P (this plane is called th e interpretation plane by M acworth [Mac73]). IG P of L can also be constructed by using the line Li and the origin. T he norm al of IG P ( called th e N IG P of the line L) A is given by: A = R x P = Ri x Pi = (a,b,—pua ~ pvb) (C .2 ) T he = symbol indicates the parallelity of the vectors which im plies com ponen twise equality up to a common scale. For U = (uxiuy,u2) and V = (vx,vy,vz), if U = V then (ux,uy,u 2) = (\v x, \ v y, \ v z). T he N IG P of a line is called the vanishing gradient of a line in [SKK83]. Note th a t (— pua — pvb) is equal to c since (pmPv, 1) is on the line Li. In fact c is proportional to the m inim um distance of th e line from th e origin of the u — v plane. T hen A is equal to: A — (a, 6 , c) (C.3) T he NIGP, A, of the line I is a very simple quantity th a t we can obtain directly from the equation of Li on th e image plane. C .l Constraints Under Perspective Projection We now derive several constraints th a t follow from the properties of lines and surfaces under perspective projection. 1If the line passes through the origin then it projects as a point on the im age plane, therefore this plane is defined for every visible line on the im age plane. 125 C .1.1 C h o o sin g a R e p r e se n ta tio n for S u rface O rien ta tio n Consider a plane in 3-D having equation: ax + by + cz + d = 0 (C-4) The norm al of this plane is iV = (a,b,c ). However, as the norm al of the plane has only two degrees of of freedom, we can norm alize the norm al vector as N = (pyq, 1), where p = a/c and q — b/c (note th a t this excludes cases where c = 0). (p, q) can be viewed as a point in the gradient space. G radient space has been useful for orthographic analysis since th e degeneracies of gradient space (norm als of th e planes parallel to the 2 axis) also corresponds to the degeneracies of the orthographic projection (those planes project as lines). However, planes th a t are unrepresentable by the gradient space m ay be present in perspective images. U nfortunately, there is no known representation for th e norm al of a plane having only two com ponents such th a t the equation of th e plane is linear in these com ponents and it is able to represent planes of any orientation. We will derive our constraints first in abstract vector notation and then give two different representations. F irst representation is th e regular gradient space; it has th e advantage of sim plicity b u t contains im portant singularities. T he second one is ju st a regular vector (p, q,r) in 3-D w ith the constraint th at: p 2 + q2 + r 2 = 1 (C.5) This can represent norm al of any plane in 3-D, w ith the added com plexity th a t equation C.5, a quadratic equation, should be included among equations to be solved. C .1 .2 S h ared B o u n d a ry C on strain t This constraint relates the orientation of two planes intersecting using th e image of th e line of intersection. Sim ilar results has been derived previously in [SKK83]. 126 Say two planes have norm als N\ and iV2, then the line, L = Rt -f P, form ed by intersection of these planes has orientation: R = N1 x N2 (C .6 ) If the im age Li = R(t + Pi has th e equation au + bv + c = 0 on th e u — v plane then the N IG P of th e line is A = (a, b, c). Also note th at A = R x P , th a t is A_Li2, therefore: A R = 0 A - N-! x N 2 = 0 (C.7) This is the shared boundary constraint in the form of a vector equation. Depending on th e representation for N\ and JV 2 the final equation changes, but th e vector equation rem ains the same. If the gradient space is used w ith N\ — (Pi,qi, 1) and JV2 = (p2,q2, 1) then the shared boundary constraint becomes: « ( ? 2 - qi) ~ b{P2 - Pi) + c(p29i ” P i92) = 0 (C .8 ) H ere a, b and c are known and unique quantities up to a scale factor. This equation defines a line in p — q space when we fix one of the norm als Ni or N 2. If th e (p,q,r) representation is used, then Ni = (pi, < 71, iq), N 2 — (p2 ,<? 2, r 2 ) and the constraint equation is: a(qir2 - q2r\) + b(rtp2 - pxr2) + c(pi^ 2 - qifr) — 0 (C.9) This equation defines a plane in term s of (p i, < 71, r i) when (p2, 9 2 , ^2 ) is fixed or vice versa. In this representation there are actually two m ore constraint equations which are obtained by substituting (pi, <?i,rq) and (p2,q2,r 2) in equation C.5. C .1 .3 P a r a lle lity T h e o rem In orthographic projection, parallel lines in 3-D project into parallel lines in the image. This, in general, is not the case under perspective projection. However, 127 if we are given th e inform ation th a t two image lines are in fact parallel in 3-D, we can infer some im portant inform ation about their orientations. T h e o re m 13 If two lines L x = R xt + Px and L 2 = Rzt + P 2 are known to be parallel in 3-D (i.e. R x = R 2 = R). Then the orientation of the lines, R, is given by R = A x x A 2 where A x and A 2 are the NIGPs of the lines L x and L 2. P r o o f This is not an entirely new result but has been used previously in [Bar83], [SKK83]. A x and A 2 are com putable from the image of the lines L x and L 2 as given by equation C.3. Also from equation C.2, A x = R x Px and A 2 = R x P2., then we get: Unless th e lines L x and L 2 are parallel to the im age plane, th eir images in tersect and the intersection point, I , is given by I = A x x A 2. Note th a t this theorem does not have any analogy in orthographic projection. C .1 .4 O rth o g o n a lity C o n stra in t This constraint is derived from the knowledge th a t two lines in a plane are orthog onal in 3-D. In orthographic projection, this hint m ay come from the observation of a skew sym m etry. For perspective projection, we will assum e th e orthogonality knowledge to be given for now. In th e next sub-section, we show how it m ay be inferred from a new form of sym m etry th a t we call th e convergent symmetry. This constraint could also be applied to curved surfaces w here we m ay have some m eans of detecting lines of m inim um and m axim um curvatures. Consider a plane II having norm al N, and two orthogonal lines on the plane A x x A 2 = (R x Px) x (R x P2) = (R - (Px x P2))R - (Px • (R x R))P2 = (R ■ (Pi x P2))R = R (C.10) L x — R xt + Pi L 2 — R 2t + P2 (C .ll) 128 These lines have th e N IG Ps A\ — ( a i, 61 ? ci) and A 2 ~ ( a 2 , b2, c2). Since A\ is th e norm al of the plane containing L x and th e origin and also L x is on th e plane II then L x is th e intersection of these two planes. Therefore Rx = A x x N R 2 = A2 x N (C.12) By th e orthogonality constraint (i.e. R X± R 2) we get: i?i • R 2 = 0 (Ax x N ) ■ (A2 x N ) = 0 (C.13) This is the orthogonality constraint in the form of a vector equation. This constraint takes slightly different form depending on the representation used. If we use the gradient space, then N = (p, q, 1) and the constraint equation is: (axa2 + cxc2)q2 + (bib2 -f cxc2)p2 - (axb2 + a2bt )pq- (bxc2 + b2C\)q — (a\C2 — a2cx)p + bxb2 + ctia2 — 0 (C.14) This is a quadratic equation in term s of the gradient (p, q) of the plane II. For m ost choices of param eters, this will represent a hyperbola on the p — q plane, as for orthographic projection, but not necessarily centered at the origin. If we use the (p ,q ,r ) representation then N = (p ,q ,r ) and the constraint equation is: ( bxb2 + a i « 2 ) ^ 2 + ( C 1 C 2 + axa2)q2 + ( C 1 C 2 + bxb2)p2 — ((bxc2 + b2Cx)q - (atc2 + a2Cx)p)r — (—axb2 + a2bx)pq = 0 (C.15) N ote th a t p and q in this representation are not th e same as for th e gradient space. This is a quadratic surface in p ~ q — r space. As in th e case of shared boundary constraint, this equation should be used in conjunction w ith the con strain t equation C.5 which is a sphere. W ith these constraints only one degree of freedom is left for th e orientation of the plane II. As in th e case of orthographic projection, the orthogonality constraint by itself does not give unique orientations, and some ad-hoc choices could be m ade such as choosing th e tips of constraint hyperbola [Kan81]. 129 C .1 .5 C o n vergen t S y m m e tr y In this section an object refers to a planar surface in 3-D bounded w ith a piecewise linear boundary, and a figure refers to the projection of the boundary. An object is called symmetric in 3-D if there are lines on the object joining th e points of the boundary, called lines of sym m etry, such th a t the locus of th e m id points of these lines forms another line, called axis of sym m etry, and th a t th e axis of sym m etry is orthogonal to the lines of sym m etry. An arrow like object and its sym m etry axis w ith th e lines of sym m etry are shown in figure C .l(a). If we project a sym m etric object using orthographic projection we get a figure having skew sym m etry as proposed by K anade [KanSl]. If we use perspective projection then we get a figure having a new sym m etry called convergent symmetry. D e fin itio n : A figure is said to be convergent symmetric if there exist point to point correspondences between all points of the figure such th at: (a) All lines joining points of correspondence, called lines of sym m etry, intersect in a common point on the image plane. (b) T he projection of the m id-points of the 3-D lines of sym m etry lie along a straight line on the image plane. U nder perspective projection, projections of parallel lines m eet at a point on th e image plane. Therefore the projections of the lines of sym m etry should meet at a point when extended on th e im age plane, th a t is they should be convergent. T he axis of sym m etry is, however, no longer defined by the locus of the mid points of the lines of sym m etry in the image plane. Instead, we require th a t the m idpoints of the lines of sym m etry, in 3-D be along a straight line. We show how this 3-D com putation can be perform ed in the following. Figure C .l (b) shows an exam ple of an arrow like object under perspective projection w ith its axis and lines of sym m etry. F irst, we give a formal definition for convergent sym m etry and then a procedure for checking it. 130 (b) (a) Figure 0.1: (a) An arrow like planar object w ith its axis of sym m etry, solid vertical line, and lines of sym m etry, dashed horizontal lines, (b) Projection of the arrow like object and its convergent sym m etry lines; dashed lines are the lines of sym m etry m eeting at the point (uc, vc), Li is one of the lines of sym m etry m eeting the boundary at points E and F. T he vertical solid line is the axis of sym m etry, L s, having N IG P of A = (a,b,c). The corresponding points would be easier to determ ine in a figure w ith several corners, as each corner m ust correspond to another corner. However, th e above definition is general and applies to any figure (including curved figures). In gen eral, of course, we can first choose the point of convergence, and then define lines of sym m etry from it. The following procedure is to check w hether the second p art of th e definition is also satisfied. For every line of sym m etry we can find th e projection of its 3-D m id point. Consider figure C .l (b). Let L\ — R\t -f Pi be one of the lines of sym m etry, and let E and F be the two corresponding points on this line w ith image coordinates of (ue,ve) and (« / , n / ) respectively. Let (uc,vc) be th e point of convergence for th e lines of symm etry. Then Pi = (u c, nc, 1) from the parallelity theorem , and Pi = (we,u e, I) as Li passes through E. W ith these values for Ri and Pi the u — v coordinates of the image of a point on the line Li is given by: ue + tuc ve + tvc. (' 1 -f-1 1 -f- t L ) (C.16) 131 T he t value th a t gives th e point F on line L\ is given by the solution to the equation h(uf ,Vf,l) = Rit + Pi (C.17) where h is a constant, such th a t the above equation is satisfied only for a particular value of t and h. T he above equation gives: tin t = = tUJZls. = h - ! ( 0 .18) Uc Vc Elim inating the constant h gives two solutions for tint: tini = (C.19) Uf - Uc Vf — vc These two values for t{nt are in fact the same since the point (uf,Vf) is on the line defined by the points (u e, v e) and (u c, v c). The image coordinates for the projection of the m idpoint of the line Li between the points E and F is obtained by substituting t = tint/2 into the equation C.16. (2 ue - uc)uf - u cue (2 ve - vc)vf - uene ^ ^ uj + ue — 2 uc ’ vf + ve — 2 vc This gives us a procedure for finding the projection of the 3-D m id-point of any given line of symm etry. To check w hether a given figure is convergent sym m etric, we simply need to find the projections of m id-points of all lines of sym m etry and check th a t they lie on a straight line, say L s (L s is th e projection of the 3-D axis of sym m etry). T he N IG P value for L s, A = (a, 6 , c), can be obtained by A = T x P (C.21) where T and P are m idpoints of any two distinct lines of sym m etry (the m id-points are given by equation C.20). C . 1.5.1 C o m p u tin g O rien ta tio n U sin g C on vergen t S y m m e tr y Now we will apply the constraint th a t the axis of sym m etry is orthogonal to the lines of sym m etry in 3-D. F irst, we state a theorem related to this. 132 T h eo re m 14 If a convergent symmetric figure is assumed to be a perspective projection of an orthogonal symmetric planar object, then the orientation of the planar object can be determined uniquely (unless the convergent symmetry is ac tually a skew symmetry with point of convergence at infinity, the axis of symmetry through the origin of the image plane and the lines of symmetry are orthogonal to the axis of symmetry on the image plane). We will give a constructive proof of this theorem in the following. Note th a t th e theorem asserts th a t th e constraints provided by convergent sym m etry are m uch stronger th an those provided by skew symm etry. T he process is sim ilar to th a t of skew sym m etry analysis, but unlike in the case of orthographic projection, in perspective projection th e axis of sym m etry intersects every line of sym m etry at a different angle th an the others on the image plane. This results in a different constraint equation at every point on the axis of sym m etry. Every equation gives a different constraints hyperbola on the p — q plane (if p — q space is used). But all of these hyperbolas pass through one point on the p — q plane and this point is the only solution to all of these constraints equation. Therefore, we get a unique answer for the surface norm al by using convergent sym m etry, except for some special cases noted in the theorem . In gradient space (p , q) representation, we can find a closed form solution. And the degeneracies of p — q space can be com pensated as will be clear later. Consider the object in figure C .l. Axis of sym m etry has the N IG P of A — (a,b,c ) and assum e th a t A is norm alized (i.e. \A\ = 1). There are infinitely m any lines of sym m etry all of which pass through the point (u c, bc) on the image plane. Say the intersection of these lines w ith the v axis has th e coordinate (0 , k) where k is a param eter having a range th a t covers the figure. Then these lines have the NIGP: Ai = (0 , k, 1 ) x (uc, vc, 1 ) — (k vc,u c, uck ) (C .2 2 ) 133 Say the normal of the plane containing the object is N = (p,q, 1), then from the orthogonality constraint we have: (A x N) ■ (At x N ) = 0 (C.23) k((—cq2 + bq — cp2 + ap)uc + aq2 — bpq — cp + a) + (— aq2 + bpq + cp — a)vc + ((—ap — c)q + bp2 + b)uc = 0 (C.24) This constraint should be satisfied independent of th e value of k , then we get two constraints of the form: (— cq2 + bq — cp2 + ap)uc + aq2 — bpq — cp + a = 0 (—aq2 + bpq + cp — a)vc -f ((—ap — c)q -f bp2 + b)uc — 0 (C.25) There is only one real solution to these equations given by: (ab2 + a3)v 2 + ( — 63 — a 2b)ucvc + (~b 2 — a 2)cuc — ac2 + a ^ (b2 + a2)cv2 + (be2 — b)vc + (b2 + a2 )cu2 + (ac2 — a)uc — _ ((«fr2 + + (b2 + a 2)c)vc + (-fr 3 — a 2b)u2 c + be2 — b . , ^ (b2 4 - a2)cv 2 + (be2 — b)vc + (b2 + a2)cu 2 + (ac2 — a)uc This gives us the norm al, N = (p, q, 1 ), of the plane containing the object in term s of th e observable, A = (a,b,c), and the intersection point, (uc,vc): of lines of sym m etry on th e image plane. As m entioned before in th e gradient space rep resentation norm al of the planes th a t are parallel to z axis are not representable, th a t is because those planes have th e th ird com ponent of the norm al vectors equal to zero, and equivalent of a vector, V = (f , g , I), under this representation is ob tained by dividing the vector by the th ird com ponent of the vector, (f/l,g/l, 1 )- However, the expressions for p and q in equation C.26 have th e property th a t the denom inator for p and q are the sam e then by m ultiplying th e N vector w ith this denom inator we get another vector, N having the same orientation as N but have no singularity as for representing planes parallel to z axis. Then the vector N' is : N f — ((ab2 + a3)v2 c + ( — 63 — a 2b)ucvc + ( — b2 — a 2)cuc — ac2 + a, 134 — ((ab2 + a3)uc + (b2 + a2)c)vc + (— 6 3 — a2b)u2 c -f be2 — b, (b2 + a2)cv2 + (be2 — b)vc 4 - (b2 4- a2)cu2 4- (ac2 — a)uc) (C.27) Unlike the skew sym m etry under orthographic projection, for a convergent sym m etric figure in perspective projection we can com pute the orientation of the planar surface uniquely. T h at is, we even do not have th e Neckers reversal, this is also in agreem ent w ith hum an perception. For exam ple the cube in figure C .2 can be reversed if one tries b u t the reversed figure does not look sym m etric at all. Therefore if we want to bias towards sym m etric objects then there is only one answer for a convergent sym m etric figure. This is another instance in which th e perspective projection can be used to give m ore inform ation than the orthographic projection. All of th e above derivations assume th at the intersection point (uc,vc) is not at infinity. In the latter case, we can obtain the solution using the lim its of the solutions. Let us say the slope of the lines of sym m etry is m , then we can obtain the solution by replacing vc by m uc in equation C.26 and taking the lim it as uc goes to infinity. Then th e solution in 3-component vector form is: N' — (am2 — 6m , — am 4- 6 , cm 2 4- c) (C.28) In th e theorem of convergent sym m etry, we m entioned th a t we get a unique orientation from convergent sym m etry except under some special cases. In fact, if lines of sym m etry are parallel to each other on the image plane, and image of the axis of sym m etry is passing through the origin of th e image plane (i.e. c = 0 ), and on th e image plane the axis of sym m etry is orthogonal to the lines of sym m etry (i.e. bja — m ), then N' becomes a zero vector. However this requires a very specific viewing angle and thus can be ignored under norm al viewing conditions. This is the case th a t convergent sym m etry acts like the skew sym m etry of orthographic projection, th at is, now it is a constraint leaving one degree of freedom, which is basically the orthogonality constraint given in equation C.13. 135 C.2 Usage of the Constraints for Polyhedral Objects In the previous section we have derived four constraints under perspective pro jection (however, not all four are independent as parallelity and orthogonality constraint are used in the convergent sym m etry). For a given figure, we need to determ ine which constraints are applicable. Note th a t th e shared boundary constraints make no regularity assum ptions about the figure and m ust always apply (ignoring any “errors” in th e line drawing). O ther constraints, however, require observation of some regularity in the im age and assum ption th a t the 3-D object obeys corresponding regularity also. Of these regularities, sym m etric convergence is quite stringent, i.e. it is unlikely to be caused by accident, though we still can not guarantee th a t the 3-D ob ject is orthogonally sym m etric. U nfortunately, sym m etric convergence is strong only when at least three lines converge on the image plane, as two lines always converge (unless they are parallel to each other on th e im age plane). Thus, a planar object having four sides to a face can always be construed to be sym m etric convergent. Observations about parallelism and orthogonality m ay also not be apparent in th e image. In orthographic projection, parallel lines rem ain parallel; in perspective projection they do not. On the other hand, however, in perspective projection we have m uch tighter constraints. Thus, one way to solve the interpretation problem is to make regularity assum ptions and verify by using th e constraints. We illustrate this by an example. Figure 0.2(a) shows th e image of a cube under perspective projection (the reader will get a b etter perception of the figure if the picture is held very close to the eye). Applying the shared boundary constraint (in the gradient space for the sake of illustration here) gives us a triangle, say G 1G 2G3 in figure C.2(b) which specifies the orientations of the three faces. Note th at in perspective projection, th e shape of the triangle m ay depend both on its position and its size (both 136 (a) Q Figure C.2: A cube under perspective projection (a), and com puted orientations for the faces shown as points on the p — q space w ith th e shared boundary con straints overlayed, dashed lines, (b). of which need to be determ ined). A dditional constraints can come from the sym m etry of th e faces. As described earlier, any quadrilateral can be viewed as being convergent sym m etric. Assuming th a t the three faces are projections of orthogonally sym m etric shapes {i.e. rectangles), we can get unique orientations for the three faces. In this exam ple, these values happen to be consistent w ith the shared boundary constraints (and with th e known correct answers from which th e exam ple was constructed). A lternately, we could have used th e parallelity constraints betw een opposite sides of the faces. For this exam ple, this regularitj' is suggested by the observation th a t groups of three lines (corresponding to parallel lines on two faces) do intersect in common points. Using this constraint, the answers tu rn out to be the same as before and hence consistent. Of course, in general, we can not expect all constraints to be satisfied sim ul taneously for all figures. If the image had been derived from a non orthogonal prism , instead of a cube, the convergent sym m etric results would not agree with the shared boundary constraints. Now, we can m ake several choices. We can ei ther m ake all faces equally non-sym m etric (by some m easure), or still achieve con sistency by m aking two faces (any two) sym m etric and th e th ird non-sym m etric. 137 In general, it should be possible to define a penalty function and find “optim al” solutions. However, we have not investigated such approaches. O ur feeling is th at the only tim e we can get strong interpretations is when some of the evidence is overwhelmingly strong and th a t this is the evidence we would use at th e exclusion of th e other constraints (those th a t require some assum ptions). C.3 Extensions to Curved Surfaces In another paper we [UN8 8 ] have described m ethods for recovery of surface ori entation of curved surfaces from contours under orthographic projection. The analysis was based on observation of a form of sym m etry th a t we called parallel symmetry. Two planar curves are defined to be parallel sym m etric if there ex ist a one to one correspondence between th e points on the curves such th a t the tangents to the curves at corresponding points are parallel. T he im portance of the parallel sym m etry is th a t, if we cut a zero gaussian curvature surface with two parallel planes. Then it can be shown th at we get two parallel sym m etric curves from the intersection of the planes w ith the surface such th a t correspond ing points of these curves are joined by the rulings of the zero gaussian curvature surface. Two curves th a t are parallel sym m etric in 3-D also project into parallel sym m etric curves in the image under orthographic projection. However, this is not the case under perspective projection. In th e following, we give conditions th a t curves in a perspective image m ust satisfy if they are projections of parallel sym m etric 3-D curves. Then we give a m ethod for com puting parallel sym m etry for certain class of images (those th a t are projections of conic surfaces) and show how this can be used for surface reconstruction. C .3.1 P a ra llel S y m m e tr y 138 p I ( S ) Figure C.3: Contours of a conic surface under perspective projection, w ith the point of convergence for the rulings P, and the line Say there are two curves a\ and a 2 on the im age plane generated by the two planar 3-D curves 0i and 02 in planes parallel to a plane, call it II, which passes through th e origin. We have the relation: at(s) = = T 7 7 \ (C '29) P 2 z \S ) where 0iz is the th ird coordinate of the curve 0t. T he curves 0\ and 02 are parallel sym m etric if and only if we can form a m onotonic correspondence function f(s) such that: P M = « / W ) (c.30) where 0-(s) is the tangent vector of the curve 0i{s). For each point of the curve a,- th e N IG P of the tangent line (i.e. the line passing from the point Q't(s) in the direction c^(s)) of the curve is A ,- = « ;(s) x c k '( s ) . Since the curves 0 i and 02 have parallel tangents at the corresponding points. From the parallelity 139 theorem the vector function I(s ) = A i(s) x A2( / ( s )) gives the orientation of the tangents of the curves /A and /?2. That is &(*) = # (/0 0 ) = H*) = A^s) x A,(/(*)) = (« ,(s) x <*!(*)) x (<*,(/(*)) x a'2(f(S))XC.31) Since the curves /A and (3 2 are planar curves resting on planes parallel to II, their tangent vectors f3[ and ( 3 ' 2 must be parallel to II, and the tangent vectors are on the plane II. Therefore every orientation given by the function I{s) (i.e. the vector from the origin to the points of I(s )) should be on the plane II. The image of I(s), Ii(s), is the curve on the image plane that can be obtained by projecting the points of I(s) on to the image plane. Since I(s) is on the plane II which passes through the origin its image has to be a line. Therefore, Ji(s) being a line is the necessary condition that two curves <*1 and a 2 are projections of the parallel symmetric curves /A and /?2. Also the normal of the plane II is just the NIGP of the line Ii{s) since II is the IGP of this line. Also the line Ii(s) is the locus of intersection points of the tangent lines of the curves cti(s) and a 2(/(s )) see figure C.3. C .3 .2 A n a ly sis o f a C on ic Surface We now concentrate on conic surfaces (or linear straight homogeneous generalized cones in generalized cones terminology) cut by two parallel planes (say parallel to plane II) to form curves, say fti(s) and /?2(s). Let a \ and o 2 be the projections of /A and ( 3 2. In this case, the curves f3\{s) and /32(s) are parallel symmetric; let the correspondence function be f(s) as before. The lines joining the corresponding points on the curves /A and /?2 are the rulings of the surface. For a conic surface, these rulings intersect in a single point in 3-D. For a cylindrical surface, these rulings are parallel to each other. In either case, the projections of the rulings intersect at a point, say IP, in the image plane. 140 Point P can be found by the intersection of th e lines joining the end-points of and a 2. Now draw lines from P such th a t they intersect curves cq and a 2; the intersection points are the corresponding points on the two curves. W ith this correspondence we can construct the curve I ( s ) and check if Ii(s) is straight as in figure C.3. If it is, we can interpret the figure as a conic surface. (Note: point P can also be found by a search process if the end-points of the curves are not reliable.) Now we have the plane II containing /(s ) , we can reconstruct f3\ and /?2 by back projecting a.\ and a 2 onto planes parallel to II (up to a scale). However, this is not sufficient to reconstruct the conic surface; th e distance between the planes containing the two curves still rem ains as a one degree of freedom. This degree of freedom can be fixed if we interpret th e surface as being cylin drical (under perspective, a conic surface can always be interpreted as being cylindrical). Given th e orientation N of the plane n containing 3-D curves /?i(s) and /32(s), the 3-D tangent /?,-(s) is given by: # ( s ) = At(s) x N (C.32) where H ,(s) is the N IG P of the line passing through cq(s) in the direction a '(s ) H ;(s) = a 2 -(s) x o4(s), therefore # (* ) = M s ) x a\{s)) x N (C.33) Now we have the orientation of /3,(s ) at any point. The orientation of the surface is given by : # ( s ) x R{s) (C.34) where R(s) is the 3-D orientation of th e rulings. Since the surface is cylindri cal, R(s) is constant (i.e. R(s) = R) and is equal to the intersection point P of th e rulings on the image plane (c.f. parallelity theorem ). T h at is R = P and the orientation of th e surface at any point is given by: ( M s ) x aq(s)) x N ) x P (C.35) 141 We conjecture th a t if th e resulting surface corresponds to a right cylindrical surface2 hum ans will accept this interpretation as being th e m ost preferred. If the figure is to be interpreted as a non-cylindrical conic surface, further assum ptions need to be made. One alternative is to assume th a t the surface belongs to a right, generalized cone. We have not studied the hum an preferences in such cases, and such experim ents are in fact difficult to perform. C.4 Conclusion We have derived some constraints on the interpretations of line drawings under perspective projection. Some of the constraints are analogous to th e constraints used in orthographic analysis. However, in perspective analysis, we typically need to use one more variable in representing orientations; this makes some of the equations more complex and non-linear. The observation of the regularities m ay also be not as clear w ith perspective as it is w ith orthographic projection. However, when such regularities can be inferred from some other, perhaps exter nal, context, our constraints can be used directly. Our m ajor observation, however, is th at when regularity is discovered in per spective, it provides much stronger constraints than under orthographic projec tion. We dem onstrated this for th e case of convergent sym m etric figures. For the case of curved surfaces, too, perspective projection provides considerable am ount of inform ation. If the surface is cylindrical, (or can be interpreted as being cylin drical), th an we can reconstruct the surface just from its contours. For conic surfaces one degree of freedom is left if we do not m ake additional assum ptions. We hope th at these observations will lead to increased use, and exploitation, of 2 A right cylindrical surface is the one where planes cutting the surface to generate the curves 13\ and /?2 are orthogonal to the orientation of the rulings. This translates to the condition that P = A T , where N is the normal of the plane II and P is the point on the im age plane at which rulings intersect 142 perspective projection rather than regarding perspective as a com plicating agent th a t can be ignored under “norm al” viewing conditions. 143

Linked assets