University of Southern California Dissertations and Theses

Articulated human body deformation from in-vivo 3d image scans

(USC Thesis Other)

Articulated human body deformation from in-vivo 3d image scans

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content ARTICULATED HUMAN BODY DEFORMATION FROM IN-VIVO 3D IMAGE SCANS by Taehyun Rhee A Dissertation Presented to the FACULTY OF THE GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Ful¯llment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (COMPUTER SCIENCE) December 2008 Copyright 2008 Taehyun Rhee Acknowledgements Thisstudyhasbeenindebtedtomanypeoplefortheirsincereadvicesandsupports. The list of individuals I wish to thank extends beyond the limits of this format. I would like to thank the following persons as well as all others who may not be included on this document. In particular, I wish to express my gratitude to my advisor, Prof. Ulrich Neu- mann for his continued encouragement and invaluable supports during my Ph.D. program. His sincere advices keep me on the right track with strong con¯dence. I give a special thank to Prof. Krishna Nayak for his sincere support and advices for medical imaging study. His wise insight have strengthen this research signi¯cantly. I am grateful to him for sharing with me his deep knowledge on the medical imaging ¯elds, which provided me a strong basis for solving challenging problems addressed in this study. I thank to Prof. Aiichiro Nakano for his sincere guidance and sound advices during my Ph.D. study. Also, I appreciate my Ph.D. quali¯cation member Prof. Karen Liu, Suya You for their good discussion and sincere comments. On top of the list, I am deeply indebted whole my research to J.P. Lewis, my technical advisor, collaborator, mathematics tutor, co-author, and friend. None of human word can present my special thank to his endless support and guidance. I will always be thankful for his wisdom, knowledge, and deep concern not only for me, but also for his instance for research as a true scientist. I will never forget his true friendship. ii I also thank to my colleagues in USC graphics lab and computer vision lab for their sincere discussion and friendship during this study: Zhengyao Mo, Ismail Oner Sebe, Zhigang Deng, Lu Wang, Jonathan Mooser, Charalambos Poullis, Sang YunLee, Qian Yi Zhou, Kyung-kook Park, Kwang-woo Lee, Changki Min, TaeEun Choe, Kwangsu Kim, Eunyoung Kim, JongMoo Choi. Also, I hope to share my specialthanktopeopleinMagneticResonanceEngineeringLabatUSC,andmany other researchers for their sincere discussion for this study. Also, I would like to acknowledge the whole sta® of IMSC, CS department, USC graduate school for their sincere support. I wish to extend mythanks to Prof. In-Sung Ihm, and JikHyun Chang a formal advisors at Sogang University. Their sincere advices and guidance provides me a sound fundamentals for my PhD study. I am very grateful to Samsung Electronics for their sincere support throughout my studies. I give special thanks to SaeHyun Kim, a vice president and chief of Research Innovation Center for his continued encouragement and con¯dence for my study. I truly appreciate many sta®s at CTO department, who have shared their time and e®orts to support my PhD program. Also, I give special thank to my formal colleagues at Samsung Electronics who gave me strong inspirations and motivations for this study. Above all, I would like to thank to my family. The encouragement and support frommybelovedwife Youn-jungand our always positiveand joyfulson Jae-yoon is a powerful source of inspiration and energy. I truly appreciate their understanding and prayers throughout my Ph.D. study. A special thought is devoted to my mom, mother in law, brother Jung-hyun Rhee and his family, brother-in-law Wonki Hong and his family, and sister-in-law YoungJoo Hong and her family, and all other relatives in Korea for their never- ending support over the paci¯c ocean. iii List of Tables 3.1 Timing results (in FPS) of each algorithm: low-resolution mesh has 5762triangleswith2972vertices, mid-resolutionmeshhas22836tri- angles with 11574 vertices, high-resolution mesh has 91460 triangles with 46036 vertices. . . . . . . . . . . . . . . . . . . . . . . . . . . 58 3.2 Texture memory to store data required in fragment program (in Kbytes); refer each texture notation to section 3.5.2. . . . . . . . . 58 4.1 MRI scan parameters . . . . . . . . . . . . . . . . . . . . . . . . . . 67 4.2 Registration results of synthetic volumes. . . . . . . . . . . . . . . . 94 4.3 Registration results of the knee volume. . . . . . . . . . . . . . . . . 95 4.4 Registration results of the hand volume. . . . . . . . . . . . . . . . 95 vi List of Figures 2.1 Human hand cloning from surface anatomy: (a) palm creases ex- tractedusingtensorvotingandsurfaceanatomy, (b)estimatedhand joints, (c) feature correspondences; red dots represent curve seg- ments, (d) cloned 3D model of female hand, (e) cloned 3D model in a di®erent view and position with inherited skinning properties. 10 2.2 Hand modeling overview: Two main parts are grouped into four stepsinthegrayboxes. Supplementaryparts, whichneedadditional tools or are beyond the scope of this work, are indicated by the dotted boxes. Numbers beside each box indicate the required user interaction level; level 2 is fully automatic, level 1 requires no user interaction but needs some parameters (experimentally ¯xed in nor- mal case; canny edge thresholds, tensor voting ¾, ¯nger extraction thresholds), level 0 needs manual input. . . . . . . . . . . . . . . . . 12 2.3 Human hand anatomy: (a) Bone Anatomy: 1. Distal phalanx, 2. Middle phalanx, 3. Proximal phalanx, 4. Metacarpals, 5. Ulna, 6. Radius, A. DIP joint, B. PIP joint, C. MCP joint, D. IP joint, E. CMC joint; (b) Surface Anatomy: 1. Distal phalanx, 2. Middle phalanx, 3. Proximal phalanx, 4. Distal palmar, A. DIP crease, B. PIP crease, C. Palmar digital crease, D. Distal palmar crease, E. Proximal palmar crease, F. Thenar crease, G. Distal wrist crease, F1-5 present thumb, index, middle, ring, little ¯nger. . . . . . . . . 17 2.4 Basic hand geometry extraction: (a) Segmented hand image, (b) Basichandgeometry;contourofthehand,¯ve¯ngertips,four¯nger valleys, and medial axis of each ¯nger . . . . . . . . . . . . . . . . . 19 2.5 Palmcreasesextraction: (a)Sobeledgeimage,(d)Firsttensorvoting result, (e) Second tensor voting result (¯nal result) . . . . . . . . . 20 2.6 Tensor Voting: (a) tensor ellipse, (b) 2D stick kernel, (c) Tensor voting example: left image shows the sparse input tokens without orientation, middle image is the tokens oriented by tensor voting but still sparse, the right image is the resulting extracted dense curve. 21 vii 2.7 Crease extraction and joint modeling: (a) Green curves are the ex- tracted palm creases (b) Blue lines are the location of ¯nger crease (c) Blue dots are the calculated joint locations . . . . . . . . . . . . 27 2.8 Curve segment matching example: (a) Curve matching example of curve segment S 2 (s 2 ;e 2 ): s 2 is on the intersection between contour and the palmar distal crease of little ¯nger, and the e 2 is on the intersection between contour and the PIP crease of little ¯nger, (b) Curve segment 2 in generic hand model . . . . . . . . . . . . . . . . 29 2.9 Handcloneresults: (a)pictureofsourcehand, (b)clonehandmodel with texture mapping, (c) skin geometry of generic hand model, (d) deformed skin geometry of clone hand, (e) joint skeleton of generic handmodel(eachcircleisajointlocation),(f)modi¯edjointskeleton of clone hand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 2.10 Hand clone result in various view points . . . . . . . . . . . . . . . 34 2.11 Hand clones of ¯ve people: the ¯rst row is a set of source hand images and extracted joint information, the second row is a set of rendered clone models with texture maps, and the third row is a set of polygon meshes of the clone models; (a) a four year-old girl of 12cm hand length (low resolution mesh), (b) a ¯ve year-old boy of 13cm hand length (low resolution mesh), (c) an average female of 16.5cm hand length (low resolution mesh), (d) an average male of 18.5cm hand length (high resolution mesh), (e) a large male of 21cm hand length (high resolution mesh). . . . . . . . . . . . . . . . . . . 35 3.1 Skinning results of each algorithm: the ¯rst row shows SSD, the second row shows PSD, and the third row shows WPSD. Areas of blue and red arrows represent unexpected results of SSD and PSD respectively. The second column shows deformed vertices of each algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.2 Packing data into textures: T i can be T v , T n , T w1 , T w2 , and T dj . V(vertex),N(normal),W(weight),J(jointindex),andD(displacement) represent each texel (RGB(A)) value of the related texture. T d con- sist of eight T dj storing displacements of each pose. . . . . . . . . . 48 3.3 Overview of WPSD on GPU: Each T i is the texture maps storing required data for the calculations (refer value i in section 3.5.2) , T i 0s are the texture maps to store output of 1st and 2nd pass, P 0 is a geometry in rest pose, P a is a deformed geometry in arbitrary pose. 52 viii 3.4 Meshesoftestdata: thetoprowisalow-resolutionmesh,thesecond rowisamid-resolutionmesh,andthebottomrowisahigh-resolution mesh.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.5 Samples poses and displacements: the ¯rst and second row show each sample pose, the third and fourth row show displacement of each sample pose with the rest pose in the ¯rst column of the third row. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 3.6 Arbitrary poses deformed by WPSD on GPU. . . . . . . . . . . . . 57 4.1 From left, (a) schematic source (neutral pose) scan, (b) target pose, (c) rigid transformation does not provide adequate initialization for non-rigid registration, (d) VSD produces an approximate but suit- able initialization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 4.2 HumanInvivoMRIscans: (a)Humanhandwithasupportingcast, (b) MRI scans of the hand, (f) MRI scans of a human knee. . . . . 68 4.3 Hierarchical Bone Registration: (a) Hand bone hierarchy, (b) Knee bone hierarchy, (c) Hierarchical bone registration of proximal pha- lanx, (d) Hierarchical bone control of middle phalanx. . . . . . . . . 70 4.4 Multiple resolutions of the representative volumes . . . . . . . . . . 74 4.5 Bone registration across poses: The top left most image is the base volumeincludingmarchingcubemeshesofextractedbonesandtheir OBBs(yellowboxes);theotherimagesshowsbasebones(redpoints) registered to the other sample poses. . . . . . . . . . . . . . . . . . 75 4.6 Jointestimation: (a)Thecenterofrotation(COR)estimation,where B j¡1 is the parent bone and B j is transformed to pose k by matrix T 0!k j , (b) The bone skeleton, (c) The estimated CORs (cayan dots) of each joint. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 4.7 Aligning a generic atlas polygon surface mesh to the volume: (a) Source polygon mesh and features, (b) Target volume and corre- sponding features, (c) Surface mesh warped to align to the volume, (d) Yellow dots (Fp) are vertices of the deformed surface, Fb is each bonesurface, andweightsoftheblue°eshareaisinterpolatedbyFp and Fb. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 4.8 Articulated volume initialization: a template volume (cyan) is ini- tialized to a target volume (purple) using VBD. . . . . . . . . . . . 82 ix 4.9 Globalvslocaldeformationfunction: fromtheleft,allocatedcontrol points (blue dots) and the displacements (white bars), the global deformation using RBFs based on TPS, the local deformation using TCP constrained by a unit sphere. . . . . . . . . . . . . . . . . . . 83 4.10 Top: comparison of the 2D clamped plate spline (left) with a thin platespline(right)interpolatingthesamepoints. Theclampedplate splineisbothsmoothandlocalized,decayingtozeroattheboundary of the unit disc. Bottom: comparison of the triharmonic clamped platespline(blackline)withGaussianRBFinterpolation(red). The plotisthedensityofalinearsectionthroughaninterpolatedvolume. Note that the control points are distributed through the volume and cannot meaningfully be visualized on this plot. The Gaussian RBF interpolation has unnecessary ripples. Increasing ¾ reduces these, but at the cost of increasing the overshoot. . . . . . . . . . . . . . . 86 4.11 Locallyadaptivenon-rigidvolumeregistration: fromthetop,¯ndthe maximum mismatching block and allocate control points; construct B s ¤ ;t ¤ (front and back blocks are omitted to simplify the ¯gure); register and deform the local block. . . . . . . . . . . . . . . . . . . 90 4.12 Progressive non-rigid volume registration results: the initialized vol- ume(cyan)isprogressivelyregisteredtothetargetvolume(magenta) using adaptively allocated control points (blue dots) at block level 3; small lines show the displacement value for the TCP function. . . 91 4.13 Nonrigidarticulatedregistrationofthehumanhand. (a)Atemplate volume in the neutral pose, (b) A target volume, (c) A template volume registered to the target volume, (d) Di®erences between reg- istration (cyan points) and target volume (magenta) with allocated controlpointsatblocklevel3(32 3 resolutionblocks), (e)Aninterior view of the deformed volume. . . . . . . . . . . . . . . . . . . . . . 97 4.14 Nonrigidarticulatedregistrationofthehumanknee. (a)Atemplate volume in the neutral pose; (b) A target volume; (c) A template volume registered to the target volume; (d) Di®erences between reg- istration (cyan points) and target volume (magenta) with allocated control points at block level 3 (32 3 resolution blocks); (e) An inte- rior view of the deformed volume; the markers (small dots) in knee volumes are successfully registered and used only for the validation. 98 4.15 Hand volume deformation by EVD: the human hand volume are deformed toarbitrary poses; translucent volumerendering visualizes the complex anatomical layers without loosing the context of the subject; a clipping plane shows the interior MRI volumes. . . . . . . 99 x 4.16 Knee volume deformation by EVD: the human knee volume are de- formed to arbitrary poses; a clipping plane shows the interior MRI volumes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 4.17 The VBD and EVD result (hand). Green (VBD) and cyan (EVD) points represents deformed volume, and the magenta region shows the di®erence between the deformation and a ground truth volume which is not included in the training set. . . . . . . . . . . . . . . . 101 4.18 The VBD and EVD result (knee). From the left, the VBD and EVD results of the knee volume deformed to ground truth volumes which arenotincludedinthetrainingset, andthegroundtruthkneevolume.101 xi Abstract Asystemforproducinganaccuratehumanbodymodelisimportantandrequiredin many ¯elds. This study describes a complete system for creating an anatomically accurate 3D human body model from living human scans of a speci¯c person. The method provides accurate and personalized tissue deformation arising from articulated motions of human body regions such as the human hand, probably the most complex articulated region in the human body. Our system approaches the problem in three stages. First, a person-speci¯c hand model is generated from a single canonically posed palm image of the hand withouthumanguidance. Tensorvotingisemployedtoextracttheprincipalcreases on the palmar surface. Joint locations are estimated using extracted features and analysis of surface anatomy. The skin geometry of a generic 3D hand model is deformed using radial basis functions guided by correspondences to the extracted surface anatomy and hand contours. The result is a 3D model of an individual's hand, with similar joint locations, contours, and skin texture. Second, a parallel skin deformation algorithm is presented suitable for SIMD (Single Instruction Multiple Data) architectures such as GPUs. It presents real- time performance of skin deformation arising from the motion. Joint weights of each vertex are automatically computed and used in skinning algorithms. Skin deformation algorithms for articulated subject such as SSD (Skeletal Subspace De- formation), PSD (Pose Space Deformation), and WPSD (Weighted Pose Space Deformation)areparallelizedinSIMDmannerandimplementedonGPUfragment processors to obtain su±cient speed-up for real-time skin deformation. xii Finally, a complete system is developed to create a deformable articulated hu- man body volume from multiple 3D MRI scans of an in vivo human body, which canproduceanatomicallyaccuratehumanvolumedeformationforarticulatedbody animation. Thesystemcombinesexistingandnewtechniquesthattogetheraddress thepracticalissuesinvolvedinproducingadetailedarticulatedvolumemodelfrom scans. These include joint estimation from the actual skeleton, bone volume reg- istration, smooth deformation that preserves the rigidity of the skeleton, and a staged registration process that incorporates a careful initialization as needed to deal with the strong local minima inherent in registering articulated bodies. A lo- callyadaptivenon-rigidregistrationalgorithmisdevelopedthatgreatlyreducesthe highdegreesoffreedomanddataprocessingarisinginregistrationofdeformablear- ticulated volumes. The registration step establishes a correspondence across scans that permits the last step, interpolation of the scans under intuitive joint control using an example based deformation approach that we extend from the surface to the volume domain. The result is a volumetric model driven by standard joint control that incorpo- rates detailed deformation based on real data. Visual presentation and quanti¯ca- tionofarticulatedhumanbodymodelanddeformationhavepotentialtocontribute many applications such as medical studies of arthritis, bio-mechanics, robotics, er- gonomics, medical education, rehabilitation, virtual reality as well as computer graphics and animation. xiii Chapter 1 Introduction 1.1 Motivations An anatomically accurate 3D model of an articulated skeletal body including the bones, muscles, tendons, and other anatomical layers is important in various ¯elds. The motion of the underlying skeleton displaces the related body parts, producing subtle deformations that arise from the complex interactions between each layer. Althoughaphysically-basedmodelofimportantanatomicallayerscanapproximate such subtle deformations, manually creating an accurate physical model including allanatomicallayersisdi±cultandmaynotexactlymimicthecomplexinteraction and compression of di®erent tissue layers of the individual body. The di±culty in manually or algorithmically de¯ning complex articulated body structures of an individual subject can be avoided by adopting a scan-based ap- proach. Sincescansofalivingsubjectatmultipleposescanbeusedasthetraining samples, accurate deformable models can be simulated from actual data. Also, a model constructed from living human scans re°ects characteristics of the subject and provides personalized information, which is often essential to create a virtual clone for medical and other applications. Volumedataobtainedfrom3Dmedicalimagescans(e.g. MRIorCT)represents 3D interior anatomy including the muscles, tendons, bones, and other important 1 structures. Furthermore, volume data captures the skin surface, even in poses with visual occlusions due to self-contact. Translucent volume rendering can succes- sively visualize all these anatomical layers without loosing the overall context of the subject. The quanti¯cation and manipulation of medical image volumes obtained from articulated vertebrate bodies has clear importance. The skeleton in articulated re- gions such as the human hand and knee is capable of a wide range of movements, resulting in large motion and deformation of the surrounding soft tissues that are poorly represented in still images. Measurement and visual presentation of such motion and deformation are important for many applications: clinical studies such as motion analysis of patients su®ering from arthritis, rehabilitation using arti¯- cial limbs, bio-mechanics, humanoid robotics, ergonomics, medical education using virtual reality, and volume graphics for translucent human body animation. However,examplebasedapproachesusingrealsubjectscanshaveseveralfurther challenges. While volume samples can be obtained from medical image scans, the use of an articulated in vivo subject presents di±culties that are not present when the data are obtained from either cadavers or non-articulated subjects (e.g. the human head): motion artifacts, patient discomfort, and other technical issues dis- cussed in section 4.3. Furthermore, the raw medical volume samples require com- plex preprocessing steps that are not needed when manually created surface model samples are used. One of the important steps in handling raw volume samples is to obtain con- sistent geometric topology across the volume samples, since the example based deformation requires displacement of each geometric point (e.g. each vertex in sur- face mesh) using the di®erent pose samples. However, in case of medical image volumes, the geometrical information is represented by volume intensity properties without any explicit geometric parametrization. Especially, creating iso-surfaces of each anatomical layer from magnetic resonance image (MRI) volumes is very 2 challenging due to obscure segments between tissue properties. Since allocating ¯ducial markers inside the in vivo human body is not practical, many previous surface-based ¯tting and registration algorithms cannot be easily extended to the volume setting. Intensity based non-rigid volume registration algorithm provides adequate solution to accomplish consistent volumetric topology between the med- ical volume samples which can synchronize intensity values of each voxel among samples. Inthisstudy, theissuesdescribedabovearesuccessivelyaddressedandresultin sophisticatedsolutionsforcreatingaperson-speci¯c3Dhumanbodymodelwhichis also be animated from in vivo human scans. The robustness of our system is tested by its application on the hand, probably the most complex articulated region in the human body. 1.2 Overview of recent work Although the detail review of related works will be discussed in each section (sec- tion2.2,3.2,and4.2),severalimportantrecentworksareintroducedheretoprovide the proper research context of this study. Model-based 3D reconstruction methods have generally been used in face mod- eling research [6,75,95]. A canonical face model is deformed into a target face model guided by feature correspondences between them. If a set of photographic images is used, corresponding 3D features are extracted from the images in various view points using the calibrated camera positions and proper user interactions. A semi-automatic technique to reconstruct a 3D creature from an anatomically basedcanonicalmodelispresentedbySimmonsetal.[93]. Anarticulatedcanonical horsemodeliscreatedbyhierarchicalsegmentsincludingbones,muscles,andskins. Sparse 3D features of a target model are acquired with a hand digitizer to achieve 3D correspondences between pre-de¯ned locations at the canonical model and the 3 targetmodel. Usingthesparsefeatures,theunderlyingstructuressuchasbonesand muscles are reconstructed. Then, the denser correspondences of the skin geometry areautomaticallygeneratedusingtheunderlyingstructures. Finally, RadicalBasis Functions [7] deform skin mesh driven by the generated corresponding features. Reconstruction methods for 3D human body shapes were presented in recent works of [4,90]. A template human body model created from 3D range scans is ¯tted to an individual human body shape guided by sparse 3D markers. An optimization function is formulated to ¯nd a set of a±ne transformations for each vertex, while minimizing ¯tting errors: data, smoothness, and marker errors. They usedCAESAR(CivilianAmericanandEuropeanSurfaceAnthropometryResource Project) data library of whole-body 3D laser scan samples [84]. Allen et al. demonstrated example based skin deformation from range scan data [3] and their e®orts are enhanced by Anguelov et al. as a data-driven method for creating deformable human body shapes of individual people [5]. A set of di®erent human bodies is acquired from 3D scans of each person, and the samples are used to generate di®erent body shapes of each individual. Pose deformation components of a canonical human body are obtained from 3D scans of a single person in multiple poses demonstrating skin deformation of the subject. From the two separate spanning spaces, which are di®erent poses of a template model and di®erent shapes of di®erent models, the system estimates an arbitrary pose of a speci¯c body shape. The deformation properties captured from a template model is transferred to the target shapes by [96]. Recently, Park et al. [73] capture subtle skin deformation using commercial motioncapturedevicesandaround350markers. Themethodscaptureandproduce detailanimation for subtleskin deformation suchas bending, bulging, jiggling, and stretching. The surveyed methods are based on surface scans and they cannot handle inner anatomical layers as well as occluded surfaces. Volume data obtained from medical 4 image scans (e.g. MRI or CT) can represent anatomical structures without any occlusions, and translucent volume rendering can successively visualize inner tissue layers without loosing the overall context of the subject. These e®orts have started from the visible human data set [70] which is the set of high resolution images captured from a cadaver model. Teran et al. [97] generated a physically based ar- ticulated body animation of muscular skeletal body based on the segmented visible human data. Although it is based on a volumetric data, the method is focussed on the meshes of speci¯c tissue surfaces. Articulated volume animation of the visible human data has been shown in [26,94] but their algorithmic methods could not produce anatomically accurate tissue deformation of the complex human tissues. Although the visible human data is high quality, it cannot provide person- speci¯c scans nor multiple pose samples. In [49], an example-based deformable human hand model is derived from medical images scans. Multiple CT scans of the human hand are obtained from several poses. However, since they just use skin surface meshes extracted from the volumes, the method could not use full advantagesofvolumegraphicsnorprovidedadequatesolutionsforvolumemodeling and animation. Based on our survey on recent related works, the articulated human volume animation and inner tissue deformation from real subject scans appear to be unad- dressed issues in the previous research. The surveyed methods could not provide anatomically accurate and realistic inner tissue deformation of the speci¯c person from actual scans. Algorithms using surface representations and surface scans can- not easily provide an accurate surface model in occlusion (e.g. palmar skin of a ¯st pose). This study presents a complete system to create an anatomically accurate human volume from in vivo medical volume scans. The results provide an occlu- sionfreeperson-speci¯c3Dhumanbodymodel,accurateinnertissuedeformations, realistic volume animations of articulated movements, while handling the practical limitations arising in obtaining volume data from living subjects. 5 1.3 Contribution Our e®orts to build a complete system can be categorized into three main studies. Each study provides adequate solutions for their speci¯c domain. First, we present an automated method to make a speci¯c human hand model from an image of the surface anatomy of a human hand (chapter 2). The human hand is an important interface with the complex shape and movement. In virtual reality and gaming applications the use of an individualized rather than generic hand representation can increase the sense of immersion and in some cases may lead to more e®ortless and accurate interaction with the virtual world. We present a method for constructing a person-speci¯c model from a single canonically posed palm image of the hand without human guidance. Tensor voting is employed to extract the principal creases on the palmar surface. Joint locations are estimated using extracted features and analysis of surface anatomy. The skin geometry of a generic 3D hand model is deformed using radial basis functions guided by corre- spondences to the extracted surface anatomy and hand contours. The result is a 3D model of an individual's hand, with similar joint locations, contours, and skin texture. Since the algorithm has no human guidance and the only information re- quired is a palmar skin image, the method is easily applied, despite of its innate limitations based on using a 2D image. Second,aparallelskinningalgorithmsuitableforSIMD(SingleInstructionMul- tiple Data) architecture like GPU is presented in chapter 3 to achieve real-time performanceofskindeformationarisingfromthearticulatedmotion. Jointweights of each vertex are automatically computed and used in skinning computation. Ar- ticulated body skinning algorithms using SSD (Skeletal Subspace Deformation), PSD (Pose Space Deformation), and WPSD (Weighted Pose Space Deformation) are parallelized in SIMD manner and implemented on GPU fragment processors to obtain su±cient speed-up for real-time simulations. In our experiments, we 6 can speed-up WPSD, PSD, and SSD around 20»30 times faster than CPU (from 1.2FPS to 25FPS speed-up of WPSD on a detailed model having 22836 triangles with 11574 vertices) using modern graphics card, thus making WPSD a possible real-timeskinningsolutionforvariousapplications: game,virtualreality,andother real-time simulations. Finally, in the chapter 4, a complete system is presented for producing an an- imatable volumetric model from in vivo MRI scans of articulated body regions such as the human hand and knee. Our system approaches the problem in stages. First the underlying skeleton is identi¯ed, along with the center of rotation of joints. Next, a volumetric initialization is applied to approximately conform a sin- gle (e.g. neutral pose) template scan to each of the other scans. This provides the basis for the third and most di±cult step, an accurate registration of the template toeachoftheotherscans. Makingthissteppracticalrequiredthedevelopmentofa hierarchicalandlocallyadaptivevolumeregistrationapproachtoreducethedegrees of freedom involved in complete registration of a number of articulated deformable volumes. The registration minimizes a di®erence measure that adaptively uses mutual information and standard voxel intensity di®erences at appropriate scales; no manually placed ¯ducial markers are required. The deformation is performed successively and locally using the clamped plate spline, a technique that produces curvature minimizing but local deformation using control points that need not be on a regular grid. The registration step establishes a correspondence across scans that permits the last step, interpolation of the scans under intuitive joint control using an example based deformation approach [49] that we extend from the surface to the volume domain. The result is a manipulable volumetric model built from scanned data of a living person. Although the computation time in the construction of the model is substantial, manual guidance is limited to several steps in the initial identi¯cation 7 of the skeleton. The model can be displayed with existing volumetric rendering systems and animated with standard keyframe techniques. In addition to this general contribution, a number of speci¯c applications are foreseeable: ² Amoveablevolumetricmodelwithrealistictreatmentofinternaltissuescould be used in medical education; ² The visual presentation of multiple scans as a single coherent model can bene¯t the patient's (and possibly the doctor's) understanding of problems such as knee injuries and arthritis; ² Volume MRI scans of the hand holding an object capture the position and compression of skin regions that are in contact with the object but visually occluded. This information is useful in ergonomics and product design; ² The registration process developed for this system may help to extend the scope of atlas-based registration to articulated subjects. 8 Chapter 2 Human Hand Modeling from Surface Anatomy 2.1 Overview The ¯rst step of our study is to provide solutions for creating a realistic person- speci¯c 3D virtual hand model from a single canonical posed palm image while minimizing manual input. Virtual reality, simulation, and entertainment applica- tions often display a proxy of the operator's hand in the virtual world. The virtual hand may be driven by camera tracking, a haptic interface, or a sensor (or actu- ator) globe, and serves to provide visual feedback on object interaction as well as to enhance realism. The operator is intended to regard this virtual hand as their hand for the purpose of interacting in the virtual world. Di®erences in shape and mechanics between the operators' real and virtual hands may therefore cause in- accuracies in simulations and decrease the sense of immersiveness. Unfortunately, due to the di±culty, time, and cost of generating a person-speci¯c hand model, immersive systems today generally use only a simple generic hand model. The goal of this study is to make a realistic person-speci¯c 3D virtual hand model, including skin geometry, texture, underlying skeleton. The process requires minimal human guidance and depends only on capturing a skin texture image whose viewing direction is perpendicular to the palm plane. By analysis of the skin 9 (a) (b) (c) (d) (e) Fig. 2.1: Human hand cloning from surface anatomy: (a) palm creases extracted using tensor voting and surface anatomy, (b) estimated hand joints, (c) feature corre- spondences; red dots represent curve segments, (d) cloned 3D model of female hand, (e) cloned 3D model in a di®erent view and position with inherited skin- ning properties. 10 texture of the hand, we extract surface anatomy information and estimate osseous structures underneath the skin. The method has two main parts. The ¯rst part is a surface anatomy feature- extraction method, based on tensor voting, to extract the main creases on the palmar skin and the hand geometry without human guidance. Joint structure is estimated from an anatomical analysis of the relationships between the surface anatomy and its osseous structure. The second part deforms a prede¯ned generic 3D hand model using scattered data interpolation based on a radial basis functions (RBFs). A curve segment matchingmethodperformsautomaticfeaturecorrespondencesbetweenthe3Dhand model and the contours detected in the hand image. These matches capture the contour of the hand image as well as the geometry of joints, ¯nger tips and ¯nger valleys. As a result, the ¯nal hand model has the same joint structure, hand geometry, contour curves, and texture map as the original hand image. Our method requires only an image of a hand and requires no manual input. Figure 2.2 describes our methods and user interaction level in each step. The details are covered in the remainder of this chapter. 2.2 Related work The subject has been considered in human computer interaction (HCI), gesture recognition, medical simulation, as well as in several important graphics papers. Relatively simple hand models have been developed for gesture interface research [105] and research related to motion analysis, hand tracking, and gesture recogni- tion. In vision-based tracking and analysis, accurate kinematical models and joint 11 Fig. 2.2: Hand modeling overview: Two main parts are grouped into four steps in the gray boxes. Supplementary parts, which need additional tools or are beyond the scope of this work, are indicated by the dotted boxes. Numbers beside each box indicate the required user interaction level; level 2 is fully automatic, level 1 requires no user interaction but needs some parameters (experimentally ¯xed in normal case; canny edge thresholds, tensor voting ¾, ¯nger extraction thresholds), level 0 needs manual input. constraints are required based on biomechanical and anatomical hand motion anal- ysis, but a hand model with simple geometry and without skin deformation is used in general [52,57]. Thompson et al. described an interactive 3D graphics workstation system for hand surgery practice that encompasses the simulation of both the kinematics and dynamics of the human hand [98]. A cadaver hand specimen was used to obtain CT images, from which 3D structures were modeled. The segmentation of each bone and joint axis was accomplished by manual input in an interactive 3D display workstation. 12 In addition to an accurate kinematics model, high quality hand geometry and skindeformationmodelsaredesirableincomputergraphicsapplications. Moccozet et al. pursue hand modeling and animation based on Dirichlet free-form deforma- tions (DFFDs) [68]. A multi-layer deformation model simulates the intermediate layerbetweenthe skeleton and the skin. Skin deformations use the palm and ¯nger creases as constraints, based on the observation that each joint of the skeleton is associated with a hand crease on the surface. However, in their paper, the creases were determined and designed manually. Kry et al. use a ¯nite element model (FEM) of the human hand to obtain models in an example-based approach. The models are then compressed using principalcomponentanalysis,andrealtimedeformationisdemonstratedusingGPU acceleration [48]. Anatomy-based modeling of the human musculature is presented by Scheepers et al. [89]. They developed muscle models considered by the in°uence of the mus- culature on surface form. The method is demonstrated by the torso and arm of a human body. Ip et al. presented an anatomy-based hand model with a hand gesture coding system called Hand Action Coding System(HACS) [37]. Target gestures are given by static images and the hand motion for the target gesture is generated from the system. However, soft-tissue, tendons and ligaments are not modeled explicitly. Albrecht et al. developed a human hand model with its underlying anatomical structure: skin, muscles, and bones [1]. Hand animation employs a hybrid muscle model. Pseudo muscles control the rotation of bones based on anatomical analysis andmechanics. Geometricmusclescontrolthedeformationoftheskinusingamass- springsystem. Thehandskingeometryisbasedona3Dscannerandthebone-joint structure is obtained from a public 3D skeleton model. Anatomical analysis and physical models achieve accurate ¯nger movements and skin deformations. Their work has some similarity to our work to generate individual hand models. Their 13 method uses an image and scattered data interpolation to deform a generic hand model, butitrequiresmanuallyde¯nedcorrespondencesbetween3Dfeaturepoints and the hand image. Furthermore, the limited numbers of feature correspondences are not enough to generate an accurate shape match. In[49],anexample-baseddeformablehumanhandmodelisderivedfrommedical images. Multiple CT scans of the hand are taken in several poses. A comparison of bone shapes and di®erent poses allows the estimation of joint locations. Hand skin deformation is achieved using the weighted pose space deformation (PSD) method. Tsang et al. describe an anatomically accurate skeletal musculotendon model of human hand. The hand skeleton can be moved using muscle activations which derive forward and inverse dynamics simulation [99]. Determination of realistic hand poses for interacting with other objects is also a challenging research topic in computer graphics and robotics. Several papers suggest interesting solutions for this issue. Rijpkema et al. developed a hand simulation system to demonstrate grasping objects by hand [83]. They used a knowledgebasedapproachtosatisfyconstraintsofthehandmotionandreducethe enormoussearchspaceofgraspingoperations. Thetargetobjectisidenti¯edbythe similarity of the shape and the grasping strategy associated with the object class is determined by the knowledge base. Koga et al. presented a method to generate humanarmmotionsperforminggivenmanipulatingtasks[46]. Anewpathplanner generates collision-free trajectories to handle moving objects and develops inverse kinematics algorithm to constraint human arm motion. An automatic grasping motion control is demonstrated by multi-sensor approach in [36]. The strategies of grasping are determined by a heuristic method and the arms are moved toward theobjectusinginversekinematics. Multi-sensorsattachedtoeachhandjointscan detect collisions with the object and control the ¯nger positions for grasping. More complex hand animations are demonstrated by playing musical instru- ments. Kim et al. presented a system to simulate ¯nger movement to play the 14 violin [44]. The movements are generated by neural network training and the ¯nger position is decided automatically by the best ¯rst search. Furthermore, ElK- oura et al. addressed problems of physically based hand control using kinematics and dynamics, and demonstrated the hand animation of a guitar player using their data-driven approach [20]. From motion captures of real human hands, a space of possible con¯gurations for ¯nger movements is extracted. One of the challenging tasks for the physically based hand control is creating an e®ective joint controller. Pollard et al. developed a controller for physically based grasping that draws from motion captures of real hands [77]. Their system calculates all parameters of the controller automatically from the given set of cap- tured examples of grasps and shows interactions between two hands actuated by the controller. It may be noted that the general approach of deforming a generic prior model to match individual shape data has been successfully applied to heads [42] and the wholebody[4,90],andthatproblemdomainknowledgewasusefullyappliedineach of these cases. Among these works, the semi-automatic technique to reconstruct 3D creature from an anatomically based canonical model should be noted [93]. An articulated canonical horse model was created by hierarchical segments including bones, muscles, and skin. The canonical model is reconstructed to a target model using the features anatomically generated from the underlying structures. The re- sultingcreaturehassimilarbones,muscles,aswellasskinmesheswiththecanonical model. The method has similarity to our idea in terms of using a canonical model to reconstruct a target shape and basic anatomical information to generate corre- sponding features. Although the denser features are generated automatically by anatomical structure, the basic features are created roughly by manual input. Anguelov et al. introduced the SCAPE (Shape Completion and Animation for PEople) method as a data-driven approach to create deformable human body shapes of individual people [5]. A set of di®erent human bodies are acquired from 15 3Dscansofeachpersonandthesamplesareusedtogeneratedi®erentbodyshapes of each individual. Furthermore, the pose deformation components of a canonical human body are obtained from 3D scans of a single person in multiple poses to demonstrate skin deformation arising from the human body animation. From the two separate space representing variations of individual body shape and deforma- tions of each pose, the SCAPE estimates individual body shapes in arbitrary poses including rigid and non-rigid skin deformation arising from the joint movements. Research in anatomy and biomechanics of human hand should be considered in this study and related information about anatomical structure can be found in many anatomical resources [19,107,108]. However, because of required knowledge about the complex anatomical structures, we cover the related work on the issue in a separate section [section 2.3]. In this section, some basic knowledge about the bone and surface anatomy is studied and important anatomical observations related to this study will be described. 2.3 Hand anatomy The human hand has a complex anatomical structure consisting of bones, mus- cles, tendons, skin, and the complex relationships between them [9,48]. Analysis of human hand anatomical structures is important in various ¯elds, including er- gonomics, HCI, hand surgery, as well as computer animation. The bones of the hand are grouped into three areas: digital, carpal, and wrist. The digital bones of four ¯ngers consist of distal, middle, and proximal phalanges with distal interphalangeal (DIP), proximal interphalangeal (PIP), and metacarpal phalangeal (MCP) joints. The digital bones of the thumb consist of the distal and proximal phalange with the interphalangeal (IP) and carpometacarpal (CMC) joints. The carpal bones are a set of eight small bones, and the wrist bones are the radius and ulna [Figure 2.3(a)]. 16 Fig. 2.3: Human hand anatomy: (a) Bone Anatomy: 1. Distal phalanx, 2. Middle phalanx,3. Proximalphalanx,4. Metacarpals,5. Ulna,6. Radius,A.DIPjoint, B.PIPjoint, C.MCPjoint, D.IPjoint, E.CMCjoint; (b)SurfaceAnatomy: 1. Distal phalanx, 2. Middle phalanx, 3. Proximal phalanx, 4. Distal palmar, A. DIP crease, B. PIP crease, C. Palmar digital crease, D. Distal palmar crease, E. Proximal palmar crease, F. Thenar crease, G. Distal wrist crease, F1-5 present thumb, index, middle, ring, little ¯nger. Inbiomechanics,notonlytheanatomicalfeaturesbutalsothemechanicaljoints of the hand are important. Several methods are suggested to de¯ne the mechanical joints. In [9,49], the axis of rotation between two bones is de¯ned as a line that does not move in relationship with either bone while the bones move around each other. On the other hand, the mechanical joint centers of the hand have been anatomically estimated as the center of curvature of the head of the bone proximal to the given joint [10,107]. Figure 2.3(b) shows the basic surface anatomy of the human hand [108]. The three palmar skin regions are the palm, ¯ngers, and thumb. The palm region has three main creases. The proximal palmar crease starts from the silhouette edge 17 of the hand near the head of the metacarpal bone of the index ¯nger and runs through the hollow of the palm. The distal palmar crease starts from the head of the metacarpal bone in the little ¯nger and runs through the hollow, passing the head of the metacarpal bone of the ring and middle ¯nger. The thenar crease is located between the proximal palmar crease and distal wrist crease longitudinally. The ¯ngers have three transverse digital creases, the DIP, PIP, and palmar digital crease. A DIP crease and palmar digital crease are located on the thumb. The distal palm area is located between the line, which starts from the radial end of the proximal palmar crease to the ulnar end of the distal palmar crease and the palmar digital creases. The MCP joints of ¯ngers are located in the distal palm area. Since the crease on the palmar skin is produced by a skin °exion fold when the hand is closed, basic creases on the palmar skin have a strong relationship with the underlying bone structure, resulting in landmarks used in hand surgery [9,19,108]. Bugbee et al. demonstrated the relationship between creases of palmar skin and superimposed osseous anatomy using radiograph [11]. Since most skin creases of the palmar skin are consistent with underlying fascia and located near the center of the curvature of the head of the bone proximal of the given joint, we observe that the creases can be used as an estimation of mechanical joints when folding the human hand. Thesurfaceanatomyofthehand, aswellasthebonestructure, isuniqueforan individual human hand [108]. Due to their uniqueness, hand surface features have been proposed for biometric use. For example, a palm print has three principal lines (distal palmar, proximal palmar, and thenar creases) that are unique and unchanging biometrics suitable to identify a person [39,91]. Also, palmistry uses these lines to indicate individual characteristics. 18 2.4 Basic hand geometry extraction (a) (b) Fig. 2.4: Basic hand geometry extraction: (a) Segmented hand image, (b) Basic hand geometry; contour of the hand, ¯ve ¯nger tips, four ¯nger valleys, and medial axis of each ¯nger In this paper, we de¯ne basic hand geometry as the contour of the hand, ¯nger tips, and ¯nger \valleys" shown in Figure 2.4(b). Since we use skin texture as the base hand image, it is generally taken within good lighting conditions, simple dark background, ¯ngers-extended natural pose, and ¯xed camera location. The camera viewing direction is perpendicular to the palm plane and the image up vector is on the axis from wrist joint to middle ¯nger tip. We assume an orthographic camera andexplainourmethodbasedontherighthandduetothesimilarityandsymmetry of the left and right hand. Generally, contour extraction is a similar problem as background extraction in computervision. Inourcase, sinceweusedaspeci¯edbackground, asimpleCanny edge algorithm [23] followed by a binary image thresholding is enough to extract a proper contour line [Figure 2.4(b)]. When we see the surface anatomy of each 19 (a) (b) (c) Fig. 2.5: Palm creases extraction: (a) Sobel edge image, (d) First tensor voting result, (e) Second tensor voting result (¯nal result) ¯nger, their shapes are almost symmetrical and each ¯nger tip is located on the intersection between the medial axis curve of each ¯nger and the hand contour curve. Since our canonical photograph pose has the ¯ngers oriented upward, we can easily locate the ¯nger extremities by scanning the medial axis curves in the y-direction. When we sort the ¯ngers with respect to x-axis, we can easily identify the ¯nger tips for each¯nger. The ¯nger valleysare located in between each ¯nger. We trace a contour curve from each ¯nger tip and de¯ne each ¯nger valley as the in°ection point of that curve. 2.5 Hand crease extraction Thepalmarskinhascomplexgeometrywithlotsofdiscontinuouswrinkles. Among the wrinkles, just a few salient creases are meaningful in terms of surface anatomy. Although humans can easily detect these creases, computers can hardly detect themwithouthumanvisualperception. Tensorvoting[32]isane±cientperceptual groupingframeworktodetectdensesalientfeaturesinnoisyinputdata. Weusethe 20 tensor voting framework [38,65] to detect principal creases and extract continuous curves for each crease in the palmar skin image. 2.5.1 Tensor voting overview Fig. 2.6: TensorVoting: (a)tensorellipse,(b)2Dstickkernel,(c)Tensorvotingexample: leftimageshowsthesparseinputtokenswithoutorientation,middleimageisthe tokens oriented by tensor voting but still sparse, the right image is the resulting extracted dense curve. Extracting main creases from the palmar skin is a similar problem to inferring salient curves from sparse and noisy data. Tensor voting can extract salient geo- metric features (e.g. point, curve, surface) in a sparse and noisy data set without humanguidance. Thetensorvotingframeworkproposedin[65]needsonlyonefree parameter ¾ to de¯ne the scale of the vote, and handles multi-dimensional data as well as 2D data for various applications. Tensor voting can be grouped into two major stages: data encoding using tensors and linear tensor voting for tensor communication. The input tokens, a set of points or oriented points in a noisy 21 environment, can be encoded into a set of geometric features which are salient, smooth and considerably continuous. In the case of 2D, each token can be encoded into a second order symmetric tensor which is equivalent to an ellipse whose major axis serves as the orientation and the length of that axis is the saliency. In Figure 2.6(a), an oriented point is representedbyathinorientedellipse. Inthevotingstage,eachtokenisaccumulated into the second order symmetric tensor form by votes cast from its neighbors' prede¯ned voting ¯elds. The voting ¯eld of oriented tokens can be derived from the2Dsticktensor(degenerateellipsewith¸ 2 =0)calledthefundamental2Dstick kernel, and it decays the saliency of the vote. In spherical coordinates, the decay of the 2D stick kernel is of the form: DF(°; µ; ¾)=e ¡( ° 2 +cµ 2 ) ¾ 2 (2.1) Where,° isthearclengthbetweenvoterandreceiver,µisthecurvaturebetween voterandreceiver,¾isthescaleofvotingwhichde¯nesthesizeoftheneighborhood of each token, and c is a constant. In the Figure 2.6(b), the osculating circle connectingvoterOandreceiverPgeneratesthesmoothestcircularpathconnecting O and P, and the length of normal vector P represents the saliency of the vote. After voting, the second order symmetric tensor T can be decomposed into its eigenvectors e 1 , e 2 with related eigenvalue ¸ 1 ¸ ¸ 2 . T =[e 1 ;e 2 ] h ¸ 1 0 0 ¸ 2 ih e 1 T e 2 T i =¸ 1 e 1 e T 1 +¸ 2 e 2 e T 2 (2.2) Equation2.2canberearrangedwithT c ,whichrepresentscurvecomponent,and T p , which represents point component encoding junction information. T =(¸ 1 ¡¸ 2 )e 1 e T 1 +¸ 2 (e 1 e T 1 +e 2 e T 2 )=T c +T p (2.3) 22 Asaresultofvotingin2Dspace,wehavetwodensevectormapswhichrepresent curve and junction features. The salient curves are located at the local extrema of map T c , and they can be extracted using non-maxima suppression. Please refer to [32,65] for details. 2.5.2 Wrinkle extraction using tensor voting In order to increase the performance of tensor voting, we segment the hand image. From the hand geometry extracted in section 2.4, we can roughly divide hand images into three regions; the palm, thumb, and four ¯ngers. In the four ¯ngers and thumb region, the medial axis for each ¯ngers and thumb are calculated. For all pixels in the four ¯ngers and thumb region, ¯nd the nearest medial axis line. The region which is related to the nearest line of the given pixel is assigned to the region for that pixel. Finally we achieve six image segments, like Figure 2.4(a). The image segments are converted into the set of edge points using the Sobel edge algorithm [23]. Although the Canny edge algorithm is generally used for edge detection, the Sobel edge algorithm shows better performance as an input of tensor voting in our experiment [Figure 2.5(a)]. Since the only free variable ¾ is not sensitive, we experimentally de¯ne the voting scale ¾ as large as possible to reduce the in°uence of local features which can produce noise. The result of ¯rst tensor voting is in Figure 2.5(b). In order to eliminate small noises and achieve smoother curves, we perform a second tensor voting using the input of the ¯rst tensor voting results. In the second voting stage, theoutlineofthe¯ngerswhicharegeneratedbythe¯rsttensorvotingiseliminated before voting, since the contour line is salient and can cause errors in the voting process. The ¯nal result of tensor voting is the set of salient wrinkle curves and each curve is smooth and mostly continuous [Figure 2.5(c)]. 23 2.5.3 Palm crease extraction Now we have images of continuous principal wrinkles of the palm and each of the ¯ngers but they do not yet correspond to the important creases of surface anatomy. In the palm region, we have three anatomically meaningful creases, the distal palmar, proximal palmar, and thenar crease [Figure 2.3]. Since the wrinkle curvesachievedfromtensorvotingarecontinuousandhavelongerlengththanother curves, we can extract these creases using a tree search algorithm that employs knowledge of hand geometry. Variables: R=region, r=start point, W=wrinkle, P=path, L=path length 1 De¯ne region R using basic hand geometry [Section 2.4]; 2 Scan R ¯nding start points r i of wrinkle curve W i ; /* Where r i =fr 1 ;:::;r n g;W i =fW 1 ;:::;W n g;r i 2R*/; 3 for all r i do 4 while Traverse curve from r i to the end using DFS do 5 if found the largest path P i then 6 W i Ã¡P i ; L i Ã¡Length(P i ); 7 if IsMaximum(L i ) then 8 Crease CÃ¡ W i ; 9 Repeat step 1 to 8 for each anatomical region R; Algorithm 1: Palm crease extraction algorithm First,weassignanatomicalregionRto¯ndthestartingpointofthecrease. For example, the root of the distal palmar crease can be located in region R d (R d 2R) in Figure 2.7(a). The region R d can be de¯ned easily using hand geometry features such as ¯nger valley 4 and the MCP joint of thumb. The MCP joint of thumb can be roughly estimated using the ¯nger line of thumb and ¯nger valley 1. The region for the root of the proximal palmar crease can be assigned by similar anatomical analysis. Since the start points of several wrinkle curves can be in the region R, we should ¯nd the main crease among these wrinkle curves. First, we extract every wrinkle curve and its arc length using depth ¯rst search (DFS), since our wrinkle 24 curve has no loops. For every start point r i (r i 2 region R), traverse wrinkle curve using DFS and assign the largest path P i as the wrinkle curve W i having length L i . The wrinkle curve W i which has the largest length among L i is assigned to the crease C within the area R. We summarize the method in Algorithm 1. Green curves in the Figure 2.7(a) are the extracted proximal palmar crease and the distal palmar crease. 2.5.4 Finger crease extraction We propose a method to extract the crease of each ¯nger from the given wrinkle images. First, label in turn each segmented image in section 2.5.2 as ¯nger region R i , (i for each ¯nger). Then, assign ¯nger line F i using the least square line ¯t of the medial axis curve of each ¯nger region R i (calculated in section 2.4). For each ¯nger, perform the following operations in Algorithm 2 to extract representative lines of ¯nger creases. The result is in Figure 2.7(b). 2.6 Hand modeling Wemadeageneric3Dhandmodel[Figure2.9(c),(e)]whichhasskingeometry,joint skeleton, andskindeformationusingMaya[2]andPoser[18]. Theskingeometryis exported from Poser and bound to the joint skeleton using Maya for skin deforma- tion and convenient joint control. The joint skeleton has accurate biomechanical structure using 21 joints (including ¯nger tips and wrist) [Figure 2.9(e)]. Each joint has a di®erent degree of freedom for better control. Although carpal and wrist bones consist of several bone segments, since the radio carpal joint plays themostimportantpartofoverallmovement,wede¯nedonethree-DOFmechanical joint for the wrist joint. DIP and PIP joints of each ¯nger and the IP and MCP joints of thumb have one DOF. MCP joints of each ¯nger have two DOF for the simplicity of control and the CMC joint of the thumb has three DOF due to its 25 Variables: R: ¯nger region, F: ¯nger line, p: current position, W: wrinkle curves, O: orthogonal line(F, p), O': Selected lines, C: Selected crease line, threshold1 and threshold2 are de¯ned after experi- ment. 1 for each ¯nger region R i and line F i do 2 while Trace F i from ¯nger tips to wrist using p i do /* p i (1·i·5; p i 2R i ) is a current tracing position */; 3 if p i is an intersection point between F i and W ik =fW i1 ;:::;W in g then 4 CalculatealineO ik =fO i1 ;:::;O in g,whichisorthogonaltoF i and passes through the intersection point; 5 for all pixels X i 2 creases of R i do 6 if jX i ¡O ik j<threshold1 then vote(O ik ); 7 if vote(O ik )>threshold2 then O 0 ik Ã¡O ik ; where O 0 ik =fO 0 i1 ;:::;O 0 im g; 8 if O 0 ik is veri¯ed by the hand geometry such as ¯nger region, tips, and valleys then C ik Ã¡O 0 ik ; where C ik =fC i1 ;:::;C iN g; Algorithm 2: Finger crease extraction algorithm complexity. For simple notation, we will call the generic 3D hand model a generic hand, the 2D hand features extracted from the picture a source hand, and the remodeled 3D hand model a clone hand. The generic 3D hand model is reshaped as a clone hand that has the same mechanical joints, contour curve, and texture map as the unique human hand. 2.6.1 Joint estimation From the careful analysis of hand surface anatomy (consisting of the basic hand geometryandcreasesextractedintheprevioussection),wecanestimatemechanical joint locations of the source hand. From the ¯nger crease lines C in and ¯nger lines F i obtained in section 2.5.4., we can ¯nd DIP and PIP joints of the four ¯ngers and 26 Fig. 2.7: Crease extraction and joint modeling: (a) Green curves are the extracted palm creases (b) Blue lines are the location of ¯nger crease (c) Blue dots are the calculated joint locations the IP and MCP joints of the thumb. Since the ¯nger lines are the medial axis, the mechanicaljointsarelocatedontheintersectionpointsbetweenthecreaselinesC in and ¯nger lines F i . If two creases are extracted near a joint, we locate the joint at the lower crease with the analysis of surface anatomy. Whenweconsidersmoothskindeformationwithcreaseconstraints,MCPjoints ofthethefour¯ngersshouldbelocatedinthedistalpalmarea[Figure2.7(c)]. From each ¯nger line F i we can calculate line segment S i , which is between the related digitalcreaseandthelineH,whichisde¯nedbystartpointsofdistalpalmarcrease H a and the start point of proximal palmar crease H b [Figure 2.7(c)]. Within each segment S i , we can arrange MCP joints at approximately one-third of S i from the line H with the analysis of surface anatomy in section 2.3. The two highest curvature points, W a and W b of the ulnar and radial side of contour curve in wrist area, can represent the anatomical boundary between the wristbones(ulnaandradius)andcarpal bones. These twopoints, W a andW b , are simplycalculatedbythehighestcurvaturepointsofcontourwithinthewristregion, 27 which is below the MCP joint of the thumb. The wrist joint can be estimated by mid-position of W a and W b [Figure 2.7(c)]. We have extracted every mechanical joint except the thumb CMC joint with analysis of surface anatomy. However, since the CMC joint of the thumb has relativelycomplexmovementscomparedwithotherjoints,and2Dsurfaceanatomy cannot give enough information, estimating the CMC joint is very di±cult. We position the thumb CMC joint based on the following anatomical analysis. First, the metacarpal bone of the thumb is parallel to line M and starts from the MCP joint of thumb [Figure 2.7(c)]. Line M can be calculated by least square line ¯tting on the middle part of contour curve segment between the highest curvature point W b and the MCP joint. The axis of the radius bone is parallel to the medial axis of wrist and passes point W b . Since the anatomical CMC joint is located on the end of the metacarpal bone, the mechanical CMC joint can be on the intersection point between the axis of the radius bone and metacarpal bone. Fromtheextracted2Djoints,wecanmodifythejointskeletonofthe3Dgeneric hand. Since the origin and the y-axis of the generic hand model is determined by the wrist joint and the vector from wrist joint to middle ¯nger tip, respectively, a simple a±ne transformation matrix M s can transfer the source hand coordinate to the generic hand coordinate. For accurate scaling, we measured hand length from the wrist to the middle ¯nger tip when taking the hand picture. The generated joint skeleton model for clone hand is shown in Figure 2.9(f). 2.6.2 Skin modeling In addition to the joints, the contour of the hand is a unique and important feature of a person's hand. Curve segment matching can allow automatic feature corre- spondence between the source hand and generic hand. From the corresponding features we can warp entire geometry using scattered data interpolation based on the radial basis functions (RBFs). 28 2.6.2.1 Curve segment matching Fig. 2.8: Curve segment matching example: (a) Curve matching example of curve seg- ment S 2 (s 2 ;e 2 ): s 2 is on the intersection between contour and the palmar distal crease of little ¯nger, and the e 2 is on the intersection between contour and the PIP crease of little ¯nger, (b) Curve segment 2 in generic hand model From the joint modeling process in section 2.6.1, we can obtain a joint skele- ton model with 21 corresponding features. The skin vertices related to the joint skeleton can be used as the corresponding feature between the generic and source hand. In addition to these 21 features, contour of the hand provides good feature correspondences, since the source hand picture is also a skin texture image taken in the speci¯c viewing direction and pose [Figure 2.9(a)]. Whenwedeformthehandskin,themeshbetweencreasesshouldbecontinuous. The contour of the hand can be divided into 30 curve segments G i =fG 1 ;:::;G 30 g basedonthehandcreasesandgeometryfromtheanalysisofhandsurfaceanatomy [Figure2.1(c)]; eachG i consistsofordered3DverticesV ij =fV i1 ;:::;V in galongthe curvesegmentG i ,wherenischosentorepresentcurvesegmentgeometrysu±ciently 29 andG i arecarefullyextractedwhenmakingagenerichandmodelusingtherelated joint skeleton. Every G i is transformed to the source hand coordinate using the inverse matrix of M s in section 2.6.1. The transformed curve segments are G 0 i =fG 0 1 ;:::;G 0 30 g and each segment G 0 i consists of transformed 2D points V 0 ij . With curve segment matching based on the arc length, we can ¯nd corresponding feature of V 0 ij in the sourcehandimage. Ouralgorithmcanaccumulatesmallerrorswithinitstraversing step but it shows acceptable results in our experiment. The method is described in Algorithm 3 and an example of curve segment 2 is in Figure 2.8. Variables: G: curve segments of the generic hand, G': transformed curve segments of the generic hand, S: curve segments of the source hand, V': transformed vertices composing a curve segment G' 1 De¯ne curve segment S i (s i ;e i ) = fS 1 ;:::;S 30 g from hand anatomy [Fig- ure 2.1(c)]; /* s i : start point of segment, e i : end point of segment */; /* Each G 0 i correspond with segment S i using its anatomical location */; 2 Calculate arc length of G 0 i and S i ; 3 Calculate scaling ° = length(S i )/ length(G 0 i ); 4 for all segment i do 5 for all vertices j in segment i, V 0 i;j =fV 0 i;1 ;:::;V 0 i;n g do Calculate D j =jV 0 i;j+1 ¡V 0 i;j j /* n: number of vertices on the G 0 i */; 6 Initialize s 0 j = p k = s i ; /* s 0 j : current start point, p k : pixels in the S i */; 7 while traverse p k from s i to e i do 8 Calculate D 0 j =jp k ¡s 0 j j; 9 if D 0 j ¸°D j then 10 Assign p k as the corresponding feature of V i;j+1 ; 11 s 0 j Ã¡p k ; Algorithm 3: Curve segment matching algorithm 30 2.6.2.2 Skin deformation Withthesetofpointlocationsx i =fx 1 ;:::;x n gandtheirvaluesf i =ff 1 (x 1 );:::;f n (x n )g we can ¯nd function R(x) which gives smooth interpolation of these data using ra- dial basis functions (RBFs) [14,54]. R(x)=P(x)+ n X i=1 ¸ i Á(jx¡x i j); x2R d (2.4) In equation 2.4, P(x) is a low-degree polynomial, ¸ i is a real valued weigh, Á is abasisfunction, andnisthenumberofcontrolpoints. OurRBFs R(x)arede¯ned by the corresponding feature points achieved in section 2.6.2.1, and its distance vector between the generic hand and source hand. The thin plate spline (TPS), Á(r) = r 2 logr is used as our basis function for smooth deformation of our skin mesh, since TPS interpolates speci¯ed points while minimizing an approximate curvature [7]. The vertices of the generic hand mesh are displaced by the RBFs resulting in a clone hand which has the same contour curve, hand geometry, and joint location as the source hand. The depth value of hand geometry was not considered in RBFs deformation, since the features for depth could not be obtained from the 2D source hand image. Instead, the generic hand depth is proportionally scaled by the hand length measured in section 2.6.1. 2.7 Experimental results Ourautomatedmethodgeneratesaperson-speci¯chumanhandmodelwithsimilar joint structure and skin geometry to the source individual's hand. The similarity between the original and cloned hand can be seen in Figure 2.9(a) and (b). While texturing and animated skin deformation are beyond the scope of this paper, some 31 supplementary properties of our process help reduce the e®ort involved in these tasks: ² The source hand image can be re-used as the skin texture map in the 3D clone hand. Due to the similarity of contour shape and scaling between the clone hand and the skin texture, the texture mapping can be executed using simple planar projection in the camera direction. ² Also, because our generic hand model has a joint skeleton bound to skin geometry, it is easy to control the ¯nal model using this joint skeleton [Fig- ure2.9(f)]. Ifthegenerichandmodelhasanimatedskindeformationorother properties, the clone hand inherits those characteristics properly. These supplementary properties are demonstrated in Figure 2.10, and Fig- ure 2.1(e) where the individualized model is textured by planar projection, and the skin deformation system from the underlying generic Maya hand model is used to deform the skin guided by rotations of the skeleton joints. We tested our method by creating ¯ve human hand models having di®erent age, sex, and scale; a four year-old girl, a ¯veyear-old boy, a large male, an average female, andanaveragemale. Wetookhandpictureinspeci¯cconditions described in section 2.4 and used two generic hand models, a low resolution mesh (13372 triangles with 6715 vertices), and a high resolution mesh (32784 triangles with 16423vertices). TheresultsareshowninFigure2.11. The¯rstrowshowsextracted joint information, the second row shows rendered clone model with texture maps, the third row shows the polygon mesh, and each column demonstrates cloning a particular source hand. Our method requires setting several thresholds to extract creases; Canny edge detection, tensor voting, and crease extraction. Once set the default thresholds are su±cient during our test provided good lighting conditions (without signi¯cant shadows on surfaces) in the source image. 32 Fig. 2.9: Hand clone results: (a) picture of source hand, (b) clone hand model with texture mapping, (c) skin geometry of generic hand model, (d) deformed skin geometry of clone hand, (e) joint skeleton of generic hand model (each circle is a joint location), (f) modi¯ed joint skeleton of clone hand 33 Fig. 2.10: Hand clone result in various view points 34 Fig. 2.11: Hand clones of ¯ve people: the ¯rst row is a set of source hand images and extracted joint information, the second row is a set of rendered clone models with texture maps, and the third row is a set of polygon meshes of the clone models; (a) a four year-old girl of 12cm hand length (low resolution mesh), (b) a ¯ve year-old boy of 13cm hand length (low resolution mesh), (c) an average female of 16.5cm hand length (low resolution mesh), (d) an average male of 18.5cm hand length (high resolution mesh), (e) a large male of 21cm hand length (high resolution mesh). 35 Chapter 3 Real-time Skin Deformation on GPU 3.1 Overview Skinning is an important part of realistic articulated body animation and has been considered as an important topic of computer graphics and animation. Generally, skinning can be categorized into algorithmic, physically-based, and example-based methods. Although widely used, simple algorithmic skinning schemes cannot cap- ture the complexity and subtlety of real skin deformation, and revised approaches will be required to increase character realism. Physical-based skinning uses biome- chanics of the skin deformation arising from the motions of muscles and tendons. Although this approach can generate physically accurate simulations of each layer, itisnotatpresentsuitableforrealtimeapplicationssuchasgamingduetothelarge computation required. Example-based methods capture some of the complexity of real skin deformation by interpolating scanned or sculpted examples of the desired skin shape in various poses. Although this requires gathering a su±cient number of samples and some pre-calculations, example-based methods can potentially be used in real-time applications due to their relatively simple real-time computation. The weighted pose space deformation (WPSD) is an example based skinning method to generate adequate skinning within limited number of sample poses [49]. Although it can generate accurate skinning, it requires more computation than the 36 original PSD since joint distances are computed independently for each vertex. As such, this method has not been suitable for real-time applications. Furthermore,becauseWPSDusesthejointweightsforeachvertexincalculating pose distance, as with SSD skinning, accurate joint weights are required to achieve goodresult. However,generally,theweightsaremanuallygeneratedbyartistswith large amount of time and in case of complicated skeletal joints this requires great skill. In this chapter, we present a parallel WPSD algorithm (including automatic determination of joint weights) suitable for SIMD architectures such as current GPUs. Joint weights of each vertex are automatically computed using sample poses. This can enhance the skinning quality not only of SSD but also WPSD, since both methods require accurate joint weight values. 3.2 Related work Skinning of articulated object has been considered as a challenging topic in com- puter graphics and animation and has been a subject of recent research for articu- lated body animation including hand simulations. Many commercial software packages generating skin deformation arising from joint movement use a method known as (linear blend) skinning, Skeletal Subspace Deformation(SSD),enveloping,etc.,basedinpartonworkpublishedbyThalmann et al. [60]. SSD is based on the weighed blending of a±ne transformations of each joint and used in many real-time applications due to its simple and fast computa- tion. However it shows erroneous results known as collapsing-joints artifacts and several papers are published to solve this problem [43,69,101,102]. Recently example-based methods [3,48,49,55,95] have permitted more complex skinning e®ects such as muscle bulges and major wrinkles, while also addressing theartifactsofsimplealgorithmicschemes. Inthesemethodsanumberofprovided 37 (scanned or sculpted) samples of the desired skin shape are simply interpolated based on the creature pose (and possibly additional control \dimensions"). These example-based methods can also be considered as a non-parametric approach to skin deformation. In common with non-parametric sampling methods in texture synthesis (and more generally in statistical regression), the amount of memory for these methods grows with the number of training samples, but arbitrary distribu- tions can be approximated. Some of the most impressive example-based results to date are Kurihara and Miyata'shandmodelderivedfrommedicalimages[49]. Sinceacquiring3Dmedical images are relatively expensive, they developed weighted pose space deformation (WPSD) to generate proper skinning with a limited number of pose samples. They de¯neddistanceofeachposeusingjointweightofeachvertextogiveproperweight to each element of a joint vector. AlthoughthejointweightforeachvertexisimportantdataforSSDandWPSD calculation, it has traditionally been manually generated by skilled artists. Least squaresbasedvertexweightestimationwasshownintheskinningmethods[69,101]. Jamesetal.describedmeshbasedskinningincludingestimationofboneparameters and vertex weight of each bone [41]. In their paper, vertex weights of each joint are calculated by NNLS (non-negative least squares) and we derive a similar approach to calculating weights for SSD and WPSD. Inrecentyears, sincetheperformanceofGPUshasbeenimprovedmorerapidly than that of CPUs, and GPU has many processing units serving as SIMD parallel architecture, many algorithms have been enhanced by GPU programming [29,59, 74]. DeformationandskinningalgorithmscanalsobeenhancedbyGPUandseveral papers have pro¯ted from this [8,40,41,48]. In the previous research, since vertex information cannot be accessed in the fragment program, GPU-based vertex deformation is usually performed by vertex 38 programs. Inthisstudy, wedevelopaparallelWPSDalgorithmusingthefragment program to gain greater parallelism and performance. Person speci¯c data modeling and its deformation is also an interesting topic in realistic articulated body simulation. We described human hand modeling from surface anatomy of the person in chapter 2 and [82]. Anguelov et al. developed shape completion and animation of people derived from the set of range scan data and example based deformation in pose and shape space [5]. Physically inspired skinning should be also recognized as another important area of articulated body animation. Physical-based skinning uses biomechanics of the skin deformation arising from the motion of muscles and tendons. Since the methods rely on the biomechanics of underneath layers, careful analysis of related anatomical structures is required. Gourret et al. simulated deformations between objects and the grasping hand using a numerical method based on ¯nite element method (FEM) [28]. The active and reactive forces are calculated during grasping and the forces drive the defor- mation for object as well as the ¯ngers. In their human factory system, the ¯nger °exion without contact is calculated by a hybrid method based on joint-dependent local deformation operators and the FEM is used when the contact is detected. Oneoftheimpressiverecentworksforphysical-basedhandskinningisexplained in the paper of Albrecht et al. They developed a human hand model with its underlying anatomical structure: skin, muscles, and bones [1]. Hand animation employsahybridmusclemodel. Pseudomusclescontroltherotationofbonesbased on anatomical analysis and mechanics. Geometric muscles control the deformation of the skin using a mass-spring system. Some recent works show rather di®erent or hybrid approach to the issue. An outside-inapproachtocreatemuscleandtendonmodelwasdevelopedbyPratscher et al [78]. From the ¯gure of manually created skin model and control skin, inner muscles and tendons are generated and reshaped to conform the outer surface. 39 Then,theouterskincanbedeformedbytheresultingmusculatureusinganymuscle deformation method. Capell et al. introduced a framework of skeletal control for dynamic deformable models in [13] and improved their work for skin rigging of elastic articulated body in [12]. They control deformations indirectly from rigging forces and the forces can be combined with other forces of elastic dynamics during the simulation. Although physical-based skinning approach can generate physically accurate simulation of each layer, it is not at present suitable for real time applications such as gaming due to the large computation required. Since our study focuses real- time skin deformation, especially for complex joint structures like human hand, we uses example-based methods rather than physically inspired skin deformation. Therefore,weentrustthefurtherreviewofthephysical-basedskinningtotherecent related papers [1,12,78,92]. 3.3 Skin deformation Example-based skinning problems can be described by following the equation, ^ v i (p a )=S(v 0;i +D i (p a )) (3.1) where i indexes each vertex, p a is an arbitrary pose, ^ v(p a ) is a vertex of a deformed target surface of arbitrary pose, v 0 is an undeformed (rest pose) vertex, S() is the SSD function, and D(p a ) is a displacement as a function of an arbitrary pose. S(v 0 ) is calculated by SSD as a linear transform blending and D i (p a ) can be calculated by interpolation in pose space using PSD. 40 (a) (b) (c) (d) (e) (f) Fig. 3.1: Skinning results of each algorithm: the ¯rst row shows SSD, the second row shows PSD, and the third row shows WPSD. Areas of blue and red arrows represent unexpected results of SSD and PSD respectively. The second column shows deformed vertices of each algorithm. 41 3.3.1 Skeletal subspace deformation (SSD) SSD [60] is based on the weighed blending of a±ne transformation of each joint by equation 3.2. v a =( n joint X j=0 w j T j )v 0 (3.2) where n joint is the number of joint, v a is a vertex in pose p a , v 0 is a vertex in the rest pose, T j is an a±ne transformation of joint j in the world coordinates, and w j is the weight of each joint which de¯nes the contribution of joint transformation to the current vertex. The w j can be assigned by the artist to control deformation and usually P n joint j=0 (w j ) = 1:0. This simple algorithm is used in many commercial graphics softwares and real-time rendering applications but shows several limita- tions, because the deformation of this method is restricted to the subspace of the a±ne transformation of the joints [55]. 3.3.2 Pose space deformation (PSD) Ifwehavesu±cientsetofsampleposestodescribemovementofarticulatedobject, we can interpolate displacement within the pose space [55]. Each sample pose consists of skin geometry with related joint skeletons, and a joint vector containing the joint skeleton can represent the pose. If we translate each sample pose using inverse SSD, displacement between the sample pose and rest pose can be calculated: d i =( n joint X j=0 w j T j ) ¡1 v i ¡v 0 (3.3) where, n joint , v 0 , w j , and T j are de¯ned as equation 3.2 and v i is a vertex in sample pose p i and d i is a displacement of the p i . Note that the inverse here is of the weighted sum of a±ne transforms. 42 After de¯ning displacement of each sample pose, displacement of an arbitrary pose can be calculated by interpolation of the sample poses' displacements using weight r achieved by RBFs (Radial Basis Functions). When we de¯ne d a as the displacement of an arbitrary pose p a , it can be calcu- lated as following: d a = n pose X i=0 r i d i (3.4) where n pose is the number of sample pose, d i is a displacement of the sample pose p i calculated in equation 3.3, d a is a displacement of arbitrary pose p a , and r i is a weight of p i for blending. The weight r i de¯nes contribution of each sample pose having a characteristic of P n pose i=0 r i =1:0 and can be calculated by normalized RBFs (Radial Basis Functions) in pose space [49,95]. RBFs (Radial Basis Functions) can smoothly interpolate pose space using: f i (p a )= npose X k=0 ¸ i;k Á k (° k ) (3.5) where f i (p a ) is RBFs of the pose p i with input of an arbitrary pose p a , n pose is the number of sample poses, ¸ is real valued weights, and Á k is the basis functions, and° k isthedistancesbetweenthep k andp a de¯nedbyEuclidiandistancebetween the joint vector of each pose. The weight r i is calculated by normalized RBFs and used as weights of the sampleposep i inequation3.4to calculatethe displacement d a ofan arbitrary pose p a : r i (p a )= f i (p a ) P n pose k=0 f k (p a ) (3.6) Gaussian radial basis function Á k (° k )=exp( ¡° 2 k 2¾ 2 ) is one possible choice of basis and constant ¾ can be speci¯ed experimentally [55]. 43 3.3.3 Weighted pose space deformation (WPSD) WPSD is developed by Kurihara et al. [49] to deform their example-based human hand model derived from medical images. In equation 3.5, ° k is the distance of two pose p a and p k having n joint number of joint for each pose and de¯ned as following function ° k (p a ;p k ). ° k (p a ;p k )= v u u t n joint X j=0 (p a;j ¡p k;j ) 2 (3.7) In equation 3.7, since the ° k is the di®erence of n joint -dimensional joint vectors of related pose, every vertex in the pose p k have unique value of ° k resulting same weight r i in every vertex of the pose p i . Furthermore, because each element of the joint vector equally contributes in distance calculation, two vectors having a same value but di®erent order generate same pose distance. For example three di®erent joint vectors a=(µ;0;0);b=(0;µ;0);c=(0;0;µ) has same distance between them and it can cause unexpected results in PSD. In [49], Kurihara et al. de¯ned distance of each pose using joint weight of each vertex l to give proper weight to each element of a joint vector. ° l;k (p a ;p k )= v u u t n joint X j=0 w l;j (p a;j ¡p k;j ) 2 (3.8) where ° l;k (p a ;p k ) is the distance between pose p a and p k on vertex l, and w l;j is weightofjointj ofvertexlusedinequation3.2. Fromthisde¯nition,moreaccurate pose distance can be achieved and it generate better skinning in an arbitrary pose; especially when it is far from the samples. Figure 3.1 shows result of three di®erent skinning methods, but we entrust the detail comparison between quality of each algorithm to their original papers [49,55,60]. 44 3.4 Computing joint weights from samples Thejointweightsofeachvertexareimportantvaluestogenerateaccurateskinning in SSD (equation 3.2) as well as WPSD (equation 3.8). In many applications, the weightsaremanuallygeneratedbyskilledartistsanditishardtogenerateaccurate values when a number of joints are involved in deforming a region. In this study, we calculate the joint weights of each vertex using sample poses to enhance the accuracy of the weight value resulting better skinning and save elaborate manual works to create weight map. In each sample pose p i we can have following equation based on SSD. ~ v i ¡e i =( n joint X j=0 w j T j )v 0 (3.9) where ~ v i is a particular vertex from sample pose p i , the right hand side is the SSD deformation of vertex v 0 from the rest pose, e i is a displacement between SSD deformationandsampleposep i ,andtheothervariablesarethesameasinequation 3.2. Ifwehavesu±cientsampleposesinvolvingthesamesetofnjoints,wecanhave m (m=number of poses) equations of the form ~ v i ¡e i =( n X j=0 v j w j ) (3.10) where v j is v 0 transformed by T j . Although the e i are unknown, we can solve for weights that minimize the e i in a least squares sense by stacking the equations 3.10 (with e i omitted) into a linear matrix system kx¡Awk 2 (3.11) where w is a n-dimensional weight vector, x is a 3m-dimensional vector con- taining the vertex ~ v i from every sample, and A is a 3m£n matrix . 45 From equation 3.11, we can calculate w with the given value of x and A to reduce the error of this equation. We use the non-negative least square (NNLS) method to solve this problem and it presents positive weight values minimizing error in equation 3.10. The calculated weight vector w is normalized to satisfy P n j=0 w j =1:0. In order to avoid singular matrix A, sample poses can express each joint varia- tions properly and the number of poses should be greater or equal to the number of overall DOF (Degree of Freedom) of the joint vector (each joint has 3 DOF). James et al. used a similar approach to estimate vertex weights in each joint [41] and we demonstrate their e®orts in our skinning method. 3.5 Parallel deformation on GPU A skinning is a deformation based on per-vertex computations. In the skinning method like SSD and WPSD, this per-vertex computation is independent with each pixel and can be parallelized by SIMD parallel architecture. We developed parallel skinning algorithm for SSD and WPSD which is suitable to GPU having SIMD architecture with read-only texture memory. 3.5.1 Parallel skin deformation The computation cost of SSD is O(N vertex £N joint ) from equation 3.1, 3.2, PSD is O(N vertex £N joint £N pose ) from equation 3.1, 3.2, 3.4, and WPSD is O(N vertex £ N joint £N pose £N pose £N pose ) from equation 3.1, 3.2, 3.4, 3.5, 3.6. Where, compu- tation cost of original PSD is de¯ned by equation 3.1, 3.2, 3.4, since r i is same in all vertices and d i can be pre-calculated. In the skinning algorithms, the computation involves each vertex in the ge- ometry, and it is independent in each vertex. Therefore, we can parallelize those 46 algorithms in a SIMD architecture calculating per-vertex deformation in each par- allel processor. In the previous research, the Eigenskin method was developed using GPU ver- tex programming [48]. The vertex program uses relatively small number of slow processing units compared with the fragment processor and per-vertex computa- tion cost of original PSD is O(N joint £N pose ). Therefore WPSD, having a higher per-vertex computation cost O(N joint £ N pose £ N pose £ N pose ), can show better improvement with parallel computation on the fragment processor. We developed parallel skinning using the GPU fragment processor and demon- strate our method using three rendering passes. In order to minimize real-time computation, we separate possible pre-calculation steps and save the result into texture memory using texture maps. Because the value in the texture memory is notchangedinthesuccessivedeformation,itcanbestoredintheread-onlymemory in pre-processing. Inthe¯rstandsecondpass,per-vertexdeformationiscalculatedinthefragment program and the results are stored in texture maps using FBO (Frame Bu®er Object) extension [31]. These texture maps are bound to the geometry of the rest pose with their texture coordinate. In the third pass, each vertex in rest pose is changedbythedeformedvertexstoredintheoutputtexturegeneratedby¯rstand second pass using vertex texture patch. 3.5.2 Packing data into textures Thefragmentprogramcannotaccessvertexinformationintheirprocessor. Instead, we can use texture memory to send data to fragment program. Information needed inthefragmentprogramispackedintotexturemapandstoredintotexturememory. Geometry information of the rest pose are stored into two RGB texture maps; a vertex texture T v and normal texture T n ; each has size S(N vertex £3(rgb)). These 47 Fig. 3.2: Packing data into textures: T i can be T v , T n , T w1 , T w2 , and T dj . V(vertex), N(normal),W(weight),J(jointindex),andD(displacement)representeachtexel (RGB(A)) value of the related texture. T d consist of eight T dj storing displace- ments of each pose. textures represent parameter v 0 in equation 3.2 and each 3D element (x, y, z) are stored into (r, g, b) value of a texel [Figure 3.2]. The joint weights calculated in section 3.4 are also stored in the texture maps. In general, the distribution of skinning e®ects in an articulated body is local to several joints [48,68], even in a region as complicated as a hand. For example, deformations arising from the PIP (Proximal Interphalangeal) joint of index ¯nger donotpropagatetotherest¯ngersanddeformationonthemiddlephalanxofindex ¯nger is only a®ected by the movement of PIP and DIP(Distal phalanx) joints. From this observation, we can save space to store joint weights from actual number of joint N joint to a smaller number of \principal joints" n joint selected by sorting on the weight value. We threshold n joint at four in our tests with an additional four elements to explain related joint index. As a result, we can save joint weight of entire geometry in two RGBA textures T w1 , T w2 each have size S(N vertex £4(rgba)) 48 andstorewholeinformationrequiredforSSD[equation3.2]tofourtextures T v ,T n , T w1 , and T w2 . The displacement values calculated by equation 3.3 can be stored in N pose dis- placement textures; N pose is the number of sample poses. In case of complex joint structureandlargeDOFmodel, weneedtoachievemanysampleposestocalculate accurate joint weights and PSD deformation. PCA (Principal Component Analy- sis) of pose space can yield an orthogonal basis called \Eigendisplacement" [48]. If we reduce the size of pose space from N pose to n pose (principal poses less than N pose ), we can reduce the number of displacement texture to store each sample poses. In our paper, we set n pose as eight in our experiment and save displacements of all poses into a RGB texture T d having size S(N vertex £8(n pose )£3(rgb)). InoriginalPSD,sinceweightr i inequation3.4is¯xedineveryvertices,wedon't need to calculate this value in the GPU. Since the size of this value is just n pose , we can simply pass them to GPU as parameters without generating any texture map. Therefore, we store whole information to calculate original PSD at this point. In order to reduce real-time computation, we pre-calculate T j in equation 3.2 and ¸ in equation 3.5 and store them into another one channel texture T x having size S(n pose £(n pose +n joint £3(x;y;z))). Asaresult,wecanstorevariablesrequiredtocalculateWPSD,PSD,andSSDin sixtexturemaps;T v ;T n ;T w1 ;T w2 ;T d ,andT x . Thevaluesinthetexturemapscanbe stored in the texture memory in pre-setup time, since they are not changed during deformation process. In current graphic card architecture, data transfer speed from CPU to GPU is slow compared with memory accessing speed within GPU. In our method, since the only data changed in each deformation and passed from CPU to GPU is a joint vector p a (size = S(N joint )) representing current arbitrary pose, memory access rate in our method is very e±cient; In case of original PSD, additional n pose number of r i is required. 49 3.5.3 Con¯gurations for fragment program Variables: T out = output texture, T v = vertex texture 1 /* Set orthographic camera with same size of quad */; 2 gluOrtho2D(-1, 1, -1, 1); 3 bind(FBO); 4 /* Bind T out and set to FBO drawbu®er */; 5 bind(T out ), SetFBOdrawbu®er(T out ); 6 bind(T v ); 7 enable(fragment program); 8 /* Set viewport to the resolution of the texture */; 9 glViewport(0, 0, texWidth, texHeight); 10 /* Render a quad into T out using FBO */; 11 glBegin(GL QUADS); 12 f glTexCoord2f(0, 0); glVertex3f(-1, -1, -0.5f); 13 glTexCoord2f(1, 0); glVertex3f( 1, -1, -0.5f); 14 glTexCoord2f(1, 1); glVertex3f( 1, 1, -0.5f); 15 glTexCoord2f(0, 1); glVertex3f(-1, 1, -0.5f); 16 g; 17 disable(fragment program); Algorithm4: Vertexreferingandrenderingintotextureonfragmentprocessor Since vertex information cannot be accessed by the fragment program, vertex deformation on GPU is usually performed by vertex program [8,48]. Although, we cannot access each vertex data in the fragment program, the e±ciency of parallel computation on fragment program is better, since fragment processor has more processing units and each of them has more computation power. In the fragment program, each fragment is assigned to each fragment processor to calculate its ¯nal color value and this is general SIMD architecture using fragment streams as their input data. WedevelopedparallelWPSDalgorithmsbasedonfragmentprogramtoenhance e±ciency of parallel computation. Geometry information like vertex positions and normals are stored in texture maps, T v and T n as described in section 3.5.2 and 50 eachvertexinformationisreferredinthefragmentprocessortocalculate¯nalcolor value. In order to assign each vertex value stored in a texture map to each fragment, we bind the geometry texture T v or T n to a quad and render it using an ortho- graphic camera having same width and height with the quad. Furthermore, since theviewportissettothesamedimensionwiththetextures,eachfragmentisexactly matched with each texel storing vertex information, and we can access each vertex information using texture coordinates of the fragment. A similar idea was devel- oped in [80] to calculate ray tracing on a fragment program and used in GPGPU (General Purpose computation on GPUs) applications [29,59,74]. TheFBO(FrameBu®erObject)extension[31]supportsrenderingintoattached texture and it saves memory and processing time, since there is no copy operation from frame bu®er to texture bu®er. We implemented our WPSD algorithm using fragment program with FBO to store the result directly into texture maps accessed by vertex program in the next pass. Basic con¯gurations of this method are sum- marized in algorithm 4. 3.5.4 GPU implementation We implemented GPU deformation using three rendering passes, and the basic architecture is described in ¯gure 3.3. In the ¯rst pass, we parallelize per-vertex deformation using GPU fragment processors. Required data to calculate deformations are stored in the textures as described in section 3.5.2 and deformation for each vertex is calculated in each fragment processor. In a given arbitrary pose consisting of a joint vector, SSD is computed by equation 3.2 using texture map T v , T w1 , T w2 and T x ; refer texture mapnotationstosection3.5.2. PSDiscomputedbyequation3.4usingT d ,T x ,after calculating r i by equation 3.6. In case of WPSD, weighed distance is computed by equation 3.8 using T w1 , T w2 , and T x . 51 Fig. 3.3: Overview of WPSD on GPU: Each T i is the texture maps storing required data for the calculations (refer value i in section 3.5.2) , T i 0s are the texture maps to storeoutputof1stand2ndpass, P 0 isageometryinrestpose, P a isadeformed geometry in arbitrary pose. 52 The result of the deformation is rendered into a texture map T v 0 using FBO (Frame Bu®er Object) [31] and passed to the third pass. Additional second pass is to calculate and store normal deformation with the similar method as the ¯rst pass and the result is stored in a texture map T n 0. Inthethirdpassbasedonvertexprogram,eachvertexoftherestposeischanged into deformed position stored in the texture map which is generated and passed from the ¯rst pass. In order to access related texture information in each vertex, wecreatedtexturecoordinatesofeachtexelinpre-processingandused theminthe vertex program. Therefore, two texture maps, T v 0 and T n 0 which are output of the ¯rst and second pass to store the deformed vertices and normals are accessed in vertex program using the texture coordinate of the current vertex. 3.6 Experimental results Wetestedourmethodsbyupperarmmodelsconsistingoffourjointssuchascollar, shoulder,elbow,andwrist; eachhasthreeDOFandwrististheendjointhavingno DOF. Three di®erent resolution meshes are used in the test to see the performance of GPU parallel computation; the high-resolution model has 91460 triangles with 46036 vertices, mid-resolution model has 22836 triangles with 11574 vertices, and low-resolution model has 5762 triangles with 2972 vertices [Figure 3.4]. Note that these models are considerably more detailed than those used in current games, so the reported frame rates would be much higher if typical game-resolution models wereused. Ontheotherhand, withtheexpectedgrowthofGPUprocessingpower, models such as these will be in wide use in a few years, and algorithms such as WPSD will be required to produce realistic deformations at this level of resolution. Eight sample poses are created by Poser [18] and the joints weights and dis- placements of each sample are derived from this models [Figure 3.5]. 53 Fig. 3.4: Meshes of test data: the top row is a low-resolution mesh, the second row is a mid-resolution mesh, and the bottom row is a high-resolution mesh. 54 Our parallel algorithm is based on three pass GPU computation. The fragment program for the 1st, and 2nd pass and the vertex program for the 3rd pass are implemented by Cg language [22]. FBO extension is used to render the output into texturewithoutcopyingframebu®erdatatotexturememory. ForaccuracyofGPU computation, the computations are performed by 32bit °oating point operations with 32bit °oating point texture maps. Table 3.2 shows total memory space to store texture maps required to the fragment program. Note that the maximum required memory space for the highest resolution model is just 6.8 Mbyte; the size of the output texture T v 0 and T n 0 is same as T v and T n . The results of GPU based deformation for SSD, PSD, and WPSD are in Fig- ure 3.1 and 3.6, and the experiment are performed in a GeForce 6800 Ultra GPU and a 3.4Ghz Pentium 4 CPU. The timing results of each algorithm in CPU and GPU are summarized in table 3.1. In average, our GPU based deformation shows around 20 to 30 times speed up compared with CPU based deformation. Approximately, GPU based WPSD shows similar speed to CPU based SSD. Therefore, real-time applications based on SSD can be substituted by WPSD on GPU without loosing their real-time performance. Since our algorithm shows speed-up for SSD, PSD as well as WPSD, applications can choose proper skinning method within their boundary. 55 Fig. 3.5: Samples poses and displacements: the ¯rst and second row show each sample pose, the third and fourth row show displacement of each sample pose with the rest pose in the ¯rst column of the third row. 56 Fig. 3.6: Arbitrary poses deformed by WPSD on GPU. 57 Method Mesh CPU(FPS) GPU(FPS) SSD low 150 1425 middle 39 630 high 5 164 PSD low 98 1230 middle 23 530 high 4.5 140 WPSD low 5 85 middle 1.2 25 high 0.29 7 Tab. 3.1: Timing results (in FPS) of each algorithm: low-resolution mesh has 5762 tri- angles with 2972 vertices, mid-resolution mesh has 22836 triangles with 11574 vertices, high-resolution mesh has 91460 triangles with 46036 vertices. Vertices T v &T n T w1 &T w2 T d T x Tot 2972 (low) 35£2 46£2 278 1 441 11574 (mid) 135£2 180£2 1080 1 1711 46036 (high) 539£2 719£2 4315 1 6832 Tab. 3.2: Texture memory to store data required in fragment program (in Kbytes); refer each texture notation to section 3.5.2. 58 Chapter 4 Articulated Human Volume Deformation from 3D Medical Image Scans 4.1 Overview Although the approach presented in the previous chapter can provide proper solu- tions for the real-time example-based skin deformation, the method is based on the manually created surface model samples. A human body model and deformation based on the real subject scans should consider several further issues. This chapter describes a complete system for producing an animatable volu- metric model from in vivo MRI scans of an articulated body region. The ability to examine volume scans of an articulated region such as the hand or knee as it moves under standard joint control is a desired capability for the medical imaging community. Current non-rigid registration of medical practice generally focuses on non-articulated regions such as the brain, and research on volume deformation has not yet produced systems that preserve the rigidity of the internal skeleton, nor that interpolate multiple scans. Presentation of multiple scans from a living subject as a single movable model is attractive in both diagnostic and educational settings. There are many challenges when producing a volumetric model of an articu- lated subject. Manual construction of an arti¯cial volumetric simulation is time 59 consuming and may not exactly mimic the complex interaction and compression of di®erent tissue layers. Further, the use of real data is preferable in many medical applications, especially when the data is of the patient. Working with volumetric data brings scaling issues, since the volume of data is considerably larger than the surface data that is the subject of most computer graphics techniques. Registra- tionmethodsforproducingcorrespondenceacrossmultiplescansmustdealnotonly with the volume of data, but also with the many degrees of freedom (DOF) aris- ing from both non-rigid tissue deformation and rotations of the underlying skeletal joints. Inaddition,theoptimizationmusthandlethestronglocalminimathatarise in matching articulated subjects (e.g., the index ¯nger in one scan aligning to a di®erent ¯nger in a second scan). There are also issues involving the use of in vivo MRIthatdonotarisewithcadaversornon-articulatedsubjects; theseconcernsare described in section 4.3. Inthisstudy, theissuesdescribedabovearesuccessivelyaddressedandresultin sophisticatedsolutionsforquantifyingandvisualizingcomplexvolumedeformation arising from a wide range of skeletal movements of the human body. Di®erently posed volumes of real human body regions (e.g. the hand and knee) are obtained by MRI scans. Skeleton models of each pose sample are created by semi-automatic hierarchical bone volume registration. Then, the kinematic joint structures of each volume samples are estimated. The volume blend deformation (VBD) algorithm is developed to generate any arbitrary pose volume while providing smooth deformation of soft-tissue and main- tainingrigidhardtissuevolumeinthenewpose. Avolumetricweightmapde¯ning joint weights of each grid point of the volume is calculated using computed surface weights and identi¯ed bone voxel grid points. In order to solve the correspondence problem across scans, a template volume is registered to each volume sample. The wide range of pose variations is ¯rst 60 approximatedbyVBDalgorithm, providinggoodinitializationforsubsequentnon- rigidvolumeregistration. Theinitializedvolumeisthenautomaticallyregisteredto thetargetvolumewhileminimizingthemutualinformation(MI)orsumofsquared intensity di®erence (SSID) without relying on ¯ducial markers. Non-rigid volume registration requires high degree of freedom (DOF) warping functions for accurate registration. To address this, an locally adaptive registra- tion algorithm that e±ciently reduces the search domain and DOF of the warping function is developed. The domain is hierarchically and spatially decomposed and dissimilar regions are locally and adaptively registered using Triharmonic Clamped Plate (TCP) spline interpolation. TPC provides smooth, shear free, and local de- formationusingirregularcontrolpoints. Thebonevoxelsalreadyregisteredineach pose provide static markers for global deformation. Accurately registered volume samples are now used to compute volumetric dis- placementsbetweenthesamplesasisrequiredforexamplebaseddeformationalgo- rithms such as pose space deformation (PSD) [49,55] that generate arbitrary poses ofanarticulatedsubject. Theresultisarapidlydeformablevolumetricatlasmodel of an articulated body containing accurate data-driven deformation of all anatomi- callayersthatcanbevisualizedbyanyvolumerenderingalgorithm. Theendresult has potential uses in many applications of medical image analysis, biomechanics, and computer graphics: 4.2 Related work The goal of quantifying the movement in a set of volume scans can be approached with non-rigid volume registration. Registration methods based only on the raw intensity data have obvious bene¯ts, since these methods operate directly on the image intensity values without using any ¯ducial markers or prior data reduction and segmentation by user [17,33,35,61,110]. 61 Non-rigid volume registration requires high degree of freedom (DOF) warping functions for smooth, continuous, and shear-free deformation of the soft-tissue vol- umes [35,64,110]. Rueckert et al. registered 3D breast MRIs while optimizing normalized mutual information (MI) [87]. The initial locations were positioned by rigid global a±ne transformations, and the volumes were smoothly warped to the targets using optimization of B-splines based free-form deformation (FFD). Although the B-splines create locally supported smooth deformations, using a regulargridofcontrolpointsresultsinmanyDOFinoptimization. Recently,adap- tive registration has been studied to reduce DOFs and computation times. Rohde et al. used the gradient of global mutual information (MI) to ¯nd mismatching regions. Wu's radial basis functions (RBF) successfully supported smooth volume deformation using irregularly allocated sparse control points [85]. Locally measured normalized MI was used by Park et al. to identify highly mis- matched regions requiring additional control points [72]. The radial basis function formulation of thin plate splines (TPS) was used for deformation. Previous non-rigid registration studies have been generally focused on non- articulated subjects such as the brain [72,85] and breast [87]. Articulated subjects involve a much wider range of deformation, and are best approached with methods that explicitly model the e®ect of movement of the underlying bones. However, rigid transformation alone is not su±cient to produce the desired smooth defor- mation, nor even to provide an adequate initialization for non-rigid deformation algorithms (Fig. 4.1). Martin-Fernandez et al. presented elastic registration of human hand X-rays using a 2D articulated registration approach. A simple wire model is manually createdbasedontheinnerskeletonstructureandtheimagesaredeformedbasedon theweightscalculatedbyadistancemeasurefromthewire[63]. Papademetrisetal. presented a rigid body registration method for serial lower-limb mouse CT images. 62 Fig. 4.1: From left, (a) schematic source (neutral pose) scan, (b) target pose, (c) rigid transformation does not provide adequate initialization for non-rigid registra- tion, (d) VSD produces an approximate but suitable initialization. They addressed the importance of the initialization step in intensity based non- rigid body registration and present a method to initialize a model to any arbitrary posture [71]. Li et al. presented articulated whole body registration of serial micro CTmouseimages. Poseinitializationisaccomplishedusingpoint-basedregistration algorithms operating on the bone point clouds. Despite the advantages of volume graphics approaches, volume animation re- search has been relatively scarce, perhaps due to the di±culty of adding the time dimension to the already challenging resources needed for static volume graphics, e.g., large memory requirements, long computation, complex scanning processes, etc. Recently, computer hardware has supported large volume data su±ciently, and hence volume animation has received more attention in computer animation and volume graphics. Volumemetamorphosisbringstraditionalmorphinganimationtothevolumetric domain by producing intermediate volumes between two objects. He et al. [34] presents wavelet-based volume morphing methods, and Lerios et al. [53] developed a feature-based volume morphing system. A second area of volume animation involves free-form deformation such as vol- umesculpting,fullbodydeformation,orelasticobjectsimulation. Physicallybased volume animation using a mass-spring model and ¯nite element model (FEM) is shown in [15], and Gibson [27] present 3D chain mail algorithms to propagate vol- ume deformation through the entire volume rapidly. 63 Kurzion and Yagel introduces methods to simulate deformation using ray dis- tortion. In their methods, the sight rays are deformed to produce the illusion of a deformed scene without actual volume object deformation [50], or each slice of mappedtextureisdeformedintheiradvancedsystemacceleratedbygraphicshard- ware [51]. Recently this idea has been enhanced to the real-time volume animation system using GPU computation. Westermann and Salama [104] produce texture space deformation systems using backward distortion of 3D texture coordinates. The 3D texture map contains appearances of volume data and the computation can be accelerated by graphics hardware, resulting in real-time volume deforma- tion. Thepreviousworkshavebeenmainlydevotedtofree-formvolumedeformation. Although these methods can generate shape deformations of non-articulated sub- jects, they are not designed for articulated body animation involving high range of motion such as human body animation using skeleton control. Furthermore, physically-based methods require speci¯c parameters such as elasticity or viscos- ity and these values are di±cult to select and intuitively control. Also, volume metamorphosis requires a speci¯c target object. Gavani et al. [25] developed a volume animation system for articulated objects. Theskeletalstructureofthearticulatedobjectisapproximatedbyvolumethinning methods called a \skeleton tree". An arbitrary pose is produced by transformed skeletalpointsandthewholevolumeisregrownbythesphericalvolumereconstruc- tion. Themethodissuccessfullyintegratedintoatraditionalanimationsystemsuch asMaya,andthevolumeismanipulatedbyabstractskeletalcontrol[24]. Although themethodproducesvolumeanimationofarticulatedbodyusingstandardskeletal control, theyhavenotconsideredseveralimportantissuesneededtocreaterealistic volume animation. Themethodisbasedonvolumereconstructionsfromtransformedimplicitskele- tons(lines,curves)withoutconsideringrealisticsoft-tissuedeformationarisingfrom 64 motion. However, skinning is a notably important issue in articulated body ani- mation; e.g. muscle bulging, and wrinkle folding. Their method does not consider soft and hard tissue properties separately, and therefore produces erroneous arti- facts such as bone bending in some postures. The skeleton is produced by volume thinning, and the joints are simply approximated by basic primitives. Therefore, the constructed joints cannot guarantee proper movements of bones. Inordertogeneraterealisticvolumeanimation,thejointstructureofarticulated object should be estimated by a bio-mechanical approach. Previous bio-mechanics and robotics research [47,49,67,88] has generated feasible joint skeleton estimation approaches, and our work bene¯ts from this earlier research. Although the surveyed methods in volume animation and articulated volume registration highlight the importance of addressing the articulated body, the meth- ods have not addressed accurate and detailed human tissue deformations such as muscle bulges and contact creases. However, skinning and accurate articulated body deformation has been a major topic in surface-based computer graphics. Al- gorithmic methods such as those inspired by the skeletal subspace deformation (SSD) or linear blend skinning algorithm [60,66,69] are widely used and suitable forreal-timeapplications, butproducesmoothdeformationsthatcannotreproduce thecreases,musclesbulges,andotherdetailsofrealskindeformation. Inthisstudy, we extend the SSD algorithm to articulated volume deformation. Volumetric joint weights are computed to generate smooth soft-tissue deformation while preserving the the hard tissue volume. However the result is constrained by the limitations of the SSD algorithm and the accuracy of deformation within the soft-tissue volume is not guaranteed. Example-based methods that interpolate between samples of the desired shape invariousposes[55,95]canapproximatecomplexdeformationoftherealskinusing 3D scans of the subject [3,5,49]. 65 These methods can be considered as non-parametric approaches to shape de- formation, and permit complex e®ects such as muscle bulges and major wrinkle deformations while addressing the artifacts of simple algorithmic methods. In this study, thegeneralexamplebasedapproachisextendedtothevolumedomainwhile addressing practical issues that arise in using medical image volume samples. 4.3 Obtaining3Dmedicalimagescansofthehuman body Visiblelightscanningofarticulated3Dobjectssuchasthehumanhandcanbechal- lenging due to the self-occlusion of surfaces in certain poses (e.g. the palm surface in the ¯st pose). Computerized tomography (CT) scanning provides the highest resolution 3D imaging with ¯ne spatial resolution and straightforward segmenta- tion [49,56], but involves exposure to harmful ionizing radiation, and its associated risks. Wechosetousemagneticresonanceimaging(MRI),becauseitisnon-invasive and involves no ionizing radiation and little risk. Compared to CT however, the image quality obtained by MRI has lower SNR, non-uniform bone signal, and over- lapping intensity ranges between di®erent tissues. As a result thresholding is not su±cient for segmentation of MRI. In addition, human MRI scans requires care- ful consideration of subject comfort, scan-time, signal-to-noise ratio (SNR), and potential for motion artifacts. Our pipeline is designed to handle image quality lower than that of previous approaches involving CT and/or cadaver scans. The number of obtainable volume samples is sometimes quite limited in the case of human subjects. However, as was demonstrated by [49] in the non-volumetric case, useful models can be built with surprisingly few scans. Our pipeline produces a fully articulated hand model using relatively few scans at critical poses. 66 Parameters Hand Knee Pulse Sequence 3D Fast SPGR 3D Fast SPGR Repetition Time (TR) 4.9 s 4.7 s Echo Time (TE) 1.4 s 2.1 s Flip Angle 15 ± 20 ± Volume Size(mm 3 ) 200£200£100 320£320£163.2 Spatial Resolution(mm 3 ) 0.78£0.78£1.0 1.25£1.25£1.2 Scan Time per Pose 5 min. 6 min. Tab. 4.1: MRI scan parameters 3D MRI volumes of the human hand were captured in ten di®erent poses and 3D volumes of the human knee were captured in four di®erent poses [Fig. 4.2] . Data were obtained from one healthy male volunteer with no known anatomic or functional abnormalities in the captured regions. A GE Signa Excite 3.0 T scanner with gradients capable of 40 mT/m amplitude and 150 T/m/s slew rate (GEHealthcare,Milwaukee,WI,USA)wasused,withrelevantimagingparameters summarized in Table 4.1. The volume data set for each pose was approximately 256£ 256£ 100 for the hand, and 256£ 256£ 136 for the knee. Inordertoimprovepatientcomfortandminimizemotionartifacts, castsforthe handweremadeusingMR-compatibleair-dryclay[Fig.4.2]. Thesecastssupported the dorsal skin surface, and soft ¯ber cotton was used to support softer skin which maybeeasilydeformed. Thesematerialshavevirtuallynomoisture,andproducea MR signal that is below the noise level. Background elimination was not required. 4.4 Joint skeleton modeling Since MRI volumes contain the actual skeletal structure, the kinematic joint loca- tions should be extracted from the raw medical images for realistic and intuitive joint control. The discovered joint structures for each volume provide a compact descriptionoftheposeofeachvolumesamplethatisusedinsubsequentarticulated 67 (a) (b) (c) Fig. 4.2: Human In vivo MRI scans : (a) Human hand with a supporting cast, (b) MRI scans of the hand, (f) MRI scans of a human knee. volume initialization and for example based deformation. The joint skeleton of an invivosubjectvolumecanbegeneratedbyanatomicalanalysisoftheboneskeleton or bio-mechanical joint estimation from multiple samples. Our method is inspired by previous joint estimation approaches based on the work by [47,49,67,88]. 4.4.1 Hard tissue (bone) extraction The ¯rst step in this process is to extract and identify each bone from the volume. In case of in vivo MRI, the image quality is quite limited. Furthermore, bone segmentation from a region such as the human hand using MRI is very di±cult due to the following issues: multple similarly shaped bones are tightly packed in a small region, non-uniform tissue properties of bones, overlapping density range between adjacent tissues, and poor signal to noise ratio. Manual segmentation, which is a time consuming task, is unavoidable, because automatic segmentation using simple method such as thresholding cannot be used. Therefore, the manual bone segmentation has been recommended in previous research by [47,64,67,88]. Joint estimation methods using pose samples require bone volumes in di®erent poses in the work of [47,49,67]. Extracting all the bones from every sample would 68 requirealargeamountoftimeande®ort. Inordertominimizetherequiredmanual work, only the bones of a single base volume are extracted using 2D graph cut algorithms by [86]. Then, the bones are registered to the other poses' samples using methods to be described next. More sophisticated segmentation algorithms such as methods by [30,109] are possible, but are not our focus. In a pre-processing step, the iso-surfaces of the volume are extracted using the marching cubes algorithm by [58]. Voxels identi¯ed to be inside the volume are storedinaseparateone-dimensionallisttermedthe representative volume tospeed subsequent processing. 4.4.2 Hierarchical bone registration The oriented bounding box (OBB) of each bone is created from three eigenvectors (principalcomponents)ofthecovariancematrixofthevolume. Duetotheanatomy of the phalanges, the eigenvalues are usually distinct and su±cient to uniquely identify the orientation of the ¯nger bones as noted by [67]. The centers of the OBBs'base(proximalend)areassignedastheoriginoftheOBBcoordinatesofthe bone. The OBBs of the bone and their representative volumes are transformed to the origin of the world coordinate frame for the registration process. The extracted bones and OBBs of the base pose (pose 0) are shown in Fig. 4.5. Wewishto¯ndT k j ,the4£4matrixthattransformsb 0 j (theOBBofbonejofpose 0intheworldcenter)tob k j (theOBBofbonej inposek). Sincetheintensityvalue ofeachboneisverysimilar,boneregistrationof3DMRIischallengingwhetherdone manually or automatically. Semi-automatic bone registration of MRI combining manually approximated initialization and automatic optimization is developed by [47] and their e®orts are developed further in our paper into an intensity based rigid body registration using hierarchical bone structure at multiple resolutions. First, hierarchical bone structures are constructed for the articulated subject. For the human hand, the carpal bones and metacarpals of the index, middle, and 69 (a) (b) (c) (d) Fig. 4.3: Hierarchical Bone Registration: (a) Hand bone hierarchy, (b) Knee bone hier- archy, (c) Hierarchical bone registration of proximal phalanx, (d) Hierarchical bone control of middle phalanx. 70 ring¯ngersaregroupedinavolumetorepresentarootbone,sincetheirmovements are slight compared with other bones in the ¯ngers. The metacarpal of the little ¯nger and its subtle movement is skipped to simplify the skeletal structure. Then, the metacarpals of thumb, and proximal phalanx of four ¯ngers are attached to the root bone as it's child bones. The middle phalanx and the distal phalanx of each ¯nger is then successively attached to their parent bones while maintaining hierarchical structure of the hand bones in Fig. 4.3 (a). In the case of the knee, the femur is assigned to root bone, and the tibia and ¯bula are attached to the femur as its child bones [Fig. 4.3 (b)]. In order to control the bones, forward kinematics are applied to the hierarchical model. When the parent bone is transformed, the successive child bones are transformed along with the movement of the parents bone. The scheme provides easy user control for manual bone initialization and registration in a highly bent posture such as a hand in the ¯st pose [Fig. 4.3 (c) and (d)]. Intensitybasedrigidbodyregistrationmethods[33,61]areusedtoregisterbones without any extrinsic ¯ducial markers. Since intensity based registration methods require good initialization to avoid local minima during optimization, manual bone initialization is generally required for complex articulated subjects [47]. In order to make the manual initialization easy, convenient user interfaces are created. The OBB coordinate frame provides rotation and translation axes to control the hi- erarchical bone structure. Real-time 3D volume visualization methods described by [21,45] are used to display the current transformation of b 0 j in the volume space of target pose k. This manually initialized transformation is denoted by ~ T k j . Aftertheapproximatemanualinitialization,thebonevolumeisregisteredtothe target pose by minimizing sum of squared intensity di®erences (SSID) in Eq. (4.1). The SSID based similarity measure showed faster and better results in our experi- ment than a similarity based on Mutual Information. This is because the volumes arescannedinthesamesettingusingthesamegreyscalemodality(MRI), whereas 71 Mutual Information would possibly outperform SSID if the scan sources were more varied. In order to reduce computation and help avoid local minima, a multi-resolution approache is used [Fig. 4.4]. At each resolution level l the following cost is mini- mized: min T n X i=1 jI l (v i )¡J l (Tv i )j 2 ; for all levels l (4.1) where T is T k j initialized by the ~ T k j , n is the number of voxels in the represen- tative volume, v i is the location of the representative voxel, Tv i is the transformed voxel location, I l () is the intensityvalue of the volume in the base pose 0 at volume level l, and J l () is the intensity value of the volume in the target pose k at the volume level l. The resolution of volume at level l is the original resolution divided by 2 l . After Eq. (4.1) convergesto a minimum, the resulting T is used as the initial position in the next resolution level. A quasi-Newton method (the BFGS algorithm by [79]) is used as the optimiza- tion solver. In order to achieve more robust results, several random restarts are used when a minimum value found. We represent matrix T as fR;tg to reduce the dimension from 16 (4x4) to 7 where R is the 4 dimensional quaternion vector representing rotation and t is the 3 dimensional translation vector. In quasi-Newton methods, the gradient of the error function is required (the Hessian is approximated within the algorithm). In Eq. (4.1), the transformation matrix is de¯ned in Euclidian space but I() and J() are in the volume intensity space (l is skipped to simplify notation), so the gradient cannot be calculated directly. Instead, the gradient can be obtained by approximating J() through its Taylor series, J((R+°)v+t+±) ¼ J(Rv+t)+rJ¢(°v+±) (4.2) ¼ J(Rv+t)+rJ R ¢°+rJ t ¢± (4.3) 72 where ° (4x1) and ± (3x1) are small changes in rotation and translation respec- tively, the gradientrJ is separated intorJ R for rotation andrJ t for translation, andRv denotesv rotated by quaternion R. In the volume space,rJ R andrJ t can be approximated by numerical di®erencing in the volume under a small change of translationandrotation. Then,theerrorfunctioninEq.(4.1)canbeapproximated as Eq. (4.4), after substituting r i ´I(v i )¡J(Rv i +t) to simplify notation: E(R;t)= n X i=1 (r i ¡rJ R ¢°¡rJ t ¢±) 2 (4.4) Thisobjectivefunctioncanberepresentedinamatrixformwherethedimension of r is (n£1),rJ R is (n£4), ° is (4£1),rJ t is (n£3), and ± is (3£1): E =(r¡rJ R °¡rJ t ±) T (r¡rJ R °¡rJ t ±) (4.5) Then, the gradient of E can be obtained as dE d± ¼ ¡2(rJ t ) T r (4.6) dE d° ¼ ¡2(rJ R ) T r (4.7) wherethesquareinEq.(4.5)isexpandedandsecond-ordertermsO(± 2 );O(° 2 );O(±¢ °) have been dropped before di®erentiation. As the result, the transformation matrix T k j that transfers the world coordi- nates origin to bone j of pose k is obtained from the optimization process. The concatenated set of all T j registered to target pose k is de¯ned as the pose vector P k that de¯nes the pose of the object during later processing. The registered bones for each pose of the hand are shown in Fig. 4.5. 73 Fig. 4.4: Multiple resolutions of the representative volumes 4.4.3 Kinematic joint estimation Although a particular bone's location in each pose can be represented by the world coordinate transformations T k j s, estimating kinematic joint locations is required for the accurate and intuitive joint control. From the sample poses, the center of rotation (COR) c j of joint j can be estimated by minimizing the error de¯ned in Eq. (4.8), because the COR does not change under rotation when viewed in the coordinates of the parent joint [49], min c j m¡1 X k=1 jT 0!k j c j ¡c j j 2 (4.8) where c j is the COR of joint j in the parent joint coordinates, m is the number of sample poses (pose 0 will be excluded), and T 0!k j is the relative (homogeneous coordinates)transformationofthejointj fromthepose0toposek afterregistering the parent bone of the pose k to the parent bone of the pose 0. The T 0!k j can be calculated by Eq. (4.9) (see Fig. 4.6), T 0!k j =(T k j¡1 ) ¡1 T k j (T 0 j ) ¡1 T 0 j¡1 (4.9) 74 Fig. 4.5: Bone registration across poses: The top left most image is the base volume including marching cube meshes of extracted bones and their OBBs (yellow boxes); the other images shows base bones (red points) registered to the other sample poses. 75 where T k j is the transformation matrix of joint j of the pose k calculated by the bone registration in section 4.4.2, and the T k j¡1 is the transformation matrix of the parent joint of j. Since Eq. (4.8) is assuming homogeneous coordinates with c = (x;y;z;1) and 4£4 matrix transformation, some care is required to keep c[4] = 1. Instead, split the matrix T 0!k j into a 3£3 rotation matrix R and 3£1 translation vector t. A single term of Eq. (4.8) is then Rc+t=c and the matrix form is: 2 6 6 6 4 R 1 ¡I R 2 ¡I . . . 3 7 7 7 5 [c]= 2 6 6 6 4 ¡t 1 ¡t 2 . . . 3 7 7 7 5 (4.10) Calling this Mc = t, the the solution is t multiplied by the pseudo-inverse of M. After ¯nding c j of each joint, the COR of pose k in world coordinates (~ c j ) can be calculated by T k j¡1 c j , and the result is shown in Fig. 4.6. 4.5 Volume blend deformation The popular Skeletal Subspace Deformation (SSD) algorithm in computer graphics and animation is based on the weighted blending of a±ne transforms determined by each joint as described in [69], v a =( n joint X j=1 w j T j )v 0 (4.11) wheren joint isthenumberofjoints,v 0 isavertexinthebasepose0,v a isadeformed vertexinanarbitraryposea,T j isahomogeneous4£4transformationmatrixthat transfers joint j from base pose 0 to the arbitrary pose a in world coordinates, and w j is a joint weight that de¯nes the contribution of joint j's transformations to the deformation. 76 (a) (b) (c) Fig. 4.6: Joint estimation: (a) The center of rotation (COR) estimation, where B j¡1 is the parent bone and B j is transformed to pose k by matrix T 0!k j , (b) The bone skeleton, (c) The estimated CORs (cayan dots) of each joint. 77 Eq. (4.11) was developed for deformation of boundary representation (surface) models. Although it can be extended to volume deformation by interpreting v as a voxel grid point rather than vertex and calculating weights w j for each voxel grid point, there are two di±culties that must be addressed. First, the hard tissue (e.g. bone) volume must not be non-rigidly deformed. Second, the weights w j have until recently been generated using manual input in the case of polygon surface models. Creatingaweightmapforvolumetricdataisfarmoredi±cult, simplydue to the increased dimensionality and amount of data. The following subsections address these issues and describe steps to automat- ically compute joint weights that support smooth soft tissue deformation while preserving the hard tissue volume. 4.5.1 Volume weights for in vivo subject Recentskindeformation(skinning)researchincomputergraphics[41,69]hasshown how to produce joint weights given a su±cient number of posed skin samples to create better skinning and reduce the elaborate manual work required to create weight maps. We discussed this method in section 3.4; in this chapter, pose p i can be substituted with sample pose k. However, the automatic determination of skinning weights in section 3.4 is im- practical if directly applied to volume data. In the case of a complex body having large DOF (e.g. the human hand), direct capture of a su±cient number of volume sampleswithadequateposevariabilityusinginvivoMRIscanisdi±cultduetothe technicalissuesdiscussedinsection4.3,aswellasthedi±cultyinproducingsteady poses (for example, independently controlling the ring ¯nger is di±cult for some people). We solve the problem of obtaining volume SSD weights while considering these practical issues. First, an atlas model consisting of a polygonal surface mesh for the subject (e.g. the human hand) is created. A su±cient number of di®erent poses of the 78 (a) (b) (c) (d) Fig. 4.7: Aligningagenericatlaspolygonsurfacemeshtothevolume: (a)Sourcepolygon mesh and features, (b) Target volume and corresponding features, (c) Surface mesh warped to align to the volume, (d) Yellow dots (Fp) are vertices of the deformed surface, Fb is each bone surface, and weights of the blue °esh area is interpolated by Fp and Fb. 79 model are generated by general computer graphics software such as Maya or Poser. Given su±cient samples, a conventional surface joint weight map is calculated by the automatic weight computation method in section 3.4. The computed surface weights encode the surface deformation properties of the subject. With the set of feature locations x i and their values f i ; x i =fx 1 ;:::;x n g;f i =ff 1 (x 1 );:::;f n (x n )g; wecan¯ndafunctionR(x)whichgivessmoothinterpolationofthesefeaturesusing radial basis functions (RBFs) [14] as equation 2.4. The vertices of the generic atlas polygon model are aligned to the iso-surface of the volume by RBF interpolation. Our RBFs are de¯ned by manually created sparse corresponding feature locations, and they interpolate the o®set between the vertices of the surface model and corresponding voxels in the iso-surface of the volume. The thin plate spline (TPS), Á(r) = jrj is used as the basis function for smooth deformation of the polygon mesh, since the TPS interpolates speci¯ed points while minimizing an approximate curvature [7]. Fig. 4.7 shows the polygon model in the base pose, the corresponding features, and the deformed polygon model ¯t to the iso-surface of the volume. The volume of the skeletal subject can be divided into hard tissue (e.g. bones) and soft tissue, and the hard tissue volume should not be deformed by movements. The hard tissue bones are extracted and identi¯ed in section 4.4.1, and w j (v) the weight of joint j for a voxel grid point v is assigned as: w j (v)= 8 < : 1 i® v2 bone j 0 otherwise (4.12) The last step is to compute weights of the soft tissue volume which is located betweentheskinsurfaceandhardtissue. Sincetheweightsfortheskinsurfaceand bone voxels have already been obtained in the previous step, the weights for the 80 soft tissue are now interpolated using scattered data interpolation methods. The previously aligned mesh vertices as well as the voxel grid points of the bone volume now become the feature locations for RBF interpolation, and their joint weights are the feature values to be interpolated. The total number of features n f is n p + n b , where n p is the number of the vertices in the generic polygon model (2 134 for hand and 473 for knee) and n b is the number of voxel grid points in the iso-surface of the bone volume (2 805 for hand and 2 872 for knee). 4.5.2 Articulated volume initialization We apply VBD algorithm (section 4.5) for articulated volume initialization to ap- proximately register the template volume to all volume samples k. The pose vector P k computed in section 4.4.2 contains the necessary transformation matrix T k j for use in equation (4.11). Since the VBD algorithm smoothly deforms the soft-tissue volume while preserving rigid bone regions, a template volume is smoothly de- formed to target sample volume k while maintaining the already registered voxels in the rigid bone area. The results of this step are shown in ¯gure 4.8. 4.6 Locally adaptive non-rigid registration Followingarticulatedvolumeinitializationweneedtode¯neanon-rigidregistration approach capable of accurate and detailed matching while operating on extensive volume data. General non-rigid volume registration methods can be de¯ned by a cost function minimizing the mismatch between the source and target volumes, max f SM(V s +D(V s ;c;f); V t ) (4.13) whereSM isasimilaritymeasuresuchasSSDorMI(incaseofSSDthevalueshould beminimized),V s isasourcevolume,V t isatargetvolume,Disawarpingfunction 81 Fig. 4.8: Articulated volume initialization: a template volume (cyan) is initialized to a target volume (purple) using VBD. to determine the needed additional deformation of volume V s , c is a control point vector assigned at the target volume frame, f is the displacement value (warping energy)oftherelatedcontrolpoint(c;farediscussedfurtherbelow). Typically,the costfunctionisoptimizedbyiterativenon-linearoptimizationandthecomputation time is determined by both the DOF of deformation function D and the volume size involved in the ¯t. We present a method to e±ciently reduce the DOF and volume size involved in the optimization while avoiding strong local minima arising from the articulated subject registration. Multi-resolution source and target volumes are created, and the volumes are hierarchically divided into multiple source and target blocks B s x;y;z and B t x;y;z where x;y;z 2 f1::n x ;1::n y ;1::n z g and the block resolution r x;y;z = volume resolution x;y;z =n x;y;z . Thenumberofblocksn x;y;z isdeterminedbytheblock hierarchy level b; the coarse level b min has n x;y;z =8 blocks. Locally su±cient (but globally sparse) control pointsc are adaptively allocated at the block B s x;y;z that has minimum local similarity (maximum mismatch). Then 82 Fig. 4.9: Global vs local deformation function: from the left, allocated control points (blue dots) and the displacements (white bars), the global deformation using RBFs based on TPS, the local deformation using TCP constrained by a unit sphere. corresponding local block B s x;y;z and B t x;y;z are registered, optimizing the cost func- tion(equation(4.13))usingthesmallnumberofcontrolpoints. Globaldeformation functions such as RBFs based on TPS [72,81] cannot provide locally constrained deformation (see ¯gure 4.9) and therefore may require an excessive number of con- trol points and DOFs to register articulated subject such as the human hand. In order to produce a smooth but locally constrained deformation over these irregular control points, the triharmonic clamped plate spline (described in section 4.6.2) is used as D. Whereas many image registration approaches minimize a functional that includes a data ¯delity term and an expensive global smoothness term, in our approach globally smooth deformation is obtained by applying a locally smooth deformation adaptively over a hierarchy of scales. 4.6.1 Volume similarity measure Similarity measures based on the raw intensity values are attractive because they make use of all the available data and do not require ¯ducial markers (which are 83 di±cult to place inside the body) or prior data reduction and segmentation by user [33,61]. For our similarity measure function SM, an adaptive combination of sum-squared intensity di®erence (SSID) and mutual information (MI) is used to measure similarity. In this context SSID is de¯ned as SSID = n X i=1 jI s (v i )¡I t (v i )j 2 (4.14) where n is number of voxel grid points, I t is the intensity of voxel grid v i in the target volume, and I s is the intensity of v i in the source volume. Since the SSID method is fast and easy to implement, it is widely used for unimodal medical image registration [33]. Also, similarity thresholds de¯ned in terms of intensity are relatively intuitive. In recent research, MI based similarity measures are widely used for multi- modal image registration. Since these methods analyze images's joint entropy, any functional relationship between the images relationship can be captured. The Shannon entropy for joint distribution is de¯ned as H(V s ;V t )=¡ X s;t P(s;t)logP(s;t) (4.15) where P(s;t) is the joint probability distribution of voxel grid points associated with image volume V s and V t . When the corresponding distributions are aligned, thejointentropyisreducedandminimizingH(V s ;V t )maximizesthesimilaritybe- tween the volumes. Mutual information considers both joint entropy and marginal entropy as; MI(V s ;V t )=H(V s )+H(V t )¡H(V s ;V t ) (4.16) 84 The idea can be understood intuitively by considering that when two images areinperfectalignment, anintensityvalueinoneimagemapstoauniqueintensity valueintheotherimage, sothejointhistogramisine®ectavalueremappingtable. As the images fall out of alignment, a value in one image overlaps increasingly arbitrary values in the second image, resulting in a histogram that is not crisp and thus has higher entropy. The marginal entropies can usually be considered constant, so maximizing MI entails ¯nding the transformation that minimizes the joint entropy H(x;y). How- ever, entropy only measurement can produce low value in case of complete mis- match. [76,100]. Pluimetal.[76]surveysanumberofvariationsofthisidea. Weuseanormalized variant of MI by [72], who observed that areas that are relatively featureless are poorcandidatesformatching. TheratioMI=min(H(V s );H(V t ))islowestinareas that have low MI but high detail; these are then the areas that registration should focus on. For our overall similarity measure function SM, MI and SSID are adaptively selected. SincetheprobabilitydensityfunctionsneededbyMIcanbeapproximated by histograms only when the histogram contains a su±cient number of values, the MI measure may not be accurate within small regions. We switch from MI at the ¼32 3 resolution level to SSID at the¼16 3 level. 4.6.2 Triharmonic clamped plate spline Because the overall registration is being adaptively split into smaller sub-problems to speed the optimization, we require a deformation that is smooth but localized; see¯gure4.9. GaussianRBFswouldbeadefaultchoiceforthispurpose,butinfact they are not ideal. The Gaussian function is not strictly localized, but this is easily remedied by using a Wendland or Wu basis [103,106], or by simple thresholding. Another issue is that while Gaussians are smooth in the sense of being in¯nitely 85 −1 −0.5 0 0.5 1 −1 0 1 0 0.5 1 −1 −0.5 0 0.5 1 −1 0 1 0 0.5 1 −1 −0.5 0 0.5 1 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5 Fig. 4.10: Top: comparison of the 2D clamped plate spline (left) with a thin plate spline (right)interpolatingthesamepoints. Theclampedplatesplineisbothsmooth and localized, decaying to zero at the boundary of the unit disc. Bottom: comparison of the triharmonic clamped plate spline (black line) with Gaussian RBF interpolation (red). The plot is the density of a linear section through an interpolated volume. Note that the control points are distributed through the volume and cannot meaningfully be visualized on this plot. The Gaussian RBF interpolation has unnecessary ripples. Increasing ¾ reduces these, but at the cost of increasing the overshoot. 86 di®erentiable, they are not smooth in the alternate sense of minimizing a curvature functional R L(f(x)) 2 dx of a di®erential operator L. The clamped plate spline introduced in [62] is a better solution to the opposing goalsofsmoothnessandlocality,asisseenin¯gure4.10. Itminimizesthestandard spline curvature functional Z < n X a 1 +a 2 +¢¢¢=m ° ° ° ° @ m f @x a 1 1 @x a 2 2 ¢¢¢ ° ° ° ° 2 dx; which would ordinarily result in a bi- or tri-harmonic solution ¢ 2m f = 0, but subject to having zero value and derivative on the boundary of the unit disc in < n . The derivation of the clamped plate spline resembles that of RBFs, with the solution being a weighted sum of the Green's function of the solution's di®erential operator, but the Green's function in this case is not a radially symmetric function but instead depends implicitly on the location relative to the origin. The function for the tri-harmonic case in three dimensions has been derived [62], 8 > < > : G(x;y) =kx¡yk(A+1=A¡2) A(x;y) = p kxk 2 kyk 2 ¡2x T y+1 kx¡yk Thexcomponentoftheresultinginterpolateddeformationatapointpiscalculated as d x = X w k G(p;c k ) (and similarly for the y;z components) where c k are the locations of the feature points to be interpolated. The weights w k are solved as a linear system 2 6 6 6 6 6 6 4 G 11 G 12 G 13 ¢¢¢ G 21 G 22 ¢¢¢ ¢¢¢ 3 7 7 7 7 7 7 5 2 6 6 6 4 w 1 w 2 . . . 3 7 7 7 5 = 2 6 6 6 4 d 1 d 2 . . . 3 7 7 7 5 87 whereG jk : =G(p j ;p k )andd k arethedesireddisplacementsatthefeaturelocations. Note that the G jk matrix is common to the x;y;z dimensions. 4.6.3 Algorithm and implementation 1 for (b min <b<b max ) do 2 compute number of block n and resolution r; 3 construct block B s xyz ;B t xyz ; 4 while (min(SM(B s xyz ;B t xyz ))< tolerance) do 5 B s;t min is assigned having min(SM(B s xyz ;B t xyz )); 6 allocate c and f to B s;t min ; 7 construct B s ¤ ;t ¤ min using surrounding blocks; 8 normalize c, f, and B s ¤ ;t ¤ min for TCP; 9 optimize cost function using c, f, and B s ¤ ;t ¤ min ; 10 deform V s using D(c;f ¤ ); Algorithm 5: Registration algorithm; b min and b max are minimum and maxi- mum block level; the other variables are same as de¯ned in the paper; the cost function in line 8 is equation (4.13). The volumes are hierarchically divided into small local blocks B s x;y;z and B t x;y;z . ThepairofcorrespondingblockswhichhasminimumsimilarityvalueSM(B s x;y;z ,B t x;y;z ) can be considered as the most mismatched local region. When such a block B s;t min is found, eight control points c are assigned at a small ¯xed distance inside the corners of the block as in ¯gure 4.11. In addition the bone voxels registered in section 4.4.2 can be used as ¯xed control points to guide registration and maintain rigidity in the bone region. To provide better support for the local registration, the block is enlarged to B s ¤ ;t ¤ min using the surrounding 3x3x3 blocks. This also provides a boundary region for the TCP to smoothly fade to zero without counteracting the desired e®ect in the center block. For this purpose, B s ¤ ;t ¤ min 's coordinates are scaled and centered at (0,0,0), thereby locating all voxel grid points in (-1..1), since TCP requires coordi- nates within the unit sphere as in ¯gure 4.11. 88 Now the minimum similarity source block is registered to the target block while optimizing the cost (equation (4.13)) for the local block, where V s = B s ¤ min and V t = B t ¤ min . The result is a set of x;y;z vectors f ¤ that give the optimal displacement values at the given control points c for warping the source block to target block. If the minimum similarity measure of all blocks at level b is less than a given tolerance, the block level is increased to create additional small blocks to register smaller details. The end result is a progressive registration of the source volume to the target volume while reducing the maximum mismatch across a succession of scales. The progressive registration result is shown in ¯gure 4.12 and the method is summarized in algorithm 5 and ¯gure 4.11; one additional loop over the multi- resolution volume is not mentioned in the algorithm summary. The multi-resolution volume, random restarts of the simplex optimizer, and the locally constrained registration algorithm together e±ciently handle the local minima that arise in matching complex articulated subjects. 4.7 Example based volume deformation The registered volumes now provide correspondence across scans, and the displace- ments of each voxel grid point between the di®erent poses can be computed. This provides the volume displacement maps required for example-based volume defor- mation, allowing us to manipulate the articulated volume to any arbitrary pose. The example-based volume deformation (EVD) has the general form v a =S(v 0 +d a ) (4.17) where v 0 is an undeformed voxel grid point in the neutral-pose template volume P 0 , d a is a displacement of the voxel grid point in an arbitrary pose a, and S is the VBD function in equation 4.11. The displacement at the arbitrary pose d a is 89 (a) (b) (c) Fig. 4.11: Locally adaptive non-rigid volume registration: from the top, ¯nd the max- imum mismatching block and allocate control points; construct B s ¤ ;t ¤ (front and back blocks are omitted to simplify the ¯gure); register and deform the local block. 90 Fig. 4.12: Progressivenon-rigidvolumeregistrationresults: theinitializedvolume(cyan) is progressively registered to the target volume (magenta) using adaptively allocated control points (blue dots) at block level 3; small lines show the dis- placement value for the TCP function. 91 obtained by interpolating the displacements at the training poses P k in a \pose space" [55]. TheposevectorP k ofvolumesampleV k containingthesetofbonejointtransfor- mationshasbeenobtainedfromboneregistrationinsection4.5.2,andthetemplate volume V 0 registered to each sample V k (section 4.6) provides corresponding voxel grid geometry. The volumetric displacements between V k and V 0 are calculated by d k =( n joint X j=1 w j T j ) ¡1 v k ¡v 0 where v k is a voxel grid point in sample pose P k and d k is the displacement of this voxel grid relative to v 0 in the sample pose P k . The inverse VBD transfers v k to its template pose P 0 and the displacement is calculated in the template pose coordinate frame. After de¯ning the displacement of each sample pose, d a , the displacement at an arbitrary pose can be smoothly interpolated using RBFs or normalized RBFs [49]. d a = n sample X k=1 r k (a)d k The weight r k (a) de¯ning the contribution of each sample pose is calculated by the normalized RBFs approach r k (a)= f k (a) P n sample t=1 f t (a) wheref t (a)istheradialbasisweightfunctionforsampletevaluatedatanarbitrary pose a. f t (a) is calculated by: f t (a)= n sample X k=1 ¸ t;k Á k (° k ) where ¸ t;k are real valued weights between pose t and k, Á k are the radial basis functions, and ° k is the distance between the pose vector P k and P a . The clamped 92 plate spline function (section 4.6.2) for high-dimensional spaces such as the pose space of a hand is not known, so the Gaussian function Á k (° k ) = exp(¡° 2 k =2¾ 2 ) is our default choice of basis [55,82]. Theendresultprovidestheabilitytorapidlymanipulatethearticulatedvolume to any arbitrary pose while including detailed deformation that interpolates actual data. A posed volume produced by example-based volume deformation (EVD) is shown in ¯gure 4.15. Both the surface and inside anatomical volume layers of the arbitrary pose are adequately reproduced by the EVD algorithm. As in [49], we ¯nd that even relatively few training samples greatly enhances the detail in the deformation. 4.8 Result The robustness of our methods is tested by their application on human knee and handMRIvolumes. Fourdi®erentkneevolumeposesandtendi®erenthandvolume poses were captured while considering practical issues as discussed in section 4.3. We approximated the knee joint as one DOF and the hand joint as 23 DOF which produce the largest articulated motion. The articulated volume initialization in section 4.5.2 smoothly manipulates the template volume to di®erent poses. Then, the initialized template volumes are progressively registered to the scan samples using our locally adaptive volume registration algorithm to obtain correspondences across scans. 4.8.1 Articulated volume registration Sincenogeometricmarkersareusedintheintensity-basedregistration,validationis usually performed by additional measurements or showing that a result performed by pairs of synthetic data for which the geometric correspondences are known [17, 35]. In this paper, three stages of validations show the robustness of our algorithm. 93 First, weregisteredsyntheticallyposedkneevolumeswithsmallposevariations without articulated initialization. The registration only relies on the locally adap- tive registration algorithm in a setting where the ground truth correspondences are known. Voxel grid points identi¯ed as inside of the base source volume V 0 are bent 10 degrees using the VBD algorithm and produce a target volume V t ; 10 de- grees bending generates volumedi®erences that are relatively larger than in typical non-articulated problems. Using the corresponded geometry between source and target volumes, the mean Euclidean distance between corresponding points can be computed [81]. The tolerance (algorithm 5) for ¯nding minimum similarity block was set to around 1.5 x base MI ( = MI(source (or initialized) volume, target vol- ume),¼ 18 SSD) for all the following tests. The Mean Euclidian Distance (MED) between source and target is reduced from 5.5 to 3.5 (mm 3 ) and the Correlation Coe±cient (CC) of the two volumes is improved from 0.63 to 0.80. Table 4.2 shows the MED, SSD, CC, and MI voxel similarity measure results. While considering spatial resolution of each voxel (1.25x1.25x1.2 mm 3 ) and volume interpolations for reconstruction, the MED of 3.5 shows accurate result within two voxels of the ground truth result. volume MED SSD CC MI source 5.5 mm 3 21.3 0.63 0.52 register 3.5 mm 3 15.6 0.80 0.73 Tab. 4.2: Registration results of synthetic volumes. Following our articulated initialization step, the knee volumes in four signi¯- cantly di®erent poses are registered. In order to compute the MED between the registeredandtargetvolume, weattachedasmallnumberofMRcompatiblemark- ers on the knee skin surfaces only for the validation purpose; small dots of knee volumes in ¯gure 4.14 shows the markers. The MED of the markers and other similarity measures with respect to the target volume is shown in table 4.3. The 94 registrationresultof3.8MEDisreasonableconsideringthevoxelspatialresolution. The results are visually validated in ¯gure 4.14. volume MED of markers SSD CC MI source 18.5 mm 3 41.1 0.31 0.11 initialize 5.8 mm 3 18.3 0.72 0.55 register 3.8 mm 3 15.7 0.79 0.65 Tab. 4.3: Registration results of the knee volume. Next,wetestedouralgorithmonaverychallengingsubject,thehumanhand(to our knowledge, there are no previous demonstrations of intensity based non-rigid volume registration of the full hand in signi¯cantly di®erent poses). In case of hu- man hand, validation test using synthetic volume and ¯ducial markers are di±cult and impractical due to it's complex skeletal structures and very subtle deformation within small areas. We evaluate the registration using similarity measures and vi- sual validation as was generally accepted in many previous papers [56,72,81,87]. The result are shown in table 4.4, ¯gure 4.12, and ¯gure 4.13. volume SSD CC MI source 31.6 0.01 0.07 initialize 19.4 0.53 0.23 register 17.8 0.60 0.28 Tab. 4.4: Registration results of the hand volume. Non-rigid volume registration takes a large amount of time [110]. Registration of articulated subjects require vastly more computation time due to the additional DOF, the larger areas of mismatch, and decreased smoothness in the deformation due motion of the skeleton. Since human hand registration requires highly con- strainedlocal deformation inthe ¯nger areas, registration using global deformation functions [72,81] did not provide adequate results in our attempts. 95 Ourlocallyadaptivealgorithmadequatelyregistersthearticulatedkneevolumes in 5 hours using around 900»1250 total allocated control points at block level 3, and the human hand volumes in an average of 4 hours using 2000» 2500 allocated control points at block levels 3, 4, 5. Although the total DOFs based on the allocated control points are very large (2700»7000), the actual DOFs are always constrainedto24(eightcontrolpointsin3D)ateveryoptimizationstep;potentially a few more points (less than 4 x 3) from the bone volume registered in section 4.4.2 can be considered as ¯xed control points. The experiments are performed by a machine having a Intel Xeon 3.0GHZ processor with 3 Gbyte memory. For comparison Park et al. [72] reported 11-hour compute times for non articu- lated CT volume registration using 48 ¯xed control points and a global registration function. 4.8.2 Example based volume deformation The registered volumes in di®erent poses provide correspondence across samples, allowing volume displacements between the di®erent poses to be computed. The EVDalgorithmmanipulatesatemplatevolumeintheneutralposetoanyarbitrary poseusingkinematicjointcontrolestimatedinsection4.5.2. Althoughtheexample based volume deformation requires complex pre-computations to handle raw med- ical volume samples, note that the EVD deformation time itself is small: the time is around 3.5 seconds to deform the human hand volume (255x255x90 voxel grid points) and 1.7 sec. to deform the knee volume (255x255x123 voxel grid points) to any arbitrary pose. The human hand and knee volume deformed to arbitrary poses are shown in ¯gures 4.15, and 4.16. The volume interior containing all anatomical layers is also adequately deformed and shown via a clipping plane. In ¯gure 4.17, and ¯gure 4.18, the VBD and EVD results are compared with a volume scan ex- cluded from the training set. A real-time volume rendering program [45] is used for display. 96 (a) (b) (c) (d) (e) Fig. 4.13: Non rigid articulated registration of the human hand. (a) A template volume in the neutral pose, (b) A target volume, (c) A template volume registered to the target volume, (d) Di®erences between registration (cyan points) and target volume (magenta) with allocated control points at block level 3 (32 3 resolution blocks), (e) An interior view of the deformed volume. 97 (a) (b) (c) (d) (e) Fig. 4.14: Nonrigidarticulatedregistrationofthehumanknee. (a)Atemplatevolumein the neutral pose; (b) A target volume; (c) A template volume registered to the target volume; (d) Di®erences between registration (cyan points) and target volume (magenta) with allocated control points at block level 3 (32 3 resolution blocks); (e) An interior view of the deformed volume; the markers (small dots) in knee volumes are successfully registered and used only for the validation. 98 Fig. 4.15: Hand volume deformation by EVD: the human hand volume are deformed to arbitrary poses; translucent volume rendering visualizes the complex anatom- ical layers without loosing the context of the subject; a clipping plane shows the interior MRI volumes. 99 Fig. 4.16: Knee volume deformation by EVD: the human knee volume are deformed to arbitrary poses; a clipping plane shows the interior MRI volumes. 100 Fig. 4.17: The VBD and EVD result (hand). Green (VBD) and cyan (EVD) points represents deformed volume, and the magenta region shows the di®erence be- tween the deformation and a ground truth volume which is not included in the training set. Fig. 4.18: The VBD and EVD result (knee). From the left, the VBD and EVD results of the knee volume deformed to ground truth volumes which are not included in the training set, and the ground truth knee volume. 101 Chapter 5 Conclusions In this study, we describes a complete system to produce an anatomically accurate articulated human body model of a speci¯c individual and it's tissue deformation, while managing the practical limitations arising in 2D/3D image scans of a living human. Our system approaches the problems and solutions in stages. 5.1 Overview and discussion 5.1.1 Human hand modeling from surface anatomy In the chapter 2, we presented an individualized hand modeling method using the surface anatomy, visible in a single photograph of that individual's hand. We ex- tractedbasichandgeometryandcreasesusingtensorvotingforthesurfaceanatomy features. Thejointstructurewasinferredfromthesurfaceanatomyfeatures. Next, we deformed skin geometry using RBFs with automated feature correspondences based on the curve segment matching. As a result, we can create a virtual hand clone that has the same contour geometry, joint skeleton, and skin texture map as the source person from a single photograph. Since the algorithm has no human guidance (other than threshold adjustment in di±cult cases) and the only informa- tion required is a palmar skin image, the method is easily applied. However, since we only use a palm picture, the method has several innate limitations. 102 We cannot measure hand thickness accurately, relying instead on an approx- imate depth proportional to hand length. This approximation may result in an erroneous depth of the joints and the mesh. Because the CMC joint of the thumb has very complex mechanisms in terms of biomechanics and anatomy, accurate CMC joint modeling from a 2D image cannot be guaranteed. Also, because this method relies on important creases on the palmar skin, we cannot apply our al- gorithms to an atypical human hand; for example, those with missing or vague important creases, or with corns, calluses, and any signi¯cant scar on the palmar surface. On the other hand, the overall approach does not prohibit the integration ofadditionalinformation,suchasadditionaltexturephotographsfromotherangles, and accurate depth data from a volumetric scanner. The limitations encountered in this chapter inspires the further study presented in the following chapters such as skin deformation algorithms in chapter 3, and creating deformable human body model from in vivo 3D scans in chapter 4. 5.1.2 Real-time skin deformation on GPU For realistic human body animation, sophisticated skin deformation algorithm is required. Inthe chapter3, wehavepresenteda parallelskindeformation algorithm suitable for SIMD architecture like the GPU. Manual creation of joints weights of complex subjects such as the human hand is di±cult and impractical. In our method, joint weights of each vertex are automatically computed by NNLS using su±cient pose samples and used in skinning computation for SSD and WPSD. Independent per-vertex deformation is parallelized on GPU using three rendering passes. In the ¯rst and second pass, per-vertex deformation is calculated by the fragment processor and the results are stored in texture maps using FBO. In the third pass using the vertex processor, each vertex of the rest pose is changed by the deformed vertex stored in the textures generated by the ¯rst and second pass. ArticulatedbodyskinningusingSSD,PSDandWPSDaree±cientlyparallelizedby 103 ourGPUbasedmethodandtheyshowaround20»30timesspeed-upcomparedwith CPU-onlymethodonadetailedmodel. Principalcomponentsoftheeigenspaceand carefulanalysisofjointdistributionscanfurtherreducethedomainofcomputation [48], and other algorithms based on the SSD, PSD and shape interpolation may be parallelized on GPU within this general approach. Although the method present proper solutions for the real-time example-based skin deformation system, the approach is based on the manually created surface modelsamples. Ahumanbodymodelinganddeformationbasedontherealsubject scans should consider several further issues. 5.1.3 Articulatedvolumeregistrationanddeformationfrom living human scans Finally, we presented a complete system (chapter 4) to produce an articulated de- formablehumanbodymodelfrominvivo3Dmedicalimagescans. Sincethesystem istobuildacompletevolumetricmodelfromrawmedicalimagescans,itincludesa number of components such as multiple in-vivo volume scans of articulated human body regions such as the human hand, intensity based bone volume registration, joint COR estimation, volume weight computation, volume blend deformation al- gorithm, articulated volume initialization, and example based volume deformation algorithm. In order to provide correspondences across scans for EVD algorithm, a template volume is registered to each volume sample using VBD initialization in combination with a locally adaptive non-rigid volume registration algorithm that e±ciently reduces computation cost while avoiding the strong local minima inher- ent in articulated body registration. The end result is a person-speci¯c articulated deformable volume model with anatomical accuracy well beyond current state of the art systems that are generally based on a single cadaver scan with simple de- formation and abstracted skeleton. 104 The approach is demonstrated on human knee and hand MRI volumes. Both subjectsarechallengingandimportantformanyapplications. Inparticular,thehu- manhandisoneofthemostcomplexarticulatedhumanbodyregions. Itisdi±cult to scan and therefore produces poor MRI images with unclear tissue boundaries. Based on our survey, our results may be the ¯rst to register full human hand MRI volumesinhighlydi®erentposesusingintensity-basednon-rigidregistration. Given the results obtained with a complex subject, we feel there is a good argument that themethodwillworkbetterformanysimplercasesasshowninourresultusingknee MRIs; previous registration papers and skin deformation papers have often shown restricted examples [3,35,55,56,72,85]. Therefore, the approach can be used for medicalimagingapplicationsrequiringnon-rigidregistrationsofarticulatedhuman body regions having complex skeletal structures. Many previous non-rigid volume registration e®orts have addressed non-articulated subjects such as the brain or breast, or have used simpli¯ed deformation that does not preserve bone rigidity. Thisworkwillhelptoextendtherangeofmedicalimageregistrationtoarticulated humanbodyregions, supportingclinicalstudyofpatientssu®eringfromjointpain. To our knowledge, fully volumetric example-based deformation from living hu- man scans has never been attempted in the previous work. A model constructed from living human scans provides personalized information, whichis often essential for medical applications. Therefore, our model can substitute for previous cadaver based static models in many applications while providing the full advantages of manipulatable volume data. Our volume registration result also contributes to surface based graphics. The speci¯cexampleof3Dscans andreconstruction ofself-occluded surfaces areanun- solvedprobleminsurfacebasedapproach. SinceMRIcancapture3Ddatawithout occlusions and the registration is only reliant on the intensity similarity measure without using geometrical surface features, a neutral pose volume registered to any 105 occluded pose volume (e.g. ¯st) can produce accurately identi¯ed surfaces in the occluded region. Our contributions are evident and many applications are feasible. On the other hand, due to the challenging problems associated with scans from living humans, several limitations were encountered. Our methods for volume registration and ex- amplebasedvolumedeformationarefullyautomatedexceptfortheboneextraction and initialization steps, which are partially automated. Manual guidance in these steps would not be necessary if the CT scans were used, since bones can be reliably segmentedwithsimplethresholding. However, CTscansinvolveionizingradiation. SincewesettheMRparameterstocovertheentirekneevolume,theMRsignals on the patella (knee cap) and tendons were not distinguishable. As we skipped the patella during the manual segmentation, the region was not properly initialized for the subsequent non rigid registration, resulting in some artifacts. Custom MRI setting or CT scans can provide better data resulting in easy segmentation and accurate registration. Although we approximated the knee joint as one DOF for simple animation control, more DOF should be considered for accurate control. An obvious question arises concerning the accuracy of using a limited number of scans in the EVD of the hand. This is not an intrinsic limitation as the system can accommodate more scans, however the number of scans needed is fewer than one might expect. Kurihara et. al [49] obtained an excellent surface deformation model of the hand with only ¯ve scans; we use 10 scans of the hand. Recent bio-engineering research showed that most actually encountered human hand poses can be reproduced using a combination of a limited number of principal poses [16]. We tested this idea using 60 di®erent meaningful poses from the \Poser" character system { the 10 dominant eigenposes covered more than 90% of the variation. Aswediscussed,mostofthelimitationsofourmethodcomefromthechallenges ofusingscansofarticulatedregionsoflivingsubjects. Althoughwecansolvemany of the addressed limitations using CT scans of cadaver volumes while increasing 106 quality of registration and ¯nal animation, results obtained from in vivo subjects have obvious advantages and are highly evaluated in medical and biomechanics research. The trade o® between the higher quality visual output from a cadaver (e.g. [70]) or CT scans versus the practical limitations arising from using living human subjects may be a good discussion issue in this study. 5.2 Future study The algorithms and methods presented in this study can possibly be used for in- terdisciplinary research related to medical imaging, ergonomics, computer aided design, robotics, virtual reality, as well as computer graphics and animation. Our solutions for several basic problems (e.g. scan based 3D modeling, correspondences across volume samples, and soft-tissue deformation algorithms) can be used for ad- vanced human body simulations in conjunction with several related studies such as sophisticated joint estimation algorithms, accurate motion generation using mo- tion captures, and high quality volume visualization. In addition to theses research topics, a number of speci¯c applications are feasible: ² The adaptive non-rigid volume registration algorithm for articulated human bodies provides an advanced medical image registration solution to quantify soft-tissue deformation of articulated subjects. The method helps to extend the range of medical image registration to articulated human body regions, supporting clinical study of patients su®ering from joint pain. ² The deformable articulated volume generates an anatomically accurate de- formable atlas model for medical education, surgical simulation, and a tem- plate model for multi-modal volume registration. Since previous atlas models have generally used polygonal mesh surface models with limited algorithmi- cally generated deformation properties, our volumetric model can provide 107 visualization which is well beyond the state-of-art-technology of current med- ical simulators. ² The deformable volume can be used for visual presentation of the interior anatomy during articulated body motion. It can support motion study of patients su®ering from arthritis, and also be used for visual simulation of arti¯cial limbs, rehabilitation, and sports science. ² Volume MRI scans of the hand holding an object capture the position and compression of skin regions that are in contact with the object, but are visu- allyoccluded. Therefore,self-occludedsurfacessuchasthepalmoftheclosed human hand are accurately reconstructed from articulated volume registra- tion. Thiscancontributetohumanhandsimulationinergonomics,industrial design, and bio-mechanics. We expect our methods can be used in various ¯elds to enhance accuracy and reality of the ¯nal output while providing sound fundamentals for related studies. 108 Bibliography [1] Irene Albrecht, Joerg Haber, and Hans Peter Seidel. Construction and an- imation of anatomically based human hand models. In Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA-03), pages 98{109, 2003. [2] Alias. Maya 6. http://www.alias.com. [3] Brett Allen, Brian Curless, and Zoran Popovi¶ c;. Articulated body deforma- tionfromrangescandata. InSIGGRAPH'02: Proceedingsofthe29thannual conference on Computer graphics and interactive techniques, pages 612{619, New York, NY, USA, 2002. ACM Press. [4] Brett Allen, Brian Curless, and Zoran Popovi¶ c. The space of human body shapes: reconstruction and parameterization from range scans. ACM Trans. Graph., 22(3):587{594, 2003. [5] Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. Scape: shape completion and animation of people. ACM Trans. Graph., 24(3):408{416, 2005. [6] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In SIGGRAPH '99: Proceedings of the 26th annual conference on Computer graphics and interactive techniques,pages187{194,NewYork,NY, USA, 1999. ACM Press/Addison-Wesley Publishing Co. [7] F. L. Bookstein. Principal warps: Thin-plate splines and the decomposition of deformations. IEEE. Trans. Pattern Analysis and Machine Intelligence, 11:567{585, 1989. [8] Mario Botsch and Leif Kobbelt. Real-time shape editing using radial basis functions. Computer Graphics Forum, 24(3):611{621, 2005. (Proceedings of Eurographics 2005). [9] Paul W Brand and Anne M Hollister. Clinical mechanics of the hand: 3rd Ed. Mosby, 1999. 109 [10] B Buchholz, TJ Armstrong, and SA Goldstein. Anthropometric data for describing the kinematics of the human hand. Ergonomics, 35(3):261{73, 1992. [11] William D. Bugbee and Micheael J. Botte. Surface anatomy of the hand: The relationships between palmar skin creases and osseous anatomy. Clinical Orthopaedics and Related Research, (296):122{126, 1993. [12] Steve Capell, Matthew Burkhart, Brian Curless, Tom Duchamp, and Zo- ran Popovi¶ c;. Physically based rigging for deformable characters. In SCA '05: Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computeranimation,pages301{310,NewYork,NY,USA,2005.ACMPress. [13] Steve Capell, Seth Green, Brian Curless, Tom Duchamp, and Zoran Popovi¶ c. Interactive skeleton-driven dynamic deformations. In SIGGRAPH '02: Pro- ceedings of the 29th annual conference on Computer graphics and interactive techniques, pages 586{593, New York, NY, USA, 2002. ACM Press. [14] Jonathan C. Carr, Richard K. Beatson, Jon B. Cherrie, Tim J. Mitchell, W. Richard Fright, Bruce C. McCallum, and Tim R. Evans. Reconstruction and representation of 3D objects with radial basis functions. In SIGGRAPH 2001, Computer Graphics Proceedings, pages67{76.ACMPress/ACMSIG- GRAPH, 2001. [15] Y. Chen, Q. Zhu, A. Kaufman, and S. Muraki. Physically-based animation of volumetric objects. In CA '98: Proceedings of the Computer Animation, pages 154{160, 1998. [16] Robert H. Clewley, John M. Guckenheimer, and Francisco J. Valero-Cuevas. Estimating e®ective degrees of freedom in motor systems. IEEE Trans. Biomedical Engineering, 55(2):430{442, 2008. [17] W. R. Crum, T. Hartkens, and D. L. G. Hill. Non-rigid image registration: theory and practice. The Britich Journal of Radiology, 77:141{153, 2004. [18] CuriousLab. Poser 5. http://www.curiouslabs.com. [19] Dartmouth-Medical-School. http://www.dartmouth.edu/»anatomy. [20] George ElKoura and Karan Singh. Handrix: animating the human hand. In SCA '03: Proceedings of the 2003 ACM SIGGRAPH/Eurographics sympo- sium on Computer animation, pages 110{119, 2003. [21] KlausEngel,MarkusHadwiger,JoeKniss,ChristofRezk-Salama,andDaniel Weiskopf. Real-Time Volume Graphics. A K PETERS, 2006. 110 [22] Randima Fernando and Mark J. Kilgard. The Cg Tutorial; The De¯nitive Guide to Programmable Real-Time Graphics. Addison-Wesley, 2003. [23] D Forsyth and J Ponce. Computer Vision A Modern Approach. Prentice Hall, 2002. [24] N. Gagvani and D. Silver. Realistic volume animation with alias. In Volume Graphics. Springer, 2000. [25] Nikhil Gagvani, D. Kenchammana-Hosekote, and D. Silver. Volume anima- tion using the skeleton tree. In VVS '98: Proceedings of the 1998 IEEE symposium on Volume visualization, pages 47{53. ACM Press, 1998. [26] Nikhil Gagvani and Deborah Silver. Animating volumetric models. Graph. Models, 63(6):443{458, 2001. [27] Sarah F. Gibson. 3D chainmail: a fast algorithm for deforming volumetric objects. In SI3D '97: Proceedings of the 1997 symposium on Interactive 3D graphics, pages 149{154. ACM Press, 1997. [28] J. P. Gourret, N. M. Thalmann, and D. Thalmann. Simulation of object and humanskinformationsinagraspingtask. InSIGGRAPH '89: Proceedings of the 16th annual conference on Computer graphics and interactive techniques, pages 21{30, New York, NY, USA, 1989. ACM Press. [29] GPGPU. Websiteforgeneral-purposecomputationusinggraphicshardware. http://gpgpu.org. [30] Leo Grady. Random walks for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 28(11):1768{1783, 2006. [31] SimonGreen. The OpenGL Framebu®er Object Extension. GameDevelopers Conference. [32] Gideon Guy and Grard Medioni. Inferring global perceptual contours from local features. International Journal of Computer Vision, 20, December 06 1996. [33] Joseph V. Hajnal, Derek L.G. Hill, and David J. Hawkes. Medical Image Registration. CRC Press, 2001. [34] Taosong He, Sidney Wang, and Arie Kaufman. Wavelet-based volume mor- phing. In VIS '94: Proceedings of the conference on Visualization '94, pages 85{92, 1994. [35] Mark Holden. A review of geometric transformations for nonrigid body reg- istration. IEEE Transactions on Medical Imaging, 27(1):111{128, 2008. 111 [36] Z. Huang, R. Boulic, and Daniel Thalmann. A multi-sensor approach for grasping and 3-D interaction. In Computer Graphics International '95, 1995. [37] Horace H. S. Ip, Sam C. S. Chan, and Maria S. W. Lam. Hand gesture animation from static postures using an anatomy-based model. In CGI '00: Proceedings of the International Conference on Computer Graphics, page 29, Washington, DC, USA, 2000. IEEE Computer Society. [38] IRIS. Tensor voting framework. http://iris.usc.edu/»tensorvt. [39] AnilK.Jain,ArunRoss,andSharathPankanti. Aprototypehandgeometry- based veri¯cation system. In Proc. of 2nd Int'l Conference on Audio and Video-based Biometric Person Authentication(AVBPA), pages 166 { 171, Washington D.C., March 1999. [40] Doug L. James and Dinesh K. Pai. DYRT: dynamic response textures for real time deformation simulation with graphics hardware. In SIGGRAPH '02: Proceedings of the 29th annual conference on Computer graphics and interactivetechniques,pages582{585,NewYork,NY,USA,2002.ACMPress. [41] Doug L. James and Christopher D. Twigg. Skinning mesh animations. ACM Trans. Graph., 24(3):399{407, 2005. [42] Kolja KÄ ahler, JÄ org Haber, Hitoshi Yamauchi, and Hans-Peter Seidel. Head shop: generating animated head models with anatomical structure. In SCA '02: Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation, pages 55{63, 2002. [43] Ladislav Kavan and Jiri Zara. Spherical blend skinning: A real-time de- formation of articulated models. In 2005 ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, pages 9{16. ACM Press, April 2005. [44] Junhwan Kim, Frederic Cordier, and Nadia Magnenat-Thalmann. Neural network-based violinist's hand animation. In Computer Graphics Interna- tional Proceedings, pages 37{41. IEEE Computer Society, June 2000. [45] Joe Kniss, Gordon Kindlmann, and Chuck Hansen. Simian. http://www.cs.utah.edu/»jmk/simian, 2007. [46] Yoshihito Koga, Koichi Kondo, James Ku®ner, and Jean-Claude Latombe. Planningmotionswithintentions. InSIGGRAPH'94: Proceedingsofthe21st annual conference on Computer graphics and interactive techniques, pages 395{408, New York, NY, USA, 1994. ACM Press. 112 [47] Satomi Komojima, Natsuki Miyata, and Jun OTA. Identi¯cation of posi- tion and orientation of hand bones fromMR Images by bone model registra- tion. InProceedingsoftheIEEE/RSJInternationalConferenceonIntelligent Robots and Systems, 2004. [48] Paul G. Kry, Doug L. James, and Dinesh K. Pai. EigenSkin: Real time large deformationcharacterskinninginhardware. In Proceedings of the 2002 ACM SIGGRAPH Symposium on Computer Animation (SCA-02), pages 153{160, 2002. [49] Tsuneya Kurihara and Natsuki Miyata. Modeling deformable human hands from medical images. In Proceedings of the 2004 ACM SIGGRAPH Sympo- sium on Computer Animation (SCA-04), pages 357{366, 2004. [50] Yair Kurzion and Roni Yagel. Space deformation using ray de°ectors. In Rendering Techniques '95 (Proceedings of the Sixth Eurographics Workshop on Rendering), pages 21{30. Springer-Verlag, 1995. [51] Yair Kurzion and Roni Yagel. Interactive space deformation with hardware- assisted rendering. IEEE Comput. Graph. Appl., 17(5):66{77, 1997. [52] S. Lee and I. Cohen. 3D hand reconstruction from a monocular view. Inter- national Conference on Pattern Recognition, August 2004. [53] Apostolos Lerios, Chase D. Gar¯nkle, and Marc Levoy. Feature-based vol- ume metamorphosis. In SIGGRAPH '95: Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, pages 449{456. ACM Press, 1995. [54] J. Lewis, H. J. Hwang, U. Neumann, and R. Enciso. Smart point landmark distribution for thin-plate splines. In Proc. SPIE Medical Imaging, 2004. [55] J. P. Lewis, Matt Cordner, and Nickson Fong. Pose space deformation: a uni¯ed approach to shape interpolation and skeleton-driven deformation. In SIGGRAPH '00: Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 165{172, New York, NY, USA, 2000. ACM Press/Addison-Wesley Publishing Co. [56] Xia Li, Todd Peterson, John Gore, and Benoit Dawant. Automatic registra- tionofwholebodyserialmicroCTimageswithacombinationofpoint-based and intensity-based registration techniques. pages 454{457, April 6{9 2006. ISBI'06: IEEE International symposium on Biomedical Imaging. [57] John Lin, Ying Wu, and Thomas S. Huang. Modeling the constraints of human hand motion. Workshop on Human Motion, 2000. 113 [58] William E. Lorensen and Harvey E. Cline. Marching cubes: A high resolu- tion 3D surface construction algorithm. In SIGGRAPH '87: Proceedings of the 14th annual conference on Computer graphics and interactive techniques, pages 163{169. ACM Press, 1987. [59] David Luebke, Mark Harris, Jens Kruger, Tim Purcell, Naga Govindaraju, Ian Buck, Cli® Woolley, and Aaron Lefohn. Gpgpu: general purpose compu- tation on graphics hardware. In GRAPH '04: Proceedings of the conference on SIGGRAPH 2004 course notes,page33,NewYork,NY,USA,2004.ACM Press. [60] Nadia Magnenat-Thalmann, Richard Laperriµ ere, and Daniel Thalmann. Joint{dependent local deformations for hand animation and object grasping. In Graphics Interface '88, pages 26{33, June 1988. [61] J. Maintz and M. Viergever. A survey of medical image registration. Medical Image Analysis, 2(1):1{36, 1998. [62] S. Marsland and C. J. Twining. Constructing di®eomorphic representations for the groupwise analysis of nonrigid registrations of medical images. IEEE Trans. Medical Imaging, 23(8):1006{1020, 2004. [63] M. A. Martin-Fernandez, E. Munoz-Moreno, M. Martin-Fernandez, and C. Alberola-Lopez. Articulated registration: Elastic registration based on a wire-model. In Medical Imaging 2005: Image Processing, pages 182{191. SPIE Press, February 12{17 2005. Proc. of the SPIE 5747. [64] T. McInerney and D. Terzopoulos. Deformable models in medical images analysis: a survey. Medical Image Analysis 1(2), pages 91{108, 1996. [65] G Medioni, M Lee, and C Tang. A Computational Framework for Segmenta- tion and Grouping. Elsevier, 2000. [66] Bruce Merry, Patrick Marais, and James Gain. Animation space: A truly linear framework for character animation. ACM Trans. Graph., 25(4):1400{ 1423, 2006. [67] Natsuki Miyata, Makiko Kouchi, Masaaki Mouchimaru, and Tsuneya Kuri- hara. Finger joint kinematics from MR Images. In Proceedings of the IEEE/RSJInternationalConferenceonIntelligentRobotsandSystems,2005. [68] LaurentMoccozetandNadiaMagnenat-Thalmann. Dirichletfree-formdefor- mations and their application to hand simulation. In Computer Animation, 1997. 114 [69] Alex Mohr, Luke Tokheim, and Michael Gleicher. Direct manipulation of interactive character skins. In SI3D '03: Proceedings of the 2003 symposium on Interactive 3D graphics, pages 27{30, New York, NY, USA, 2003. ACM Press. [70] National Library of Medicine. Visible human project. http://www.nlm.nih.gov/research/visible, 1986. [71] X. Papademetris, D.P. Dione, L.W. Dobrucki, L.H. Staib, and A.J. Sinusas. Articulatedrigidregistrationforseriallower-limbmouseimaging. pages919{ 926, 2005. MICCAI'05. [72] H. Park, P. H. Bland, K. K. Brock, and C. R. Meyer. Adaptive registration using local information measures. Medical Image Analysis 8(4), pages 465{ 473, 2004. [73] Sang Il Park and Jessica K. Hodgins. Capturing and animating skin defor- mation in human motion. ACM Trans. Graph., 25(3):881{889, 2006. [74] MattPharrandRandimaFernando. GPU Gems 2; Programming Techniques forHigh-PerformanceGraphicsandGeneral-PurposeComputation. Addison- Wesley, 2005. [75] Fr¶ ed¶ eric Pighin, Jamie Hecker, Dani Lischinski, Richard Szeliski, and David H. Salesin. Synthesizing realistic facial expressions from photographs. Computer Graphics, 32(Annual Conference Series):75{84, 1998. [76] Josien P. W. Pluim, J. B. Antoine Maintz, and Max A. Viergever. Mutual informationbasedregistrationofmedicalimages: Asurvey. IEEE Trans Med Imaging, 22(8):986{1004, August 2003. [77] Nancy S. Pollard and Victor B. Zordan. Physically based grasping con- trol from example. In SCA '05: Proceedings of the 2005 ACM SIG- GRAPH/Eurographics symposium on Computer animation, pages 311{318, 2005. [78] Michael Pratscher, Patrick Coleman, Joe Laszlo, and Karan Singh. Outside- in anatomy based character rigging. In SCA '05: Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation, pages 329{338, New York, NY, USA, 2005. ACM Press. [79] William H. Press, Brian P. Flannery, Saul A. Teukolsky, and William T. Vetterling. Numerical Recipes in C: The Art of Scienti¯c Computing, 2nd ed. Cambridge University Press, 1992. 115 [80] Timothy J. Purcell, Ian Buck, William R. Mark, and Pat Hanrahan. Ray tracing on programmable graphics hardware. ACM Transactions on Graph- ics, 21(3):703{712, July 2002. (Proc. of ACM SIGGRAPH 2002). [81] Taehyun Rhee, J.P. Lewis, Krishna Nayak, and Ulrich Neumann. Adaptive non-rigid registration of 3D knee MRI in di®erent pose spaces. March 14{17 2008. ISBI'08: IEEE International symposium on Biomedical Imaging. [82] TaehyunRhee,UlrichNeumann,andJ.P.Lewis. Humanhandmodelingfrom surface anatomy. In I3DG '06: Proc. of ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, 2006. [83] HansRijpkemaandMichaelGirard. Computeranimationofknowledge-based human grasping. In SIGGRAPH '91: Proceedings of the 18th annual confer- ence on Computer graphics and interactive techniques, pages 339{348, New York, NY, USA, 1991. ACM Press. [84] K. M. Robinette, H. Daanen, and E. Paquet. The CAESAR project: a 3-D surface anthropometry survey. In 3DIM99, pages 380{386, 1999. [85] Gustavo K. Rohde, Akram Aldroubi, and Benoit M. Dawant. The adap- tive bases algorithm for intensity-based nonrigid image registration. IEEE Transaction on Medical Imaging, 22:1470{1479, 2003. [86] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. GrabCut: inter- active foreground extraction using iterated graph cuts. In SIGGRAPH '04: ACM SIGGRAPH 2004 Papers, pages 309{314. ACM Press, 2004. [87] D. Rueckert, L. I. Sonoda, C. Hayes, D. L. Hill, M. O. Leach, and D. J. Hawkes. Nonrigid registration using free-form deformations: application to breast MR images. IEEE Trans Med Imaging, 18(8):712{721, August 1999. [88] Jae Hun Ryu, Nasuki Miyata, Mikiko Kouch, Masaaki Mochimary, and Kwan H. Lee. Analysis of skin movements with respect to bone motions using MR Images. International Journal of CAD/CAM, 3(2):61{66, 2003. [89] Ferdi Scheepers, Richard E. Parent, Wayne E. Carlson, and Stephen F. May. Anatomy-based modeling of the human musculature. In SIGGRAPH '97: Proc. of the 24th annual conference on Computer graphics and interactive techniques, pages 163{172, 1997. [90] HyewonSeo,FredericCordier,andNadiaMagnenat-Thalmann. Synthesizing animatable body models with parameterized shape modi¯cations. In SCA '03: Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, pages 120{125, 2003. 116 [91] W. Shu and D. Zhang. Automated personal identi¯cation by palmprint. Optical Engineering, 37(8):2359{2362, 1998. [92] Eftychios Sifakis, Igor Neverov, and Ronald Fedkiw. Automatic determina- tion of facial muscle activations from sparse motion capture marker data. ACM Trans. Graph., 24(3):417{425, 2005. [93] Maryann Simmons, Jane Wilhelms, and Allen Van Gelder. Model-based re- construction for creature animation. In SCA '02: Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation, pages 139{146, New York, NY, USA, 2002. ACM Press. [94] VikasSinghandDeborahSilver. Interactivevolumemanipulationwithselec- tiverenderingforimprovedvisualization. In VV '04: Proceedings of the 2004 IEEE Symposium on Volume Visualization and Graphics (VV'04), pages 95{ 102, 2004. [95] Peter-Pike J. Sloan, Charles F. Rose, and Michael F. Cohen. Shape by ex- ample. In SI3D '01: Proceedings of the 2001 symposium on Interactive 3D graphics, pages 135{143, New York, NY, USA, 2001. ACM Press. [96] Robert W. Sumner and Jovan Popovi¶ c;. Deformation transfer for triangle meshes. ACM Trans. Graph., 23(3):399{405, 2004. [97] Joseph Teran, Eftychios Sifakis, Silvia S. Blemker, Victor Ng-Thow-Hing, Cynthia Lau, and Ronald Fedkiw. Creating and simulating skeletal muscle from the visible human data set. IEEE Transactions on Visualization and Computer Graphics, 11(3):317{328, 2005. [98] DavidE.Thompson, WilliamL.BufordJr., LoydM.myers, DavidJ.Giurin- tano, and John A. Brewer III. A hand biomechanics workstation. Computer Graphics (SIGGRAPH '88 Proceedings), 22(4):335{343, August 1988. [99] Winnie Tsang, Karan Singh, and Eugene Fiume. Helping hand: an anatom- ically accurate inverse dynamics solution for unconstrained hand motion. In SCA '05: Proceedings of the 2005 ACM SIGGRAPH/Eurographics sympo- sium on Computer animation, pages 319{328. ACM Press, 2005. [100] P.ViolaandIIIW.M.Wells. Alignmentbymaximizationofmutualinforma- tion. In ICCV '95: Proc. of International Conference on Computer Vision, page 16. IEEE Computer Society, 1995. [101] Xiaohuan Corina Wang and Cary Phillips. Multi-weight enveloping: least- squares approximation techniques for skin animation. In SCA '02: Proceed- ings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation, pages 129{138, New York, NY, USA, 2002. ACM Press. 117 [102] JasonWeber. Run-timeskindeformation. InProceedingsofGameDevelopers Conference, 2000. [103] H. Wendland. Piecewise polynomial, positive de¯nite and compactly sup- portedradialfunctionsofminimaldegree. Advances in Computational Math- ematics, 4(1):389{396, 1995. [104] RÄ udiger Westermann and Christof Rezk-Salama. Real-time volume deforma- tions. Comput. Graph. Forum, 20(3), 2001. [105] Ying Wu and Thomas S. Huang. Human hand modeling, analysis, and ani- mationinthecontextofHCI. InternationalConferenceonImageProcessing, 1999. [106] Z. Wu. Solving pde with radial basis function and the error estimation. In Lecture Notes on Pure and APplied Mathematics, page 202. 1998. [107] YYoum,TEGillespie,AEFlatt,andBLSpraque. Kinematicinvestigation of normal MCP joint. Journal of Biomechanics, 11:109{118, 1978. [108] HanLiangYu, RobertAChase, andBerishStrauch. Atlas of Hand Anatomy and Clinical Implications. Mosby, 2004. [109] Xiaoru Yuan, Nan Zhang, Minh X. Nguyen, and Baoquan Chen. Volume cutout. The Visual Computer (Special Issue. Paci¯c Graphics 2005), 21(8{ 10):745{754, 2005. [110] LyubomirZagorchevand ArdeshirGoshtasby. A comparativestudyoftrans- formation functions for nonrigid image registration. IEEE Transactions on Image Processing, 15(3):529{538, 2006. 118

Abstract (if available)

Abstract A system for producing an accurate human body model is important and required in many fields. This study describes a complete system for creating an anatomically accurate 3D human body model from living human scans of a specific person. The method provides accurate and personalized tissue deformation arising from articulated motions of human body regions such as the human hand, probably the most complex articulated region in the human body.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

3D face surface and texture synthesis from 2D landmarks of a single face sketch

PDF

Anatomically based human hand modeling and simulation

PDF

Object detection and recognition from 3D point clouds

PDF

3D object detection in industrial site point clouds

PDF

Complete human digitization for sparse inputs

PDF

3D deep learning for perception and modeling

PDF

Multi-scale dynamic capture for high quality digital humans

PDF

Feature-preserving simplification and sketch-based creation of 3D models

PDF

3D inference and registration with application to retinal and facial image analysis

PDF

Acquisition of human tissue elasticity properties using pressure sensors

PDF

Data-driven 3D hair digitization

PDF

Human appearance analysis and synthesis using deep learning

PDF

Technology for improved 3D dynamic MRI

PDF

Autostereoscopic 3D diplay rendering from stereo sequences

PDF

Semantic structure in understanding and generation of the 3D world

PDF

Real-time simulation of hand anatomy using medical imaging

PDF

Representation problems in brain imaging

PDF

Face recognition and 3D face modeling from images in the wild

PDF

Plant substructuring and real-time simulation using model reduction

PDF

Scalable dynamic digital humans

Asset Metadata

Creator Rhee, Taehyun (author)

Core Title Articulated human body deformation from in-vivo 3d image scans

School Viterbi School of Engineering

Degree Doctor of Philosophy

Degree Program Computer Science

Publication Date 09/17/2008

Defense Date 04/22/2008

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag GPU,human hand,MRI,OAI-PMH Harvest,registration,skin deformation,vlume deformation

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Neumann, Ulrich (committee chair), Nakano, Aiichiro (committee member), Nayak, Krishna S. (committee member)

Creator Email thrhee@gmail.com,trhee@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-m1606

Unique identifier UC1133338

Identifier etd-Rhee-2304 (filename),usctheses-m40 (legacy collection record id),usctheses-c127-93589 (legacy record id),usctheses-m1606 (legacy record id)

Legacy Identifier etd-Rhee-2304.pdf

Dmrecord 93589

Document Type Dissertation

Rights Rhee, Taehyun

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Repository Name Libraries, University of Southern California

Repository Location Los Angeles, California

Repository Email cisadmin@lib.usc.edu