University of Southern California Dissertations and Theses

Spherical harmonic and point illumination basis for reflectometry and relighting

(USC Thesis Other)

Spherical harmonic and point illumination basis for reflectometry and relighting

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Spherical Harmonic and Point Illumination Basis for Reflectometry and Relighting by Borom Tunwattanapong A Dissertation Presented to the FACULTY OF THE GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (Computer Science) August 2014 Copyright 2014 Borom Tunwattanapong Acknowledgements This dissertation is accomplished by a collection of effort, guidance, support and encourage- ment. I would like to express my sincerest gratitude to everyone that help making this possible. I would like thank my advisor Prof. Paul Debevec for giving me an opportunity to work at the ICT graphics lab, guiding me to progress in the right direction and supporting me in every ways. I would like to thank my co-advisor Prof. Abhijeet Ghosh for his guidance in research, generous support in many ways and kind encouragement during tough times. I would like to thank Prof. Shanghua Teng for giving me an opportunity to study in the Ph.D. program at University of Southern California, and contributing his precious time to advise me as my dissertation committee member. I would also like to thank Prof. Andreas Kratky and Prof. Ulrich Neumann for their precious time and suggestion as my dissertation committee members. I would like to express my sincerest gratitude to everyone at the ICT graphics lab: Jay Busch for her cheerful support, Graham Fyffe and Andrew Jones for their expertise and kind assistance, Paul Graham and Xueming Yu for their collaboration, Katheleen Haase and Valerie Dauphin for their kindest help. I would also like to thank the rest of the members: Matt Cheng, Pieter Peer, Bruce Lamond, Cyrus Wilson, Alex Ma, Ryosuke Ichikari, Oleg Alexander and Koki Nagano. Thank you for all of your generous support. I would like to thank my parents for their love, support and encouragement throughout the entire time in the program. Without them, I would not be able to come this far. Additionally, I would like to thank all of my friends for giving me a precious experience during this journey. ii Finally, I would like to thank my fiance, Warunee, for her love and tenderness. She always supports and encourages me in every way. I am very grateful to have her by my side. iii Table of Contents Acknowledgements ii List of Figures vi Abstract viii Chapter 1 Introduction 1 1.1 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Chapter 2 Related work 6 2.1 Reflectance and Geometry Acquisition . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.1 Spatially Varying BRDF Capture . . . . . . . . . . . . . . . . . . . . . 7 2.1.2 Using Extended Light Sources . . . . . . . . . . . . . . . . . . . . . . . 8 2.1.3 Reflectance from Spherical Illumination . . . . . . . . . . . . . . . . . . 8 2.1.4 Specular Photometric Stereo from Point Illumination . . . . . . . . . . . 9 2.2 Relighting Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.2.1 Image-based Relighting . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.2.2 Relighting Basis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.2.3 Lighting Edit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Chapter 3 Practical Image-Based Relighting and Editing with Spherical-Harmonics and Local Lights 13 3.1 Relighting Basis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.1.1 Optimizing Weights . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.1.2 Point vs Gaussian Lights . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.1.3 Data Acquisition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.2 Lighting Edits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.3 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Chapter 4 Acquiring Reflectance and Shape from Continuous Spherical Harmonic Illumination 29 4.1 Setup and Acquisition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 4.2 Reflectometry from Spherical Harmonics . . . . . . . . . . . . . . . . . . . . . 33 4.2.1 Acquiring Spherical Harmonic Responses . . . . . . . . . . . . . . . . . 35 iv 4.2.2 Building the Reflectance Table . . . . . . . . . . . . . . . . . . . . . . . 35 4.2.3 Estimating the Specular Surface Normal . . . . . . . . . . . . . . . . . . 37 4.2.4 Estimating the Angle of Anisotropy . . . . . . . . . . . . . . . . . . . . 38 4.2.5 Estimating Roughness and Anisotropy . . . . . . . . . . . . . . . . . . . 39 4.2.6 Estimating Specular Albedo . . . . . . . . . . . . . . . . . . . . . . . . 41 4.2.7 Estimating Diffuse Albedo and Normal . . . . . . . . . . . . . . . . . . 41 4.2.8 Reflectometry Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 43 4.3 Geometry Reconstruction from Diffuse and Specular Reflections Discussion . . . 45 4.3.1 Stereo Reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 4.4 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Chapter 5 Acquiring Reflectance from Monitor-based Basis Illumination 53 5.1 Setup and Acquisition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 5.2 Acquisition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 5.2.1 Basis Illumination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 5.3 Reflectometry from Fourier Transform . . . . . . . . . . . . . . . . . . . . . . . 56 5.3.1 Estimating Diffuse and Specular Albedo . . . . . . . . . . . . . . . . . . 57 5.3.2 Estimating Specular Reflection Vector . . . . . . . . . . . . . . . . . . . 58 5.3.3 Estimating Specular Roughness . . . . . . . . . . . . . . . . . . . . . . 59 5.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 5.4.1 Monitor Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 Chapter 6 Acquiring Reflectance and Shape from Rapid Point Source Illumination 63 6.1 Hardware Setup and Capture Process . . . . . . . . . . . . . . . . . . . . . . . . 63 6.2 Deriving Reflectance and Geometry . . . . . . . . . . . . . . . . . . . . . . . . 65 6.2.1 Specular Photometric Stereo . . . . . . . . . . . . . . . . . . . . . . . . 65 6.3 Cost V olume Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 6.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 Chapter 7 Conclusion and Future Work 74 7.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 7.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 7.2.1 Performance Capture for Relighting . . . . . . . . . . . . . . . . . . . . 76 7.2.2 Multi-view Reflectance Merging . . . . . . . . . . . . . . . . . . . . . . 76 7.2.3 Estimating Specular Occlusion and Roughness . . . . . . . . . . . . . . 77 BIBLIOGRAPHY 78 v List of Figures 1.1 Digitizing real-world objects . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 Flowchart of the proposed framework . . . . . . . . . . . . . . . . . . . . . . 3 3.1 Face relighting comparison under the Grace cathedral environment . . . . . . . 14 3.2 Factorization of Grace Cathedral environmental illumination . . . . . . . . . . 14 3.3 Different error functions for the proposed optimization procedure . . . . . . . . 15 3.4 Combination of SH lighting and local lights . . . . . . . . . . . . . . . . . . . 16 3.5 Comparison of local lighting in two different specular regions . . . . . . . . . . 18 3.6 Examples of lighting intensity edits in the Grace Cathedral environment . . . . 20 3.7 Examples of lighting angular width modulation in the Grace Cathedral environ- ment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.8 Relighting result for a fixed lighting budget of 20 lighting conditions . . . . . . 23 3.9 RMS error plot of approximation error for different combinations of SH and local lights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.10 Relighting result in Grace Cathedral EM for different local light positions . . . 24 3.11 Generalizing our approach to fewer lighting conditions . . . . . . . . . . . . . 25 3.12 Relighting comparison under the Grace Cathedral environment . . . . . . . . . 27 3.13 Relighting and editing examples in various environment maps . . . . . . . . . 28 4.1 Spinning Spherical Reflectance Acquisition Apparatus . . . . . . . . . . . . . 30 4.2 Capture in progress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 4.3 Cross-section of the LED arm . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 4.4 The spherical harmonic functions up to the 3rd order . . . . . . . . . . . . . . 34 4.5 Responses of diffuse and specular to zonal harmonics . . . . . . . . . . . . . . 35 4.6 Plot of responses of an isotropic lobe to zonal harmonics . . . . . . . . . . . . 36 4.7 Responses of an anisotropic lobe to SH . . . . . . . . . . . . . . . . . . . . . . 39 4.8 Comparing separated specular albedo . . . . . . . . . . . . . . . . . . . . . . 44 4.9 Multi-channel input data stereo reconstruction . . . . . . . . . . . . . . . . . . 46 4.10 Estimated reflectance maps for a plastic ball. . . . . . . . . . . . . . . . . . . . 47 4.11 Maps, rendering, and photo for brushed metal. . . . . . . . . . . . . . . . . . . 48 4.12 Rendering of sunglasses with geometry and reflectance derived from the system 49 4.13 3D geometry of sunglasses . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 4.14 Rendering of a digital camera . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 vi 4.15 Analysis of shape and reflectance measurement for different types of spheres . . 52 5.1 Monitor-based Scanner hardware setup . . . . . . . . . . . . . . . . . . . . . . 54 5.2 Fourier transform basis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 5.3 Monitor-based scanner diffuse specular albedo . . . . . . . . . . . . . . . . . . 58 5.4 Monitor-based scanner surface normal . . . . . . . . . . . . . . . . . . . . . . 59 5.5 Monitor-based scanner results . . . . . . . . . . . . . . . . . . . . . . . . . . 60 5.6 Viewing-angle brightness variation . . . . . . . . . . . . . . . . . . . . . . . . 62 6.1 Point illumination facial capture setup . . . . . . . . . . . . . . . . . . . . . . 64 6.2 Rough geometry and diffuse-specular separation . . . . . . . . . . . . . . . . . 65 6.3 Specular images and specular normal map . . . . . . . . . . . . . . . . . . . . 66 6.4 Optimizing geometry from differing specular highlights . . . . . . . . . . . . . 69 6.5 Recovered reflectance maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 6.6 Final result and rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 6.7 Additional results and renderings . . . . . . . . . . . . . . . . . . . . . . . . . 73 vii Abstract This dissertation presents a system for acquiring spatially-varying reflectance information and relighting various surface types by observing the objects under active basis illumination. Most types of real-world objects are illuminated with a succession of spherical harmonic illumination conditions. From the response of the object to the harmonics, we can separate diffuse and specular reflections, estimate world-space diffuse and specular normals and compute anisotropic roughness parameters for each view of the object. Additionally, this work proposes a system to acquire the spatially-varying reflectance prop- erties of flat samples with minor surface orientation variation using an LCD monitor. The sys- tem illuminates the samples with low frequency Fourier transform illumination to encode the reflectance function of the samples, in the form of Fourier transform coefficients. Then the re- flectance properties of the samples are recovered by analyzing the Fourier transform coefficients of different frequencies. For objects with complicated reflectance or geometry, this work proposes a system that practi- cally acquires relightable and editable models of the objects. The system employs a combination of spherical harmonics and local illumination which reduces the number of required photographs by an order of magnitude compared to the traditional techniques. Finally, for faces, this work proposes a novel technique to rapidly capture and estimate re- flectance properties using an array of cameras and flashes. The reflectance properties can also be used to reconstruct complete 3-D models of the face. viii Chapter 1 Introduction The virtual world, a world without frontiers where anything is possible, used to be a place that people could only dream of. In this age of technology, where more and more people have high performance mobile computing devices and constant internet access, an emerging experience in the virtual world at anytime is no longer a dream. The virtual world is not only used for expressing non-existing objects, but, many times, it is also used for representing objects in the real world. The reason could be that people still need an entity with which they can recognize in order to be able to associate the virtual world with the real world. Alternatively, it could be due to the fact that the virtual world is the best place to present something since anyone can access it. The next challenge lies just ahead. The contents in the virtual world have to be as realistic as possible in order to represent real-world objects seamlessly. Digitally recording realistic models of real-world objects represents a longstanding problem in the realm of computer graphics and vision, with countless applications existing in the fields of cultural heritage preservation, industrial design, visual effects, on-line commerce and interactive entertainment. The main goal is acquiring digital models which can be used to render how the object would look from any viewpoint, reflecting the light properties inherit in any environment, allowing the digital model to represent an object faithfully in the virtual world. 1 (a) Real-world object (b) Reflectance information (c) Virtual world object (d) Geometry Figure 1.1: The process of digitizing real-world objects, starting with the object(a). The re- flectance (b) and geometry (d) is captured from the scanning system which then produces a real- istic rendering of the digitized object (c). The challenge is that there are countless variations of real-world objects regarding their shape and how they interact with light. For example, some objects have a very finely detailed shape, like hair or cloth, which requires micro-level detail in order to be able to model the object accurately. Some objects are transparent, like glass, or highly reflective, like mirrors, which also requires special techniques in order to measure and accurately reproduce their shape and reflectance. This makes it extremely challenging for a single scanner or scanning technique to be able to handle these variations within shapes and surface types. For certain types of objects, some scanning techniques can represent the shape and reflectance with high levels of accuracy. For example, if the object has a diffuse-dominant reflection and tex- ture variation, computer vision techniques like structure from motion or photogrammetry can recover both the shape and reflectance with highly defined detail. If the object has less texture 2 variation, but still has a diffuse dominance reflection, employing structured light with stereo re- construction could also solve the problem. On some occasions, when the digital model does not need to be rendered from different view points, an image-based rendering technique is the perfect solution. This technique has been pro- posed for rendering certain types of objects that express a highly complicated shape or reflectance which is very hard to reconstruct accurately with a physically-based model. By employing this technique, a highly realistic rendering of target objects under different illumination conditions, also called relighting, can be created. In its current state, the technique has some limitations wherein the number of required pho- tographs is dependent on the surface material of the object. If the surface is highly specular, the number of required photographs increases. The relightable model is also difficult to modify, because of the fact that they are photographs of the object under various lighting directions. Figure 1.2: Flowchart diagram of the proposed framework. Each chapter of this dissertation is represented as a processing module in the figure. 3 1.1 Contribution This dissertation presents a framework for high-fidelity reflectance and geometry acquisition as well as relightable models of real-world objects. The framework employs various basis illumina- tion conditions on the object to analyze and extract reflectance informations. The contributions are: 1. A reflectance and geometry scanning system from continuous spherical harmonic illumina- tion. (a) A controllable semi-circular LED arm which is capable of projecting high resolution spherical illuminations (b) A view independent reflectance analysis technique to recover diffuse and specular albedo, surface normals and anisotropic roughness. 2. A framework for acquiring a relightable and editable model from spherical harmonic and point-source illumination. (a) A practical technique for image-based relighting that greatly reduces the number of required images necessary for achieving high quality editable relighting results (b) An optimization procedure that combines low order spherical harmonics with a set of local lights to generate a close approximation to a given environment illumination (c) An image-based approach for artistic editing of the relit result including the intensity and angular width modulation of the incident illumination 3. A system to recover reflectance information of faces from flash photographs. (a) An algorithm that estimates specular surface normals from flash photographs 4 (b) A system assessing the cost volume associated with different surface displacements 4. A framework for acquiring the reflectance properties of flat samples using Fourier transform illumination on an LCD monitor (a) A system that illuminates the sample with Fourier transform illumination and capture images for reflectance analysis (b) An algorithm that directly estimates the per-pixel reflectance properties of the sample based on the Fourier transform illumination 5 Chapter 2 Related work An extensive body of work in the graphics and vision literature addresses the acquisition of ge- ometry and reflectance from images under controlled and uncontrolled lighting conditions. Two recent overviews are Weyrich et al. [61], which covers a wide variety of techniques for acquiring and representing BRDFs and SVBRDFs over object surfaces, and Ihrke et al. [22], which focused on the acquisition of purely specular and transparent objects. Image-based relighting has also been widely researched over the recent years in computer graphics, both in terms of reflectance acquisition techniques as well as the choice of basis for encoding of reflectance functions for relighting. In the following, some of the most relevant work in capturing opaque objects with spatially- varying diffuse and specular reflectance components are highlighted, as well as relighting tech- niques under directional lighting with no assumption of geometry or reflectance properties. 6 2.1 Reflectance and Geometry Acquisition 2.1.1 Spatially Varying BRDF Capture SVBRDFs can be captured exhaustively using point light sources (e.g. [3, 33]), but this requires a large number of high-dynamic range photographs to capture every possible combination of in- cident and radiant angle of light. Similar to our approach, many techniques (e.g. [5, 9, 20, 46]) look instead at BRDF slices of spatially-varying materials observed from a single viewpoint to infer parameters of a reflectance model, which can be used to extrapolate reflectance to novel viewpoints. Other approaches [48, 29, 66] use sparse sets of viewpoint and lighting directions and extrapolate BRDFs per surface point assuming that the reflectance varies smoothly over the object. This approach is also used by Dong et al. [6], which employs a dedicated BRDF measure- ment system to sample representative surface BRDFs, which are extrapolated to the surface of the entire object based on its appearance under a moderate number of environmental lighting con- ditions. None of these techniques, however, produces independent measurements of diffuse and specular reflectance parameters for each observed surface point, and thus may miss important sur- face reflectance detail. Holroyd et al. [21] describes a complete system for high-precision shape and reflectance measurement of 3D objects using a pair of co-axial camera-projector units. Their setup uses phase-shifted structured light leveraging Helmholz reciprocity [67] for high-quality geometry estimation of nonconvex objects, and a clustering technique to derive SVBRDFs across object surfaces from a relatively sparse sampling of viewpoints. While their system can pro- duce high-quality results for many objects, it would likely have trouble estimating geometry and reflectance where sharp specular reflections are dominant, such as sunglasses lenses. Recently, Aittala et al. [1] analyzed SVBRDF of flat samples using Fourier transform illumination on LCD 7 screen. Their setup is restricted to only planar objects and required multi resolution optimization to estimate reflectance properties. 2.1.2 Using Extended Light Sources Ikeuchi [23] extended the original photometric stero approach of Woodham [64] to specular sur- faces, using a set of angularly varying area light sources to estimate the specular surface orienta- tion. Nayar et al. [34] used an extended light source technique to measure orientations of hybrid surfaces with both diffuse and specular reflectance, but they did not characterize the BRDF of the specular component. Gardner et al. [9] employed a moving linear light source to derive BRDF models of spatially-varying materials, including highly specular materials, but still required hun- dreds of images of the moving light to record sharp reflections. Hawkins et al. [16] recorded diffuse and specular reflectance behavior of objects with high angular resolution using a surround- ing spherical dome and a laser to excite the various surface BRDFs through Helmholz reciprocity, but achieved limited spatial resolution and required a high-powered laser equipment. Recently, Wang et al. [55] used step-edge illumination to estimate dual scale reflectance properties of highly glossy surfaces, but did not estimate per-pixel BRDFs. 2.1.3 Reflectance from Spherical Illumination Ma et al. [30] used spherical gradient illumination representing the 0th and 1st order spherical harmonics in an LED sphere to perform view-independent photometric stereo for diffuse and/or specular objects, and used polarization difference imaging to independently model diffuse and specular reflections of faces. Ghosh et al. [11] extended this approach by adding 2nd order spher- ical harmonics to estimate spatially varying specular roughness and anisotropy at each pixel. 8 Unfortunately, the use of an LED sphere with limited resolution made the reflectance analysis ap- plicable only to relatively rough specular materials such as human skin, and the use of polarization for component separation becomes complicated for metallic surfaces and near the Brewster angle. To avoid using polarization for reflectance component separation, Lamond et al. [27] modulated gradient illumination patterns with phase-shifted high-frequency patterns to separate diffuse and specular reflections and measure surface normals of 3D objects. Our work reformulates and gen- eralizes this frequency-based component separation approach to increasing orders of spherical harmonic illumination. Noting that BRDFs can usefully represented by spherical harmonic func- tions (e.g. [60]), Ghosh et al. [13] used spherical harmonic illumination projected to a zone of a hemisphere for reflectance measurement, but only for single BRDFs from flat samples. Their ap- proach also did not separate diffuse and specular reflectance using the measurements and required very high orders of zonal basis function to record sharp specular reflectance. Instead, in this work we propose employing up to 5th order spherical harmonics for both diffuse-specular separation as well as estimating reflectance statistics such as specular roughness and anisotropy. 2.1.4 Specular Photometric Stereo from Point Illumination Photometric stereo [63] has been applied to recover dynamic facial performances using simulta- neous illumination from a set of red, green and blue lights [18, 26]. However, these techniques are either data intensive or do not recover reflectance information. An exception is Georghiades [10], who recovers shape and both diffuse and specular reflectance information for a face lit by mul- tiple unknown point lights. The problem is formulated as uncalibrated photometric stereo and a constant specular roughness parameter is estimated over the face, achieving a medium scale reconstruction of the facial geometry. Zickler et al. [65] showed that photometric invariants al- low photometric stereo to operate on specular surfaces when the illuminant color is known. The 9 practicality of photometric surface orientations in computer graphics has been demonstrated by Rushmeier et al. [47] for creating bump maps, and Nehab et al. [35] for embossing such surface orientations for improved 3D geometric models. Hertzmann and Seitz [19] showed that with exemplar reflectance properties, photometric stereo can be applied accurately to materials with complex BRDF’s, and Goldman et al. [14] presented simultaneous estimation of normals and a set of material BRDFs. However, all of these require multiple lighting conditions per viewpoint, which is prohibitive to acquire using near-instant capture with commodity DSLRs. Most of the above techniques have exploited diffuse surface reflectance for surface shape re- covery. This is because typically specular highlights are not view-independent and shift across the subject as the location of the light and camera changes. Zickler et al. [67] exploits Helmholtz reciprocity to overcome this limitation for pairs of cameras and light sources. Significant work [2, 62, 5] analyzes specular reflections to provide higher-resolution surface orientations for translu- cent surfaces. Ma et al. [30] and Ghosh et al. [12] perform photometric stereo using spherical gradient illumination and polarization difference imaging to isolate specular reflections, record- ing specular surface detail from a small number of images. While these techniques can produce high quality facial geometry, they require a complex acquisition setup such as an LED sphere and many photographs. In our work, we aim to record comparable facial geometry and reflectance with off-the-shelf components and near-instant capture. 2.2 Relighting Techniques 2.2.1 Image-based Relighting Haeberli [15] introduced the idea of relighting images as a linear combination of a set of basis images. Using a device called a Light Stage, Debevec et al. [5] recorded reflectance functions of 10 a face at relatively high angular resolution and achieved very realistic relighting results with the acquired data. Subsequent versions of the apparatus improved on the acquisition time [17, 58] or generalized to a non-regular sampling of lighting directions [31]. Matusik et al. [32] extended the approach for viewpoint independent relighting. However, these approaches are data-intensive in both acquisition and storage. Additionally, inclusion and editing of the data in production pipelines requires significant effort. More recently, Peers et al. [41] proposed a reflectance transfer approach for image based relighting of facial performance. However, it is difficult to generalize the approach for relighting arbitrary objects. 2.2.2 Relighting Basis Nimeroff et al. [38] proposed a set of steerable basis functions for image-based relighting and demonstrated relighting results for low frequency natural sky lighting. Spherical harmonics were first proposed by Westin et al. [59] as a basis for reflectance functions in computer graphics. Ra- mamoorthi and Hanrahan proposed using spherical harmonics to efficiently capture diffuse low frequency illumination [43], and extended the analysis to higher order SH for glossy and specular reflections [44]. Similar analysis has been done for relighting with pre-computed radiance trans- fer (PRT) [50, 24]. A significant drawback of the SH basis is Gibbs ringing or aliasing artifacts which occurs around high frequency features, such as highlights. Hence, Ng et al. [36, 37] pro- posed using a non-linear Haar wavelet basis for high frequency relighting. Subsequent work has proposed other non-linear representations such as spherical radial basis functions (SRBFs) [52] for even better compression of reflectance functions in a PRT context. In our work, we demonstrate good qualitative relighting results with a budget of around only 20 lighting conditions. For such a restricted budget of lighting conditions, linear measurements with bases such as wavelets or SRBFs would not appropriately localize leading to a somewhat 11 low frequency reconstruction. While a compressive sensing approach [40, 49] would seem an obvious option, even these approaches need to make direct measurements of the low frequency light transport and would not be suitable with such a small image capture budget. Recent low rank approximation techniques such as Kernel Nystrom [56] and Krylov sub- space [39] methods achieve good quality results with a moderate capture budget. However, it is unclear how to practically extend these techniques from a limited projector-camera setup to relighting under full environmental illumination. 2.2.3 Lighting Edit Another important consideration is the editability of the captured data as demonstrated by recent research in the area of lighting design and interfaces for editing point lights [25], as well as natural illumination [42]. SH being a global basis, does not provide a good mechanism of local lighting control which is desirable for artistic editing effects. And editing of reflectance functions based on wavelet coefficients would still be non-trivial. Instead, our proposed technique combines the efficient low frequency approximation of SH lighting with local light sources for high frequency lighting and intuitive control for artistic editing. Our approach is similar in spirit to the work of Davidovic et al. [4] who employ visibility clus- tering for low-rank approximation of global effects and local lights for high rank approximation in the context of global illumination, whereas we focus on global and local control of directional lights for image-based relighting. 12 Chapter 3 Practical Image-Based Relighting and Editing with Spherical-Harmonics and Local Lights We present a practical technique for image-based relighting under environmental illumination which greatly reduces the number of required photographs compared to traditional techniques, while still achieving high quality editable relighting results. The proposed method employs an optimization procedure to combine spherical harmonics, a global lighting basis, with a set of local lights. Our choice of lighting basis captures both low and high frequency components of typical surface reflectance functions while generating close approximations to the ground truth with an order of magnitude less data. This technique benefits the acquisition process by reducing the number of required photographs, while simplifying the modification of reflectance data and enabling artistic lighting edits for post-production effects. Here, we demonstrate two desirable lighting edits, modifying light intensity and angular width, employing the proposed lighting basis. Sections of this chapter have been published in Combining spherical harmonics and point-source illumination for efficient image-based relighting[54]. 13 (a) EM (b) 20 SH lights (c) 20 local lights (d) Our technique (e) Ground truth Figure 3.1: Face relighting comparison under the Grace cathedral environment (a). (b) 20 SH lights fail to capture the high frequency highlight of the bright orange alter correctly while also introducing a slight color banding artifact on the left side of the face. (c) 20 local lights capture the high frequency lighting effects, but do not accurately capture the low frequency lighting. (d) Our proposed technique for combining SH lighting with local lights (20 lighting conditions) captures both low frequency and high frequency lighting effects. (e) Ground truth image-based relighting consisting of 156 lights. 3.1 Relighting Basis (a) EM (b) 9 SH (c) residual EM (d) local lights Figure 3.2: Factorization of Grace Cathedral environmental illumination (a) into 2 nd order SH lighting (b) and a residual EM representing high frequency lighting (c). This residual energy is represented in our approach with a set of Gaussian local lights (d). In this work, we propose a practical image-based relighting approach that combines low fre- quency SH lighting for global contribution of environmental illumination with a set of local light sources. Building on the work of Ramamoorthi and Hanrahan [43], we employ low order SH lighting for approximating low frequency lighting. Given a total budget of N lighting conditions, we employ a fixed number of m low order SH lights and an additional set of n local lights such that N = m+ n. We place the n local lights uniformly spaced apart on the upper hemisphere. The rationale for this is that most bright light sources tend to be located in the upper hemisphere 14 (a) Original space (b) PCA space (c) Ground truth Figure 3.3: Different error functions for the proposed optimization procedure. (a) Sub-optimal relighting result after 600 iterations of the optimization when using error function on the original space of the EM. (b) The optimization achieves much better convergence with the same number of iterations using error function in PCA space. (c) Ground truth relit image. in an environment map (EM). Our approach then proceeds as follows: we first compute an m SH reconstruction of the EM (Fig. 3.2, (b)). Then we generate a residual EM by subtracting the m SH reconstruction from the original EM (Fig. 3.2, (c)). The residual map normally contains the bright high frequency content of the EM. It may also contain some negative pixels from the subtraction which we clamp to zero in order to preserve the contributions of the local lights in the final combination. We then sample the residual EM into the basis of local lights in order to compute the energy of each light. In this way, the local lights account for the energy unaccounted by the SH lighting. However, direct combination of SH lighting and these local lights achieves sub-optimal relighting (see Fig. 3.4, (a)). Hence, we propose an optimization approach to find the optimal combination weights of SH and local lights (see Fig. 3.4, (b)). 15 (a) Un-optimized (b) Optimized (c) Ground truth Figure 3.4: Combination of SH lighting and local lights in the Grace cathedral environment. (a) Combi- nation with un-optimized weights. (b) Combination with optimized weights. (c) Ground truth relighting consisting of 253 lights. 3.1.1 Optimizing Weights We employ an optimization procedure to find the appropriate combination weights which repre- sents the contribution of each basis to the final relit result. Given a set of SH and local lighting basis b =fb 1 ;b 2 ;:::;b N g, and an EM y, the optimization proceeds by iteratively reducing the difference between the original EM and its reconstruction based on the weighted combination of SH and local lights. Mathematically, this can be expressed as: min x f(x); f(x)=kAx yk 2 where f(x) is the objective function to be minimized, x is the N 1 set of solution weights, and A is the projection of the EM y into the space spanned by the chosen lighting basis b. Each column i of the projection matrix A is given as A[i]= c i b i ;i2[1:::N], where c i =b i y. Instead of directly solving the above optimization in the original space of the EM, we compute the error term by first projecting the EM y into Principal Component Analysis (PCA) space A 0 of the chosen SH and local lighting basis to obtain an N 1 feature vector y 0 of the EM in this PCA space. This can be formally described as: 16 min x f(x); f(x)=kA 0T Ax y 0 k 2 where A 0 = PCA(A), and y 0 = A 0T y. All of the projections of SH and local lights are done using only the luminance channel. In each iteration of the optimization we calculate the distance of the target feature vector y 0 to the feature vector of the EM reconstructed with the combination weights of the current iteration. We initialize all the weights x to one in the first iteration. We found that employing these feature vector distances in PCA space of the chosen basis as an error function for the optimization gives rise to fewer local minima problems, faster convergence and better results than computing the optimization in the original space (see Fig. 3.3). We employ a bounded version of FMINSEARCH function in MATLAB, which uses the Nelder-Mead simplex algorithm, to find the optimal weights x that reduce the distance between two feature vectors. Subsequently, we generate the final relit result by applying the optimized weights x to the SH and local lights with energies obtained from the initial factorization of the EM described in Fig. 3.2. We chose the FMINSEARCH function for the optimization over lin- ear least squares as it enables a more general error term formulation and gave better results for more complex lighting environments. We also restrict all the m SH coefficients to share the same weight in the optimization for a total of K= n+ 1 weights that are optimized. This further limits the dimensions for the optimization for better convergence as well as providing a single global control of the low frequency lighting in addition to a few local lights for any subsequent lighting edits. For most results presented in the paper, we set the total image capture budget to N = 20 lighting conditions, with m= 9 (2 nd order) SH lights and n= 11 local lights. 17 (a) Point lights (b) Gaussian lights (a) Point lights (b) Gaussian lights Figure 3.5: Comparison of local lighting in two different specular regions of the Plant dataset. (a): Point lights produce unnaturally sharp highlights. (b): Gaussian lights produce more realistic glossy highlights. Note that an alternate possible approach for combining these SH and local lights would be orthonormalization of the space spanned by these individual lighting bases. While this approach may result in a valid relighting result, such an orthonormalization process converts the local light- ing bases into bases with global support like the SH basis functions, thus negatively impacting localized lighting edits with the proposed lighting basis (Section 3.2). 3.1.2 Point vs Gaussian Lights We compared using different types of local lights for the combination with SH lighting including point lights and narrow width Gaussians. The intuition behind using Gaussian light sources is that they result in smoother specular highlights (see Fig. 3.5, b) compared to point-source lights for specular scenes (see Fig. 3.5, a). In addition, most light sources in natural environments are extended light sources rather than true point lights. This makes the narrow Gaussian lights a more appropriate choice for the proposed combination with SH lighting (see Fig. 3.2, d). 3.1.3 Data Acquisition We now briefly discuss the data acquisition and then present some results with the proposed technique in Section 3.3. We employ an LED sphere lighting system with 156 lights similar to [58] in order to obtain data for the face example, while using data available online from another 18 variant of the device with greater lighting density [7] for the other examples in this paper. Such a lighting system can project spherical harmonic illumination on a subject after scaling it to the[0;1] range. This implies that we can directly capture (scaled) SH coefficients from photographing a subject under these lighting conditions. For the local lights, we capture photographs of the subject while illuminating it with banks of lights that have a sharp Gaussian fall-off in intensity. Finally, we employ traditional dense reflectance function data from photographs captured with individual lighting directions for ground truth relighting. 3.2 Lighting Edits Our technique greatly reduces the number of images required for realistic relighting, which also makes post-editing of the relit result more intuitive. Our choice of basis enables direct manip- ulation of the combination weights as well as local light intensities, while keeping them to a manageable number for a user as argued in [25]. Figure 3.6 presents two examples of such direct manipulations for light intensity modulation that we can achieve with our technique. In Edit 1, we have reduced the contribution weight of the local light corresponding to the rim lighting of the orange alter in the Grace Cathedral EM. In Edit 2, we have increased the contribution weight of the local light corresponding to the bright windows above the alter, while slightly reducing the global SH contribution leading to sharper shadowing and a more dramatic look. We also present a novel image-based technique for modulating the angular width of a local light with the proposed lighting basis. For increasing the angular width of a chosen local light, we propose to interpolate between the high frequency lighting of the local light and the low fre- quency SH lighting using ratio images. Ratio images have been employed by Peers et al. [41] for 19 (a) Optimized (b) Edit 1 (c) Edit 2 Figure 3.6: Examples of lighting intensity edits in the Grace Cathedral environment. (a) Original opti- mized result. (b) Edit 1: The effect of the rim lighting from the orange alter has been reduced. (c) Edit 2: The effect of the sky light from the windows above the alter has been increased casting a sharper shadow under the subject’s nose. image-based lighting transfer. In this work, we employ ratio images for artistic editing of image- based relighting. In particular, we propose to interpolate between two ratio images - one ratio representing high frequency illumination obtained by dividing the local light illumination by uni- form spherical illumination (Fig. 3.7, top-row (b)), and another ratio representing low frequency lighting obtained by dividing a spherical linear gradient along the direction of the local light by uniform spherical illumination (Fig. 3.7, top-row (c)). Given that our lighting basis includes low order SH lighting scaled to [0,1], we can create a spherical linear gradient along any direction by steering the 1 st order SH conditions to the desired direction. Finally, we relight the subject with the interpolated ratio image after scaling it with the intensity and color of the chosen local light. An example of such an edit to the angular width of local light can be see in Fig. 3.7 (top-row), where the rim lighting of the orange alter has been broadened to create a softer lighting from the side. 20 Another lighting edit that is possible is reducing the angular width of a local light. Here, we employ the ratio image obtained by dividing the local light illumination by uniform spherical illumination and exponentiate it to effectively reduce the width of the Gaussian local light. The exponentiated ratio image is once again scaled by the intensity and color of the original local light to obtain the edited relit result. An example of such an edit to the angular width of local light can be seen in Figure 3.7 (bottom-row), where the width of top light corresponding to the window has be reduced with ratio image exponentiation to create a dramatic harsh lighting effect. The above lighting edits can also be seen in the accompanying video. Such artistic control of intensity and angular width modulation of lighting is very desirable in post-production work-flows where the ability to easily edit a lighting result is an important consideration besides achieving high quality relighting. 3.3 Results and Discussion The input to our algorithm is an environment map and photographs of a subject in the chosen basis of N lighting conditions (m SH + n local lights). An advantage of the optimization procedure in our technique is that it is defined on the global lighting EM, and is hence applicable for relighting any dataset captured from any viewpoint with just a single optimization. In our implementation, the optimization procedure takes 2 minutes to converge on a dual core 2.5 GHz laptop with 4GB RAM. Figure 3.1 presents the result of the optimization procedure for a subject relit in the Grace Cathedral EM. As can be seen, 20 SH lighting conditions fail to correctly capture the high fre- quency highlights while resulting in a slight ringing artifact on the left side of the face. The 20 local lights on the other hand fail to correctly capture the low frequency lighting correctly. 21 (a) Initial (b) Ratio 1 (c) Ratio 2 (d) Edited Figure 3.7: Examples of lighting angular width modulation in the Grace Cathedral environment. (top- row) Softening of the rim light corresponding to the orange alter using interpolation between ratio images 1 and 2. (bottom-row) Sharpening of the top Gaussian local light corresponding to the window using exponentiation of ratio image 1 to obtain ratio image 2. Our proposed optimized combination achieves a good result for the same lighting capture budget (N= 20) compared to the ground truth. We present relighting results of additional datasets in the Grace Cathedral EM in Figure 3.12. Particularly visible here is the Gibbs ringing of SH lighting resulting in color banding artifacts (Figure 3.12, (a)) on specular surfaces, as well as the hot spots due to local lights on the Helmet and the sword of the Kneeling knight (Figure 3.12, (b)). Our proposed combination of SH and local lights achieves the best qualitative results for a similar budget of lighting conditions for these scenes. Fig. 3.13 presents more examples of relighting and editing with our technique in different lighting environments. Here, the edited results have been obtained with a few simple operations on selected local lights and global SH lighting. Additional 22 (a) 4 SH + 16 local lights (b) 9 SH + 11 local lights (c) 16 SH + 4 local lights (d) Ground truth Figure 3.8: Relighting result for a fixed lighting budget of 20 lighting conditions. (a) 1 st order SH lighting with 16 local lights. (b) 2 nd order SH lighting with 11 local lights. (c) 3 rd order SH lighting with 4 local lights. (d) Ground truth relighting with 253 lights. Figure 3.9: RMS error plot of approximation error for different combinations of SH and local lights (left) and for different total budget of lighting conditions (right). EM relighting examples can be found in the supplemental document. The accompanying video includes results of a lighting animation sequence where we achieve consistent results across the sequence with rotation of coefficients and weights of the SH and local lights. In Figure 3.8, we analyze the optimal combination ratio of SH lighting with local lights for a fixed total budget of lighting conditions. In this example, we fixed the total number of light- ing conditions N = 20. For this budget, we empirically found the best qualitative results to be obtained with configuration (b) with a similar number of SH (m= 9) and local lights (n= 11). Configuration (b) also resulted in a lower RMS error in the approximation of the EM compared to configurations (a) and (c) (Fig. 3.9, left). 23 In Figure 3.9 (right), we plot the approximation error for different total budget of lighting conditions. As expected, with increase in the total number of lighting conditions N from 10 to 25, there is a decrease in the approximation error. As can be seen from the plot, a reasonable approximation of the EM can be obtained with as few as N = 20 lighting conditions with m= 9 SH lights and n= 11 local lights. (a) Distribution 1 (b) Distribution 2 Figure 3.10: Relighting result in Grace Cathedral EM for different local light positions. Top-row: Local light distributions. Bottom-row: Relit result. The light positions in (b) are slightly shifted with respect to the light positions in (a). In Figure 3.10, we present relighting results for two different positions of local lights on the upper hemisphere that are slightly shifted with respect to each other. As expected, there is a slight 24 difference in the two cases due to the discretization of high frequency lighting. However, it should be noted that despite the shift in the light positions the optimization scheme preserves the original relit result to a large extent demonstrating the robustness of the technique. (a) 9 SH (b) 10 local lights (c) 4 SH + 6 local lights (d) Ground truth Figure 3.11: Generalizing our approach to fewer lighting conditions. (a) 2 nd order SH lighting does not preserve high frequency specular highlights. (b) 10 local lights preserve high frequency highlights but suffer from low frequency bias. (c) Combining 1 st order SH lighting with 6 local lights (10 lighting conditions) achieves reasonable results. (d) Ground truth relighting with 253 lights. We also present combining SH lighting with local lights for an even more restricted budget of just 10 lighting conditions in order to test the generalization of the approach. As seen in Figure 3.11, our optimization procedure enables a reasonable relighting result even for such a small budget of lighting conditions (c). Here, we combine 1 st order SH lighting (4 images) with 6 local lights to achieve the relit result. For a similar budget of lighting conditions, the relit result suffers from low pass filtering with pure SH lighting (a), and significant low frequency bias with 10 local lights (b). This scenario would be particularly interesting for dynamic performance relighting applications where the available budget for lighting conditions is very limited even when employing high speed photography [7]. Given that the technique relies on a small set of local lights for high frequency lighting, it may suffer from aliasing artifacts for highly specular surfaces or scenes with complex light transport. The reliance on low order SH lighting for global lighting can result in slight overestimation of 25 low frequency lighting in the dark regions of EMs with very high dynamic range illumination. For environment maps with very low frequency illumination, our technique may not provide any advantage in relighting quality over pure SH lighting. However, the proposed lighting basis should still be useful for artistic editing of the lighting in such cases. While the technique currently works for lighting animation (see accompanying video), it may lead to temporally inconsistent results for time varying illumination if the optimization is run independently for each time step. However, it should be possible to enforce temporal coherence in this scenario with temporal regularization of the optimization. 26 (a) 4 th order SH (b) 20 local lights (c) Our technique (d) Ground truth Figure 3.12: Relighting comparison under the Grace Cathedral environment. (a) 4 th order SH relighting (25 lighting conditions) fails to accurately capture the high frequency highlights while also introducing color banding artifacts on specular surfaces. (b) 20 point lights capture the high frequency lighting effects, but do not accurately capture the low frequency lighting. (c) Our proposed technique for combining SH lighting with local lights (20 lighting conditions) captures both low frequency and high frequency lighting effects. (d) Ground truth image-based relighting consisting of 253 lights. Top-row: Helmet. Second-row: Kneeling knight. Bottom-three-rows: Plant. 27 (a) Environment maps (b) Optimized (c) Edited Figure 3.13: Relighting and editing examples in various environment maps (a). (b) Relighting result with proposed combination of SH and local lights (20 lighting conditions). (c) Edited relighting result. Top-row: Face in the Pisa EM. The orange bounce light from the wall has been sharpened while the direct skylight on the left has been diffused in (c). Center-row: Kneeling knight in Eucalyptus Grove EM. The global SH lighting has been reduced and the top sky light has been sharpened to create a more dramatic effect in (c). Bottom-row: Plant in the Kitchen EM. The direct window light from the right has been removed and the house light from the back has been intensified in (c). 28 Chapter 4 Acquiring Reflectance and Shape from Continuous Spherical Harmonic Illumination This chapter presents a novel technique for acquiring the geometry and spatially-varying re- flectance properties of 3D objects by observing them under continuous spherical harmonic il- lumination conditions. The technique is general enough to characterize either entirely specular or entirely diffuse materials, or any varying combination across the surface of the object. We em- ploy a novel computational illumination setup consisting of a rotating arc of controllable LEDs which sweep out programmable spheres of incident illumination during 1-second exposures. We illuminate the object with a succession of spherical harmonic illumination conditions, as well as photographed environmental lighting for validation. From the response of the object to the harmonics, we can separate diffuse and specular reflections, estimate world-space diffuse and specular normals, and compute anisotropic roughness parameters for each view of the object. We then use the maps of both diffuse and specular reflectance to form correspondences in a mul- tiview stereo algorithm, which allows even highly specular surfaces to be corresponded across views. The algorithm yields a complete 3D model and a set of merged reflectance maps. We use Sections of this chapter have been published in Acquiring reflectance and shape from continuous spherical har- monic illumination[53]. 29 this technique to digitize the shape and reflectance of a variety of objects difficult to acquire with other techniques and present validation renderings which match well to photographs in similar lighting. 4.1 Setup and Acquisition Figure 4.1: Spinning Spherical Reflectance Acquisition Apparatus consists of a light arc, object and cameras Our lighting apparatus is designed to illuminate an object at its center with any series of continuous spherical incident illumination conditions. The light is produced by a 1m diameter semi-circular arc (Fig. 4.1, a) of 105 white LEDs (Luxeon Rebels) which rotates about its central vertical axis using a motion control motor. As seen in cross-section in 4.3, each LED is focused toward the center with a clear plastic optical element which is aimed through two spaced-apart layers of diffusion. The diffusion allows the LEDs to form a smooth arc of light when they are all on, but baffles between the optics help each LED have only a local effect on the arc (top graphs 30 (a) Exposure for one rotation (b) Object lit by sphere of light Figure 4.2: Capture in progress shows the object being lit by the apparatus in Fig. 4.3). Since the arc spins through space more slowly near its top and bottom than at the equator, it would naturally produce more light per solid angle near the poles than from the equator. To counteract this, a curved aperture slit is applied to the arc which is 1cm wide at the equator and tapers toward 0cm wide at the poles proportional to the cosine of the angle to the center. The object to be scanned sits on a small platform at the center of the arc. The platform is motorized to rotate around the vertical axis yielding additional views of the object. Typically, the object is rotated to eight positions, 45 degrees apart. One version of the platform is a dark cylinder which can light up from LEDs mounted inside it; this version can measure an additional transparency map for objects such as eyeglasses. In front of the object and outside the arm is an array of five machine vision cameras (PointGrey Grasshopper 2.0)(Fig. 4.2, b) arranged in a plus sign configuration, with each camera being spaced about fifteen degrees apart from its neighbor(s). Each camera has a narrow field of view lens framed and focused on the object. 31 Figure 4.3: Cross-section of the LED arm and plots of measured vertical intensity profiles for six of the 105 LEDs (blue curves) and the nearly constant intensity achieved along the arm with all LEDs driven to equal intensity (red curve). In this work, we spin the arm at one revolution per second during which the intensities of the 105 LEDs are modulated to trace out arbitrary spherical illumination environments. Differences in LED intensity due to manufacturing are compensated for by calibrating the intensity of each LED as reflected in a chrome ball placed in the center of the device. We use pulse width modulation to achieve 256 levels of intensity with 400 divisions around the equator, allowing a resolution of 400105 pixel lighting environments to be produced. We expose each of the cameras for the full second of each rotation to record a full sphere of incident illumination as the arc rotates (Fig. 4.1, a). Additionally, we do one rotation pass where the object is only illuminated from the back and dark on the front in order to obtain masks for visual hull for subsequent stereo processing. The use of a spinning arm largely eliminates problematic interreflections which would occur if one were to surround the object with a projection surface. Our spherical lighting patterns minimize the brightness disparity between diffuse and specu- lar reflections compared to techniques with more concentrated illumination sources: a perfectly 32 specular sphere and perfectly diffuse sphere of the same albedo appear the same brightness under uniform light. Nonetheless, to optimally estimate reflectance properties of both high albedo and low albedo surfaces with low noise, we employ high dynamic range photography with 3 exposures per lighting condition, one and a half stops apart. The motion of the arc creates a vertical sliver of reflection occlusion when it passes in front of each camera (see the dark line down the center of the sphere in Fig. 4.2, d). In processing, we estimate this missing reflectance information from data in the neighboring views. Typical datasets in this work are captured in approximately ten minutes. 4.2 Reflectometry from Spherical Harmonics This section describes our algorithm to estimate per-pixel reflectance parameters for a single viewpoint of an object observed under SH illumination functions y m l (w)= y m l (x;y;z) up to 5th order where w is the unit vector (x;y;z) (Fig. 4.4). We assume a traditional reflectance model consisting of a Lambertian diffuse lobe D and a specular lobe S with roughness and anisotropy. Holding the view vector fixed, we parameterize the reflectance functions D(w) and S(w) by the unit vector w indicating the incident illumination direction. Of course, we do not observe D(w) and S(w) directly, but rather the responses of their sum f(w)= D(w)+ S(w) to the SH illumination functions. We denote these reponses f m l = R W f(w)y m l (w)dw. A key to our reflectometry technique is the observation by Ramamoorthi and Hanrahan [43] that a Lambertian diffuse lobe exhibits the vast majority of its energy in only the 0th, 1st, and 2nd- order spherical harmonic bands. We observe that this implies that the 3rd order SH coefficients and above respond only to the specular lobe S(w), so that S m l f m l for l 3, as seen in Fig. 4.5. We then estimate the specular lobe’s albedo, reflection vector, roughness, and anisotropy 33 l=0 l=1 l=2 l=3 m=3 m=2 m=1 m=0 m=1 m=2 m=3 Figure 4.4: The spherical harmonic functions y m l (w) up to 3rd order, seen from the top as reflected in a mirrored sphere, with x= y= 0 and z= 1 in the center. Positive values are shown in magenta and negative values are shown in green. Each harmonic locally resembles the harmonic above it around x= y= 0. parameters from the higher order responses by comparing to the higher-order responses of lobes from a reflectance model such as [57]. From the specular reflectance parameters, we can estimate the responses S 0 0 , S m 1 of the specular lobe to the lower-order harmonics, and subtract this response from the observations to estimate the response of just the diffuse lobe D(w) to the 0th and 1st order harmonics D 0 0 , D m 1 . From those, we can estimate the diffuse albedo and diffuse surface normal as in [30], yielding a complete model of diffuse and specular reflectance per pixel from a small number of observations. Specifically, our reflectance measurement process is as follows: 34 D S D+S lobe l=0 l=1 l=2 l=3 l=4 l=5 Figure 4.5: Responses of diffuse D, specular S, and mixed D+ S reflectance lobes to the first six zonal harmonics y 0 l (w); negative values are shown in absolute value. (Non-zonal responses are 0). Above second order, the diffuse response is small and the higher order responses to the mixed lobe closely match the response to the specular lobe on its own. 4.2.1 Acquiring Spherical Harmonic Responses We use our acquisition setup to acquire the responses of the object to the thirty-six SH illumination conditions up to the 5th order. Since the device cannot produce negative light, we offset and scale the SH functions above 0th order to produce two lighting patterns, one with pixel values between 0 and 255 and a complementary condition with pixel values from 255 to 0. The difference of these two images yields the response to the spherical harmonic. One could acquire fewer images by using the harmonics scaled 0 to 255, but our approach distributes camera noise more evenly throughout the range of intensities. 4.2.2 Building the Reflectance Table We compute the response of our chosen reflectance model’s specular lobe to the SH illumination basis over its range of valid roughness values. In this work, we arbitrarily choose the Ward [57] 35 model’s specular lobe f s a 1 ;a 2 and choose anisotropic roughness parametersa 1 a 2 ranging from 0 (perfectly sharp) to 0:35 (very rough) in increments of 0:005. We view the surface at normal incidence along the z-axis, choose a unit specular albedor s = 1, and align the axis of anisotropy to 0 degrees along the x-axis. We then numerically integrate the lobes against the SH basis to determine the coefficient table R m l (a 1 ;a 2 ) for each order l across the range ofa. 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0 0.5 1 R(0,m) responses 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0 10 20 R(0,3)/R(0,5) R(0,0) R(0,1) R(0,2) R(0,3) R(0,4) R(0,5) R(0,3)/R(0,5) (a) Plot of responses (b) Roughnessa 1 (c) Roughnessa 1 Figure 4.6: (a) Plots of the responses R 0 l (a;a) (with l= 0 in black) of an isotropic specular lobe to the first six zonal harmonics y 0 l as roughness increases from a = 0 to a = 0:35. The dotted line plots the ratio R 0 3 =R 0 5 which is used in determining specular roughness. (b, c) The rendered reflectance tables which map measured responses (u;v) = ( f 0 5 = f 0 3 ; f 2 3 = f 0 3 ) back to anisotropic roughness valuesa 1 (b) anda 2 (c). The response of the lobes to the SH basis has useful properties for reflectance measurement. Since isotropic lobes are radially symmetric, only the zonal SH basis functions y 0 l yield a nonzero response when a 1 =a 2 . Even when the lobe is anisotropic, it is symmetrical across both the x and y axes in that f s a 1 ;a 2 (x;y;z)= f s a 1 ;a 2 (x;y;z) so its response to the basis functions y 1 l (for l 1) are both zero since y 1 l (x;y;z)=y 1 l (x;y;z) and y 1 l (x;y;z)=y 1 l (x;y;z) (seen in Fig. 4.4). Furthermore, the response to the basis functions y 2 l (for l 2) are zero since they also have the property y 2 l (x;y;z)=y 2 l (x;y;z). The responses to y 2 l will be nonzero, however, when the lobe is anisotropic, since for small x and y, y 2 l (x;y;z) is positive whenjxj<jyj and negative when jxj>jyj, so lobes stretched more along x will have a negative response and lobes stretched more 36 along y will have a positive response to y 2 l . We will use this response to measure the anisotropy of the observed specular lobes. Fig. 4.6(a) shows the reflectance data R 0 l (a 1 ;a 2 ) for 0 l 5 for isotropic lobes where a 1 =a 2 . In practice, the elements of interest in our reflectance table R are the zonal responses R 0 3 and R 0 5 (for roughness) and the tessoral response R 2 3 (for anisotropy). When measured from a real surface, they will all be scaled by the unknown specular albedor s , so we divide each of them by R 0 3 to obtain the two independent measurements u= R 0 5 =R 0 3 and v= R 2 3 =R 0 3 which generally correlate with specular roughness and anisotropy, respectively, as will be discussed further in Sec. 4.2.4 and Sec. 4.2.5. Our calculations tell us values(u;v) for given values(a 1 ;a 2 ). When it is time for reflectance measurement, we will need to evaluate the inverse of this mapping. Fortunately, the mapping is smooth and monotonic over our ranges of roughness, which allows us to construct a fast inverse table lookup. For each ofa1 anda2, we scan convert the mesh of the values they take on into the range of(u;v) (Fig. 4.6 (b & c)). Then, when we measure values u and v, we can quickly look up the anisotropic roughness parametersa1 anda2 to which they correspond. 4.2.3 Estimating the Specular Surface Normal The specular lobe of our pixel’s reflectance function will be centered around a reflection vector r, implying a specular surface normal n s halfway between r and the view vector. We search for this specular peak to be located at the maximum of the l= 3 order SH reconstruction of the function: r= argmax w l å m=l f 3 m y 3 m (w) (4.1) Since we assume the specular lobe is relatively narrow, we note that rotating the lobe to align with the zonal +z axis will maximize its response to the zonal harmonic y 0 3 , which assumes its 37 global maximum along the+z axis. Around this same location, all other SH functions of the third order are close to zero. Thus, we observe that a narrow specular lobe’s projection into the 3rd order SH basis will resemble a rotated version of the 3rd order zonal harmonic itself, and thus will have a clear global maximum. Since finding where the SH reconstruction of a function attains its maximum is complicated to perform analytically, we use the quick-to-converge hill climbing approach of Sloan [51] to find r starting from six possible initial estimates. From r, we can calculate the world-space surface normal n s , and can rotate the other SH responses f m l using the (2l+ 1)(2l+ 1) SH rotation matrices so that the reconstructed specular lobe aligns with the zonal +z axis, yielding a set of rotated SH coefficients ˆ f m l . 4.2.4 Estimating the Angle of Anisotropy The responses ˆ f m l now measure the lobe as if it were aligned with the zonal axis, but if the lobe is anisotropic, the angle of anisotropy y could be anywhere. We would like to determine this angle, and further rotate the harmonics around the zonal axis to align the direction of anisotropy with the x axis so that it will match with our reflectance table. To measure the angle of anisotropy of a distribution g(x;y) centered about the origin, one typically computes the covariances s= R g(x;y)(x 2 y 2 )dxdy and t= R g(x;y)(2xy)dxdy, where s characterizes the spread of values along the x-axis versus the y-axis and t characterizes the spread of values along versus the x= y diagonal versus the x=y diagonal. Then, the angle of anisotropy can be found as 1 2 tan 1 (s=t). Conveniently, we observe that the ˆ f 2 3 and ˆ f 2 3 harmonic responses essentially perform these integrations for us (visualized in Fig. 4.7), up to a scaling factor, since from the spherical harmonic formulae in the neighborhood of the zonal peak z= 1 are: 38 y 2 3 (x;y;z)= 1 4 r 105 p 2xyz 1 4 r 105 p 2xy (4.2) and y 2 3 (x;y;z)= 1 4 r 105 p (x 2 y 2 )z 1 4 r 105 p (x 2 y 2 ) (4.3) Thus, the angle of anisotropy can be computed as y = 1 2 tan 1 ( ˆ f 2 3 = ˆ f 2 3 ). If we were to rotate the harmonic responses around the z axis by y to form ˆ ˆ f m l , we note that the ˆ ˆ f 2 3 response be- comes zero and the ˆ ˆ f 2 3 response becomes q ( ˆ f 2 3 ) 2 +( ˆ f 2 3 ) 2 . We have simply calculated the angle and magnitude of the vector ( ˆ f 2 3 ; ˆ f 2 3 ). This magnitude is our indication of how anisotropic the specular lobe is relative to its roughness. S(x;y) S(x;y) ˆ y 2 3 S(x;y) ˆ y 2 3 S(x;y) ˆ y 0 3 S(x;y) ˆ y 0 5 Figure 4.7: Responses of an anisotropic lobe S to the rotated SH functions ˆ y m l whose responses together determine estimates of the anisotropic angley, roughnessesa 1 anda 2 , and the specular albedor s . 4.2.5 Estimating Roughness and Anisotropy Ideally, to measure the roughness of the rotated specular lobe ˆ ˆ S(x;y;z), we would compute its second moment as the following integral: Z ˆ ˆ S(x;y;z)(x 2 + y 2 )dxdy (4.4) 39 Notably, the zonal harmonics for l> 1 perform something similar to this integration, visual- ized in the right images of Fig. 4.7. For example, for l= 3, near the apex z= 1: y 0 3 (x;y;z)= 1 4 r 7 p z(2(x 2 + y 2 )) 1 2 r 7 p 1 4 r 7 p (x 2 + y 2 ) (4.5) Thus the zonal harmonic approximation has an (x 2 + y 2 ) term but also a constant term. If we knew the response of the specular lobe S to the 0th order harmonic y 0 0 , we could subtract off the response to the constant term. Unfortunately, we only know the response of y 0 0 to D+ S. To resolve this problem, we can look to a higher zonal harmonic we capture such as y 0 5 : y 0 5 (x;y;z)= 1 16 r 11 p z(8 56(x 2 + y 2 )+ 63(x 4 + 2x 2 y 2 + y 4 )) (4.6) 1 2 r 11 p 7 2 r 11 p (x 2 + y 2 ) (4.7) To make this approximation, we set the 4th order terms to zero (and drop z 1). This approx- imation also contains a constant term and an(x 2 +y 2 ) term, with linearly independent coefficients from the y 0 3 , approximation, meaning that one can theoretically determine the integral of the spec- ular lobe S against the constant term and the(x 2 +y 2 ) term from the responses ˆ ˆ f 0 3 and ˆ ˆ f 0 5 , yielding measures of the specular lobe’s albedo and roughness. However, since these polynomial expansions are approximate, and since our reflectance model’s parameters may or may not relate closely to these same measures of roughness and anisotropy, we employ a lookup table to determine which specular model parameters will produce the same re- sponses as we find in our captured data. But the derivations above explain why the measurements we seek are contained in the available data. 40 We return to our reflectance function’s rotated SH responses ˆ ˆ f 0 3 , ˆ ˆ f 0 5 , and ˆ ˆ f 2 3 . As discussed ear- lier in Sec. 4.2.2, these will all be proportionately scaled by the specular albedo, so we normalize them by dividing by the 3rd order zonal response ˆ ˆ f 0 3 to obtain(u;v)=( ˆ ˆ f 0 5 = ˆ ˆ f 0 3 ; ˆ ˆ f 2 3 = ˆ ˆ f 0 3 ). From the precomputed reflectance table of Sec. 4.2.2, look up which anisotropic roughness parametersa 1 anda 2 yield a specular lobe in our reflectance model which has the same response to the rotated harmonics. Note that we could in principle similarly use the 4th order zonal harmonic y 0 4 instead of the 5th order harmonic. In practice, we prefer the 5th order harmonic because of it provides a greater contrast to the 3rd order measurement and hence better conditioned meaurements. Also, the 4th order measurement, being an even order harmonic, still has some diffuse response in the signal making it less suitable for this purpose than the the 5th order measurement (Fig. 4.5). 4.2.6 Estimating Specular Albedo As shown above, the responses ˆ ˆ f 0 3 and ˆ ˆ f 0 5 of our specular lobe to the rotated zonal harmonics contain information about both specular roughness and albedo. With the roughness valuesa 1 and a 2 determined, we refer to our tabulation R (Fig. 4.6) to determine the 3rd order zonal response to a unit specular albedo lobe with our lobe’s roughness parameters. Dividing our lobe’s 3rd order zonal response yields our estimate of the specular albedor s = ˆ ˆ f 0 3 =R 0 3 (a 1 ;a 2 ). 4.2.7 Estimating Diffuse Albedo and Normal We have now fully characterized our specular lobe with specular albedo r s , normal n s , angle of anisotropy y, and anisotropic roughness parameters a 1 and a 2 . From r s , a 1 , and a 2 , we can determine the 0th and 1st order responses to our lobe from the table R m l (a 1 ;a 2 ). We use spherical harmonic rotation to rotate the 1st order responses by the angley around the z axis and then rotate 41 Reflectance function Roughness(Ward BRDF) R 0 3 (a) R 0 5 (a) 0 (Mirror-like Specular) 1.0 1.0 0.005 0.9691 0.9254 0.01 0.8886 0.7435 0.02 0.6356 0.3165 0.03 0.3678 0.0708 Table 4.1: Examples of responses from different isotropic lobe a to the 3rd and 5th order har- monics. 42 to align the lobe’s center to the reflection vector from the view vector to n s . We subtract these rotated responsesr s ˆ R m l (a 1 ;a 2 ) from our observations f m l (which include the response to both the diffuse and specular lobes) to estimate the response to the diffuse lobe on its own. From the 0th and 1st order responses D m l to the diffuse lobe, it is straightforward to estimate the diffuse normal n d as (D 1 1;D 0 1 ;D 1 1 ) (which should be normalized) and albedo as 1 p D 0 0 =y 0 0 , where the 1 p factor divides out the integral of a unit Lambertian lobe over the sphere and y 0 0 is the constant value of the 0th order harmonic 1 2 q 1 p . 4.2.8 Reflectometry Discussion Our diffuse and specular albedo separation technique is inspired from frequency based diffuse- specular separation work like [28]. Ramamoorthi and Hanrahan [45] propose that the first two order of spherical harmonics can estimate diffuse reflectance with 99% accuracy, which means that most of diffuse component has lower frequency than 2nd order spherical harmonics. In our work, we use sphrical harmonics order higher than 2 to separate diffuse and specular albedo. Figure 4.8 shows that, by using higher order harmonics, the high frequency specular component got separated better. We do not consider the effect of Fresnel gain, which makes grazing lobes considerably brighter. This could be accounted for using the Fresnel terms from a physically based reflectance model and estimates of the index of refraction of the materials. In our setup, we capture enough view- points to view most surface normals at an angle where Fresnel gain is minor, and our map merging process assigns frontal reflectance data the highest weight. Our technique for measuring specular surface normal is different from Ma et al. [30] and Our technique for measuring roughness and anisotropy is different from Ghosh et al. [11], where they use the lower order harmonics for reflectance analysis. They also require diffuse reflection to be 43 0th order 1th order 2th order 3th order 4th order 5th order Figure 4.8: Comparing separated specular albedo from different order of spherical harmonics from the 0th order to 5th order. The higher order harmonics show better separation where the shiny specular reflection from the lens is maintained while the diffuse reflection from the frames and background is gone. 44 eliminated though polarization difference imaging, and integrate against the equatorial region of the zonal harmonic rather than the zonal peaks. We believe our technique is more general since it can be applied to higher-order harmonics (which exclude diffuse reflections without polarization) and can obtain better-conditioned estimates of sharp specular behavior. 4.3 Geometry Reconstruction from Diffuse and Specular Reflections Discussion Once the reflectance maps are acquired and processed for each of the viewpoints, we have esti- mates of the spatially varying diffuse and specular albedo and diffuse and specular normal maps (which correspond to diffuse and specular reflected directions on the illumination sphere). Our acquisition setup has five color cameras in a ”plus sign” arrangement with fifteen degrees be- tween views, and a motorized platform rotates the object about the vertical axis at 45 increments to provide views all around the object. We estimate a 3D surface model for the object using the maps from all of these views using a geometry reconstruction algorithm that leverages multi-view stereo correspondence and the surface normal estimates from the reflectance analysis. 4.3.1 Stereo Reconstruction We solve for the object’s shape beginning with an initial approximate base mesh of the geometry obtained from either sparse stereo correspondence and Poisson reconstruction (e.g. [8]), shape- from-silhouettes, or simply a cylinder at the approximate location of the object being scanned. We define a displacement map over the vertices of this base mesh, refining its estimated shape by displacing the base vertices along their normals. 45 Diffuse albedo Specular albedo Surface normal Figure 4.9: Stereo reconstruction takes multiple channels of data, diffuse albedo, specular albedo and surface normal, as inputs. In our setup, we have 5 cameras mounted ridigly to our structure, and thus the calibration between these 5 cameras is precise. However, we have multiple copies of these cameras with dif- ferent scan object rotations, which are not precise enough for stereo matching. Thus we compute matching costs within sets, and compute a weighted average of the costs, weighted by w k;i for the center camera k of each set. The matching cost for each set is normalized cross correlation (NCC) with a 3 3 window aligned in space to the surface normal n i . The NCC cost is a weighted average over the cameras (weighted by w k;i ) and scaled by the window variance in a primary view to avoid undue influence from noisy low-intensity pixel values. The primary view is the view within the set that is most facing n i . 46 (a) Diffuse (b) Specular (c) Spec normal (d) Roughness Figure 4.10: Estimated reflectance maps for a plastic ball. We average the NCC cost over 10 data channels (diffuse albedo RGB, mean specular albedo, mean diffuse normal XYZ, and mean specular normal XYZ), and truncate the cost to 1 for ro- bustness to outliers. Vertex positions outside of the visual hull are given a constant cost of 1. After the geometry reconstruction is complete, we blend the various channels of reflectance data using similar weights as the photometric normal. Please refer to more detail in the stereo reconstruction section in [53]. 4.4 Results and Discussion We now present some results of scanning objects with complex varying reflectance using our tech- nique. All viewpoints were recorded under SH lighting up to fifth order, for 2 36 photographs per camera per view (though the 2 14 images from the 2nd and 4th orders are not used). Figure 4.10 shows recovered maps for the shiny red plastic ball seen in Fig. 4.2 under uniform illumination. The maps show good diffuse/specular separation and the normals trace out the spherical shape, except in areas of reflection occlusion near the bottom. The roughness is low and consistent across the ball, but becomes especially low around the periphery presumably due to Fresnel gain, which can be observed in the specular map. 47 (a) Diffuse (b) Specular (c) Specular normal (d) Anisotropic angle (e) Roughnessa 1 (f) Roughnessa 2 (g) Rendering (h) Photograph Figure 4.11: Maps, rendering, and photo for brushed metal. 48 (a) Acquisition Setup (b) Reflectance Mapsr d ,r s , n s ,a (c) Rendered 3D Model Figure 4.12: (a) A pair of sunglasses lit by continuous spherical harmonic illumination (harmonic y 2 3 (w)) in our capture setup. (b) Recovered diffuse, specular, specular normal, and specular roughness maps. (c) A rendering of the sunglasses with geometry and reflectance derived from the SH illumination and multiview reconstruction. Fig. 4.11 shows maps recovered for an arrangement of five slightly bent anisotropic brushed metal petals at different angles. The maps exhibit four different angles of anisotropy. The major- axis roughness a 1 is consistent for all five petals, and the minor-axis roughness a 2 is sharper overall. The rendering and validation photograph, both under point-source illumination, are still reasonably consistent, though the tails of the specular lobes are wider in the photograph. This is likely because a the tendency of the Ward model lobe to fall to zero too quickly; a different reflectance model might better represent this reflectance. Fig. 4.12 shows the scanning process for a pair of sunglasses with several materials of varying roughness on the frames and strongly hued lenses with a mirror-like reflection. The reflectance maps correctly show very little diffuse reflection on the lenses and spatially-varying roughness on the ear pieces. The glasses were scanned in five poses for twenty-five viewpoints total. The diffuse and specular albedo and diffuse and specular normals were used in the multiview shape reconstruction shown in Fig. 4.13, providing merged maps in a cylindrical texture space from all 25 views. The geometry successfully reconstructs both the diffuse ear pieces and the mirror- like lenses in a single process. Fig. 4.13 also provides a ground truth comparison of the rendered 49 Figure 4.13: Two views of the 3D geometry for sunglasses (left), with a validation rendering (lower middle) and photo (right) with environmental illumination created by the LED arm. glasses to the real pair of glasses lit by environmental illumination, and an animation of the glasses is shown in the accompanying video. Fig. 4.14 shows maps and geometry for a digital camera with several different colors and roughnesses of metal and plastic and an anisotropic brushed metal bezel around the lens. The maps successfully differentiate the materials and allow the renderings to give a faithful impression of the original device. Finally, Fig. 4.15 presents an error analysis for shape and reflectance measurement for differ- ent types of spheres with the presented acqusition setup. As can be seen, our technique is able to correctly separate diffuse and specular reflectance for a mirror sphere, a red metallic rough spec- ular sphere, and a diffuse sphere. Fig. 4.15 also presents plots of deviation in measured surface orientation and surface geometry compared to an ideal sphere, as well as rendering under point light illumination compared to validation photographs. As can be seen, the reconstruction error is low near the equatorial regions of the spheres that are unoccluded and well sampled by the camera viewpoints and higher near the poles due to occulsions (bottom) and insufficient views for stereo (top). The reconstruction error near the poles could be reduced with data aquisition from additional viewpoints. 50 (a) diffuse (b) specular (c) specular normal (d) roughness (e) rendering (f) photograph Figure 4.14: Reflectance maps for a 3D model of a digital camera and a rendering and photograph with environment lighting. 51 Mirrored Sphere Rough Specular Sphere Diffuse Sphere Figure 4.15: Analysis of shape and reflectance measurement for different types of spheres. Left two columns: Mirrored Sphere. Middle two columns: Rough Specular Red Sphere. Right two columns: Diffuse Sphere. Top row:(left) diffuse albedo (right) diffuse normal. Second row:(left) specular albedo (right) specular reflection vector. Third row:(left) specular reflection vector de- viation from ideal sphere (blue= 0 , yellow= 5 , red 10 ) (right) specular roughness (dark is sharp specular, light is broad specular). Fourth row:(left) reconstructed geometry (right) geome- try deviation from ideal sphere (blue= 0mm, yellow= 0:25mm, red 0:5mm; sphere diameter ranges from 7cm to 8cm). Fifth row:(left) validation photograph (right) point light rendering. 52 Chapter 5 Acquiring Reflectance from Monitor-based Basis Illumination This chapter presents an alternative option from that of the previous chapter for acquiring the spatially-varying reflectance properties of flat samples with minor surface orientation variation using an LCD monitor. We propose a hardware setup which can be built from off-the-shelf LCD monitors and cameras. The system does not have moving parts, which can greatly reduce the ac- quisition time compared to other systems. We illuminate the samples with low frequency Fourier transform illumination to encode the reflectance function of the samples, in the form of Fourier transform coefficients. Then we recover the reflectance properties, which include diffuse and specular albedo, surface normals and specular glossiness, by analyzing the Fourier transform co- efficients of different frequencies. The results in per-pixel reflectance maps are able to be used to recreate the samples under any kind of lighting environment. 5.1 Setup and Acquisition Our setup consists of a 30-inch LCD monitor, a monitor mount and a camera. We use a com- puter monitor instead of a TV monitor because of the dynamic back-light feature in regular TV 53 Figure 5.1: Our hardware setup consists of an LCD screen and a camera with shift lens monitors. This feature changes the intensity of the LED back light of the monitor dynamically depending on the scene, in order to increase the contrast ratio, which makes it not suitable for use as an illumination source for our setup. The monitor is mounted 45 from the sample. We point the camera directly down at the sample to maximize the depth of field and use a shift lens to avoid blocking the screen as shown in figure 5.1. The sample is placed close to the screen to increase coverage. Therefore, a local illumination model is used in order to estimate reflectance information. This also introduces a problem from an LCD viewing angle stability standpoint, which is discussed in the next section. 54 5.2 Acquisition This section describes the acquisition process of our system. We use a vision research camera which can capture images at 9 frames per second. At this speed, the refresh rate of the moni- tor, 60Hz, does not interfere with the acquisition. However, if a higher speed of acquisition is necessary, one should keep in mind exposing the camera for at least 2-3 refresh cycles of the monitor for each illumination. We project the basis illumination on the monitor screen starting from low frequency and shoot successively using the camera. The detail of the basis illumination is discussed in the next section. 5.2.1 Basis Illumination The key idea underpinning the use of basis illumination to extract reflectance information is that we are able to observe the results of a convolution between the illumination and the reflectance function when we take a photograph of the sample. E(x)= Z W L(x;~ w i ) f(x;~ w i ;~ w r )(~ w i ~ n)d~ w i (5.1) E(x)= Z W L(x;~ w i )F(x;~ w i )(~ w i ~ n)d~ w i (5.2) From the rendering equation 5.1, if the viewing direction is fixed, the BRDF f(x;~ w i ;~ w r ) can be changed into a reflectance function F(x;~ w i ) in 5.2. We can see that the observed pixel E(x) is a convolution result of F(x;~ w i ) and the basis illumination L(x) over the hemisphere. By observing many convolution results, also called basis coefficients, under different illumination conditions, we are able to use them to reconstruct the original reflectance function. However, it requires hundreds of basis coefficients in order to reconstruct a reflectance function, especially 55 one with the high frequency specular function, that is accurate enough for relighting. Therefore, we make as assumption that the reflectance function of an opaque surface consists of only diffuse and specular reflections. We can then fit any physically-based BRDF models to the measure basis coefficients. As a result, we employ 2-D Fourier transform as our illumination basis, b(x;y)= e i2p(ux+vy) (5.3) where i is unit imaginary number, u is the number of cycles (frequency) along the x-axis and v is the number of cycles along the y-axis of the LCD screen. Fourier transform is translation and scaling invariant, which is suitable for capturing the reflectance function. We illuminate the sample with Fourier patterns from 0 to 5Hz (5 cycles across the screen), as shown in figure 5.2. The patterns are shifted to a 0 to 1 range before being displayed on the LCD screen. We also capture the inverse patterns in order to increase the signal to noise ratios. The intuition behind 5Hz is that, it is high frequency enough to separate diffuse and specular reflections. There is also enough information from the basis coefficients to estimate reflectance functions by making some assumptions about the functions. 5.3 Reflectometry from Fourier Transform This section presents our reflectance analysis algorithm that is able to extract spatially-varying reflectance parameters of the samples, observed under Fourier transform illumination (Fig 5.2). The main idea of this technique is similar to section 4.3, where we use basis functions to ex- tract reflectance information in the form of basis coefficients. Then, we use these coefficients to 56 Figure 5.2: Our illumination patterns, Fourier transform basis generated for equation 5.3, start from 0 to 5Hz. The patterns are shifted to be within the 0 to 1 range to display on the LCD monitor. reconstruct the original reflectance function, with the assumption that the function is a smooth non-occluded function with only one specular lobe and has no inter-reflection. 5.3.1 Estimating Diffuse and Specular Albedo For each frequency of the Fourier transform, we can estimate the amplitude of the signal by simply calculating the absolute value of the complex coefficients, A u;v =kF u;v k= q R(F u;v ) 2 +I(F u;v ) 2 (5.4) 57 where A u;v is the amplitude and frequency u;v, F u;v is the complex coefficients,R(F u;v ) is the real number part of the coefficient andI(F u;v ) is the imaginary part of the coefficient. Although we can recover the amplitude of the signal, the recovered amplitude is not of the real signal, but rather the response of the real signal to that specific frequency. The reason is, we generate the coefficients by taking the forward shifting of the frequency, and subtract with the inverse shifting of the same frequency. Any low frequency signal that does not change between the forward and the inverse is eliminated. We recover the real signal, by observing at the amplitude of different frequencies and estimating the one of the original function. This recovered amplitude is a specular albedo shown in figure 5.3(c). We can then recover the diffuse albedo by subtracting the specular albedo from the constant illumination. The constant illumination and recovered diffuse albedo are shown in 5.3(a) and (b) respectively. (a) Uniform illumination (b) Separated diffuse albedo (c) Separated specular albedo Figure 5.3: Results from diffuse and specular albedo estimation by recovering the amplitude of the signal from Fourier transform 5.3.2 Estimating Specular Reflection Vector The key to estimating reflection vectors, is to use the phase of each Fourier frequency component. The phase of each frequency tells us where the center of the function lies in spatial space. In other 58 words, the phase of each frequency indicates which part of the LCD screen is reflected on the sample. These phases can be calculated from this equation P u;v = arctan( R(F u;v ) I(F u;v ) ) (5.5) Figure 5.5(a) shows the phase of one of the frequencies from the Fourier transform. By combining phases from many frequencies, we can accurately recover the reflection direction in spatial space of the LCD screen 5.5(b) which is then converted to a world space reflection map 5.5(c). (a) Phase image (b) Screen-space reflection (c) Surface normal Figure 5.4: The process for estimating surface normals of the sample starting with calculating the phase image (a) for each frequency, then finding the reflection map (b) in screen coordinate and converting it to a surface normal map (c). 5.3.3 Estimating Specular Roughness Specular roughness, or specular glossiness, refers to how shiny or dull the texture appears. The main idea in estimating specular roughness concerns the differences of response of specular func- tions to each frequency of Fourier transform. Low specular roughness responds stably from low to high frequency, while high specular roughness responds significantly less in high frequency 59 Fourier transform. Therefore, we are able to recover specular roughness using similar techniques to section 4.2. We first find the amplitude of the responses to different frequencies of Fourier trans- form. Then, based on the changes of the amplitudes, we compare these with known reflectance models and find the parameters that best describe the measure data. (a) Diffuse albedo (b) Specular albedo (c) Surface normal Figure 5.5: Reflectance map results from the monitor-based scanner. The first row is a piece of yellow cloth with some specularity. The second row is a rough specular surface with green plastic tape on the upper left and white label paper on the bottom 5.4 Discussion In this alternate setup to section 4.1, we develop a similar technique by projecting continuous basis illumination on the sample to estimate reflectance properties. However, due to the partial coverage 60 of the LCD monitor instead of full hemisphere coverage as in section 4.1, the low frequency components, like diffuse albedo, are affected the most. This problem can be solved by capturing a calibration object with a uniform Lambertian surface and applying the correction ratio to the sample. Our technique for estimating reflectance properties is different from Aittala et al. [1], where they also use Fourier transform on LCD to estimate reflectance properties. The main difference is that, instead of estimating each reflectance property individually per pixel, they run the op- timization to fit the whole BRDF to the Fourier coefficients. They also use a multi-resolution scheme to improve their optimization performance, which requires a smoothing constraint that can sometimes introduce artifacts. Our technique for estimating specular roughness also differs from Ghosh et al. [11], where they extract the roughness directly from the second order statistic of spherical harmonics. Our technique observes the changes in the responses to different frequencies of Fourier transform. 5.4.1 Monitor Calibration There are many aspects which make the monitor not an ideal illumination device and it needs to be calibrated. Non-linearity Regular consumer LCD monitors normally have non-linearity characteristic, where a pixel of value 50 is not half the intensity of a pixel of value 100. This can be calibrated by projecting uniform pixel values from 0 to 255 and taking a photograph of each value. Then, plot the intensity of each value out, fit a line to the curve and find the scaling value from the curve to the line. The scaling values are applied to every pattern that is projected on the screen. 61 Viewing-angle Brightness Variation Most LCD monitors display different brightness when viewed from different angles, as seen in figure 5.6. This is because the back-light system and the polarizer inside the screen do not work uniformly across the screen. We compensate for this effect by using a photograph of a mirror ball that is placed at the sample location and illuminated with the screen, to calculate for a scaling factor to make the illumination more uniform across the screen. Figure 5.6: An image of a hemispherical chrome ball shot from the setup showing the variations in brightness on the screen when viewed from different angles 62 Chapter 6 Acquiring Reflectance and Shape from Rapid Point Source Illumination This chapter presents a technique for acquiring facial reflectance and geometry using multiple consumer cameras and flashes. The flashes are fired in rapid succession, synchronized with a set of cameras, which produce uniformly spread illumination around the face. The reflectance information is derived from this set of photographs. The final surface detail is obtained from an optimization procedure which finds the shape that best explains the reflectance information and photographs. 6.1 Hardware Setup and Capture Process The main focus of this hardware setup is to record the subject under different point source illumi- nation conditions from as many views as possible within a fix time budget. The criteria of the time budget is, the entire acquisition process cannot exceed a latency of eye-blink reflex from visual stimuli, which is varying between 60-100 milliseconds (ms) based on different individuals [?]. In this setup, we use consumer camera ring flashes with diffusers as our point source illumi- nations. We capture the subject from 24 different views, which cover 180 horizontally and 90 63 vertically in front of the subject. Each camera takes a picture with 5ms exposure time, synchro- nize with the flash. We also use a custom-built trigger circuit which triggers the camera rapidly and consecutively, such that the whole capture process finishes with in 66ms. Some of the views share the same flash, with the intuition that both views see the same diffuse reflection but different specular highlights. These photographs are processed in order to get a rough based mesh and the separated specular signal of the subject. For more detail discussion on the choice of equipments, the intuition behind the camera and flash arrangement and diffuse-specular separation, please refer to Paul Graham’s Ph.D. thesis ”A Framework for High-Resolution, High-Fidelity, Inexpensive Facial Scanning”. Figure 6.1: Facial capture setup, consisting of 24 entry-level DSLR cameras and six with diffused ring flashes, all one meter from the face. 64 6.2 Deriving Reflectance and Geometry From the acquisition process, we obtain 24 flash-lit photographs, from different views and illumi- nation. Some of the views are lit from the same flash which creates consistent diffuse reflection, making them suitable for passive stereo reconstruction. We use PMVS2 to construct a rough esti- mate geometry of the subject. The geometry is then used for diffuse-specular separation, yielding the separated specular images in Fig. 6.2 (c). For more detail discussion on diffuse-specular separation, please refer to Paul Graham’s Ph.D. thesis ”A Framework for High-Resolution, High-Fidelity, Inexpensive Facial Scanning”. (a) Rough geometry from PMVS2 (b) Diffuse component (c) Specular component Figure 6.2: Results from rough geometry estimation and diffuse-specular separation. 6.2.1 Specular Photometric Stereo Given multiple observed pixel values p i of a surface point under differing illumination directions ~ l i , it is possible to recover the surface normal~ n and albedo r by leveraging certain assumptions about the reflectance properties of the surface. This process is known as photometric stereo [63]. 65 Figure 6.3: (Left) The result specular normal map from specular photometric stereo using specular images on the right as inputs. (Right) Specular components from different views The photometric stereo equations are presented with a distant light assumption, and light intensity p. If the actual distances r i to the light sources are known, and the intensities I i are known, then the pixel values can be adjusted to conform to the assumptions by multiplying them by pr 2 i =I i before proceeding with photometric stereo. We review the photometric stereo equations for exposition. In the Lambertian case, the light- ing equation is L ~ b = P, where L= ~ l 1 ~ l 2 ~ l k > , ~ b =r~ n, and P= p 1 p 2 p k > . Importantly, any i with p i = 0 are omitted, as the lighting equation does not hold. The solution via pseudoin- verse is: ~ b =(L > L) 1 L > P: (6.1) 66 In the Blinn-Phong case, the lighting equation is expressed in terms of halfway vectors ~ h i instead of lighting directions, and is more complicated. p= (a+ 8) 8 r (~ n ~ h) a (6.2) The dot product has an exponenta, and an associated normalization factor to conserve energy, leading to the following: H~ g = Q; where H = ~ h 1 ~ h 2 ~ h k > ; ~ h i = ~ v i + ~ l i j~ v i + ~ l i j ; Q= p 1 a 1 p 1 a 2 p 1 a k > ; and~ g = (a+ 8) 8 r 1 a ~ n; (6.3) with v i the direction towards the viewer, and a the Blinn-Phong exponent. The solution via pseudoinverse has the same form: ~ g =(H > H) 1 H > Q: (6.4) We now consider the dense photometric stereo case, with a large number of evenly spaced halfway vectors. In the limit (k!¥) we integrate ~ h over the hemisphereW instead of summing over ~ h i : 1 k (H > H)! 1 R W dw ~ h Z W ~ h ~ h > dw ~ h = 1 3 I; (6.5) 1 k H > Q! 1 R W dw ~ h Z W ~ hp( ~ h) 1 a dw ~ h = 1 3 (a+ 8) 8 r 1 a ~ n; (6.6) 67 which remains in agreement with (6.4). However, if the exponenta is unknown, and an erroneous value ˆ a is employed, we observe: ˆ g =(H > H) 1 H > ˆ Q! 3 a= ˆ a+ 2 (a+ 8) 8 r 1 ˆ a ~ n: (6.7) While~ g and ˆ g differ in magnitude, they share the same direction. This allows us to state the Enough Lights Theorem: Theorem 1 Enough Lights Theorem: In the limit of dense, evenly spaced halfway vectors, spec- ular photometric stereo recovers the surface normal without knowledge of the specular exponent. In practice, noise in the data and small errors in the diffuse-specular separation often crush the specular component to zero, even when the ideal Blinn-Phong model has a small positive value. One might try omitting these zero values (as in the Lambertian case), but most of the halfway vectors would go unused, rendering Theorem 1 unapplicable. Therefore care must be taken not to erroneously omit halfway vectors when evaluating (6.4). Our approach is to include all available halfway vectors within a hemisphere defined by the surface normal of the Lambertian reflectance component, discarding halfway vectors that are occluded with respect to the base mesh. Remark- ably, the integral in (6.7) attains the same value regardless of the orientation of the hemisphere W, partly because the ~ h~ n term is not clamped to zero. In practice, the orientation of the hemi- sphere must be close enough to the true surface normal so that the values which should have been clamped to zero are small nonetheless, motivating our choice of using the Lambertian reflectance component surface normal. We use an ˆ a value of 50 to compute the specular surface normals in our results, which is typical of human skin and appears to be close enough to obtain reasonable surface normals given the density of illumination provided by our apparatus. 68 6.3 Cost Volume Construction We build a cost volume representing diffuse photoconsistency and specular reflection consistency using a face sweep algorithm on the GPU, analogous to plane sweep algorithms used for tradi- tional stereo. Our final refined face mesh will be represented as a displacement map, displacing vertices of the base mesh along its surface normal directions. We leverage the GPU by processing the cost volume in small increments of displacement across the entire face, from2:5mm below the base mesh to 2:5mm above the base mesh in 50mm increments. At each increment, we com- pute one layer of the cost volume by rasterizing the face mesh with the uniform displacement for the cost layer. Using a fragment shader implemented in GLSL that operates on a single point in space, we perform the diffuse-specular separation, color-subspace photometric stereo, and specular photo- metric stereo steps described in Secs. 6.2.1. We compute diffuse photoconsistency by relighting the diffuse component for each lighting condition using the diffuse albedo and normal, and com- puting the sum-of-squared-difference cost against the corresponding input images, weighted by the cosine between the chroma normal and view vector. (a) (b) (c) (d) Figure 6.4: Optimizing Geometry from Differing Specular Highlights. (a-c) Three adjacent spec- ular highlights on a forehead, color-coded, illuminating different sets of surface normals. (d) The sum of the specular highlights projected onto the optimized model, fitting the highlights together like a puzzle to minimize the specular variance per pixel. 69 Along with diffuse cost, we also add a measure of specular reflection consistency, as we want the specular reflections to plausibly belong to the same surface. (See Fig. 6.4.) This is complicated by the fact that each image sees different specular highlights corresponding to the different halfway vectors. However, because the highlights are evenly distributed, the specular highlights projected from the different views should fit together like interleaved pieces of a puzzle, with minimal overlap, when the surface geometry is correct. Thus, we can minimize the extent to which the specular highlights from the different halfway vectors fall on top of each other when projected onto the model. We compute this cost as the weighted angular variance of the halfway vector of each view, weighted by the specular intensity observed in the view. Misalignments will generally result in overlapping highlights, leading to greater variance. The final cost saved to the cost volume sums the diffuse cost and specular cost, where the weights are the overall diffuse intensity and specular intensity in the image data. For robustness, we clamp the cost to an upper threshold. 6.4 Results We used our system to acquire a variety of subjects in differing facial expressions. Figures ?? shows the high-resolution geometry and several renderings under novel viewpoint and lighting conditions using our algorithm. The recovered reflectance maps used to create one of the faces are shown in Fig. 6.5. While not entirely without artifacts (which could be fixed with modest manual effort), our algorithm produces geometric quality which is competitive with more complex systems and reflectance maps not available from single-shot methods. 70 (a) Diffuse albedo (b) Diffuse normals (c) Specular albedo (d) Specular normals Figure 6.5: Recovered reflectance maps from the system running entirely on GPU. These maps will be used to calculate cost volumn, which is passed on to the face sweep process to generate the final geometry. 71 Figure 6.6: (Upper left) Final geometry result from the cost volume optimization process. (Upper right, Lower left, Lower right) Renderings of recovered geometry and reflectance maps under novel viewpoints and lightings. 72 Figure 6.7: Additional results from the system. (Left) Recovered geometry from the cost volume optimization process. (Right) Renderings of the subjects under novel viewpoints and lightings with recovered geometry and reflectance maps. 73 Chapter 7 Conclusion and Future Work 7.1 Conclusion This dissertation presents a 3-D digitizing framework to acquire the shape and reflectance of ob- jects which can be used to faithfully reproduce such objects from any viewpoints and illumination condition. This framework is designed to both handle various types of objects and to produce end results compatible with many applications. We focus on obtaining complete 3-D models and high fidelity relightable information. In chapter 3, we propose a framework to acquire a relightable and editable model of objects from spherical harmonic and point-source illumination. This framework is suitable for any kind of object on applications that require only relightable models. We use LightStage to produce the basis illumination. This proposed technique requires an order of magnitude of fewer numbers of images than the traditional techniques. This technique greatly improves the acquisition speed, which makes this suitable for dynamic performance acquisition. The spherical harmonics and reduced number of images also give artists more possibilities to alter the relightable models. For applications that require a complete 3-D model with reflectance information, we propose a digitizing system in chapter 4 to acquire the shape and reflectance of small objects. This system 74 can handle objects with material that ranges from shiny specular to dull rubber. The objects are captured under continuous spherical harmonics illumination created by a semi-circular LED rotating around the object. View-independent reflectance maps are derived from this process and then used in the multi-view stereo reconstruction process to create a complete 3-D shape of the object. The 3-D models and reflectance information can be used to recreate the objects from any viewpoint and under any type of illumination. We extend the idea of using continuous basis illumination from chapter 4 and adopt it on a cheaper setup that requires only an LCD monitor and a camera to capture the reflectance in- formation of flat samples in chapter 5. The monitor is used as an illumination source to create continuous Fourier transform illumination and the sample is put close to the monitor to have more coverage. From this setup, we can estimate the reflectance information of the sample which can be used to relight the sample under any type of illumination. Lastly, in order to quickly acquire the shape and reflectance of faces which is impractical from the previous 3 frameworks, we propose a system to acquire the reflectance information of faces from flash photographs in chapter 6. The system consists of multiple consumer cameras and flashes mounted to cover the front of the subject. The cameras are shot consecutively, synchronous with the flashes, so that the face is captured from multiple viewpoints under different illumination conditions. We then use these images to extract reflectance information, which is in turn used to reconstruct a 3-D model of the face as a final result. 75 7.2 Future Work 7.2.1 Performance Capture for Relighting The proposed system in chapter 3 greatly reduces the acquisition time for capturing relightable models of a subject. By combining this technique with a high speed camera, we can acquire relightable models of the subject performing dynamically. This also means that the relightable models of the performance could also be edited easily with the benefit of a smaller size of data. It would be interesting to see how the weight optimization process turns out, especially the consis- tency of the weights across multiple frames, in the dynamic performance sequence. 7.2.2 Multi-view Reflectance Merging The reflectance merging process in chapter 4 uses the final 3-D geometry output to merge re- flectance maps from multiple views giving higher weights to the view direction right in front. We could use this same assumption, to merge reflectance maps from many different scans of the same object. This way, we could combine the strength of many scans, where the object is placed or framed differently, to generate more accurate results. Following from chapter 5, we could also use the same idea to merge the reflectance maps from different orientations of a flat sample. Currently, the LCD monitor only covers a portion of the hemisphere, which could lead to the introduction of errors when the specular function falls outside of the monitor coverage. By combining reflectance maps from multiple views, we could eliminate that problem and produce more accurate final results. 76 7.2.3 Estimating Specular Occlusion and Roughness Our rapid shape and reflectance acquisition system in chapter 6 produces an accurate geometry of faces using reflectance maps derived from flash photographs. However, the recover specular albedo still has artifacts due to the sparseness of the flash. We could improve this by calculating the specular albedo based on the specular occlusion of the final geometry. This would produce less errors than the current method. The specular roughness could also be derived by using the estimated specular albedo, the final geometry and all the flash photographs in conjunction. 77 BIBLIOGRAPHY [1] Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. Practical SVBRDF capture in the fre- quency domain. 32(4):12 pp., 2013. [2] T.B. Chen, M. Goesele, and H. P. Seidel. Mesostructure from specularities. In CVPR, pages 1825–1832, 2006. [3] Kristin J. Dana, Bram van Ginneken, Shree K. Nayar, and Jan J. Koenderink. Reflectance and texture of real-world surfaces. ACM Trans. Graph., 18(1):1–34, January 1999. [4] T. Davidovic, J. Krivnek, M. Hasan, P. Slusallek, and K. Bala. Combining global and local lights for high-rank illumination effects. ACM Transactions on Graphics(Proc. SIGGRAPH Asia), 29(5), December 2010. [5] Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. Acquiring the reflectance field of a human face. In Proceedings of ACM SIGGRAPH 2000, pages 145–156, 2000. [6] Yue Dong, Jiaping Wang, Xin Tong, John Snyder, Yanxiang Lan, Moshe Ben-Ezra, and Baining Guo. Manifold bootstrapping for svbrdf capture. ACM Trans. Graph., 29:98:1– 98:10, July 2010. [7] P. Einarsson, C-F. Chabert, A. Jones, W-C Ma, B. Lamond, T. Hawkins, M. Bolas, S. Syl- wan, and P. Debevec. Relighting human locomotion with flowed reflectance fields. In Rendering Techniques, pages 183–194, 2006. [8] Yasutaka Furukawa and Jean Ponce. Dense 3D motion capture for human faces. In Proc. of CVPR 09, 2009. [9] Andrew Gardner, Chris Tchou, Tim Hawkins, and Paul Debevec. Linear light source reflec- tometry. In ACM TOG, pages 749–758, 2003. [10] A. Georghiades. Recovering 3-D shape and reflectance from a small number of photographs. In Rendering Techniques, pages 230–240, 2003. 78 [11] Abhijeet Ghosh, Tongbo Chen, Pieter Peers, Cyrus A. Wilson, and Paul E. Debevec. Es- timating specular roughness and anisotropy from second order spherical gradient illumina- tion. Comput. Graph. Forum, 28(4):1161–1170, 2009. [12] Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu, and Paul Debevec. Multiview face capture using polarized spherical gradient illumination. In Pro- ceedings of the 2011 SIGGRAPH Asia Conference, SA ’11, pages 129:1–129:10, New York, NY , USA, 2011. ACM. [13] Abhijeet Ghosh, Wolfgang Heidrich, Shruthi Achutha, and Matthew O’Toole. A basis illu- mination approach to brdf measurement. Int. J. Comput. Vision, 90(2):183–197, November 2010. [14] Dan B. Goldman, Brian Curless, Aaron Hertzmann, and Steven M. Seitz. Shape and spatially-varying brdfs from photometric stereo. In ICCV, pages 341–348, 2005. [15] Paul Haeberli. Synthetic lighting for photography, Jan 1992. [16] Tim Hawkins, Per Einarsson, and Paul Debevec. A dual light stage. In Proc. EGSR, pages 91–98, 2005. [17] Tim Hawkins, Andreas Wenger, Chris Tchou, Andrew Gardner, Fredrik G¨ oransson, and Paul Debevec. Animatable facial reflectance fields. In Rendering Techniques 2004., pages 309–320, 2004. [18] C. Hernandez, G. V ogiatzis, G. J. Brostow, B. Stenger, and R. Cipolla. Non-rigid photomet- ric stereo with colored lights. In Proc. IEEE International Conference on Computer Vision, pages 1–8, 2007. [19] Aaron Hertzmann and Steven M. Seitz. Example-based photometric stereo: Shape recon- struction with general, varying brdfs. PAMI, 27(8):1254–1264, 2005. [20] Michael Holroyd, Jason Lawrence, Greg Humphreys, and Todd Zickler. A photometric approach for estimating normals and tangents. ACM Trans. Graph., 27(5):133:1–133:9, December 2008. [21] Michael Holroyd, Jason Lawrence, and Todd Zickler. A coaxial optical scanner for synchronous acquisition of 3d geometry and surface reflectance. ACM Trans. Graph., 29(4):99:1–99:12, July 2010. [22] Ivo Ihrke, Kiriakos N. Kutulakos, Hendrik P. A. Lensch, Marcus Magnor, and Wolfgang Heidrich. Transparent and specular object reconstruction. Computer Graphics Forum, 29(8):2400–2426, 2010. [23] Katsushi Ikeuchi. Determining surface orientations of specular surfaces by using the photo- metric stereo method. IEEE Trans. Pattern Anal. Mach. Intell., 3(6):661–669, June 1981. 79 [24] J. Kautz, P.-P. Sloan, and J. Snyder. Fast, arbitrary brdf shading for low-frequency lighting using spherical harmonics. In Rendering Techniques 02: 13th Eurographics Workshop on Rendering, pages 291–296, 2002. [25] W. Kerr and F. Pellacini. Toward evaluating lighting design interface paradigms for novice users. ACM Transactions on Graphics (Proc. SIGGRAPH), 28(3), 2009. [26] M. Klaudiny, A. Hilton, and J. Edge. High-detail 3D capture of facial performance. In International Symposium 3D Data Processing, Visualization and Transmission (3DPVT), 2010. [27] Bruce Lamond, Pieter Peers, Abhijeet Ghosh, and Paul Debevec. Image-based separation of diffuse and specular reflections using environmental structured illumination. In Proc. IEEE International Conf. Computational Photography, 2009. [28] Bruce Lamond, Pieter Peers, Abhijeet Ghosh, and Paul Debevec. Image-based separation of diffuse and specular reflections using environmental structured illumination. In IEEE International Conference on Computational Photography, 2009. [29] Hendrik P. A. Lensch, Jan Kautz, Michael Goesele, Wolfgang Heidrich, and Hans-Peter Seidel. Image-based reconstruction of spatial appearance and geometric detail. ACM TOG, 22(2):234–257, 2003. [30] Wan-Chun Ma, Tim Hawkins, Pieter Peers, Charles-Felix Chabert, Malte Weiss, and Paul Debevec. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. In Rendering Techniques, pages 183–194, 2007. [31] V . Masselus, P. Dutr´ e, and F. Anrys. The free-form light stage. In Rendering Techniques, pages 247–256, 2002. [32] W. Matusik, H. Pfister, A. Ngan, P. Beardsley, R. Ziegler, and L. MCMillan. Image-based 3d photography using opacity hulls. In In SIGGRAPH 2002 Conference Proceedings, pages 427–437, 2002. [33] David Kirk Mcallister. A generalized surface appearance representation for computer graphics. PhD thesis, The University of North Carolina at Chapel Hill, 2002. AAI3061704. [34] S.K. Nayar, K. Ikeuchi, and T. Kanade. Determining shape and reflectance of hybrid surfaces by photometric sampling. IEEE Trans. Robotics and Automation, 6(4):418–431, 1990. [35] Diego Nehab, Szymon Rusinkiewicz, James Davis, and Ravi Ramamoorthi. Efficiently combining positions and normals for precise 3D geometry. ACM TOG, 24(3):536–543, 2005. [36] R. Ng, R. Ramamoorthi, and P. Hanrahan. All-frequency shadows using non-linear wavelet lighting approximation. ACM Transactions on Graphics (Proc. SIGGRAPH), 22(3):376– 381, July 2003. 80 [37] R. Ng, R. Ramamoorthi, and P. Hanrahan. Triple product wavelet integrals for all-frequency relighting. ACM Transactions on Graphics (Proc. SIGGRAPH), 23(3):477–487, August 2004. [38] Jeffry Nimeroff, Eero Simoncelli, and Julie Dorsey. Efficient re-rendering of naturally illu- minated environments. In Eurographics Workshop on Rendering ’94, pages 239–246, 1994. [39] Matthew O’Toole and Kiriakos N. Kutulakos. Optical computing for fast light transport analysis. ACM Trans. Graph. (Proc. SIGGRAPH Asia), 29:164:1–164:12, December 2010. [40] Pieter Peers, Dhruv K. Mahajan, Bruce Lamond, Abhijeet Ghosh, Wojciech Matusik, Ravi Ramamoorthi, and Paul Debevec. Compressive light transport sensing. ACM Transactions on Graphics, 28(3):1–18, 2009. [41] Pieter Peers, Naoki Tamura, Wojciech Matusik, and Paul Debevec. Post-production facial performance relighting using reflectance transfer. ACM TOG, 25(3), 2007. [42] Fabio Pellacini. Envylight: An interface for editing natural illumination. ACM Transactions on Graphics (Proc. SIGGRAPH), 29(4), 2010. [43] R. Ramamoorthi and P. Hanrahan. An efficient representation for irradiance environment maps. In Proc. of ACM SIGGRAPH ’01, pages 497–500, 2001. [44] R. Ramamoorthi and P. Hanrahan. Frequency space environment map rendering. In Proc. of ACM SIGGRAPH ’02, pages 517–526, 2002. [45] Ravi Ramamoorthi and Pat Hanrahan. An efficient representation for irradiance environment maps. In SIGGRAPH ’01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 497–500, New York, NY , USA, 2001. ACM. [46] Peiran Ren, Jiaping Wang, John Snyder, Xin Tong, and Baining Guo. Pocket reflectometry. ACM Trans. Graph., 30(4):45:1–45:10, July 2011. [47] Holly Rushmeier, Gabriel Taubin, and Andr´ e Gu´ eziec. Applying shape from lighting varia- tion to bump map capture. In Rendering Techniques, pages 35–44, 1997. [48] Yoichi Sato, Mark D. Wheeler, and Katsushi Ikeuchi. Object shape and reflectance modeling from observation. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques, SIGGRAPH ’97, pages 379–387, New York, NY , USA, 1997. ACM Press/Addison-Wesley Publishing Co. [49] Pradeep Sen and Soheil Darabi. Compressive dual photography. Computer Graphics Forum (Proc. EUROGRAPHICS), 28(2), 2009. [50] P.-P. Sloan, J. Kautz, and J. Snyder. Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. ACM Transactions on Graphics (Proc. SIGGRAPH), 21(3):527–536, 2002. 81 [51] Peter-Pike Sloan. Stupid spherical harmonics (sh) tricks. Game Developer’s Conference, Feb 2008. http://www.ppsloan.org/publications/. [52] Y-T. Tsai and Z-C. Shih. All-frequency precomputed radiance transfer using spherical radial basis functions and clustered tensor approximation. ACM Transactions on Graphics (Proc. SIGGRAPH), 25(3):967–976, 2006. [53] Borom Tunwattanapong, Graham Fyffe, Paul Graham, Jay Busch, Xueming Yu, Abhijeet Ghosh, and Paul Debevec. Acquiring reflectance and shape from continuous spherical har- monic illumination. ACM Trans. Graph., 32(4):109:1–109:12, July 2013. [54] Borom Tunwattanapong, Abhijeet Ghosh, and Paul Debevec. Combining spherical harmon- ics and point-source illumination for efficient image-based relighting. In ACM SIGGRAPH 2010 Posters, SIGGRAPH ’10, pages 119:1–119:1, New York, NY , USA, 2010. ACM. [55] Chun-Po Wang, Noah Snavely, and Steve Marschner. Estimating dual-scale properties of glossy surfaces from step-edge lighting. ACM Trans. Graph. (Proc. SIGGRAPH Asia), 30(6), 2011. [56] Jiaping Wang, Yue Dong, Xin Tong, Zhouchen Lin, and Baining Guo. Kernel nystrom method for light transport. ACM Trans. Graph. (Proc. SIGGRAPH), 28:29:1–29:10, July 2009. [57] Gregory J. Ward. Measuring and modeling anisotropic reflection. SIGGRAPH Comput. Graph., 26(2):265–272, 1992. [58] Andreas Wenger, Andrew Gardner, Chris Tchou, Jonas Unger, Tim Hawkins, and Paul De- bevec. Performance relighting and reflectance transformation with time-multiplexed illumi- nation. ACM TOG, 24(3):756–764, 2005. [59] S. Westin, J. Arvo, and K. Torrence. Predicting reflectance functions from complex surfaces. In Proceedings of SIGGRAPH 92, Computer Graphics Proceedings, Annual Conference Series, pages 255–264, 1992. [60] Stephen H. Westin, James R. Arvo, and Kenneth E. Torrance. Predicting reflectance func- tions from complex surfaces. SIGGRAPH Comput. Graph., 26(2):255–264, July 1992. [61] Tim Weyrich, Jason Lawrence, Hendrik P. A. Lensch, Szymon Rusinkiewicz, and Todd . Principles of appearance acquisition and representation. Found. Trends. Comput. Graph. Vis., 4(2):75–191, February 2009. [62] Tim Weyrich, Wojciech Matusik, Hanspeter Pfister, Bernd Bickel, Craig Donner, Chien Tu, Janet McAndless, Jinho Lee, Addy Ngan, Henrik Wann Jensen, and Markus Gross. Analysis of human faces using a measurement-based skin reflectance model. ACM Transactions on Graphics, 25(3):1013–1024, July 2006. 82 [63] R. J. Woodham. Photometric stereo: A reflectance map technique for determining surface orientation from image intensity. In Proc. SPIE’s 22nd Annual Technical Symposium, vol- ume 155, 1978. [64] R. J. Woodham. Photometric method for determining surface orientation from multiple images. Optical Engineering, 19(1):139–144, 1980. [65] Todd Zickler, Satya P. Mallick, David J. Kriegman, and Peter N. Belhumeur. Color sub- spaces as photometric invariants. Int. J. Comput. Vision, 79(1):13–30, August 2008. [66] Todd Zickler, Ravi Ramamoorthi, Sebastian Enrique, and Peter N. Belhumeur. Reflectance sharing: Predicting appearance from a sparse set of images of a known shape. PAMI, 28(8):1287–1302, 2006. [67] Todd E. Zickler, Peter N. Belhumeur, and David J. Kriegman. Helmholtz stereopsis: Ex- ploiting reciprocity for surface reconstruction. Int. J. Comput. Vision, 49(2-3):215–227, 2002. 83

Abstract (if available)

Abstract This dissertation presents a system for acquiring spatially‐varying reflectance information and relighting various surface types by observing the objects under active basis illumination. Most types of real‐world objects are illuminated with a succession of spherical harmonic illumination conditions. From the response of the object to the harmonics, we can separate diffuse and specular reflections, estimate world‐space diffuse and specular normals and compute anisotropic roughness parameters for each view of the object. ❧ Additionally, this work proposes a system to acquire the spatially‐varying reflectance properties of flat samples with minor surface orientation variation using an LCD monitor. The system illuminates the samples with low frequency Fourier transform illumination to encode the reflectance function of the samples, in the form of Fourier transform coefficients. Then the reflectance properties of the samples are recovered by analyzing the Fourier transform coefficients of different frequencies. ❧ For objects with complicated reflectance or geometry, this work proposes a system that practically acquires relightable and editable models of the objects. The system employs a combination of spherical harmonics and local illumination which reduces the number of required photographs by an order of magnitude compared to the traditional techniques. ❧ Finally, for faces, this work proposes a novel technique to rapidly capture and estimate reflectance properties using an array of cameras and flashes. The reflectance properties can also be used to reconstruct complete 3‐D models of the face.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

A framework for high‐resolution, high‐fidelity, inexpensive facial scanning

PDF

Compositing real and virtual objects with realistic, color-accurate illumination

PDF

3D object detection in industrial site point clouds

PDF

Recording, reconstructing, and relighting virtual humans

PDF

Object detection and recognition from 3D point clouds

PDF

Rapid creation of photorealistic virtual reality content with consumer depth cameras

PDF

Disentangling the network: understanding the interplay of topology and dynamics in network analysis

PDF

Point-based representations for 3D perception and reconstruction

PDF

3D deep learning for perception and modeling

PDF

Automatic image and video enhancement with application to visually impaired people

PDF

Sensing with sound: acoustic tomography and underwater sensor networks

PDF

City-scale aerial LiDAR point cloud visualization

PDF

Efficient coding techniques for high definition video

PDF

Plant substructuring and real-time simulation using model reduction

PDF

Design and analysis of large scale antenna systems

PDF

Scalable dynamic digital humans

PDF

A notation for rapid specification of information visualization

PDF

Hybrid methods for robust image matching and its application in augmented reality

PDF

Novel computational techniques for connectome analysis based on diffusion MRI

PDF

CUDA deformers for model reduction

Asset Metadata

Creator Tunwattanapong, Borom (author)

Core Title Spherical harmonic and point illumination basis for reflectometry and relighting

School Viterbi School of Engineering

Degree Doctor of Philosophy

Degree Program Computer Science

Publication Date 09/03/2014

Defense Date 08/03/2014

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag 3D scanning,basis illumination,computer graphics,face scanning,OAI-PMH Harvest,reflectance properties,spherical harmonics

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Debevec, Paul (committee chair), Ghosh, Abhijeet (committee chair), Kratky, Andreas (committee member), Neumann, Ulrich (committee member), Teng, Shang-Hua (committee member)

Creator Email boromt@gmail.com,tunwatta@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c3-468133

Unique identifier UC11287527

Identifier etd-Tunwattana-2875.pdf (filename),usctheses-c3-468133 (legacy record id)

Legacy Identifier etd-Tunwattana-2875.pdf

Dmrecord 468133

Document Type Dissertation

Format application/pdf (imt)

Rights Tunwattanapong, Borom

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA