Synthesis and analysis of human faces using multi-view, multi-illumination image ensembles

by 1970- Lee, Jinho

Synthesis of realistic human faces is still one of the biggest challenges in computer graphics. Similarly, analysis of human faces in images has been a traditional topic of research in computer vision. Both research areas related to human faces have many practical applications including face modeling, face tracking, face relighting, and face recognition in diverse industry fields such as games, virtual reality, digital photography, biometrics and security. According to a recent research trend, photographs are used to guide the realistic face synthesis and three-dimensional (3D) face models are exploited to overcome the limitations of traditional image-based face analysis. In this dissertation we explore the problem of model-based 3D face synthesis and analysis using various kind of image ensembles including multi-view image silhouettes, sequential video frames, and single photographs. We design and develop various image-model difference metrics that can be used in an optimization framework to recover the optimal model parameters of our 3D face model from the input images. Our prior 3D face model is a linear combination of eigenheads obtained by applying Principal Component Analysis (PCA) to a training set of laser-scanned 3D faces. This model is extended to incorporate the texture and illumination space of the dataset we acquire using our face scanning dome. In the first part of this dissertation we present a novel method for 3D face reconstruction from a set or sequence of 2D binary silhouettes. Experiments with a ii multi-camera rig as well as monocular video sequences demonstrate the advantages of our 3D modeling framework. In order to determine the set of optimal views required for silhouette-based shape reconstruction, we build on our modeling framework and extend it by aggressive pruning of the view-sphere with view clustering and various imaging constraints. A multi-view optimization search is performed using both model-based (eigenheads) and data-driven (visual hull) methods, which yield comparable best views. In the second part of this dissertation we present a novel framework that acquires the 3D shape, texture, pose and illumination of a face from a single photograph. Using a custom-built face scanning system which is equipped with 16 digital cameras and 146 directional LED light sources, a large-scale dataset was collected, which consists of 3D face scans and light reflection images of a diverse group of human subjects. From this dataset, we derive a novel measurement-based illumination model that implicitly incorporates cast-shadows and specularities. The derivation of the resulting bilinear illumination model is based on the multilinear analysis using higher-order singular value decomposition. We propose a novel fitting framework that estimates the parameters of the morphable model by minimizing the distance of the input image to the dynamically changing illumination subspace. The proposed fitting framework can deal with complex face reflectance and lighting environments in an efficient and robust manner. We leverage our modeling and fitting methods to solve two challenging problems in computer graphics and computer vision - face relighting and face recognition. iii