Reconstructing Detailed Dynamic Face Geometry from Monocular Video
Detailed facial performance geometry can be reconstructed using dense camera and light setups in controlled studios. However, a wide range of important applications cannot employ these approaches, including all movie productions shot from a single principal camera. For post-production, these require dynamic monocular face capture for appearance modification. We present a new method for capturing face geometry from monocular video. Our approach captures detailed, dynamic, spatio-temporally coherent 3D face geometry without the need for markers. It works under uncontrolled lighting, and it successfully reconstructs expressive motion including high-frequency face detail such as folds and laugh lines. After simple manual initialization, the capturing process is fully automatic, which makes it versatile, lightweight and easy-to-deploy. Our approach tracks accurate sparse 2D features between automatically selected key frames to animate a parametric blend shape model, which is further refined in pose, expression and shape by temporally coherent optical flow and photometric stereo. We demonstrate performance capture results for long and complex face sequences captured indoors and outdoors, and we exemplify the relevance of our approach as an enabling technology for model-based face editing in movies and video, such as adding new facial textures, as well as a step towards enabling everyone to do facial performance capture with a single affordable camera.
Data description: We provide 3 sequences captured indoors with a Canon EOS 550D at 25 fps in HD quality (1920 x 1088 pixels) and another one captured outdoors with a GoPro Hero at 30 fps in HD quality (but cropped to 1020x880 pixels). The sequences range between 562~1000 frames. Our tracked 3D geometry consists of 200K vertices, except for the one tracked outdoors which consists of 50K vertices. Note that each mesh was colored with the corresponding pixel in the image.
purposes only and any use of it for non-scientific means is not
allowed. This includes publishing any scientific results
obtained with our data in non-scientific literature, such as
tabloid press. We ask the researcher to respect our actors and
not to use the data for any distasteful manipulations (such as
hideous deformations, exploding heads, manipulations that might
be culturally sensitive, etc.). Any dissemination of this data
outside of your institute is forbidden; distribution within the
affiliated institution is allowed.
Downloads: The data provided below is password protected. To request the data, you must send us an email to gvvperfcapeva-at-mpi-inf.mpg.de stating the following:
Compressed rar files: Subject1 Subject2 Subject3 Subject4_Outdoors