On-set Performance Capture of Multiple Actors With A Stereo Camera
State-of-the-art marker-less performance capture algorithms reconstruct detailed human skeletal motion and space-time coherent surface geometry. Despite being a big improvement over marker-based motion capture methods, they are still rarely applied in practical VFX productions as they require ten or more cameras and a studio with controlled lighting or a green screen background. If one was able to capture performances directly on a general set using only the primary stereo camera used for principal photography, many possibilities would open up in virtual production and pre-visualization, the creation of virtual actors, and video editing during post-production. We describe a new algorithm which works towards this goal. It is able to track skeletal motion and detailed surface geometry of one or more actors from footage recorded with a stereo rig that is allowed to move. It succeeds in general sets with uncontrolled background and uncontrolled illumination, and scenes in which actors strike non-frontal poses. It is one of the first performance capture methods to exploit detailed BRDF information and scene illumination for accurate pose tracking and surface refinement in general scenes. It also relies on a new foreground segmentation approach that combines appearance, stereo, and pose tracking results to segment out actors from the background. Appearance, segmentation, and motion cues are combined in a new pose optimization framework that is robust under uncontrolled lighting, uncontrolled background and very sparse camera views.
purposes only and any use of it for non-scientific means is not
allowed. This includes publishing any scientific results
obtained with our data in non-scientific literature, such as
tabloid press. We ask the researcher to respect our actors and
not to use the data for any distasteful manipulations (such as
hideous deformations, exploding heads, manipulations that might
be culturally sensitive, etc.). Any dissemination of this data
outside of your institute is forbidden; distribution within the
affiliated institution is allowed.
Downloads: The data provided below is password protected. To request the data, you must send us an email to gvvperfcapeva-at-mpi-inf.mpg.de stating the following:
Compressed zip file: Data