Graphics, Vision & Video

On-set Performance Capture of Multiple Actors With A Stereo Camera

SIGGRAPH Asia 2013

Abstract

State-of-the-art marker-less performance capture algorithms reconstruct detailed human skeletal motion and space-time coherent surface geometry. Despite being a big improvement over marker-based motion capture methods, they are still rarely applied in practical VFX productions as they require ten or more cameras and a studio with controlled lighting or a green screen background. If one was able to capture performances directly on a general set using only the primary stereo camera used for principal photography, many possibilities would open up in virtual production and pre-visualization, the creation of virtual actors, and video editing during post-production. We describe a new algorithm which works towards this goal. It is able to track skeletal motion and detailed surface geometry of one or more actors from footage recorded with a stereo rig that is allowed to move. It succeeds in general sets with uncontrolled background and uncontrolled illumination, and scenes in which actors strike non-frontal poses. It is one of the first performance capture methods to exploit detailed BRDF information and scene illumination for accurate pose tracking and surface refinement in general scenes. It also relies on a new foreground segmentation approach that combines appearance, stereo, and pose tracking results to segment out actors from the background. Appearance, segmentation, and motion cues are combined in a new pose optimization framework that is robust under uncontrolled lighting, uncontrolled background and very sparse camera views.

Paper pdf (3.4M) / (28M)	Video avi (160M)	Presentation not yet available	Dataset available!

Videos

Bibtex

@inproceedings{WSVT13,
  author    = {Chenglei Wu and Carsten Stoll and Levi Valgaerts and Christian Theobalt},
  title     = {On-set Performance Capture of Multiple Actors With A Stereo Camera},
  booktitle = {ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2013)},
  volume    = {32},
  number    = {6},
  pages     = {161:1--161:11},
  month     = {November},
  year      = {2013},
  url       = {http://doi.acm.org/10.1145/2508363.2508418},
  doi       = {10.1145/2508363.2508418}
}

Dataset

Terms of use: The data we provide is meant for research purposes only and any use of it for non-scientific means is not allowed. This includes publishing any scientific results obtained with our data in non-scientific literature, such as tabloid press. We ask the researcher to respect our actors and not to use the data for any distasteful manipulations (such as hideous deformations, exploding heads, manipulations that might be culturally sensitive, etc.). Any dissemination of this data outside of your institute is forbidden; distribution within the affiliated institution is allowed.

If you use our data, you are required to cite the following paper (see bibtex above):
"On-set Performance Capture of Multiple Actors With A Stereo Camera".

Downloads: The data provided below is password protected. To request the data, you must send us an email to gvvperfcapeva-at-mpi-inf.mpg.de stating the following:

• Your name, title, and institution
• Your intended use of the data
• A statement saying that you accept our terms of use