Live User-Guided Intrinsic Video for Static Scenes

ISMAR 2017

A. Meka 1,2 G. Fox 1,2 M. Zollhöfer 1 C. Richardt 1,3 C.Theobalt 1
1MPI Informatics 2Saarbruecken Graduate School of Computer Science 3University of Bath


We present a novel real-time approach for user-guided intrinsic decomposition of static scenes captured by an RGB-D sensor. In the first step, we acquire a three-dimensional representation of the scene using a dense volumetric reconstruction framework. The obtained reconstruction serves as a proxy to densely fuse reflectance estimates and to store user-provided constraints in three-dimensional space. User constraints, in the form of constant shading and reflectance strokes, can be placed directly on the real-world geometry using an intuitive touch-based interaction metaphor, or using interactive mouse strokes. Fusing the decomposition results and constraints in three-dimensional space allows for robust propagation of this information to novel views by re-projection. We leverage this information to improve on the decomposition quality of existing intrinsic video decomposition techniques by further constraining the ill-posed decomposition problem. In addition to improved decomposition quality, we show a variety of live augmented reality applications such as recoloring of objects, relighting of scenes and editing of material appearance.

Paper Video Poster Talk Data



title = {{Live User-Guided Intrinsic Video For Static Scene}},
author = {Meka, Abhimitra and Fox, Gereon and Zoll{\"o}fer, Michael and Richardt, Christian and Theobalt, Christian},
journal = {IEEE Transactions on Visualization and Computer Graphics},
volume = {PP},
number = {99},
year = {2017}