Live Intrinsic Video

SG 2016

A. Meka 1 M. Zollhöfer 1 C. Richardt 1,2 C. Theobalt 1
1 Max Planck Institute for Informatics 2 Intel Visual Computing Institute


Intrinsic video decomposition refers to the fundamentally ambiguous task of separating a video stream into its constituent layers, in particular reflectance and shading layers. Such a decomposition is the basis for a variety of video manipulation applications, such as realistic recoloring or retexturing of objects. We present a novel variational approach to tackle this underconstrained inverse problem at real-time frame rates, which enables on-line processing of live video footage. The problem of finding the intrinsic decomposition is formulated as a mixed variational L2-Lp optimization problem based on an objective function that is specifically tailored for fast optimization. To this end, we propose a novel combination of sophisticated local spatial and global spatio-temporal priors resulting in temporally coherent decompositions at real-time frame rates without the need for explicit correspondence search. We tackle the resulting high-dimensional, non-convex optimization problem via a novel data-parallel iteratively reweighted least squares solver that runs on commodity graphics hardware. Real-time performance is obtained by combining a local-global solution strategy with hierarchical coarse-to-fine optimization. Compelling real-time augmented reality applications, such as recoloring, material editing and retexturing, are demonstrated in a live setup. Our qualitative and quantitative evaluation shows that we obtain high-quality real-time decompositions even for challenging sequences. Our method is able to outperform state-of-the-art approaches in terms of runtime and result quality - even without user guidance such as scribbles.

Paper Video Additional Video Talk Data



author = {Meka, Abhimitra and Zollhoefer, Michael and Richardt, Christian and Theobalt, Christian},
title = {Live Intrinsic Video},
journal = {ACM Transactions on Graphics (Proceedings SIGGRAPH)},
volume = {35},
number = {4},
year = {2016} }