How Not to Be Seen - Object Removal from Videos of Crowded Scenes

M. Granados1       J. Tompkin2       K. Kim1       O. Grau3       J. Kautz2       C. Theobalt1      

1MPI Informatik      2UCL      3BBC R&D

Input frame Input frame
Example of input video frames

Inpainted frame Inpainted frame
Result of our algorithm where the person in front was removed


Removing dynamic objects from videos is an extremely challenging problem that even visual effects professionals often solve with time-consuming manual frame-by-frame editing. We propose a new approach to video completion that can deal with complex scenes containing dynamic background and non-periodical moving objects. We build upon the idea that the spatio-temporal hole left by a removed object can be filled with data available on other regions of the video where the occluded objects were visible. Video completion is performed by solving a large combinatorial problem that searches for an optimal pattern of pixel offsets from occluded to unoccluded regions. Our contribution includes an energy functional that generalizes well over different scenes with stable parameters, and that has the desirable convergence properties for a graph-cut-based optimization. We provide an interface to guide the completion process that both reduces computation time and allows for efficient correction of small errors in the result. We demonstrate that our approach can effectively complete complex, high-resolution occlusions that are greater in difficulty than what existing methods have shown.


In Computer Graphics Forum, 31(2):219-228, 2012: Full text | Supplementary video


The video sequences shown in the paper are available in H.264 lossless format:

park-simple sequence
park-simple: [Input] [Mask] [Result]

park-complex sequence
park-complex: [Input] [Mask] [Result]

museum sequence
museum: [Input] [Mask] [Result]

duo sequence
duo: [Input] [Mask] [Result]