Intrinsic Dynamic Shape Prior for
Dense Non-Rigid Structure from Motion

Abstract

While dense non-rigid structure from motion (NRSfM) has been extensively studied from the perspective of the reconstructability problem over the recent years, almost no attempts have been undertaken to bring it into the practical realm. The reasons for the slow dissemination are the severe ill-posedness, high sensitivity to motion and deformation cues, and the difficulty to obtain reliable point tracks in the vast majority of practical scenarios.
To fill this gap, we propose a new framework that first extracts prior knowledge from an input image sequence with NRSfM. Our Dynamic Shape Prior Reconstruction (DSPR) approach then uses the obtained 3D reconstructions as a dynamic shape prior for sequential surface recovery in scenarios with recurrence. DSPR can be combined with existing dense NRSfM techniques while its energy functional is optimised with multi-start gradient descent at real-time frame rates for new incoming point tracks. The proposed versatile framework with a new core NRSfM approach outperforms several other methods in the ability to handle inaccurate and noisy point tracks, provided we have access to a representative (in terms of the deformation variety) image sequence. Comprehensive experiments highlight convergence properties and the accuracy of DSPR under different disturbing effects. We also perform a joint study of tracking and reconstruction and show applications to shape compression and heart reconstruction under occlusions. We achieve state-of-the-art metrics (accuracy and compression ratios) in different scenarios.