Non-Rigid Neural Radiance Fields:
Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video

Abstract

In this tech report, we present the current state of our ongoing work on reconstructing Neural Radiance Fields (NERF) of general non-rigid scenes via ray bending. Non-rigid NeRF (NR-NeRF) takes RGB images of a deforming object (e.g., from a monocular video) as input and then learns a geometry and appearance representation that not only allows to reconstruct the input sequence but also to re-render any time step into novel camera views with high fidelity. In particular, we show that a consumer-grade camera is sufficient to synthesize convincing bullet-time videos of short and simple scenes. In addition, the resulting representation enables correspondence estimation across views and time, and provides rigidity scores for each point in the scene. We urge the reader to watch the supplemental videos for qualitative results. We will release our code.

Downloads


Videos


  • Sequences 1 (w/o view dependence)

  • Sequences 1 (w/ approximate view dependence)

  • Sequences 1 (w/ exact view dependence)








  • Sequences 2 (w/o view dependence)

  • Sequences 2 (w/ approximate view dependence)

  • Sequences 2 (w/ exact view dependence)








  • Comparison with Rigid NeRF

  • Forced Background Stabilization




Citation

BibTeX, 1 KB

@misc{tretschk2020nonrigid,
      title={Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video}, 
      author={Tretschk, Edgar and Tewari, Ayush and Golyanik, Vladislav and Zollh\"{o}fer, Michael and Lassner, Christoph and Theobalt, Christian},
      year={2020},
      eprint={2012.12247},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
				

Acknowledgments

This work was supported by the ERC Consolidator Grant 4DRepLy (770784), and an Oculus research grant.

Contact

For questions, clarifications, please get in touch with:
Edgar Tretschk tretschk@mpi-inf.mpg.de

This page is Zotero translator friendly. Page last updated Imprint. Data Protection.