Neural Monocular 3D Human Motion Capture
with Physical Awareness

ACM Transactions on Graphics
(Proceedings of SIGGRAPH 2021)
(with audio)


We present a new trainable system for physically-plausible markerless 3D human motion capture, which achieves state-of-the-art results in a broad range of challenging scenarios. Unlike most neural methods for human motion capture, our approach, which we dub “physionical”, is aware of physical and environmental constraints. It combines in a fully-differentiable way several key innovations, i.e., 1) a proportional-derivative controller, with gains predicted by a neural network, that reduces delays even in the presence of fast motions, 2) an explicit rigid body dynamics model and 3) a novel optimisation layer that prevents physically implausible foot-floor penetration as a hard constraint. The inputs to our system are 2D joint keypoints, which are canonicalised in a novel way so as to reduce the dependency on intrinsic camera parameters — both at train and test time. This enables more accurate global translation estimation without generalisability loss. Our model can be finetuned only with 2D annotations when the 3D annotations are not available. It produces smooth and physically-principled 3D motions in an interactive frame rate in a wide variety of challenging scenes, including newly recorded ones. Its advantages are especially noticeable on in-the-wild sequences that significantly differ from common 3D pose estimation benchmarks such as Human 3.6M and MPI-INF-3DHP. Qualitative results are provided in the supplementary video.



BibTeX, 1 KB

	author = {Shimada, Soshi and Golyanik, Vladislav and Xu, Weipeng and P\'{e}rez, Patrick and Theobalt, Christian},
	title = {Neural Monocular 3D Human Motion Capture},
	journal = {ACM Transactions on Graphics}, 
	month = {aug},
	volume = {40},
	number = {4}, 
	articleno = {83},
	year = {2021}, 
	publisher = {ACM}, 
	keywords = {Monocular 3D Human Motion Capture, Physical Awareness, Global 3D, Physionical Approach}


All data captures and evaluations were performed at MPII by MPII. The authors from MPII were supported by the ERC Consolidator Grant 4DRepLy (770784). We also acknowledge support from Valeo.


For questions, clarifications, please get in touch with:
Soshi Shimada
Vladislav Golyanik

This page is Zotero translator friendly. Page last updated Imprint. Data Protection.