Expressive Body Capture: 3D Hands, Face, and Body from a Single Image

To facilitate the analysis of human actions, interactions and emotions, we compute a 3D model of human body pose, hand pose, and facial expression from a single monocular image. To achieve this, we use thousands of 3D scans to train a new, unified, 3D model of the human body, SMPL-X, that extends SMPL with fully articulated hands and an expressive face. Learning to regress the parameters of SMPL-X directly from images is challenging without paired images and 3D ground truth. Consequently, we follow the approach of SMPLify, which estimates 2D features and then optimizes model parameters to fit the features. We improve on SMPLify in several significant ways: (1) we detect 2D features corresponding to the face, hands, and feet and fit the full SMPL-X model to these; (2) we train a new neural network pose prior using a large MoCap dataset; (3) we define a new interpenetration penalty that is both fast and accurate; (4) we automatically detect gender and the appropriate body models (male, female, or neutral); (5) our PyTorch implementation achieves a speedup of more than 8x over Chumpy. We use the new method, SMPLify-X, to fit SMPL-X to both controlled images and images in the wild. We evaluate 3D accuracy on a new curated dataset comprising 100 images with pseudo ground-truth. This is a step towards automatic expressive human capture from monocular RGB data. The models, code, and data are available for research purposes at https://smpl-x.is.tue.mpg.de.

PDF Abstract CVPR 2019 PDF CVPR 2019 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
3D Human Reconstruction AGORA SMPLify-X FB-NMVE 333.1 # 3
B-NMVE 263.3 # 3
FB-NMJE 326.5 # 3
B-NMJE 256.5 # 3
FB-MVE 236.5 # 3
B-MVE 187.0 # 3
F-MVE 48.9 # 1
LH/RH-MVE 48.3/51.4 # 1
FB-MPJPE 231.8 # 3
B-MPJPE 182.1 # 3
F-MPJPE 52.9 # 1
LH/RH-MPJPE 46.5/49.6 # 1
3D Multi-Person Mesh Recovery AGORA SMPLify-X FB-NMVE 333.1 # 3
B-NMVE 263.3 # 3
FB-NMJE 326.5 # 3
B-NMJE 256.5 # 3
FB-MVE 236.5 # 3
B-MVE 187.0 # 3
F-MVE 48.9 # 1
LH/RH-MVE 48.3/51.4 # 1
FB-MPJPE 231.8 # 3
B-MPJPE 182.1 # 3
F-MPJPE 52.9 # 1
LH/RH-MPJPE 46.5/49.6 # 1

Results from Other Papers


Task Dataset Model Metric Name Metric Value Rank Source Paper Compare
3D Human Reconstruction Expressive hands and faces dataset (EHF) SMPLify-X PA V2V (mm), whole body 65.3 # 4
PA V2V (mm), body only 75.4 # 4
PA V2V (mm), left hand 11.6 # 3
PA V2V (mm), face 4.9 # 2
TR V2V (mm), whole body 93.0 # 4
TR V2V (mm), body only 116.1 # 4
TR V2V (mm), left hand 23.8 # 1
TR V2V (mm), face 11.5 # 3
MPJPE-14 87.6 # 4
MPJPE, left hand 12.2 # 2
mean P2S 36.8 # 4
median P2S 23.0 # 4

Methods


No methods listed for this paper. Add relevant methods here