SPEC: Seeing People in the Wild with an Estimated Camera

Due to the lack of camera parameter information for in-the-wild images, existing 3D human pose and shape (HPS) estimation methods make several simplifying assumptions: weak-perspective projection, large constant focal length, and zero camera rotation. These assumptions often do not hold and we show, quantitatively and qualitatively, that they cause errors in the reconstructed 3D shape and pose. To address this, we introduce SPEC, the first in-the-wild 3D HPS method that estimates the perspective camera from a single image and employs this to reconstruct 3D human bodies more accurately. First, we train a neural network to estimate the field of view, camera pitch, and roll given an input image. We employ novel losses that improve the calibration accuracy over previous work. We then train a novel network that concatenates the camera calibration to the image features and uses these together to regress 3D body shape and pose. SPEC is more accurate than the prior art on the standard benchmark (3DPW) as well as two new datasets with more challenging camera views and varying focal lengths. Specifically, we create a new photorealistic synthetic dataset (SPEC-SYN) with ground truth 3D bodies and a novel in-the-wild dataset (SPEC-MTP) with calibration and high-quality reference bodies. Both qualitative and quantitative analysis confirm that knowing camera parameters during inference regresses better human bodies. Code and datasets are available for research purposes at https://spec.is.tue.mpg.de.

PDF Abstract ICCV 2021 PDF ICCV 2021 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
3D Human Pose Estimation 3DPW SPEC PA-MPJPE 53.2 # 74
3D Multi-Person Pose Estimation AGORA SPEC B-NMVE 126.8 # 1
B-NMJE 133.7 # 1
B-MVE 106.5 # 1
B-MPJPE 112.3 # 1
3D Human Pose Estimation SPEC-MTP SPEC W-MPJPE 124.3 # 2
W-PVE 147.1 # 2

Results from Other Papers

Task Dataset Model Metric Name Metric Value Rank Uses Extra
Training Data
Source Paper Compare
3D Human Pose Estimation AGORA SPEC B-NMVE 126.8 # 8
B-NMJE 133.7 # 8
B-MVE 106.5 # 8
B-MPJPE 112.3 # 8


No methods listed for this paper. Add relevant methods here