PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop

Regression-based methods have recently shown promising results in reconstructing human meshes from monocular images. By directly mapping raw pixels to model parameters, these methods can produce parametric models in a feed-forward manner via neural networks. However, minor deviation in parameters may lead to noticeable misalignment between the estimated meshes and image evidences. To address this issue, we propose a Pyramidal Mesh Alignment Feedback (PyMAF) loop to leverage a feature pyramid and rectify the predicted parameters explicitly based on the mesh-image alignment status in our deep regressor. In PyMAF, given the currently predicted parameters, mesh-aligned evidences will be extracted from finer-resolution features accordingly and fed back for parameter rectification. To reduce noise and enhance the reliability of these evidences, an auxiliary pixel-wise supervision is imposed on the feature encoder, which provides mesh-image correspondence guidance for our network to preserve the most related information in spatial features. The efficacy of our approach is validated on several benchmarks, including Human3.6M, 3DPW, LSP, and COCO, where experimental results show that our approach consistently improves the mesh-image alignment of the reconstruction. The project page with code and video results can be found at https://hongwenzhang.github.io/pymaf.

PDF Abstract ICCV 2021 PDF ICCV 2021 Abstract

Results from the Paper


Ranked #5 on 3D Human Pose Estimation on AGORA (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
3D Human Pose Estimation 3DPW PyMAF PA-MPJPE 58.9 # 100
MPJPE 92.8 # 101
MPVPE 110.1 # 75
3D Human Pose Estimation AGORA PyMAF B-NMVE 87.3 # 5
B-NMJE 92.6 # 5
B-MVE 78.6 # 5
B-MPJPE 83.3 # 6
3D Human Pose Estimation EMDB PyMAF Average MPJPE (mm) 131.065 # 11
Average MPJPE-PA (mm) 82.8502 # 11
Average MVE (mm) 159.956 # 11
Average MVE-PA (mm) 98.1305 # 9
Average MPJAE (deg) 28.4555 # 8
Average MPJAE-PA (deg) 25.7033 # 1
Jitter (10m/s^3) 81.8447 # 8
3D Human Pose Estimation Human3.6M PyMAF Average MPJPE (mm) 57.7 # 252
PA-MPJPE 40.5 # 71

Methods


No methods listed for this paper. Add relevant methods here