HybrIK-X: Hybrid Analytical-Neural Inverse Kinematics for Whole-body Mesh Recovery

12 Apr 2023  ·  Jiefeng Li, Siyuan Bian, Chao Xu, Zhicun Chen, Lixin Yang, Cewu Lu ·

Recovering whole-body mesh by inferring the abstract pose and shape parameters from visual content can obtain 3D bodies with realistic structures. However, the inferring process is highly non-linear and suffers from image-mesh misalignment, resulting in inaccurate reconstruction. In contrast, 3D keypoint estimation methods utilize the volumetric representation to achieve pixel-level accuracy but may predict unrealistic body structures. To address these issues, this paper presents a novel hybrid inverse kinematics solution, HybrIK, that integrates the merits of 3D keypoint estimation and body mesh recovery in a unified framework. HybrIK directly transforms accurate 3D joints to body-part rotations via twist-and-swing decomposition. The swing rotations are analytically solved with 3D joints, while the twist rotations are derived from visual cues through neural networks. To capture comprehensive whole-body details, we further develop a holistic framework, HybrIK-X, which enhances HybrIK with articulated hands and an expressive face. HybrIK-X is fast and accurate by solving the whole-body pose with a one-stage model. Experiments demonstrate that HybrIK and HybrIK-X preserve both the accuracy of 3D joints and the realistic structure of the parametric human model, leading to pixel-aligned whole-body mesh recovery. The proposed method significantly surpasses the state-of-the-art methods on various benchmarks for body-only, hand-only, and whole-body scenarios. Code and results can be found at https://jeffli.site/HybrIK-X/

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
3D Human Pose Estimation 3DPW HybrIK (HRNet-W48) PA-MPJPE 41.8 # 19
MPJPE 71.6 # 26
MPVPE 82.3 # 21
3D Human Reconstruction AGORA HybrIK-X FB-NMVE 120.5 # 1
FB-NMJE 115.7 # 1
FB-MVE 112.1 # 1
FB-MPJPE 107.6 # 1
3D Human Pose Estimation AGORA HybrIK-X B-NMVE 73.7 # 3
B-NMJE 72.3 # 1
B-MVE 68.5 # 3
B-MPJPE 67.2 # 1
3D Human Pose Estimation AGORA HybrIK B-NMVE 81.2 # 4
B-NMJE 84.6 # 4
B-MVE 73.9 # 4
B-MPJPE 77 # 4
3D Human Pose Estimation Human3.6M HybrIK (HRNet-W48) Average MPJPE (mm) 47 # 127
PA-MPJPE 29.8 # 6
3D Human Pose Estimation MPI-INF-3DHP HybrIK (HRNet-W48) AUC 47.3 # 48
MPJPE 91 # 44
PCK 87.1 # 38

Methods


No methods listed for this paper. Add relevant methods here