FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integration

13 Aug 2021  ·  Yu Rong, Takaaki Shiratori, Hanbyul Joo ·

Most existing monocular 3D pose estimation approaches only focus on a single body part, neglecting the fact that the essential nuance of human motion is conveyed through a concert of subtle movements of face, hands, and body. In this paper, we present FrankMocap, a fast and accurate whole-body 3D pose estimation system that can produce 3D face, hands, and body simultaneously from in-the-wild monocular images. The core idea of FrankMocap is its modular design: We first run 3D pose regression methods for face, hands, and body independently, followed by composing the regression outputs via an integration module. The separate regression modules allow us to take full advantage of their state-of-the-art performances without compromising the original accuracy and reliability in practice. We develop three different integration modules that trade off between latency and accuracy. All of them are capable of providing simple yet effective solutions to unify the separate outputs into seamless whole-body pose estimation results. We quantitatively and qualitatively demonstrate that our modularized system outperforms both the optimization-based and end-to-end methods of estimating whole-body pose.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
3D Human Pose Estimation 3DPW FrankMocap PA-MPJPE 60 # 107
MPJPE 94.3 # 104
3D Human Reconstruction Expressive hands and faces dataset (EHF) FrankMocap PA V2V (mm), whole body 57.5 # 4
PA V2V (mm), body only 52.7 # 2
PA V2V (mm), left hand 12.8 # 2
TR V2V (mm), whole body 76.9 # 3
TR V2V (mm), body only 80.1 # 3
TR V2V (mm), left hand 32.1 # 4
MPJPE-14 62.3 # 2
MPJPE, left hand 13.2 # 3
mean P2S 31.6 # 3
median P2S 19.2 # 3

Methods


No methods listed for this paper. Add relevant methods here