Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimation

We propose a robust and accurate method for estimating the 3D poses of two hands in close interaction from a single color image. This is a very challenging problem, as large occlusions and many confusions between the joints may happen. State-of-the-art methods solve this problem by regressing a heatmap for each joint, which requires solving two problems simultaneously: localizing the joints and recognizing them. In this work, we propose to separate these tasks by relying on a CNN to first localize joints as 2D keypoints, and on self-attention between the CNN features at these keypoints to associate them with the corresponding hand joint. The resulting architecture, which we call "Keypoint Transformer", is highly efficient as it achieves state-of-the-art performance with roughly half the number of model parameters on the InterHand2.6M dataset. We also show it can be easily extended to estimate the 3D pose of an object manipulated by one or two hands with high performance. Moreover, we created a new dataset of more than 75,000 images of two hands manipulating an object fully annotated in 3D and will make it publicly available.

PDF Abstract CVPR 2022 PDF CVPR 2022 Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
hand-object pose HO-3D Keypoint-Trans Average MPJPE (mm) 25.5 # 3
ST-MPJPE 25.7 # 4
PA-MPJPE 10.8 # 5
OME 68.0 # 5
ADD-S 21.4 # 3
3D Hand Pose Estimation HO-3D Keypoint Transformer Average MPJPE (mm) 25.5 # 5
ST-MPJPE (mm) 25.7 # 9
PA-MPJPE (mm) 10.8 # 10
3D Interacting Hand Pose Estimation InterHand2.6M Keypoint Transformer MPJPE Test 12.78 # 5
MRRPE Test 29.63 # 4
MPVPE Test - # 6