Towards Accurate Multi-person Pose Estimation in the Wild

We propose a method for multi-person detection and 2-D pose estimation that achieves state-of-art results on the challenging COCO keypoints task. It is a simple, yet powerful, top-down approach consisting of two stages. In the first stage, we predict the location and scale of boxes which are likely to contain people; for this we use the Faster RCNN detector. In the second stage, we estimate the keypoints of the person potentially contained in each proposed bounding box. For each keypoint type we predict dense heatmaps and offsets using a fully convolutional ResNet. To combine these outputs we introduce a novel aggregation procedure to obtain highly localized keypoint predictions. We also use a novel form of keypoint-based Non-Maximum-Suppression (NMS), instead of the cruder box-level NMS, and a novel form of keypoint-based confidence score estimation, instead of box-level scoring. Trained on COCO data alone, our final system achieves average precision of 0.649 on the COCO test-dev set and the 0.643 test-standard sets, outperforming the winner of the 2016 COCO keypoints challenge and other recent state-of-art. Further, by using additional in-house labeled data we obtain an even higher average precision of 0.685 on the test-dev set and 0.673 on the test-standard set, more than 5% absolute improvement compared to the previous best performing method on the same dataset.

PDF Abstract CVPR 2017 PDF CVPR 2017 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Multi-Person Pose Estimation COCO G-RMI* AP 0.685 # 9
Multi-Person Pose Estimation COCO G-RMI AP 0.649 # 12
Keypoint Detection COCO test-challenge G-RMI* AR 75.1 # 6
ARM 69.7 # 6
AP 69.1 # 5
AP50 85.9 # 6
AP75 75.2 # 5
APL 82.4 # 6
AR50 90.7 # 6
AR75 80.7 # 6
ARL 74.5 # 6
Keypoint Detection COCO test-dev G-RMI APL 70.0 # 15
APM 62.3 # 11
AP50 85.5 # 11
AP75 71.3 # 9
AR 69.7 # 10
AR50 88.7 # 6
AR75 75.5 # 6
ARL 77.1 # 6
ARM 64.4 # 6
Pose Estimation COCO test-dev G-RMI AP 64.9 # 27
AP50 85.5 # 26
AP75 71.3 # 23
APL 70.0 # 26
AR 69.7 # 24
Multi-Person Pose Estimation COCO test-dev G-RMI AP 64.9 # 12
APL 70.0 # 7
APM 62.3 # 8
AP50 85.5 # 6
AP75 71.3 # 6

Methods