Unified Keypoint-based Action Recognition Framework via Structured Keypoint Pooling

CVPR 2023  ·  Ryo Hachiuma, Fumiaki Sato, Taiki Sekii ·

This paper simultaneously addresses three limitations associated with conventional skeleton-based action recognition; skeleton detection and tracking errors, poor variety of the targeted actions, as well as person-wise and frame-wise action recognition. A point cloud deep-learning paradigm is introduced to the action recognition, and a unified framework along with a novel deep neural network architecture called Structured Keypoint Pooling is proposed. The proposed method sparsely aggregates keypoint features in a cascaded manner based on prior knowledge of the data structure (which is inherent in skeletons), such as the instances and frames to which each keypoint belongs, and achieves robustness against input errors. Its less constrained and tracking-free architecture enables time-series keypoints consisting of human skeletons and nonhuman object contours to be efficiently treated as an input 3D point cloud and extends the variety of the targeted action. Furthermore, we propose a Pooling-Switching Trick inspired by Structured Keypoint Pooling. This trick switches the pooling kernels between the training and inference phases to detect person-wise and frame-wise actions in a weakly supervised manner using only video-level action labels. This trick enables our training scheme to naturally introduce novel data augmentation, which mixes multiple point clouds extracted from different videos. In the experiments, we comprehensively verify the effectiveness of the proposed method against the limitations, and the method outperforms state-of-the-art skeleton-based action recognition and spatio-temporal action localization methods.

PDF Abstract CVPR 2023 PDF CVPR 2023 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Violence and Weaponized Violence Detection Crowd Violence Dataset Structured Keypoint Pooling Accuracy 94.7 # 1
Skeleton Based Action Recognition HMDB51 Structured Keypoint Pooling Accuracy 70.9 # 1
Video Classification Hockey Fight Detection Dataset Structured Keypoint Pooling Accuracy 99.5 # 1
Skeleton Based Action Recognition Kinetics-Skeleton dataset Structured Keypoint Pooling (PPNv2 skeletons) Accuracy 43.1 # 5
Skeleton Based Action Recognition Kinetics-Skeleton dataset Structured Keypoint Pooling (HRNet skeletons) Accuracy 50.3 # 2
Skeleton Based Action Recognition Kinetics-Skeleton dataset Structured Keypoint Pooling (PPNv2 skeletons+objects) Accuracy 52.3 # 1
Violence and Weaponized Violence Detection Movies-Fight Dataset Structured Keypoint Pooling Accuracy 99.0 # 1
Activity Recognition RWF-2000 Structured Keypoint Pooling Accuracy 93.4 # 1
Action Recognition Skeleton-Mimetics Structured Keypoint Pooling Accuracy 21.2 # 1
Skeleton Based Action Recognition UCF101 Structured Keypoint Pooling Accuracy 87.8 # 1
Weakly-supervised Temporal Action Localization UCF101-24 Structured Keypoint Pooling mAP@0.2 61.8 # 1

Methods


No methods listed for this paper. Add relevant methods here