Constructing Stronger and Faster Baselines for Skeleton-based Action Recognition

29 Jun 2021  ·  Yi-Fan Song, Zhang Zhang, Caifeng Shan, Liang Wang ·

One essential problem in skeleton-based action recognition is how to extract discriminative features over all skeleton joints. However, the complexity of the recent State-Of-The-Art (SOTA) models for this task tends to be exceedingly sophisticated and over-parameterized. The low efficiency in model training and inference has increased the validation costs of model architectures in large-scale datasets. To address the above issue, recent advanced separable convolutional layers are embedded into an early fused Multiple Input Branches (MIB) network, constructing an efficient Graph Convolutional Network (GCN) baseline for skeleton-based action recognition. In addition, based on such the baseline, we design a compound scaling strategy to expand the model's width and depth synchronously, and eventually obtain a family of efficient GCN baselines with high accuracies and small amounts of trainable parameters, termed EfficientGCN-Bx, where "x" denotes the scaling coefficient. On two large-scale datasets, i.e., NTU RGB+D 60 and 120, the proposed EfficientGCN-B4 baseline outperforms other SOTA methods, e.g., achieving 91.7% accuracy on the cross-subject benchmark of NTU 60 dataset, while being 3.15x smaller and 3.21x faster than MS-G3D, which is one of the best SOTA methods. The source code in PyTorch version and the pretrained models are available at https://github.com/yfsong0709/EfficientGCNv1.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Skeleton Based Action Recognition NTU RGB+D EfficientGCN-B4 Accuracy (CV) 96.1 # 35
Accuracy (CS) 92.1 # 22
Skeleton Based Action Recognition NTU RGB+D EfficientGCN-B2 Accuracy (CV) 95.5 # 44
Accuracy (CS) 90.9 # 32
Skeleton Based Action Recognition NTU RGB+D EfficientGCN-B0 Accuracy (CV) 94.7 # 53
Accuracy (CS) 89.9 # 43
Skeleton Based Action Recognition NTU RGB+D 120 EfficientGCN-B4 Accuracy (Cross-Subject) 88.7 # 16
Accuracy (Cross-Setup) 89.1 # 22
Skeleton Based Action Recognition NTU RGB+D 120 EfficientGCN-B2 Accuracy (Cross-Subject) 87.9 # 21
Accuracy (Cross-Setup) 88.0 # 30
Skeleton Based Action Recognition NTU RGB+D 120 EfficientGCN-B0 Accuracy (Cross-Subject) 85.9 # 31
Accuracy (Cross-Setup) 84.3 # 41

Methods