Spatio-temporal Relation Modeling for Few-shot Action Recognition

We propose a novel few-shot action recognition framework, STRM, which enhances class-specific feature discriminability while simultaneously learning higher-order temporal representations. The focus of our approach is a novel spatio-temporal enrichment module that aggregates spatial and temporal contexts with dedicated local patch-level and global frame-level feature enrichment sub-modules. Local patch-level enrichment captures the appearance-based characteristics of actions. On the other hand, global frame-level enrichment explicitly encodes the broad temporal context, thereby capturing the relevant object features over time. The resulting spatio-temporally enriched representations are then utilized to learn the relational matching between query and support action sub-sequences. We further introduce a query-class similarity classifier on the patch-level enriched features to enhance class-specific feature discriminability by reinforcing the feature learning at different stages in the proposed framework. Experiments are performed on four few-shot action recognition benchmarks: Kinetics, SSv2, HMDB51 and UCF101. Our extensive ablation study reveals the benefits of the proposed contributions. Furthermore, our approach sets a new state-of-the-art on all four benchmarks. On the challenging SSv2 benchmark, our approach achieves an absolute gain of $3.5\%$ in classification accuracy, as compared to the best existing method in the literature. Our code and models are available at

PDF Abstract CVPR 2022 PDF CVPR 2022 Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Few Shot Action Recognition HMDB51 STRM 1:1 Accuracy 77.3 # 1
Few Shot Action Recognition Kinetics-100 STRM Accuracy 86.7 # 2
Few Shot Action Recognition Something-Something-100 STRM 1:1 Accuracy 68.1 # 2
Few Shot Action Recognition UCF101 STRM 1:1 Accuracy 96.8 # 1


No methods listed for this paper. Add relevant methods here