Elaborative Rehearsal for Zero-shot Action Recognition

ICCV 2021  ·  ShiZhe Chen, Dong Huang ·

The growing number of action classes has posed a new challenge for video understanding, making Zero-Shot Action Recognition (ZSAR) a thriving direction. The ZSAR task aims to recognize target (unseen) actions without training examples by leveraging semantic representations to bridge seen and unseen actions. However, due to the complexity and diversity of actions, it remains challenging to semantically represent action classes and transfer knowledge from seen data. In this work, we propose an ER-enhanced ZSAR model inspired by an effective human memory technique Elaborative Rehearsal (ER), which involves elaborating a new concept and relating it to known concepts. Specifically, we expand each action class as an Elaborative Description (ED) sentence, which is more discriminative than a class name and less costly than manual-defined attributes. Besides directly aligning class semantics with videos, we incorporate objects from the video as Elaborative Concepts (EC) to improve video semantics and generalization from seen actions to unseen actions. Our ER-enhanced ZSAR model achieves state-of-the-art results on three existing benchmarks. Moreover, we propose a new ZSAR evaluation protocol on the Kinetics dataset to overcome limitations of current benchmarks and demonstrate the first case where ZSAR performance is comparable to few-shot learning baselines on this more realistic setting. We will release our codes and collected EDs at https://github.com/DeLightCMU/ElaborativeRehearsal.

PDF Abstract ICCV 2021 PDF ICCV 2021 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Zero-Shot Action Recognition HMDB51 ER-ZSAR Top-1 Accuracy 35.3 # 18
Zero-Shot Action Recognition Kinetics ER-ZSAR (ST+Obj) Top-1 Accuracy 42.1 # 12
Top-5 Accuracy 73.1 # 9
Zero-Shot Action Recognition Kinetics ER-ZSAR (ST) Top-1 Accuracy 37.1 # 13
Top-5 Accuracy 69.3 # 10
Zero-Shot Action Recognition Olympics ER-ZSAR Top-1 Accuracy 60.2 # 3
Zero-Shot Action Recognition UCF101 ER-ZSAR Top-1 Accuracy 51.8 # 18


No methods listed for this paper. Add relevant methods here