Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Language

Learning to classify video data from classes not included in the training data, i.e. video-based zero-shot learning, is challenging. We conjecture that the natural alignment between the audio and visual modalities in video data provides a rich training signal for learning discriminative multi-modal representations. Focusing on the relatively underexplored task of audio-visual zero-shot learning, we propose to learn multi-modal representations from audio-visual data using cross-modal attention and exploit textual label embeddings for transferring knowledge from seen classes to unseen classes. Taking this one step further, in our generalised audio-visual zero-shot learning setting, we include all the training classes in the test-time search space which act as distractors and increase the difficulty while making the setting more realistic. Due to the lack of a unified benchmark in this domain, we introduce a (generalised) zero-shot learning benchmark on three audio-visual datasets of varying sizes and difficulty, VGGSound, UCF, and ActivityNet, ensuring that the unseen test classes do not appear in the dataset used for supervised training of the backbone deep models. Comparing multiple relevant and recent methods, we demonstrate that our proposed AVCA model achieves state-of-the-art performance on all three datasets. Code and data are available at \url{}.

PDF Abstract CVPR 2022 PDF CVPR 2022 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
GZSL Video Classification ActivityNet-GZSL (cls) AVCA HM 9.92 # 4
ZSL Video Classification ActivityNet-GZSL (cls) AVCA ZSL 7.58 # 1
GZSL Video Classification ActivityNet-GZSL(main) AVCA HM 12.13 # 3
ZSL 9.13 # 3
GZSL Video Classification UCF-GZSL (cls) AVCA HM 41.34 # 4
ZSL 37.72 # 4
ZSL Video Classification UCF-GZSL (cls) AVCA ZSL 37.72 # 1
GZSL Video Classification UCF-GZSL(main) AVCA HM 27.15 # 4
ZSL 20.0 # 4
ZSL Video Classification VGGSound-GZSL (cls) AVCA ZSL 6.91 # 1
GZSL Video Classification VGGSound-GZSL (cls) AVCA HM 8.31 # 4
ZSL 6.91 # 4
GZSL Video Classification VGGSound-GZSL(main) AVCA HM 6.31 # 4
ZSL 6.00 # 4


No methods listed for this paper. Add relevant methods here