XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learning

25 Nov 2022  ·  Pritam Sarkar, Ali Etemad ·

We present XKD, a novel self-supervised framework to learn meaningful representations from unlabelled videos. XKD is trained with two pseudo objectives. First, masked data reconstruction is performed to learn modality-specific representations from audio and visual streams. Next, self-supervised cross-modal knowledge distillation is performed between the two modalities through a teacher-student setup to learn complementary information. We introduce a novel domain alignment strategy to tackle domain discrepancy between audio and visual modalities enabling effective cross-modal knowledge distillation. Additionally, to develop a general-purpose network capable of handling both audio and visual streams, modality-agnostic variants of XKD are introduced, which use the same pretrained backbone for different audio and visual tasks. Our proposed cross-modal knowledge distillation improves video action classification by $8\%$ to $14\%$ on UCF101, HMDB51, and Kinetics400. Additionally, XKD improves multimodal action classification by $5.5\%$ on Kinetics-Sound. XKD shows state-of-the-art performance in sound classification on ESC50, achieving top-1 accuracy of $96.5\%$.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Self-Supervised Action Recognition Linear HMDB51 XKD (ViT-B/112/16) Top-1 Accuracy 57.4 # 6
Self-Supervised Action Recognition HMDB51 XKD (ViT-B/112/16) Top-1 Accuracy 69 # 8
Self-Supervised Action Recognition HMDB51 XKD-Modality-Agnostic (ViT-B/112/16) Top-1 Accuracy 65.9 # 14
Self-Supervised Action Recognition Kinetics-400 XKD (ViT-B/112/16) Top-1 accuracy % 77.6 # 1
Top-5 Accuracy % 92.9 # 1
Self-Supervised Action Recognition Linear UCF101 XKD (ViT-B/112/16) Top-1 Accuracy 83.8 # 7
Self-Supervised Action Recognition UCF101 XKD-Modality-Agnostic (ViT-B/112/16) 3-fold Accuracy 93.4 # 12
Self-Supervised Action Recognition UCF101 XKD (ViT-B/112/16) 3-fold Accuracy 94.1 # 9
Pre-Training Dataset Kinetics400 # 1