HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition

11 Jan 2024  ·  Licai Sun, Zheng Lian, Bin Liu, JianHua Tao ·

Audio-Visual Emotion Recognition (AVER) has garnered increasing attention in recent years for its critical role in creating emotion-ware intelligent machines. Previous efforts in this area are dominated by the supervised learning paradigm. Despite significant progress, supervised learning is meeting its bottleneck due to the longstanding data scarcity issue in AVER. Motivated by recent advances in self-supervised learning, we propose Hierarchical Contrastive Masked Autoencoder (HiCMAE), a novel self-supervised framework that leverages large-scale self-supervised pre-training on vast unlabeled audio-visual data to promote the advancement of AVER. Following prior arts in self-supervised audio-visual representation learning, HiCMAE adopts two primary forms of self-supervision for pre-training, namely masked data modeling and contrastive learning. Unlike them which focus exclusively on top-layer representations while neglecting explicit guidance of intermediate layers, HiCMAE develops a three-pronged strategy to foster hierarchical audio-visual feature learning and improve the overall quality of learned representations. To verify the effectiveness of HiCMAE, we conduct extensive experiments on 9 datasets covering both categorical and dimensional AVER tasks. Experimental results show that our method significantly outperforms state-of-the-art supervised and self-supervised audio-visual methods, which indicates that HiCMAE is a powerful audio-visual emotion representation learner. Codes and models will be publicly available at https://github.com/sunlicai/HiCMAE.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Dynamic Facial Expression Recognition DFEW HiCMAE-T(V) WAR 71.24 # 9
UAR 59.14 # 9
Dynamic Facial Expression Recognition DFEW HiCMAE-S(V) WAR 72.20 # 8
UAR 61.37 # 6
Dynamic Facial Expression Recognition DFEW HiCMAE-B(V) WAR 73.10 # 6
UAR 61.92 # 5
Dynamic Facial Expression Recognition MAFW HiCMAE-B(V) WAR 54.84 # 3
UAR 42.10 # 3
Dynamic Facial Expression Recognition MAFW HiCMAE-T(V) WAR 52.86 # 8
UAR 39.70 # 8
Dynamic Facial Expression Recognition MAFW HiCMAE-S(V) WAR 53.41 # 7
UAR 41.31 # 5