Contrastive Learning of General-Purpose Audio Representations

21 Oct 2020  ·  Aaqib Saeed, David Grangier, Neil Zeghidour ·

We introduce COLA, a self-supervised pre-training approach for learning a general-purpose representation of audio. Our approach is based on contrastive learning: it learns a representation which assigns high similarity to audio segments extracted from the same recording while assigning lower similarity to segments from different recordings. We build on top of recent advances in contrastive learning for computer vision and reinforcement learning to design a lightweight, easy-to-implement self-supervised model of audio. We pre-train embeddings on the large-scale Audioset database and transfer these representations to 9 diverse classification tasks, including speech, music, animal sounds, and acoustic scenes. We show that despite its simplicity, our method significantly outperforms previous self-supervised systems. We furthermore conduct ablation studies to identify key design choices and release a library to pre-train and fine-tune COLA models.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Spoken Command Recognition Speech Command v2 COLA Accuracy 95.5 # 4
Speaker Identification VoxCeleb1 COLA Top-1 (%) 37.7 # 9
Accuracy 37.7 # 9