Multi-Format Contrastive Learning of Audio Representations

11 Mar 2021  ·  Luyu Wang, Aaron van den Oord ·

Recent advances suggest the advantage of multi-modal training in comparison with single-modal methods. In contrast to this view, in our work we find that similar gain can be obtained from training with different formats of a single modality. In particular, we investigate the use of the contrastive learning framework to learn audio representations by maximizing the agreement between the raw audio and its spectral representation. We find a significant gain using this multi-format strategy against the single-format counterparts. Moreover, on the downstream AudioSet and ESC-50 classification task, our audio-only approach achieves new state-of-the-art results with a mean average precision of 0.376 and an accuracy of 90.5%, respectively.

PDF Abstract


Results from the Paper

Ranked #9 on Audio Classification on ESC-50 (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Audio Classification AudioSet Multi-Format Contrastive Test mAP 0.376 # 22
Audio Classification ESC-50 Multi-Format Contrastive Top-1 Accuracy 90.5 # 9