Play It Back: Iterative Attention for Audio Recognition

20 Oct 2022  ·  Alexandros Stergiou, Dima Damen ·

A key function of auditory cognition is the association of characteristic sounds with their corresponding semantics over time. Humans attempting to discriminate between fine-grained audio categories, often replay the same discriminative sounds to increase their prediction confidence. We propose an end-to-end attention-based architecture that through selective repetition attends over the most discriminative sounds across the audio sequence. Our model initially uses the full audio sequence and iteratively refines the temporal segments replayed based on slot attention. At each playback, the selected segments are replayed using a smaller hop length which represents higher resolution features within these segments. We show that our method can consistently achieve state-of-the-art performance across three audio-classification benchmarks: AudioSet, VGG-Sound, and EPIC-KITCHENS-100.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Audio Classification AudioSet PlayItBackX3 Test mAP 0.477 # 9
Audio Classification EPIC-KITCHENS-100 PlayItBackX3 Top-1 Verb 47 # 1
Top-5 Verb 78.7 # 1
Top-1 Noun 23.1 # 1
Top-5 Noun 45.1 # 1
Top-1 Action 15.9 # 1
Top-5 Action 29.2 # 1
Audio Classification VGGSound PlayItBackX3 Top 1 Accuracy 53.7 # 2
Top 5 Accuracy 79.2 # 2
Mean AP 56.1 # 1
AUC 97.8 # 1
d-prime 2.846 # 1

Methods


No methods listed for this paper. Add relevant methods here