Masked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Input

26 Oct 2022  ·  Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, Kunio Kashino ·

Masked Autoencoders is a simple yet powerful self-supervised learning method. However, it learns representations indirectly by reconstructing masked input patches. Several methods learn representations directly by predicting representations of masked patches; however, we think using all patches to encode training signal representations is suboptimal. We propose a new method, Masked Modeling Duo (M2D), that learns representations directly while obtaining training signals using only masked patches. In the M2D, the online network encodes visible patches and predicts masked patch representations, and the target network, a momentum encoder, encodes masked patches. To better predict target representations, the online network should model the input well, while the target network should also model it well to agree with online predictions. Then the learned representations should better model the input. We validated the M2D by learning general-purpose audio representations, and M2D set new state-of-the-art performance on tasks such as UrbanSound8K, VoxCeleb1, AudioSet20K, GTZAN, and SpeechCommandsV2. We additionally validate the effectiveness of M2D for images using ImageNet-1K in the appendix.

PDF Abstract

Results from the Paper

 Ranked #1 on Speaker Identification on VoxCeleb1 (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Audio Classification ESC-50 M2D ratio=0.7 Top-1 Accuracy 95.0 # 10
Accuracy (5-fold) 95.0 # 10
Keyword Spotting Google Speech Commands M2D Google Speech Commands V2 35 98.5 # 2
Music Genre Classification GTZAN M2D ratio=0.7 Accuracy 83.9 # 1
Music Genre Classification GTZAN M2D ratio=0.6 Accuracy 83.3 # 2
Speaker Identification VoxCeleb1 MSM-MAE Top-1 (%) 95.3 # 1
Accuracy 95.3 # 1
Speaker Identification VoxCeleb1 M2D ratio=0.6 Top-1 (%) 94.8 # 2
Accuracy 94.8 # 2


No methods listed for this paper. Add relevant methods here