Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning

6 Mar 2020  ·  Elad Amrani, Rami Ben-Ari, Daniel Rotman, Alex Bronstein ·

One of the key factors of enabling machine learning models to comprehend and solve real-world tasks is to leverage multimodal data. Unfortunately, annotation of multimodal data is challenging and expensive. Recently, self-supervised multimodal methods that combine vision and language were proposed to learn multimodal representations without annotation. However, these methods often choose to ignore the presence of high levels of noise and thus yield sub-optimal results. In this work, we show that the problem of noise estimation for multimodal data can be reduced to a multimodal density estimation task. Using multimodal density estimation, we propose a noise estimation building block for multimodal representation learning that is based strictly on the inherent correlation between different modalities. We demonstrate how our noise estimation can be broadly integrated and achieves comparable results to state-of-the-art performance on five different benchmark datasets for two challenging multimodal tasks: Video Question Answering and Text-To-Video Retrieval. Furthermore, we provide a theoretical probabilistic error bound substantiating our empirical results and analyze failure cases. Code:

PDF Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Zero-Shot Video Retrieval LSMDC El. Amrani et. al. text-to-video R@1 4.2 # 10
text-to-video R@5 11.6 # 10
text-to-video R@10 17.1 # 10
Zero-Shot Video Retrieval MSR-VTT El. Amrani et. al. text-to-video R@1 8.0 # 25
text-to-video R@5 21.3 # 24
text-to-video R@10 29.3 # 24
Visual Question Answering (VQA) MSRVTT-QA SSML Accuracy 0.35 # 28
Video Retrieval MSVD SSML text-to-video R@1 20.3 # 20
text-to-video R@5 49.0 # 19
text-to-video R@10 63.3 # 19
text-to-video Median Rank 6.0 # 15
text-to-video R@50 -- # 2
text-to-video Mean Rank -- # 15

Results from Other Papers

Task Dataset Model Metric Name Metric Value Rank Source Paper Compare
Visual Question Answering (VQA) MSVD-QA SSML Accuracy 0.351 # 30


No methods listed for this paper. Add relevant methods here