Cross-modal Active Complementary Learning with Self-refining Correspondence

Recently, image-text matching has attracted more and more attention from academia and industry, which is fundamental to understanding the latent correspondence across visual and textual modalities. However, most existing methods implicitly assume the training pairs are well-aligned while ignoring the ubiquitous annotation noise, a.k.a noisy correspondence (NC), thereby inevitably leading to a performance drop. Although some methods attempt to address such noise, they still face two challenging problems: excessive memorizing/overfitting and unreliable correction for NC, especially under high noise. To address the two problems, we propose a generalized Cross-modal Robust Complementary Learning framework (CRCL), which benefits from a novel Active Complementary Loss (ACL) and an efficient Self-refining Correspondence Correction (SCC) to improve the robustness of existing methods. Specifically, ACL exploits active and complementary learning losses to reduce the risk of providing erroneous supervision, leading to theoretically and experimentally demonstrated robustness against NC. SCC utilizes multiple self-refining processes with momentum correction to enlarge the receptive field for correcting correspondences, thereby alleviating error accumulation and achieving accurate and stable corrections. We carry out extensive experiments on three image-text benchmarks, i.e., Flickr30K, MS-COCO, and CC152K, to verify the superior robustness of our CRCL against synthetic and real-world noisy correspondences.

PDF Abstract NeurIPS 2023 PDF NeurIPS 2023 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Cross-modal retrieval with noisy correspondence CC152K CRCL Image-to-text R@1 41.8 # 4
Image-to-text R@5 67.4 # 7
Image-to-text R@10 76.5 # 6
Text-to-image R@1 41.6 # 5
Text-to-image R@5 68.0 # 6
Text-to-image R@10 78.4 # 1
R-Sum 373.7 # 3
Cross-modal retrieval with noisy correspondence COCO-Noisy CRCL Image-to-text R@1 79.6 # 4
Image-to-text R@5 96.1 # 11
Image-to-text R@10 98.7 # 6
Text-to-image R@1 64.7 # 1
Text-to-image R@5 90.6 # 3
Text-to-image R@10 95.9 # 4
R-Sum 525.6 # 4
Cross-modal retrieval with noisy correspondence Flickr30K-Noisy CRCL Image-to-text R@1 77.9 # 7
Image-to-text R@5 95.4 # 3
Image-to-text R@10 98.3 # 1
Text-to-image R@1 60.9 # 2
Text-to-image R@5 84.7 # 3
Text-to-image R@10 90.6 # 2
R-Sum 507.8 # 2

Methods


No methods listed for this paper. Add relevant methods here