Ensemble of MRR and NDCG models for Visual Dialog

NAACL 2021  ·  Idan Schwartz ·

Assessing an AI agent that can converse in human language and understand visual content is challenging. Generation metrics, such as BLEU scores favor correct syntax over semantics. Hence a discriminative approach is often used, where an agent ranks a set of candidate options. The mean reciprocal rank (MRR) metric evaluates the model performance by taking into account the rank of a single human-derived answer. This approach, however, raises a new challenge: the ambiguity and synonymy of answers, for instance, semantic equivalence (e.g., `yeah' and `yes'). To address this, the normalized discounted cumulative gain (NDCG) metric has been used to capture the relevance of all the correct answers via dense annotations. However, the NDCG metric favors the usually applicable uncertain answers such as `I don't know. Crafting a model that excels on both MRR and NDCG metrics is challenging. Ideally, an AI agent should answer a human-like reply and validate the correctness of any answer. To address this issue, we describe a two-step non-parametric ranking approach that can merge strong MRR and NDCG models. Using our approach, we manage to keep most MRR state-of-the-art performance (70.41% vs. 71.24%) and the NDCG state-of-the-art performance (72.16% vs. 75.35%). Moreover, our approach won the recent Visual Dialog 2020 challenge. Source code is available at https://github.com/idansc/mrr-ndcg.

PDF Abstract NAACL 2021 PDF NAACL 2021 Abstract


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Visual Dialog VisDial v1.0 test-std 5xFGA + LS*+ MRR 0.7124 # 1
Mean Rank 2.96 # 2
R@1 58.28 # 1
R@10 94.45 # 1
R@5 87.55 # 1
Visual Dialog VisDial v1.0 test-std 5xFGA + LS NDCG 64.04 # 2
Visual Dialog VisDial v1.0 test-std Two-Step MRR 0.7041 # 2
Mean Rank 3.66 # 1
NDCG 72.16 # 1
R@1 58.18 # 2
R@10 90.83 # 2
R@5 83.85 # 2
Visual Dialog Visual Dialog v1.0 test-std 2 Step: Factor Graph Attention + VD-Bert NDCG (x 100) 72.83 # 15
MRR (x 100) 69.92 # 4
R@1 58.3 # 1
R@5 81.55 # 16
R@10 89.6 # 29
Mean 3.84 # 60


No methods listed for this paper. Add relevant methods here