Visual Coreference Resolution in Visual Dialog using Neural Module Networks

Visual dialog entails answering a series of questions grounded in an image, using dialog history as context. In addition to the challenges found in visual question answering (VQA), which can be seen as one-round dialog, visual dialog encompasses several more. We focus on one such problem called visual coreference resolution that involves determining which words, typically noun phrases and pronouns, co-refer to the same entity/object instance in an image. This is crucial, especially for pronouns (e.g., `it'), as the dialog agent must first link it to a previous coreference (e.g., `boat'), and only then can rely on the visual grounding of the coreference `boat' to reason about the pronoun `it'. Prior work (in visual dialog) models visual coreference resolution either (a) implicitly via a memory network over history, or (b) at a coarse level for the entire question; and not explicitly at a phrase level of granularity. In this work, we propose a neural module network architecture for visual dialog by introducing two novel modules - Refer and Exclude - that perform explicit, grounded, coreference resolution at a finer word level. We demonstrate the effectiveness of our model on MNIST Dialog, a visually simple yet coreference-wise complex dataset, by achieving near perfect accuracy, and on VisDial, a large and challenging visual dialog dataset on real images, where our model outperforms other approaches, and is more interpretable, grounded, and consistent qualitatively.

PDF Abstract ECCV 2018 PDF ECCV 2018 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Visual Dialog VisDial v0.9 val CorefNMN (ResNet-152) MRR 64.1 # 3
Mean Rank 4.45 # 8
R@1 50.92 # 7
R@10 88.81 # 8
R@5 80.18 # 9
Visual Dialog VisDial v0.9 val CorefNMN MRR 63.6 # 5
Mean Rank 4.53 # 10
R@1 50.24 # 9
R@10 88.51 # 10
R@5 79.81 # 10
Common Sense Reasoning Visual Dialog v0.9 NMN [kottur2018visual] 1 in 10 R@5 80.1 # 1
Visual Dialog Visual Dialog v1.0 test-std CorefNMN (ResNet-152) NDCG (x 100) 54.70 # 67
MRR (x 100) 61.50 # 41
R@1 47.55 # 40
R@5 78.10 # 39
R@10 88.80 # 37
Mean 4.40 # 42