Reasoning Visual Dialogs with Structural and Partial Observations

We propose a novel model to address the task of Visual Dialog which exhibits complex dialog structures. To obtain a reasonable answer based on the current question and the dialog history, the underlying semantic dependencies between dialog entities are essential. In this paper, we explicitly formalize this task as inference in a graphical model with partially observed nodes and unknown graph structures (relations in dialog). The given dialog entities are viewed as the observed nodes. The answer to a given question is represented by a node with missing value. We first introduce an Expectation Maximization algorithm to infer both the underlying dialog structures and the missing node values (desired answers). Based on this, we proceed to propose a differentiable graph neural network (GNN) solution that approximates this process. Experiment results on the VisDial and VisDial-Q datasets show that our model outperforms comparative methods. It is also observed that our method can infer the underlying dialog structure for better dialog reasoning.

PDF Abstract CVPR 2019 PDF CVPR 2019 Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Visual Dialog VisDial v0.9 val GNN MRR 0.6285 # 14
Mean Rank 4.57 # 11
R@1 48.95 # 10
R@10 88.36 # 11
R@5 79.65 # 11
Visual Dialog Visual Dialog v1.0 test-std GNN NDCG (x 100) 52.82 # 63
MRR (x 100) 61.37 # 39
R@1 47.33 # 39
R@5 77.98 # 39
R@10 87.83 # 38
Mean 4.57 # 32


No methods listed for this paper. Add relevant methods here