VinVL+L: Enriching Visual Representation with Location Context in VQA
In this paper, we describe a novel method - VinVL+L - that enriches the visual representations (i.e. object tags and region features) of the State-of-the-Art Vision and Language (VL) method - VinVL - with Location information. To verify the importance of such metadata for VL models, we (i) trained a Swin-B model on the Places365 dataset and obtained additional sets of visual and tag features; both were made public to allow reproducibility and further experiments, (ii) did an architectural update to the existing VinVL method to include the new feature sets, and (iii) provide a qualitative and quantitative evaluation. By including just binary location metadata, the VinVL+L method provides incremental improvement to the State-of-the-Art VinVL in Visual Question Answering (VQA). The VinVL+L achieved an accuracy of 64.85% and increased the performance by +0.32% in terms of accuracy on the GQA dataset; the statistical significance of the new representations is verified via Approximate Randomization. The code and newly generated sets of features are available at https://github.com/vyskocj/VinVL-L.
PDF AbstractCode
Results from the Paper
Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Benchmark |
---|---|---|---|---|---|---|
Visual Question Answering (VQA) | GQA Test2019 | VinVL+L | Accuracy | 64.85 | # 10 | |
Binary | 82.59 | # 5 | ||||
Open | 49.19 | # 12 | ||||
Consistency | 94.0 | # 5 | ||||
Plausibility | 84.91 | # 31 | ||||
Validity | 96.62 | # 7 | ||||
Distribution | 4.59 | # 118 |