Harnessing the Power of Multi-Task Pretraining for Ground-Truth Level Natural Language Explanations

8 Dec 2022  ·  Björn Plüster, Jakob Ambsdorf, Lukas Braach, Jae Hee Lee, Stefan Wermter ·

Natural language explanations promise to offer intuitively understandable explanations of a neural network's decision process in complex vision-language tasks, as pursued in recent VL-NLE models. While current models offer impressive performance on task accuracy and explanation plausibility, they suffer from a range of issues: Some models feature a modular design where the explanation generation module is poorly integrated with a separate module for task-answer prediction, employ backbone models trained on limited sets of tasks, or incorporate ad hoc solutions to increase performance on single datasets. We propose to evade these limitations by applying recent advances in large-scale multi-task pretraining of generative Transformer models to the problem of VL-NLE tasks. Our approach outperforms recent models by a large margin, with human annotators preferring the generated explanations over the ground truth in two out of three evaluated datasets. As a novel challenge in VL-NLE research, we propose the problem of multi-task VL-NLE and show that jointly training on multiple tasks can increase the explanation quality. We discuss the ethical implications of high-quality NLE generation and other issues in recent VL-NLE research.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Explanation Generation e-SNLI-VE OFA-X-MT Human Explanation Rating 80.4 # 2
Visual Entailment e-SNLI-VE OFA-X-MT Accuracy 78.9 # 2
Explanation Generation e-SNLI-VE OFA-X Human Explanation Rating 85.7 # 1
Visual Entailment e-SNLI-VE OFA-X Accuracy 80.9 # 1
Explanation Generation VCR OFA-X-MT Human Explanation Rating 77.3 # 1
Explanation Generation VCR OFA-X Human Explanation Rating 68.9 # 2
Visual Question Answering (VQA) VCR (Q-A) test OFA-X Accuracy 71.2 # 10
Visual Question Answering (VQA) VCR (Q-A) test OFA-X-MT Accuracy 62 # 11
Visual Question Answering (VQA) VQA-X OFA-X-MT Accuracy 92.6 # 1
Explanation Generation VQA-X OFA-X-MT Human Explanation Rating 87.8 # 2
Explanation Generation VQA-X OFA-X Human Explanation Rating 89.5 # 1
Visual Question Answering (VQA) VQA-X OFA-X Accuracy 91.2 # 2

Methods