PaLI-X: On Scaling up a Multilingual Vision and Language Model

29 May 2023  ·  Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, Siamak Shakeri, Mostafa Dehghani, Daniel Salz, Mario Lucic, Michael Tschannen, Arsha Nagrani, Hexiang Hu, Mandar Joshi, Bo Pang, Ceslee Montgomery, Paulina Pietrzyk, Marvin Ritter, AJ Piergiovanni, Matthias Minderer, Filip Pavetic, Austin Waters, Gang Li, Ibrahim Alabdulmohsin, Lucas Beyer, Julien Amelot, Kenton Lee, Andreas Peter Steiner, Yang Li, Daniel Keysers, Anurag Arnab, Yuanzhong Xu, Keran Rong, Alexander Kolesnikov, Mojtaba Seyedhosseini, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, Radu Soricut ·

We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture. Our model achieves new levels of performance on a wide-range of varied and complex tasks, including multiple image-based captioning and question-answering tasks, image-based document understanding and few-shot (in-context) learning, as well as object detection, video question answering, and video captioning. PaLI-X advances the state-of-the-art on most vision-and-language benchmarks considered (25+ of them). Finally, we observe emerging capabilities, such as complex counting and multilingual object detection, tasks that are not explicitly in the training mix.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data 		Result Benchmark
Chart Question Answering ChartQA PaLI-X (Single-task FT w/ OCR) 1:1 Accuracy 72.3 # 9
Compare
Chart Question Answering ChartQA PaLI-X (Multi-task FT) 1:1 Accuracy 70.6 # 11
Compare
Chart Question Answering ChartQA PaLI-X (Single-task FT) 1:1 Accuracy 70.9 # 10
Compare
Visual Question Answering (VQA) DocVQA test PaLI-X (Single-task FT) ANLS 0.80 # 22
Compare
Visual Question Answering (VQA) DocVQA test PaLI-X (Single-task FT w/ OCR) ANLS 0.868 # 13
Compare
Visual Question Answering (VQA) DocVQA test PaLI-X (Multi-task FT) ANLS 0.809 # 20
Compare
Visual Question Answering (VQA) InfographicVQA PaLI-X (Multi-task FT) ANLS 50.7 # 11
Compare
Visual Question Answering (VQA) InfographicVQA PaLI-X (Single-task FT) ANLS 49.2 # 12
Compare
Visual Question Answering (VQA) InfographicVQA PaLI-X (Single-task FT w/ OCR) ANLS 54.8 # 9
Compare
Visual Question Answering (VQA) InfoSeek PaLI-X Accuracy 24 # 2
Compare
Temporal/Casual QA NExT-QA PaLI-X WUPS 38.3 # 1
Compare
Visual Question Answering (VQA) OK-VQA PaLI-X (Single-task FT) Accuracy 66.1 # 2
Compare
Fine-Grained Image Recognition OVEN PaLI-X Accuracy 23.1 # 1
Compare

Methods
Add Remove

No methods listed for this paper. Add relevant methods here
Contact us on: hello@paperswithcode.com . Papers With Code is a free resource with all data licensed under CC-BY-SA.
Terms Data policy Cookies policy