Otter: A Multi-Modal Model with In-Context Instruction Tuning
Large language models (LLMs) have demonstrated significant universal capabilities as few/zero-shot learners in various tasks due to their pre-training on vast amounts of text data, as exemplified by GPT-3, which boosted to InstrctGPT and ChatGPT, effectively following natural language instructions to accomplish real-world tasks. In this paper, we propose to introduce instruction tuning into multi-modal models, motivated by the Flamingo model's upstream interleaved format pretraining dataset. We adopt a similar approach to construct our MultI-Modal In-Context Instruction Tuning (MIMIC-IT) dataset. We then introduce Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following ability and in-context learning. We also optimize OpenFlamingo's implementation for researchers, democratizing the required training resources from 1$\times$ A100 GPU to 4$\times$ RTX-3090 GPUs, and integrate both OpenFlamingo and Otter into Huggingface Transformers for more researchers to incorporate the models into their customized training and inference pipelines.
PDF AbstractCode
Datasets
Introduced in the Paper:
MIMIC-IT
Used in the Paper:
Visual Question Answering
InfiMM-Eval
MMC4
BenchLMM
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Benchmark |
|---|---|---|---|---|---|---|
| Visual Question Answering | BenchLMM | Otter-7B | GPT-3.5 score | 39.13 | # 8 | |
| Visual Question Answering (VQA) | InfiMM-Eval | Otter | Overall score | 22.69 | # 10 | |
| Deductive | 22.49 | # 11 | ||||
| Abductive | 33.64 | # 10 | ||||
| Analogical | 13.33 | # 10 | ||||
| Params | 7B | # 1 |
