Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 3.0 (200k) and GPT-4 Turbo (128k). Finally, we highlight real-world use cases, such as Gemini 1.5 collaborating with professionals on completing their tasks achieving 26 to 75% time savings across 10 different job categories, as well as surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.
PDF AbstractCode
Datasets
MMLU
GSM8K
HumanEval
MATH
HellaSwag
TextVQA
DROP
BIG-bench
MM-Vet
NewsQA
DocVQA
ChartQA
YouCook2
AI2D
MathVista
ActivityNet-QA
GPQA
VATEX
FLEURS
BBQ
Video-MME
EgoSchema
QASPER
MGSM
EQA
Multilingual LibriSpeech
CoVoST2
InfographicVQA
BlINK
DUDE
MM-Vet v2
TVBench
Vinoground
MMNeedle
VNBench
SME
Zero-shot Video Question Answering on LongVideoBench
GATITOS
Results from the Paper
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Uses Extra Training Data |
Benchmark |
|---|---|---|---|---|---|---|---|
| Long-Context Understanding | MMNeedle | Gemini Pro 1.5 | 1 Image, 2*2 Stitching, Exact Accuracy | 90.34 | # 2 | ||
| 1 Image, 4*4 Stitching, Exact Accuracy | 39.85 | # 3 | |||||
| 1 Image, 8*8 Stitching, Exact Accuracy | 29.81 | # 1 | |||||
| 10 Images, 1*1 Stitching, Exact Accuracy | 89.94 | # 2 | |||||
| 10 Images, 2*2 Stitching, Exact Accuracy | 45.21 | # 2 | |||||
| 10 Images, 4*4 Stitching, Exact Accuracy | 6.09 | # 3 | |||||
| 10 Images, 8*8 Stitching, Exact Accuracy | 0.62 | # 2 | |||||
| Visual Question Answering | MM-Vet | Gemini 1.5 Pro (gemini-1.5-pro-002) | GPT-4 score | 76.9±0.1 | # 3 | ||
| Visual Question Answering | MM-Vet | Gemini 1.5 Pro (gemini-1.5-pro) | GPT-4 score | 65.8±0.1 | # 18 | ||
| Visual Question Answering | MM-Vet v2 | Gemini 1.5 Pro | GPT-4 score | 66.9±0.2 | # 6 | ||
| Question Answering | NewsQA | Google/Gemini 1.5 Flash | F1 | 79.91 | # 8 | ||
| EM | 68.75 | # 8 | |||||
| FS-MEVQA | SME | Gemini-1.5 Pro | BLEU-4 | 41.87 | # 3 | ||
| METEOR | 34.61 | # 3 | |||||
| ROUGE-L | 55.90 | # 2 | |||||
| CIDEr | 276.14 | # 2 | |||||
| SPICE | 40.58 | # 2 | |||||
| Detection | 1.40 | # 3 | |||||
| ACC | 40.88 | # 3 | |||||
| #Learning Samples (N) | 16 | # 1 | |||||
| Video Question Answering | TVBench | Gemini 1.5 Pro | Average Accuracy | 47.6 | # 8 | ||
| Zero-Shot Video Question Answer | Video-MME | Gemini 1.5 Pro | Accuracy (%) | 81.3 | # 1 | ||
| Zero-Shot Video Question Answer | Video-MME | Gemini 1.5 Flash | Accuracy (%) | 75.0 | # 4 | ||
| Zero-Shot Video Question Answer | Video-MME (w/o subs) | Gemini 1.5 Flash | Accuracy (%) | 66.3 | # 4 | ||
| Zero-Shot Video Question Answer | Video-MME (w/o subs) | Gemini 1.5 Pro | Accuracy (%) | 71.9 | # 2 | ||
| Temporal Relation Extraction | Vinoground | Gemini-1.5-Pro | Text Score | 35.8 | # 9 | ||
| Video Score | 22.6 | # 17 | |||||
| Group Score | 10.2 | # 10 | |||||
| Temporal Relation Extraction | Vinoground | Gemini-1.5-Pro (CoT) | Text Score | 37 | # 7 | ||
| Video Score | 27.6 | # 11 | |||||
| Group Score | 12.4 | # 7 | |||||
| Zero-Shot Video Question Answer | VNBench | Gemini | Accuracy | 66.7 | # 2 | ||
| Zero-Shot Video Question Answer | Zero-shot Video Question Answering on LongVideoBench | Gemini 1.5 Pro | Accuracy (% ) | 66.7 | # 1 |
