Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

PDF Abstract NeurIPS 2020 PDF NeurIPS 2020 Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Natural Language Inference ANLI test GPT-3 A1 36.8 # 8
A2 34 # 16
A3 40.2 # 14
Common Sense Reasoning ARC (Challenge) GPT-3 175B (1 shot) Accuracy 53.2 # 17
Common Sense Reasoning ARC (Challenge) GPT-3 175B (0 shot) Accuracy 51.4 # 19
Common Sense Reasoning ARC (Easy) GPT-3 175B (0 shot) Accuracy 68.8 # 18
Common Sense Reasoning ARC (Easy) GPT-3 175B (1 shot) Accuracy 71.2 # 14
Question Answering BoolQ GPT-3 (zero-shot) Accuracy 60.5 # 32
Question Answering BoolQ GPT-3 175B (few-shot) Accuracy 76.4 # 20
Natural Language Inference CommitmentBank GPT-3 175B (Few-Shot) F1 52 # 5
Accuracy 75.6 # 7
Zero-Shot Learning COPA GPT-3 Accuracy 73.0 # 4
Question Answering COPA GPT-3 175B (Few-Shot) Accuracy 92 # 6
Question Answering CoQA GPT-3 175B (Few-Shot) Overall 85 # 1
Question Answering DROP Test GPT-3 175B (few-Shot) F1 36.5 # 13
Sentence Completion HellaSwag GPT-3 (zero-shot) Accuracy 78.9 # 18
Zero-Shot Learning HellaSwag GPT-3 Accuracy 51.0 # 1
Sentence Completion HellaSwag GPT-3 175B (Few-Shot) Accuracy 79.3 # 15
Language Modelling LAMBADA GPT-3 2.7B (Zero-Shot) Accuracy 67.1 # 23
Perplexity 4.60 # 6
Language Modelling LAMBADA GPT-3 6.7B (Zero-Shot) Accuracy 70.3 # 20