Language Models are Unsupervised Multitask Learners

Preprint 2019 Alec Radford • Jeffrey Wu • Rewon Child • David Luan • Dario Amodei • Ilya Sutskever

Natural language processing tasks, such as question answering, machine translation, reading comprehension, and summarization, are typically approached with supervised learning on taskspecific datasets. We demonstrate that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of webpages called WebText. When conditioned on a document plus questions, the answers generated by the language model reach 55 F1 on the CoQA dataset - matching or exceeding the performance of 3 out of 4 baseline systems without using the 127,000+ training examples.

PDF

Evaluation


Task Dataset Model Metric name Metric value Global rank Extra
data
Compare
Question Answering Children's Book Test GPT-2 Accuracy-CN 93.30% # 1
Question Answering Children's Book Test GPT-2 Accuracy-NE 89.05% # 1
Document Summarization CNN / Daily Mail GPT-2 ROUGE-1 29.34 # 3
Document Summarization CNN / Daily Mail GPT-2 ROUGE-2 8.27 # 3
Document Summarization CNN / Daily Mail GPT-2 ROUGE-L 26.58 # 3
Language Modelling enwiki8 GPT-2 Bit per Character (BPC) 0.93 # 1
Language Modelling enwiki8 GPT-2 Number of params 1542M # 1
Language Modelling One Billion Word GPT-2 PPL 42.16 # 13
Language Modelling One Billion Word GPT-2 Number of params 1.54B # 1
Language Modelling Penn Treebank (Word Level) GPT-2 Test perplexity 35.76 # 1
Language Modelling Penn Treebank (Word Level) GPT-2 Params 1542M # 1
Language Modelling Text8 GPT-2 Bit per Character (BPC) 0.98 # 1
Language Modelling WikiText-103 GPT-2 Test perplexity 17.48 # 1
Language Modelling WikiText-103 GPT-2 Number of params 1542M # 1
Language Modelling WikiText-2 GPT-2 Test perplexity 18.34 # 1
Language Modelling WikiText-2 GPT-2 Number of params 1542M # 1
Common Sense Reasoning Winograd Schema Challenge GPT-2 Score 70.70 # 1