Natural language processing tasks, such as question answering, machine translation, reading comprehension, and summarization, are typically approached with supervised learning on taskspecific datasets. We demonstrate that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of webpages called WebText. When conditioned on a document plus questions, the answers generated by the language model reach 55 F1 on the CoQA dataset - matching or exceeding the performance of 3 out of 4 baseline systems without using the 127,000+ training examples.
PDF| Task | Dataset | Model | Metric name | Metric value | Global rank | Extra data |
Compare |
|---|---|---|---|---|---|---|---|
| Question Answering | Children's Book Test | GPT-2 | Accuracy-CN | 93.30% | # 1 |
|
|
| Question Answering | Children's Book Test | GPT-2 | Accuracy-NE | 89.05% | # 1 |
|
|
| Document Summarization | CNN / Daily Mail | GPT-2 | ROUGE-1 | 29.34 | # 3 |
|
|
| Document Summarization | CNN / Daily Mail | GPT-2 | ROUGE-2 | 8.27 | # 3 |
|
|
| Document Summarization | CNN / Daily Mail | GPT-2 | ROUGE-L | 26.58 | # 3 |
|
|
| Language Modelling | enwiki8 | GPT-2 | Bit per Character (BPC) | 0.93 | # 1 |
|
|
| Language Modelling | enwiki8 | GPT-2 | Number of params | 1542M | # 1 |
|
|
| Language Modelling | One Billion Word | GPT-2 | PPL | 42.16 | # 13 |
|
|
| Language Modelling | One Billion Word | GPT-2 | Number of params | 1.54B | # 1 |
|
|
| Language Modelling | Penn Treebank (Word Level) | GPT-2 | Test perplexity | 35.76 | # 1 |
|
|
| Language Modelling | Penn Treebank (Word Level) | GPT-2 | Params | 1542M | # 1 |
|
|
| Language Modelling | Text8 | GPT-2 | Bit per Character (BPC) | 0.98 | # 1 |
|
|
| Language Modelling | WikiText-103 | GPT-2 | Test perplexity | 17.48 | # 1 |
|
|
| Language Modelling | WikiText-103 | GPT-2 | Number of params | 1542M | # 1 |
|
|
| Language Modelling | WikiText-2 | GPT-2 | Test perplexity | 18.34 | # 1 |
|
|
| Language Modelling | WikiText-2 | GPT-2 | Number of params | 1542M | # 1 |
|
|
| Common Sense Reasoning | Winograd Schema Challenge | GPT-2 | Score | 70.70 | # 1 |
|