Muppet: Massive Multi-task Representations with Pre-Finetuning
We propose pre-finetuning, an additional large-scale learning stage between language model pre-training and fine-tuning. Pre-finetuning is massively multi-task learning (around 50 datasets, over 4.8 million total labeled examples), and is designed to encourage learning of representations that generalize better to many different tasks. We show that pre-finetuning consistently improves performance for pretrained discriminators (e.g.~RoBERTa) and generation models (e.g.~BART) on a wide range of tasks (sentence prediction, commonsense reasoning, MRC, etc.), while also significantly improving sample efficiency during fine-tuning. We also show that large-scale multi-tasking is crucial; pre-finetuning can hurt performance when few tasks are used up until a critical point (usually above 15) after which performance improves linearly in the number of tasks.
PDF Abstract EMNLP 2021 PDF EMNLP 2021 Abstract
GLUE
SST
SQuAD
MultiNLI
QNLI
CoLA
Reddit
CNN/Daily Mail
HellaSwag
BoolQ
RACE
CommonsenseQA
ANLI
SWAG
Reddit TIFU
Results from the Paper
Ranked #3 on
Text Summarization
on GigaWord
(using extra training data)
