AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model

In this work, we demonstrate that multilingual large-scale sequence-to-sequence (seq2seq) models, pre-trained on a mixture of denoising and Causal Language Modeling (CLM) tasks, are more efficient few-shot learners than decoder-only models on various tasks. In particular, we train a 20 billion parameter multilingual seq2seq model called Alexa Teacher Model (AlexaTM 20B) and show that it achieves state-of-the-art (SOTA) performance on 1-shot summarization tasks, outperforming a much larger 540B PaLM decoder model. AlexaTM 20B also achieves SOTA in 1-shot machine translation, especially for low-resource languages, across almost all language pairs supported by the model (Arabic, English, French, German, Hindi, Italian, Japanese, Marathi, Portuguese, Spanish, Tamil, and Telugu) on Flores-101 dataset. We also show in zero-shot setting, AlexaTM 20B outperforms GPT3 (175B) on SuperGLUE and SQuADv2 datasets and provides SOTA performance on multilingual tasks such as XNLI, XCOPA, Paws-X, and XWinograd. Overall, our results present a compelling case for seq2seq models as a powerful alternative to decoder-only models for Large-scale Language Model (LLM) training.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Question Answering BoolQ AlexaTM 20B Accuracy 69.4 # 41
Natural Language Inference CommitmentBank AlexaTM 20B Accuracy 67.9 # 14
Question Answering COPA AlexaTM 20B Accuracy 78.0 # 39
Question Answering MultiRC AlexaTM 20B F1 59.6 # 21
Common Sense Reasoning ReCoRD AlexaTM 20B F1 88.4 # 15
Natural Language Inference RTE AlexaTM 20B Accuracy 68.6% # 60
Coreference Resolution Winograd Schema Challenge AlexaTM 20B Accuracy 68.3 # 37
Word Sense Disambiguation Words in Context AlexaTM 20B Accuracy 53.3 # 23