Text-to-Text Pre-Training for Data-to-Text Tasks

INLG (ACL) 2020  ·  Mihir Kale, Abhinav Rastogi ·

We study the pre-train + fine-tune strategy for data-to-text tasks. Our experiments indicate that text-to-text pre-training in the form of T5, enables simple, end-to-end transformer based models to outperform pipelined neural architectures tailored for data-to-text generation, as well as alternative language model based pre-training techniques such as BERT and GPT-2. Importantly, T5 pre-training leads to better generalization, as evidenced by large improvements on out-of-domain test sets. We hope our work serves as a useful baseline for future research, as transfer learning becomes ever more prevalent for data-to-text tasks.

PDF Abstract INLG (ACL) 2020 PDF INLG (ACL) 2020 Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Data-to-Text Generation MULTIWOZ 2.1 T5-Base BLEU 35.1 # 1
Data-to-Text Generation ToTTo T5-3B BLEU 49.5 # 1
PARENT 58.4 # 1
Data-to-Text Generation WebNLG T5-Base BLEU 64.7 # 6
Data-to-Text Generation WebNLG Full T5-Large BLEU 57.1 # 5