Unified Language Model Pre-training for Natural Language Understanding and Generation

NeurIPS 2019 Li DongNan YangWenhui WangFuru WeiXiaodong LiuYu WangJianfeng GaoMing ZhouHsiao-Wuen Hon

This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks. The model is pre-trained using three types of language modeling tasks: unidirectional, bidirectional, and sequence-to-sequence prediction... (read more)

PDF Abstract

Evaluation Results from the Paper


#2 best model for Text Summarization on GigaWord (using extra training data)

     Get a GitHub badge
TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK USES EXTRA
TRAINING DATA
COMPARE
Document Summarization CNN / Daily Mail UniLM (Abstractive Summarization) ROUGE-1 43.08 # 4
Document Summarization CNN / Daily Mail UniLM (Abstractive Summarization) ROUGE-2 20.43 # 2
Document Summarization CNN / Daily Mail UniLM (Abstractive Summarization) ROUGE-L 40.34 # 2
Abstractive Text Summarization CNN / Daily Mail UniLM ROUGE-1 43.08 # 2
Abstractive Text Summarization CNN / Daily Mail UniLM ROUGE-2 20.43 # 2
Abstractive Text Summarization CNN / Daily Mail UniLM ROUGE-L 40.34 # 2
Text Summarization GigaWord UniLM ROUGE-1 38.90 # 2
Text Summarization GigaWord UniLM ROUGE-2 20.05 # 2
Text Summarization GigaWord UniLM ROUGE-L 36.00 # 2
Question Generation SQuAD1.1 UniLM BLEU-4 22.78 # 2