Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding

20 Apr 2019  ·  Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao ·

This paper explores the use of knowledge distillation to improve a Multi-Task Deep Neural Network (MT-DNN) (Liu et al., 2019) for learning text representations across multiple natural language understanding tasks. Although ensemble learning can improve model performance, serving an ensemble of large DNNs such as MT-DNN can be prohibitively expensive. Here we apply the knowledge distillation method (Hinton et al., 2015) in the multi-task learning setting. For each task, we train an ensemble of different MT-DNNs (teacher) that outperforms any single model, and then train a single MT-DNN (student) via multi-task learning to \emph{distill} knowledge from these ensemble teachers. We show that the distilled MT-DNN significantly outperforms the original MT-DNN on 7 out of 9 GLUE tasks, pushing the GLUE benchmark (single model) to 83.7\% (1.5\% absolute improvement\footnote{ Based on the GLUE leaderboard at as of April 1, 2019.}). The code and pre-trained models will be made publicly available at

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Natural Language Inference MultiNLI MT-DNN-ensemble Matched 87.9 # 15
Mismatched 87.4 # 10
Semantic Textual Similarity SentEval MT-DNN-ensemble MRPC 92.7/90.3 # 1
SICK-R - # 3
SICK-E - # 3
STS 91.1/90.7* # 1
Sentiment Analysis SST-2 Binary classification MT-DNN-ensemble Accuracy 96.5 # 14