Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation

24 Feb 2020 • Yige Xu • Xipeng Qiu • Ligao Zhou • Xuanjing Huang

Fine-tuning pre-trained language models like BERT has become an effective way in NLP and yields state-of-the-art results on many downstream tasks. Recent studies on adapting BERT to new tasks mainly focus on modifying the model structure, re-designing the pre-train tasks, and leveraging external data and knowledge... (read more)

PDF Abstract

Results from the Paper

Edit

Submit results from this paper to get state-of-the-art GitHub badges and help the community compare results to other papers.

METHOD	TYPE
Residual Connection	Skip Connections
Attention Dropout	Regularization
Linear Warmup With Linear Decay	Learning Rate Schedules
Weight Decay	Regularization
GELU	Activation Functions
Dense Connections	Feedforward Networks
Adam	Stochastic Optimization
WordPiece	Subword Segmentation
Softmax	Output Functions
Dropout	Regularization
Multi-Head Attention	Attention Modules
Layer Normalization	Normalization
Scaled Dot-Product Attention	Attention Mechanisms
BERT	Language Models

METHOD

TYPE

Residual Connection