Transformers

DistilBERT is a small, fast, cheap and light Transformer model based on the BERT architecture. Knowledge distillation is performed during the pre-training phase to reduce the size of a BERT model by 40%. To leverage the inductive biases learned by larger models during pre-training, the authors introduce a triple loss combining language modeling, distillation and cosine-distance losses.

Source: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

Papers


Paper Code Results Date Stars

Tasks


Task Papers Share
Text Classification 18 7.79%
Sentiment Analysis 18 7.79%
Language Modelling 16 6.93%
Classification 15 6.49%
Question Answering 12 5.19%
Sentence 9 3.90%
Quantization 7 3.03%
Model Compression 6 2.60%
General Classification 6 2.60%

Components


Component Type
BERT
Language Models

Categories