Autoencoding Transformers
DistilBERT
Introduced by Sanh et al. in DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighterDistilBERT is a small, fast, cheap and light Transformer model based on the BERT architecture. Knowledge distillation is performed during the pre-training phase to reduce the size of a BERT model by 40%. To leverage the inductive biases learned by larger models during pre-training, the authors introduce a triple loss combining language modeling, distillation and cosine-distance losses.
Source: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Papers
| Paper | Code | Results | Date | Stars |
|---|
Tasks
| Task | Papers | Share |
|---|---|---|
| Sentiment Analysis | 6 | 9.84% |
| General Classification | 6 | 9.84% |
| Language Modelling | 6 | 9.84% |
| Text Classification | 4 | 6.56% |
| Natural Language Understanding | 3 | 4.92% |
| Question Answering | 3 | 4.92% |
| Knowledge Distillation | 3 | 4.92% |
| Emotion Recognition | 2 | 3.28% |
| Fake News Detection | 2 | 3.28% |
BERT