Autoencoding Transformers
DeBERTa
Introduced by He et al. in DeBERTa: Decoding-enhanced BERT with Disentangled AttentionDeBERTa is a Transformer-based neural language model that aims to improve the BERT and RoBERTa models with two techniques: a disentangled attention mechanism and an enhanced mask decoder. The disentangled attention mechanism is where each word is represented unchanged using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disentangle matrices on their contents and relative positions. The enhanced mask decoder is used to replace the output softmax layer to predict the masked tokens for model pre-training. In addition, a new virtual adversarial training method is used for fine-tuning to improve model’s generalization on downstream tasks.
Source: DeBERTa: Decoding-enhanced BERT with Disentangled Attention
Papers
| Paper | Code | Results | Date | Stars |
|---|
Tasks
| Task | Papers | Share |
|---|---|---|
| Language Modelling | 8 | 18.18% |
| Question Answering | 3 | 6.82% |
| Sentiment Analysis | 3 | 6.82% |
| Semantic Textual Similarity | 2 | 4.55% |
| Aspect-Based Sentiment Analysis | 2 | 4.55% |
| Self-Supervised Learning | 2 | 4.55% |
| Ensemble Learning | 2 | 4.55% |
| Natural Language Inference | 2 | 4.55% |
| Natural Language Understanding | 2 | 4.55% |
Disentangled Attention Mechanism
