Transformers
BART
Introduced by Lewis et al. in BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionBART is a denoising autoencoder for pretraining sequence-to-sequence models. It is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Transformer-based neural machine translation architecture. It uses a standard seq2seq/NMT architecture with a bidirectional encoder (like BERT) and a left-to-right decoder (like GPT). This means the encoder's attention mask is fully visible, like BERT, and the decoder's attention mask is causal, like GPT2.
Source: BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
Papers
| Paper | Code | Results | Date | Stars |
|---|
Tasks
| Task | Papers | Share |
|---|---|---|
| Retrieval | 147 | 13.44% |
| Question Answering | 90 | 8.23% |
| Language Modelling | 75 | 6.86% |
| Text Generation | 66 | 6.03% |
| Abstractive Text Summarization | 45 | 4.11% |
| Sentence | 41 | 3.75% |
| Decoder | 39 | 3.56% |
| Text Summarization | 29 | 2.65% |
| Information Retrieval | 25 | 2.29% |
Adam
BPE
Dense Connections
Dropout
GELU
Layer Normalization
Multi-Head Attention
Residual Connection
Scaled Dot-Product Attention
Softmax
