Transformers
ELECTRA
Introduced by Clark et al. in ELECTRA: Pre-training Text Encoders as Discriminators Rather Than GeneratorsELECTRA is a transformer with a new pre-training approach which trains two transformer models: the generator and the discriminator. The generator replaces tokens in the sequence - trained as a masked language model - and the discriminator (the ELECTRA contribution) attempts to identify which tokens are replaced by the generator in the sequence. This pre-training task is called replaced token detection, and is a replacement for masking the input.
Source: ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Papers
| Paper | Code | Results | Date | Stars |
|---|
Tasks
| Task | Papers | Share |
|---|---|---|
| Language Modelling | 27 | 15.52% |
| Question Answering | 15 | 8.62% |
| Reading Comprehension | 10 | 5.75% |
| Natural Language Inference | 6 | 3.45% |
| Natural Language Understanding | 6 | 3.45% |
| Text Classification | 5 | 2.87% |
| Named Entity Recognition (NER) | 5 | 2.87% |
| Multi-Task Learning | 5 | 2.87% |
| Machine Reading Comprehension | 5 | 2.87% |
Adam
Attention Dropout
Dense Connections
Dropout
GELU
Layer Normalization
Linear Warmup With Linear Decay
Multi-Head Attention
Residual Connection
Scaled Dot-Product Attention
Softmax
Weight Decay
WordPiece
