Transformers
Extended Transformer Construction
Introduced by Ainslie et al. in ETC: Encoding Long and Structured Inputs in TransformersExtended Transformer Construction, or ETC, is an extension of the Transformer architecture with a new attention mechanism that extends the original in two main ways: (1) it allows scaling up the input length from 512 to several thousands; and (2) it can ingesting structured inputs instead of just linear sequences. The key ideas that enable ETC to achieve these are a new global-local attention mechanism, coupled with relative position encodings. ETC also allows lifting weights from existing BERT models, saving computational resources while training.
Source: ETC: Encoding Long and Structured Inputs in Transformers
Papers
| Paper | Code | Results | Date | Stars |
|---|
Tasks
| Task | Papers | Share |
|---|---|---|
| Language Modelling | 2 | 7.69% |
| Question Answering | 2 | 7.69% |
| Reinforcement Learning (RL) | 2 | 7.69% |
| Multi-Armed Bandits | 2 | 7.69% |
| Thompson Sampling | 2 | 7.69% |
| Pseudo Label | 1 | 3.85% |
| Traffic Classification | 1 | 3.85% |
| Disease Prediction | 1 | 3.85% |
| Outlier Detection | 1 | 3.85% |
Contrastive Predictive Coding
Dense Connections
Global-Local Attention
Layer Normalization
Multi-Head Attention
Position-Wise Feed-Forward Layer
Relative Position Encodings
Residual Connection
