Activation Functions
Gated Linear Unit
Introduced by Dauphin et al. in Language Modeling with Gated Convolutional NetworksA Gated Linear Unit, or GLU computes:
$$ \text{GLU}\left(a, b\right) = a\otimes \sigma\left(b\right) $$
It is used in natural language processing architectures, for example the Gated CNN, because here $b$ is the gate that control what information from $a$ is passed up to the following layer. Intuitively, for a language modeling task, the gating mechanism allows selection of words or features that are important for predicting the next word. The GLU also has non-linear capabilities, but has a linear path for the gradient so diminishes the vanishing gradient problem.
Source: Language Modeling with Gated Convolutional Networks
Papers
| Paper | Code | Results | Date | Stars |
|---|
Tasks
| Task | Papers | Share |
|---|---|---|
| Question Answering | 24 | 7.52% |
| Language Modelling | 22 | 6.90% |
| Text Generation | 18 | 5.64% |
| Abstractive Text Summarization | 11 | 3.45% |
| Machine Translation | 9 | 2.82% |
| Semantic Parsing | 8 | 2.51% |
| Natural Language Understanding | 8 | 2.51% |
| Multi-Task Learning | 5 | 1.57% |
| Data-to-Text Generation | 5 | 1.57% |
Usage Over Time
Components
| Component | Type |
|
|---|---|---|
| 🤖 No Components Found | You can add them if they exist; e.g. Mask R-CNN uses RoIAlign |
