Activation Functions

Gated Linear Unit

Introduced by Dauphin et al. in Language Modeling with Gated Convolutional Networks

A Gated Linear Unit, or GLU computes:

$$ \text{GLU}\left(a, b\right) = a\otimes \sigma\left(b\right) $$

It is used in natural language processing architectures, for example the Gated CNN, because here $b$ is the gate that control what information from $a$ is passed up to the following layer. Intuitively, for a language modeling task, the gating mechanism allows selection of words or features that are important for predicting the next word. The GLU also has non-linear capabilities, but has a linear path for the gradient so diminishes the vanishing gradient problem.

Source: Language Modeling with Gated Convolutional Networks


Paper Code Results Date Stars


Task Papers Share
Language Modelling 91 9.25%
Question Answering 59 6.00%
Sentence 41 4.17%
Text Generation 41 4.17%
Retrieval 31 3.15%
Translation 27 2.74%
Machine Translation 21 2.13%
Semantic Parsing 19 1.93%
Natural Language Understanding 18 1.83%


Component Type
🤖 No Components Found You can add them if they exist; e.g. Mask R-CNN uses RoIAlign