Universal Transformer

Introduced by Dehghani et al. in Universal Transformers

The Universal Transformer is a generalization of the Transformer architecture. Universal Transformers combine the parallelizability and global receptive field of feed-forward sequence models like the Transformer with the recurrent inductive bias of RNNs. They also utilise a dynamic per-position halting mechanism.

Source: Universal Transformers

Read Paper See Code

Papers

Paper	Code	Results	Date	Stars

Tasks

Task	Papers	Share
Sentence	4	10.26%
Language Modelling	3	7.69%
Semantic Communication	3	7.69%
Text Generation	2	5.13%
Instance Segmentation	1	2.56%
Semantic Segmentation	1	2.56%
Reinforcement Learning (RL)	1	2.56%
Semantic Similarity	1	2.56%
Semantic Textual Similarity	1	2.56%

Usage Over Time

This feature is experimental; we are continuously improving our matching algorithm.

Components

Component	Type	Add Remove
Adam	Stochastic Optimization
Attention Dropout	Regularization
Dense Connections	Feedforward Networks	(optional)
Depthwise Separable Convolution	Convolutions	(optional)
Dropout	Regularization
Layer Normalization	Normalization
Multi-Head Attention	Attention Modules
ReLU	Activation Functions
Residual Connection	Skip Connections
Scaled Dot-Product Attention	Attention Mechanisms
Softmax	Output Functions

Categories

Add Remove

Transformers

Autoregressive Transformers