OmniNet: Omnidirectional Representations from Transformers

This paper proposes Omnidirectional Representations from Transformers (OmniNet). In OmniNet, instead of maintaining a strictly horizontal receptive field, each token is allowed to attend to all tokens in the entire network. This process can also be interpreted as a form of extreme or intensive attention mechanism that has the receptive field of the entire width and depth of the network. To this end, the omnidirectional attention is learned via a meta-learner, which is essentially another self-attention based model. In order to mitigate the computationally expensive costs of full receptive field attention, we leverage efficient self-attention models such as kernel-based (Choromanski et al.), low-rank attention (Wang et al.) and/or Big Bird (Zaheer et al.) as the meta-learner. Extensive experiments are conducted on autoregressive language modeling (LM1B, C4), Machine Translation, Long Range Arena (LRA), and Image Recognition. The experiments show that OmniNet achieves considerable improvements across these tasks, including achieving state-of-the-art performance on LM1B, WMT'14 En-De/En-Fr, and Long Range Arena. Moreover, using omnidirectional representation in Vision Transformers leads to significant improvements on image recognition tasks on both few-shot learning and fine-tuning setups.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Language Modelling One Billion Word OmniNetP (Large) PPL 21.6 # 2
Number of params 100M # 21
Language Modelling One Billion Word OmniNetB (Large) PPL 22 # 4
Language Modelling One Billion Word OmniNetT (Large) PPL 21.5 # 1
Number of params 100M # 21
Machine Translation WMT2014 English-French OmniNetP BLEU score 42.6 # 17
Hardware Burden None # 1
Operations per network pass None # 1
Machine Translation WMT2014 English-German OmniNetP BLEU score 29.8 # 16
Hardware Burden None # 1
Operations per network pass None # 1
Machine Translation WMT2017 Chinese-English OmniNetP BLEU 23.0 # 3
Machine Translation WMT2017 English-Finnish OmniNetP BLEU 20.9 # 1
Machine Translation WMT2017 English-French OmniNetP BLEU 43.1 # 1
Machine Translation WMT2017 English-German OmniNetP BLEU 29.0 # 1
Machine Translation WMT2017 Russian-English OmniNetP BLEU 36.2 # 1


No methods listed for this paper. Add relevant methods here