Pay Attention to MLPs

17 May 2021  ·  Hanxiao Liu, Zihang Dai, David R. So, Quoc V. Le ·

Transformers have become one of the most important architectural innovations in deep learning and have enabled many breakthroughs over the past few years. Here we propose a simple network architecture, gMLP, based on MLPs with gating, and show that it can perform as well as Transformers in key language and vision applications... Our comparisons show that self-attention is not critical for Vision Transformers, as gMLP can achieve the same accuracy. For BERT, our model achieves parity with Transformers on pretraining perplexity and is better on some downstream NLP tasks. On finetuning tasks where gMLP performs worse, making the gMLP model substantially larger can close the gap with Transformers. In general, our experiments show that gMLP can scale as well as Transformers over increased data and compute. read more

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Image Classification ImageNet gMLP-B Top 1 Accuracy 81.6 # 188
Number of params 73M # 82
Natural Language Inference MultiNLI gMLP-large Matched 86.2 # 19
Mismatched 86.5 # 12
Question Answering SQuAD2.0 gMLP-large F1 78.3 # 210
Sentiment Analysis SST-2 Binary classification gMLP-large Accuracy 94.8 # 20