TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Text Classification	arXiv	BigBird	Accuracy	92.31	# 1
Text Summarization	arXiv	BigBird-Pegasus	ROUGE-1	46.63	# 1
Text Summarization	arXiv	BigBird-Pegasus	ROUGE-2	19.02	# 1
Text Summarization	arXiv	BigBird-Pegasus	ROUGE-L	41.77	# 1
Document Summarization	BBC XSum	BigBird-Pegasus	ROUGE-1	47.12	# 1
Document Summarization	BBC XSum	BigBird-Pegasus	ROUGE-2	24.05	# 1
Document Summarization	BBC XSum	BigBird-Pegasus	ROUGE-L	38.8	# 1
Text Summarization	BigPatent	BigBird-Pegasus	ROUGE-1	60.64	# 1
Text Summarization	BigPatent	BigBird-Pegasus	ROUGE-2	42.46	# 1
Text Summarization	BigPatent	BigBird-Pegasus	ROUGE-L	50.01	# 1
Document Summarization	CNN / Daily Mail	BigBird-Pegasus	ROUGE-1	43.84	# 4
Document Summarization	CNN / Daily Mail	BigBird-Pegasus	ROUGE-2	21.11	# 2
Document Summarization	CNN / Daily Mail	BigBird-Pegasus	ROUGE-L	40.74	# 1
Linguistic Acceptability	CoLA	BigBird	Accuracy	58.5%	# 13
Chromatin-Profile Prediction	DeepSea	BigBird	TF	96.1	# 1
Chromatin-Profile Prediction	DeepSea	BigBird	HM	88.7	# 1
Chromatin-Profile Prediction	DeepSea	BigBird	DHS	92.1	# 1
Question Answering	HotpotQA	BigBird-etc	Joint F1	73.6	# 3
Question Answering	HotpotQA	BigBird-etc	Ans	81.2	# 2
Question Answering	HotpotQA	BigBird-etc	Sup	89.1	# 1
Text Classification	Hyperpartisan	BigBird	Accuracy	92.2	# 1
Text Classification	IMDb	BigBird	Accuracy (2 classes)	95.2	# 5
Text Classification	IMDb	BigBird	Accuracy (10 classes)	-	# 3
Semantic Textual Similarity	MRPC	BigBird	F1	91.5	# 4
Natural Language Inference	MultiNLI	BigBird	Matched	87.5	# 13
Question Answering	Natural Questions	BigBird-etc	F1 (Long)	77.7	# 1
Question Answering	Natural Questions	BigBird-etc	F1 (Short)	57.8	# 1
Text Classification	Patents	BigBird	Accuracy	69.3	# 1
Text Summarization	Pubmed	BigBird-Pegasus	ROUGE-1	46.32	# 2
Text Summarization	Pubmed	BigBird-Pegasus	ROUGE-2	20.65	# 1
Text Summarization	Pubmed	BigBird-Pegasus	ROUGE-L	42.33	# 2
Natural Language Inference	QNLI	BigBird	Accuracy	92.2%	# 15
Question Answering	Quora Question Pairs	BigBird	Accuracy	88.6%	# 15
Natural Language Inference	RTE	BigBird	Accuracy	75.0%	# 13
Sentiment Analysis	SST-2 Binary classification	BigBird	Accuracy	94.6	# 19
Semantic Textual Similarity	STS Benchmark	BigBird	Spearman Correlation	.878	# 5
Question Answering	TriviaQA	BigBird-etc	F1	80.9	# 2
Question Answering	WikiHop	BigBird-etc	Test	82.3	# 1
Text Classification	Yelp-5	BigBird	Accuracy	72.16%	# 3

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/text-classification-on-arxiv)](https://paperswithcode.com/sota/text-classification-on-arxiv?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/text-summarization-on-arxiv)](https://paperswithcode.com/sota/text-summarization-on-arxiv?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/document-summarization-on-bbc-xsum)](https://paperswithcode.com/sota/document-summarization-on-bbc-xsum?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/text-summarization-on-bigpatent)](https://paperswithcode.com/sota/text-summarization-on-bigpatent?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/chromatin-profile-prediction-on-deepsea)](https://paperswithcode.com/sota/chromatin-profile-prediction-on-deepsea?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/text-classification-on-hyperpartisan)](https://paperswithcode.com/sota/text-classification-on-hyperpartisan?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/question-answering-on-natural-questions)](https://paperswithcode.com/sota/question-answering-on-natural-questions?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/text-classification-on-patents)](https://paperswithcode.com/sota/text-classification-on-patents?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/question-answering-on-wikihop)](https://paperswithcode.com/sota/question-answering-on-wikihop?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/text-summarization-on-pubmed-1)](https://paperswithcode.com/sota/text-summarization-on-pubmed-1?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/question-answering-on-triviaqa)](https://paperswithcode.com/sota/question-answering-on-triviaqa?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/question-answering-on-hotpotqa)](https://paperswithcode.com/sota/question-answering-on-hotpotqa?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/text-classification-on-yelp-5)](https://paperswithcode.com/sota/text-classification-on-yelp-5?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/document-summarization-on-cnn-daily-mail)](https://paperswithcode.com/sota/document-summarization-on-cnn-daily-mail?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/semantic-textual-similarity-on-mrpc)](https://paperswithcode.com/sota/semantic-textual-similarity-on-mrpc?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/text-classification-on-imdb)](https://paperswithcode.com/sota/text-classification-on-imdb?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/semantic-textual-similarity-on-sts-benchmark)](https://paperswithcode.com/sota/semantic-textual-similarity-on-sts-benchmark?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/linguistic-acceptability-on-cola)](https://paperswithcode.com/sota/linguistic-acceptability-on-cola?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/natural-language-inference-on-multinli)](https://paperswithcode.com/sota/natural-language-inference-on-multinli?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/natural-language-inference-on-rte)](https://paperswithcode.com/sota/natural-language-inference-on-rte?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/natural-language-inference-on-qnli)](https://paperswithcode.com/sota/natural-language-inference-on-qnli?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/question-answering-on-quora-question-pairs)](https://paperswithcode.com/sota/question-answering-on-quora-question-pairs?p=big-bird-transformers-for-longer-sequences)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/big-bird-transformers-for-longer-sequences/sentiment-analysis-on-sst-2-binary)](https://paperswithcode.com/sota/sentiment-analysis-on-sst-2-binary?p=big-bird-transformers-for-longer-sequences)`

Big Bird: Transformers for Longer Sequences

NeurIPS 2020 • Manzil Zaheer • Guru Guruganesh • Avinava Dubey • Joshua Ainslie • Chris Alberti • Santiago Ontanon • Philip Pham • Anirudh Ravula • Qifan Wang • Li Yang • Amr Ahmed

Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP. Unfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence length due to their full attention mechanism... (read more)

PDF Abstract NeurIPS 2020 PDF NeurIPS 2020 Abstract

Code

Add Remove Mark official

google-research/bigbird official

206

tensorflow/models

69,588

huggingface/transformers

44,907

Tasks

Add Remove

LINGUISTIC ACCEPTABILITY

NATURAL LANGUAGE INFERENCE

QUESTION ANSWERING

SEMANTIC TEXTUAL SIMILARITY

TEXT CLASSIFICATION

TEXT SUMMARIZATION

Datasets

GLUE

SST

IMDb Movie Reviews Pubmed

MultiNLI

QNLI

MRPC

CNN/Daily Mail

Natural Questions

TriviaQA

CoLA

HotpotQA

SuperGLUE

Quora

WikiHop BigPatent

XSum

Results from the Paper

Edit

Ranked #1 on Text Classification on arXiv

Get a GitHub badge

TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK	BENCHMARK
Text Classification	arXiv	BigBird	Accuracy	92.31	# 1	Compare
Text Summarization	arXiv	BigBird-Pegasus	ROUGE-1	46.63	# 1	Compare
			ROUGE-2	19.02	# 1	Compare
			ROUGE-L	41.77	# 1	Compare
Document Summarization	BBC XSum	BigBird-Pegasus	ROUGE-1	47.12	# 1	Compare
			ROUGE-2	24.05	# 1	Compare
			ROUGE-L	38.8	# 1	Compare
Text Summarization	BigPatent	BigBird-Pegasus	ROUGE-1	60.64	# 1	Compare
			ROUGE-2	42.46	# 1	Compare
			ROUGE-L	50.01	# 1	Compare
Document Summarization	CNN / Daily Mail	BigBird-Pegasus	ROUGE-1	43.84	# 4	Compare
			ROUGE-2	21.11	# 2	Compare
			ROUGE-L	40.74	# 1	Compare
Linguistic Acceptability	CoLA	BigBird	Accuracy	58.5%	# 13	Compare
Chromatin-Profile Prediction	DeepSea	BigBird	TF	96.1	# 1	Compare
			HM	88.7	# 1	Compare
			DHS	92.1	# 1	Compare
Question Answering	HotpotQA	BigBird-etc	Joint F1	73.6	# 3	Compare
			Ans	81.2	# 2	Compare
			Sup	89.1	# 1	Compare
Text Classification	Hyperpartisan	BigBird	Accuracy	92.2	# 1	Compare
Text Classification	IMDb	BigBird	Accuracy (2 classes)	95.2	# 5	Compare
Text Classification	IMDb	BigBird	Accuracy (10 classes)	-	# 3	Compare
Semantic Textual Similarity	MRPC	BigBird	F1	91.5	# 4	Compare
Natural Language Inference	MultiNLI	BigBird	Matched	87.5	# 13	Compare
Question Answering	Natural Questions	BigBird-etc	F1 (Long)	77.7	# 1	Compare
Question Answering	Natural Questions	BigBird-etc	F1 (Short)	57.8	# 1	Compare
Text Classification	Patents	BigBird	Accuracy	69.3	# 1	Compare
Text Summarization	Pubmed	BigBird-Pegasus	ROUGE-1	46.32	# 2	Compare
			ROUGE-2	20.65	# 1	Compare
			ROUGE-L	42.33	# 2	Compare
Natural Language Inference	QNLI	BigBird	Accuracy	92.2%	# 15	Compare
Question Answering	Quora Question Pairs	BigBird	Accuracy	88.6%	# 15	Compare
Natural Language Inference	RTE	BigBird	Accuracy	75.0%	# 13	Compare
Sentiment Analysis	SST-2 Binary classification	BigBird	Accuracy	94.6	# 19	Compare
Semantic Textual Similarity	STS Benchmark	BigBird	Spearman Correlation	.878	# 5	Compare
Question Answering	TriviaQA	BigBird-etc	F1	80.9	# 2	Compare
Question Answering	WikiHop	BigBird-etc	Test	82.3	# 1	Compare
Text Classification	Yelp-5	BigBird	Accuracy	72.16%	# 3	Compare

Methods used in the Paper

Add Remove

METHOD	TYPE
🤖 No Methods Found	Help the community by adding them if they're not listed; e.g. Deep Residual Learning for Image Recognition uses ResNet

Edit Social Preview

Big Bird: Transformers for Longer Sequences Edit social preview

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods used in the Paper Edit Add Remove