Transformers

BART is a denoising autoencoder for pretraining sequence-to-sequence models. It is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Transformer-based neural machine translation architecture. It uses a standard seq2seq/NMT architecture with a bidirectional encoder (like BERT) and a left-to-right decoder (like GPT). This means the encoder's attention mask is fully visible, like BERT, and the decoder's attention mask is causal, like GPT2.

Source: BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Papers


Paper Code Results Date Stars

Tasks


Task Papers Share
Retrieval 86 10.14%
Question Answering 61 7.19%
Language Modelling 60 7.08%
Text Generation 59 6.96%
Abstractive Text Summarization 44 5.19%
Sentence 37 4.36%
Text Summarization 26 3.07%
Translation 18 2.12%
Information Retrieval 16 1.89%

Categories