BARThez: a Skilled Pretrained French Sequence-to-Sequence Model

Inductive transfer learning has taken the entire NLP field by storm, with models such as BERT and BART setting new state of the art on countless NLU tasks. However, most of the available models and research have been conducted for English... (read more)

PDF Abstract

Datasets


Introduced in the Paper:

OrangeSum

Mentioned in the Paper:

GLUE

Results from the Paper


 Ranked #1 on Text Summarization on OrangeSum (using extra training data)

     Get a GitHub badge
TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK USES EXTRA
TRAINING DATA
RESULT BENCHMARK
Text Summarization OrangeSum BARThez (OrangeSum abstract) ROUGE-1 31.44 # 2
Text Summarization OrangeSum mBARThez (OrangeSum abstract) ROUGE-1 32.67 # 1

Methods used in the Paper


METHOD TYPE
Sigmoid Activation
Activation Functions
Tanh Activation
Activation Functions
LSTM
Recurrent Neural Networks
Seq2Seq
Sequence To Sequence Models
BPE
Subword Segmentation
Adam
Stochastic Optimization
Softmax
Output Functions
Layer Normalization
Normalization
Dense Connections
Feedforward Networks
Multi-Head Attention
Attention Modules
Dropout
Regularization
GELU
Activation Functions
Linear Warmup With Linear Decay
Learning Rate Schedules
Attention Dropout
Regularization
Weight Decay
Regularization
Residual Connection
Skip Connections
Scaled Dot-Product Attention
Attention Mechanisms
BART
Transformers
WordPiece
Subword Segmentation
BERT
Language Models