About

Transfer learning is a methodology where weights from a model trained on one task are taken and either used (a) to construct a fixed feature extractor, (b) as weight initialization and/or fine-tuning.

( Image credit: Subodh Malgonde )

Benchmarks

TREND DATASET BEST METHOD PAPER TITLE PAPER CODE COMPARE

Subtasks

Greatest papers with code

Talking-Heads Attention

5 Mar 2020tensorflow/models

We introduce "talking-heads attention" - a variation on multi-head attention which includes linearprojections across the attention-heads dimension, immediately before and after the softmax operation. While inserting only a small number of additional parameters and a moderate amount of additionalcomputation, talking-heads attention leads to better perplexities on masked language modeling tasks, aswell as better quality when transfer-learning to language comprehension and question answering tasks.

LANGUAGE MODELLING QUESTION ANSWERING TRANSFER LEARNING

BARThez: a Skilled Pretrained French Sequence-to-Sequence Model

23 Oct 2020huggingface/transformers

We show BARThez to be very competitive with state-of-the-art BERT-based French language models such as CamemBERT and FlauBERT.

 Ranked #1 on Text Summarization on OrangeSum (using extra training data)

NATURAL LANGUAGE UNDERSTANDING SELF-SUPERVISED LEARNING TEXT SUMMARIZATION TRANSFER LEARNING

fairseq S2T: Fast Speech-to-Text Modeling with fairseq

11 Oct 2020huggingface/transformers

We introduce fairseq S2T, a fairseq extension for speech-to-text (S2T) modeling tasks such as end-to-end speech recognition and speech-to-text translation.

END-TO-END SPEECH RECOGNITION MACHINE TRANSLATION MULTI-TASK LEARNING SPEECH RECOGNITION SPEECH-TO-TEXT TRANSLATION

TAPAS: Weakly Supervised Table Parsing via Pre-training

ACL 2020 huggingface/transformers

In this paper, we present TAPAS, an approach to question answering over tables without generating logical forms.

QUESTION ANSWERING SEMANTIC PARSING TRANSFER LEARNING

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

arXiv 2019 huggingface/transformers

Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP).

QUESTION ANSWERING SEMANTIC TEXTUAL SIMILARITY TRANSFER LEARNING

HuggingFace's Transformers: State-of-the-art Natural Language Processing

9 Oct 2019huggingface/transformers

Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks.

TEXT GENERATION TRANSFER LEARNING

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

NeurIPS 2019 huggingface/transformers

As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging.

KNOWLEDGE DISTILLATION LANGUAGE MODELLING LINGUISTIC ACCEPTABILITY NATURAL LANGUAGE INFERENCE QUESTION ANSWERING SEMANTIC TEXTUAL SIMILARITY SENTIMENT ANALYSIS TRANSFER LEARNING

Movement Pruning: Adaptive Sparsity by Fine-Tuning

NeurIPS 2020 huggingface/transformers

Magnitude pruning is a widely used strategy for reducing model size in pure supervised learning; however, it is less effective in the transfer learning regime that has become standard for state-of-the-art natural language processing applications.

NETWORK PRUNING TRANSFER LEARNING