Knowledge Distillation
1018 papers with code • 5 benchmarks • 4 datasets
Knowledge distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have higher knowledge capacity than small models, this capacity might not be fully utilized.
Benchmarks
These leaderboards are used to track progress in Knowledge Distillation
| Trend | Dataset | Best Model | Paper | Code | Compare |
|---|---|---|---|---|---|
|
|||||
|
|||||
|
|||||
|
|||||
|
Libraries
Use these libraries to find Knowledge Distillation models and implementations
ImageNet
COCO
CIFAR-100
PASCAL VOC
Most implemented papers
Focal Loss for Dense Object Detection
facebookresearch/detectron
•
•
ICCV 2017
Our novel Focal Loss focuses training on a sparse set of hard examples and prevents the vast number of easy negatives from overwhelming the detector during training.
Distilling the Knowledge in a Neural Network
labmlai/annotated_deep_learning_paper_implementations
•
•
A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.
Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
google-research/bert
•
•
ICLR 2020
Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training.
FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e. g., pitch, energy and more accurate duration) as conditional inputs.
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
huggingface/transformers
•
•
NeurIPS 2019
As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging.
Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks
adityac94/Grad_CAM_plus_plus
•
•
Over the last decade, Convolutional Neural Network (CNN) models have been highly successful in solving complex vision problems.
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
UKPLab/sentence-transformers
•
•
EMNLP 2020
The training is based on the idea that a translated sentence should be mapped to the same location in the vector space as the original sentence.
TinyBERT: Distilling BERT for Natural Language Understanding
huawei-noah/Pretrained-Language-Model
•
•
Findings of the Association for Computational Linguistics 2020
To accelerate inference and reduce model size while maintaining accuracy, we first propose a novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-based models.
Distilling Knowledge via Knowledge Review
Jia-Research-Lab/ReviewKD
•
•
CVPR 2021
Knowledge distillation transfers knowledge from the teacher network to the student one, with the goal of greatly improving the performance of the student network.
Sequence-Level Knowledge Distillation
harvardnlp/seq2seq-attn
•
•
EMNLP 2016
We demonstrate that standard knowledge distillation applied to word-level prediction can be effective for NMT, and also introduce two novel sequence-level versions of knowledge distillation that further improve performance, and somewhat surprisingly, seem to eliminate the need for beam search (even when applied on the original teacher model).
