Model Compression
171 papers with code • 0 benchmarks • 1 datasets
Model Compression is an actively pursued area of research over the last few years with the goal of deploying state-of-the-art deep networks in low-power and resource limited devices without significant drop in accuracy. Parameter pruning, low-rank factorization and weight quantization are some of the proposed methods to compress the size of deep networks.
Benchmarks
INRIA Aerial Image Labeling
Greatest papers with code
Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
google-research/bert
•
•
ICLR 2020
Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training.
What Do Compressed Deep Neural Networks Forget?
google-research/google-research
•
•
However, this measure of performance conceals significant differences in how different classes and images are impacted by model compression techniques.
The State of Sparsity in Deep Neural Networks
google-research/google-research
•
•
We rigorously evaluate three state-of-the-art techniques for inducing sparsity in deep neural networks on two large-scale learning tasks: Transformer trained on WMT 2014 English-to-German, and ResNet-50 trained on ImageNet.
Training with Quantization Noise for Extreme Model Compression
A standard solution is to train networks with Quantization Aware Training, where the weights are quantized during training and the gradients approximated with the Straight-Through Estimator.
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
pytorch/vision
•
•
(2) Smaller DNNs require less bandwidth to export a new model from the cloud to an autonomous car.
Model compression via distillation and quantization
NervanaSystems/distiller
•
•
ICLR 2018
Deep neural networks (DNNs) continue to make significant advances, solving tasks from image classification to translation or reinforcement learning.
AMC: AutoML for Model Compression and Acceleration on Mobile Devices
NervanaSystems/distiller
•
•
ECCV 2018
Model compression is a critical technique to efficiently deploy neural network models on mobile devices which have limited computation resources and tight power budgets.
BinaryBERT: Pushing the Limit of BERT Quantization
huawei-noah/Pretrained-Language-Model
•
•
ACL 2021
In this paper, we propose BinaryBERT, which pushes BERT quantization to the limit by weight binarization.
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks
FLHonker/Awesome-Knowledge-Distillation
•
•
To achieve faster speeds and to handle the problems caused by the lack of data, knowledge distillation (KD) has been proposed to transfer information learned from one model to another.
Contrastive Representation Distillation
HobbitLong/RepDistiller
•
•
ICLR 2020
We demonstrate that this objective ignores important structural knowledge of the teacher network.
Ranked #9 on
Knowledge Distillation
on ImageNet