Network Pruning

115 papers with code • 5 benchmarks • 5 datasets

Network Pruning is a popular approach to reduce a heavy network to obtain a light-weight form by removing redundancy in the heavy network. In this approach, a complex over-parameterized network is first trained, then pruned based on come criterions, and finally fine-tuned to achieve comparable performance with reduced parameters.

Source: Ensemble Knowledge Distillation for Learning Improved and Efficient Networks

Greatest papers with code

Movement Pruning: Adaptive Sparsity by Fine-Tuning

huggingface/transformers NeurIPS 2020

Magnitude pruning is a widely used strategy for reducing model size in pure supervised learning; however, it is less effective in the transfer learning regime that has become standard for state-of-the-art natural language processing applications.

Fine-tuning Network Pruning +1

What Do Compressed Deep Neural Networks Forget?

google-research/google-research 13 Nov 2019

However, this measure of performance conceals significant differences in how different classes and images are impacted by model compression techniques.

Fairness Interpretability Techniques for Deep Learning +4

Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding

NervanaSystems/distiller 1 Oct 2015

To address this limitation, we introduce "deep compression", a three stage pipeline: pruning, trained quantization and Huffman coding, that work together to reduce the storage requirement of neural networks by 35x to 49x without affecting their accuracy.

Network Pruning Quantization

Network Pruning via Transformable Architecture Search

D-X-Y/NAS-Projects NeurIPS 2019

The maximum probability for the size in each distribution serves as the width and depth of the pruned network, whose parameters are learned by knowledge transfer, e. g., knowledge distillation, from the original networks.

Knowledge Distillation Network Pruning +2

Rethinking the Value of Network Pruning

Eric-mingjie/rethinking-network-pruning ICLR 2019

Our observations are consistent for multiple network architectures, datasets, and tasks, which imply that: 1) training a large, over-parameterized model is often not necessary to obtain an efficient final model, 2) learned "important" weights of the large model are typically not useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited "important" weights, is more crucial to the efficiency in the final model, which suggests that in some cases pruning can be useful as an architecture search paradigm.

Fine-tuning Network Pruning +1

FastDepth: Fast Monocular Depth Estimation on Embedded Systems

dwofk/fast-depth 8 Mar 2019

In this paper, we address the problem of fast depth estimation on embedded systems.

Monocular Depth Estimation Network Pruning

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

google-research/lottery-ticket-hypothesis ICLR 2019

Based on these results, we articulate the "lottery ticket hypothesis:" dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations.

Network Pruning

Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion

NVlabs/DeepInversion CVPR 2020

We introduce DeepInversion, a new method for synthesizing images from the image distribution used to train a deep neural network.

Continual Learning Network Pruning +1

What is the State of Neural Network Pruning?

jjgo/shrinkbench 6 Mar 2020

Neural network pruning---the task of reducing the size of a network by removing parameters---has been the subject of a great deal of work in recent years.

Network Pruning

EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning

anonymous47823493/EagleEye ECCV 2020

Many algorithms try to predict model performance of the pruned sub-nets by introducing various evaluation methods.

Fine-tuning Network Pruning