|TREND||DATASET||BEST METHOD||PAPER TITLE||PAPER||CODE||COMPARE|
Network pruning has become the de facto tool to accelerate deep neural networks for mobile and edge applications.
This analysis of training a BNN at scale outlines the limitations and benefits compared to a conventional neural network.
We present a novel network pruning algorithm called Dynamic Sparse Training that can jointly find the optimal network parameters and sparse network structure in a unified optimization process with trainable pruning thresholds.
Artificial Neural Networks (NN) are widely used for solving complex problems from medical diagnostics to face recognition.
Network parameter reduction methods have been introduced to systematically deal with the computational and memory complexity of deep networks.
In order to contrast the explosion in size of state-of-the-art machine learning models that can be attributed to the empirical advantages of over-parametrization, and due to the necessity of deploying fast, sustainable, and private on-device models on resource-constrained devices, the community has focused on techniques such as pruning, quantization, and distillation as central strategies for model compression.
In most cases our method finds better selections than even the best individual pruning saliency.
The challenge of speeding up deep learning models during the deployment phase has been a large, expensive bottleneck in the modern tech industry.
As a result, we find that the data parallelism in training sparse neural networks is no worse than that in training densely parameterized neural networks, despite the general difficulty of training sparse neural networks.