Search Results for author:
Found 27 papers, 17 papers with code
DropCompute: simple and more robust distributed synchronous training via compute variance reduction
Thus, these methods are limited by the delays caused by straggling workers.
Energy awareness in low precision neural networks
Power consumption is a major obstacle in the deployment of deep neural networks (DNNs) on end devices.
Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formats
Reducing the computational footprint of the entire training process requires the quantization of the neural gradients, i. e., the loss gradients with respect to the outputs of intermediate neural layers.
Beyond Quantization: Power aware neural networks
Power consumption is a major obstacle in the deployment of deep neural networks (DNNs) on end devices.
Logarithmic Unbiased Quantization: Practical 4-bit Training in Deep Learning
Based on this, we suggest a logarithmic unbiased quantization (LUQ) method to quantize both the forward and backward phase to 4-bit, achieving state-of-the-art results in 4-bit training.
MixSize: Training Convnets With Mixed Image Sizes for Improved Accuracy, Speed and Scale Resiliency
Although trained on images of a specific size, it is well established that CNNs can be used to evaluate a wide range of image sizes at test time, by adjusting the size of intermediate feature maps.
Task Agnostic Continual Learning Using Online Variational Bayes with Fixed-Point Updates
The optimal Bayesian solution for this requires an intractable online Bayes update to the weights posterior.
Neural gradients are near-lognormal: improved quantized and sparse training
While training can mostly be accelerated by reducing the time needed to propagate neural gradients back throughout the model, most previous works focus on the quantization/pruning of weights and activations.
Augment Your Batch: Improving Generalization Through Instance Repetition
Large-batch SGD is important for scaling training of deep neural networks.
The Knowledge Within: Methods for Data-Free Model Compression
Then, we demonstrate how these samples can be used to calibrate and fine-tune quantized models without using any real data in the process.
At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?
However, asynchronous training has its pitfalls, mainly a degradation in generalization, even after convergence of the algorithm.
Mix & Match: training convnets with mixed image sizes for improved accuracy, speed and scale resiliency
Although trained on images of aspecific size, it is well established that CNNs can be used to evaluate a wide range of image sizes at test time, by adjusting the size of intermediate feature maps.
ACIQ: Analytical Clipping for Integer Quantization of neural networks
We analyze the trade-off between quantization noise and clipping distortion in low precision networks.
Augment your batch: better training with larger batches
We analyze the effect of batch augmentation on gradient variance and show that it empirically improves convergence for a wide variety of deep neural networks and datasets.
Post-training 4-bit quantization of convolution networks for rapid-deployment
Convolutional neural networks require significant memory bandwidth and storage for intermediate computations, apart from substantial computing resources.
Scalable Methods for 8-bit Training of Neural Networks
Armed with this knowledge, we quantize the model parameters, activations and layer gradients to 8-bit, leaving at a higher precision only the final step in the computation of the weight gradients.
Task Agnostic Continual Learning Using Online Variational Bayes
However, research for scenarios in which task boundaries are unknown during training has been lacking.
Norm matters: efficient and accurate normalization schemes in deep networks
Over the past few years, Batch-Normalization has been commonly used in deep networks, allowing faster training and high performance for a wide variety of applications.
On the Blindspots of Convolutional Networks
Deep convolutional network has been the state-of-the-art approach for a wide variety of tasks over the last few years.
Fix your classifier: the marginal value of training the last weight layer
Neural networks are commonly used as models for classification for a wide variety of tasks.
The Implicit Bias of Gradient Descent on Separable Data
We examine gradient descent on unregularized logistic regression problems, with homogeneous linear predictors on linearly separable datasets.
Train longer, generalize better: closing the generalization gap in large batch training of neural networks
Following this hypothesis we conducted experiments to show empirically that the "generalization gap" stems from the relatively small number of updates rather than the batch size, and can be completely eliminated by adapting the training regime used.
Exponentially vanishing sub-optimal local minima in multilayer neural networks
We prove that, with high probability in the limit of $N\rightarrow\infty$ datapoints, the volume of differentiable regions of the empiric loss containing sub-optimal differentiable local minima is exponentially vanishing in comparison with the same volume of global minima, given standard normal input of dimension $d_{0}=\tilde{\Omega}\left(\sqrt{N}\right)$, and a more realistic number of $d_{1}=\tilde{\Omega}\left(N/d_{0}\right)$ hidden units.
Spatial contrasting for deep unsupervised learning
Convolutional networks have marked their place over the last few years as the best performing model for various visual tasks.
Semi-supervised deep learning by metric embedding
Deep networks are successfully used as classification models yielding state-of-the-art results when trained on a large number of labeled samples.
Deep unsupervised learning through spatial contrasting
Convolutional networks have marked their place over the last few years as the best performing model for various visual tasks.
Deep metric learning using Triplet network
Deep learning has proven itself as a successful set of models for learning useful semantic representations of data.
