Search Results for author:
Found 21 papers, 15 papers with code
Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and Tasks
Many recent methods aim to merge neural networks (NNs) with identical architectures trained on different tasks to obtain a single multi-task model.
Towards Cheaper Inference in Deep Networks with Lower Bit-Width Accumulators
The majority of the research on the quantization of Deep Neural Networks (DNNs) is focused on reducing the precision of tensors visible by high-level frameworks (e. g., weights, activations, and gradients).
Minimum Variance Unbiased N:M Sparsity for the Neural Gradients
We show that while minimization of the MSE works fine for pruning the weights and activations, it catastrophically fails for the neural gradients.
Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks
Finally, to solve the problem of switching between different structure constraints, we suggest a method to convert a pre-trained model with unstructured sparsity to an N:M fine-grained block sparsity model with little to no training.
MixSize: Training Convnets With Mixed Image Sizes for Improved Accuracy, Speed and Scale Resiliency
Although trained on images of a specific size, it is well established that CNNs can be used to evaluate a wide range of image sizes at test time, by adjusting the size of intermediate feature maps.
Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming
Instead, these methods only use the calibration set to set the activations' dynamic ranges.
Training of Quantized Deep Neural Networks using a Magnetic Tunnel Junction-Based Synapse
A recent example is the GXNOR framework for stochastic training of ternary (TNN) and binary (BNN) neural networks.
The Knowledge Within: Methods for Data-Free Model Compression
Then, we demonstrate how these samples can be used to calibrate and fine-tune quantized models without using any real data in the process.
Mix & Match: training convnets with mixed image sizes for improved accuracy, speed and scale resiliency
Although trained on images of aspecific size, it is well established that CNNs can be used to evaluate a wide range of image sizes at test time, by adjusting the size of intermediate feature maps.
Augment your batch: better training with larger batches
We analyze the effect of batch augmentation on gradient variance and show that it empirically improves convergence for a wide variety of deep neural networks and datasets.
Scalable Methods for 8-bit Training of Neural Networks
Armed with this knowledge, we quantize the model parameters, activations and layer gradients to 8-bit, leaving at a higher precision only the final step in the computation of the weight gradients.
Fix your classifier: the marginal value of training the last weight layer
Neural networks are commonly used as models for classification for a wide variety of tasks.
Train longer, generalize better: closing the generalization gap in large batch training of neural networks
Following this hypothesis we conducted experiments to show empirically that the "generalization gap" stems from the relatively small number of updates rather than the batch size, and can be completely eliminated by adapting the training regime used.
Spatial contrasting for deep unsupervised learning
Convolutional networks have marked their place over the last few years as the best performing model for various visual tasks.
Playing SNES in the Retro Learning Environment
The environment is expandable, allowing for more video games and consoles to be easily added to the environment, while maintaining the same interface as ALE.
Ranked #1 on
SNES Games
on F-Zero
Deep unsupervised learning through spatial contrasting
Convolutional networks have marked their place over the last few years as the best performing model for various visual tasks.
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
Quantized recurrent neural networks were tested over the Penn Treebank dataset, and achieved comparable accuracy as their 32-bit counterparts using only 4-bits.
Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time.
Binarized Neural Networks
We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time and when computing the parameters' gradient at train-time.
Expectation Backpropagation: Parameter-Free Training of Multilayer Neural Networks with Continuous or Discrete Weights
Using online EP and the central limit theorem we find an analytical approximation to the Bayes update of this posterior, as well as the resulting Bayes estimates of the weights and outputs.
