Search Results for author:
Found 65 papers, 33 papers with code
How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teachers
We prove that such a random NN interpolator typically generalizes well if there exists an underlying narrow ``teacher NN" that agrees with the labels.
Towards Cheaper Inference in Deep Networks with Lower Bit-Width Accumulators
The majority of the research on the quantization of Deep Neural Networks (DNNs) is focused on reducing the precision of tensors visible by high-level frameworks (e. g., weights, activations, and gradients).
The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting -- An Analytical Model
Previous works have analyzed separately how forgetting is affected by either task similarity or overparameterization.
How do Minimum-Norm Shallow Denoisers Look in Function Space?
Neural network (NN) denoisers are an essential building block in many common tasks, ranging from image reconstruction to image generation.
The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networks
Finally, we prove that if a function is sufficiently smooth (in a Sobolev sense) then it can be approximated arbitrarily well using shallow ReLU networks that correspond to stable solutions of gradient descent.
DropCompute: simple and more robust distributed synchronous training via compute variance reduction
Thus, these methods are limited by the delays caused by straggling workers.
Continual Learning in Linear Classification on Separable Data
We analyze continual learning on a sequence of separable linear classification tasks with binary labels.
Explore to Generalize in Zero-Shot RL
Our insight is that learning a policy that effectively $\textit{explores}$ the domain is harder to memorize than a policy that maximizes reward for a specific task, and therefore we expect such learned behavior to generalize well; we indeed demonstrate this empirically on several domains that are difficult for invariance-based approaches.
Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyond
Using this result, we characterize settings where GD provably converges to the EoS in scalar networks.
Alias-Free Convnets: Fractional Shift Invariance via Polynomial Activations
Although CNNs are believed to be invariant to translations, recent works have shown this is not the case, due to aliasing effects that stem from downsampling layers.
The Role of Codeword-to-Class Assignments in Error-Correcting Codes: An Empirical Study
Error-correcting codes (ECC) are used to reduce multiclass classification tasks to multiple binary classification subproblems.
How catastrophic can catastrophic forgetting be in linear regression?
In specific settings, we highlight differences between forgetting and convergence to the offline solution as studied in those areas.
Optimal Fine-Grained N:M sparsity for Activations and Neural Gradients
We show that while minimization of the MSE works fine for pruning the activations, it catastrophically fails for the neural gradients.
Logarithmic Unbiased Quantization: Simple 4-bit Training in Deep Learning
Based on this, we suggest a \textit{logarithmic unbiased quantization} (LUQ) method to quantize all both the forward and backward phase to 4-bit, achieving state-of-the-art results in 4-bit training without overhead.
The Implicit Bias of Minima Stability: A View from Function Space
First, we extend the existing knowledge on minima stability to non-differentiable minima, which are common in ReLU nets.
Logarithmic Unbiased Quantization: Practical 4-bit Training in Deep Learning
Based on this, we suggest a logarithmic unbiased quantization (LUQ) method to quantize both the forward and backward phase to 4-bit, achieving state-of-the-art results in 4-bit training.
Regularization Guarantees Generalization in Bayesian Reinforcement Learning through Algorithmic Stability
In the Bayesian reinforcement learning (RL) setting, a prior distribution over the unknown problem parameters -- the rewards and transitions -- is assumed, and a policy that optimizes the (posterior) expected return is sought.
Physics-Aware Downsampling with Deep Learning for Scalable Flood Modeling
Background: Floods are the most common natural disaster in the world, affecting the lives of hundreds of millions.
A statistical framework for efficient out of distribution detection in deep neural networks
Our method achieves comparable or better results than state-of-the-art methods on well-accepted OOD benchmarks, without retraining the network parameters or assuming prior knowledge on the test distribution -- and at a fraction of the computational cost.
On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent
Recent work has highlighted the role of initialization scale in determining the structure of the solutions that gradient methods converge to.
Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks
Finally, to solve the problem of switching between different structure constraints, we suggest a method to convert a pre-trained model with unstructured sparsity to an N:M fine-grained block sparsity model with little to no training.
MixSize: Training Convnets With Mixed Image Sizes for Improved Accuracy, Speed and Scale Resiliency
Although trained on images of a specific size, it is well established that CNNs can be used to evaluate a wide range of image sizes at test time, by adjusting the size of intermediate feature maps.
Why Cold Posteriors? On the Suboptimal Generalization of Optimal Bayes Estimates
Recent works have shown that the predictive accuracy of Bayesian deep learning models exhibit substantial improvements when the posterior is raised to a 1/T power with T<1.
Task Agnostic Continual Learning Using Online Variational Bayes with Fixed-Point Updates
The optimal Bayesian solution for this requires an intractable online Bayes update to the weights posterior.
Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy
We provide a detailed asymptotic study of gradient flow trajectories and their implicit optimization bias when minimizing the exponential loss over "diagonal linear networks".
Beyond Signal Propagation: Is Feature Diversity Necessary in Deep Neural Network Initialization?
Deep neural networks are typically initialized with random weights, with variances chosen to facilitate signal propagation and stable gradients.
Neural gradients are near-lognormal: improved quantized and sparse training
While training can mostly be accelerated by reducing the time needed to propagate neural gradients back throughout the model, most previous works focus on the quantization/pruning of weights and activations.
Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming
Instead, these methods only use the calibration set to set the activations' dynamic ranges.
Kernel and Rich Regimes in Overparametrized Models
We provide a complete and detailed analysis for a family of simple depth-$D$ models that already exhibit an interesting and meaningful transition between the kernel and rich regimes, and we also demonstrate this transition empirically for more complex matrix factorization models and multilayer non-linear networks.
Training of Quantized Deep Neural Networks using a Magnetic Tunnel Junction-Based Synapse
A recent example is the GXNOR framework for stochastic training of ternary (TNN) and binary (BNN) neural networks.
Is Feature Diversity Necessary in Neural Network Initialization?
Standard practice in training neural networks involves initializing the weights in an independent fashion.
The Knowledge Within: Methods for Data-Free Model Compression
Then, we demonstrate how these samples can be used to calibrate and fine-tune quantized models without using any real data in the process.
Post training 4-bit quantization of convolutional networks for rapid-deployment
Convolutional neural networks require significant memory bandwidth and storage for intermediate computations, apart from substantial computing resources.
A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case
In this paper, we characterize the norm required to realize a function $f:\mathbb{R}^d\rightarrow\mathbb{R}$ as a single hidden-layer ReLU network with an unbounded number of units (infinite width), but where the Euclidean norm of the weights is bounded, including precisely characterizing which functions can be realized with finite norm.
At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?
However, asynchronous training has its pitfalls, mainly a degradation in generalization, even after convergence of the algorithm.
Mix & Match: training convnets with mixed image sizes for improved accuracy, speed and scale resiliency
Although trained on images of aspecific size, it is well established that CNNs can be used to evaluate a wide range of image sizes at test time, by adjusting the size of intermediate feature maps.
Kernel and Rich Regimes in Overparametrized Models
A recent line of work studies overparametrized neural networks in the "kernel regime," i. e. when the network behaves during training as a kernelized linear predictor, and thus training with gradient descent has the effect of finding the minimum RKHS norm solution.
A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off
Reducing the precision of weights and activation functions in neural network training, with minimal impact on performance, is essential for the deployment of these models in resource-constrained environments.
Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models
With an eye toward understanding complexity control in deep learning, we study how infinitesimal regularization or gradient descent optimization lead to margin maximizing solutions in both homogeneous and non-homogeneous models, extending previous work that focused on infinitesimal regularization only in homogeneous models.
ACIQ: Analytical Clipping for Integer Quantization of neural networks
We analyze the trade-off between quantization noise and clipping distortion in low precision networks.
How do infinite width bounded norm networks look in function space?
We consider the question of what functions can be captured by ReLU networks with an unbounded number of units (infinite width), but where the overall network Euclidean norm (sum of squares of all weights in the system, except for an unregularized bias term for each unit) is bounded; or equivalently what is the minimal norm required to approximate a given function.
Augment your batch: better training with larger batches
We analyze the effect of batch augmentation on gradient variance and show that it empirically improves convergence for a wide variety of deep neural networks and datasets.
Post-training 4-bit quantization of convolution networks for rapid-deployment
Convolutional neural networks require significant memory bandwidth and storage for intermediate computations, apart from substantial computing resources.
Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate
We prove that SGD converges to zero loss, even with a fixed (non-vanishing) learning rate - in the special case of homogeneous linear classifiers with smooth monotone loss functions, optimized on linearly separable data.
Implicit Bias of Gradient Descent on Linear Convolutional Networks
We show that gradient descent on full-width linear convolutional networks of depth $L$ converges to a linear predictor related to the $\ell_{2/L}$ bridge penalty in the frequency domain.
Scalable Methods for 8-bit Training of Neural Networks
Armed with this knowledge, we quantize the model parameters, activations and layer gradients to 8-bit, leaving at a higher precision only the final step in the computation of the weight gradients.
The Global Optimization Geometry of Shallow Linear Neural Networks
We examine the squared error loss landscape of shallow linear neural networks.
Task Agnostic Continual Learning Using Online Variational Bayes
However, research for scenarios in which task boundaries are unknown during training has been lacking.
Convergence of Gradient Descent on Separable Data
We show that for a large family of super-polynomial tailed losses, gradient descent iterates on linear networks of any depth converge in the direction of $L_2$ maximum-margin solution, while this does not hold for losses with heavier tails.
Norm matters: efficient and accurate normalization schemes in deep networks
Over the past few years, Batch-Normalization has been commonly used in deep networks, allowing faster training and high performance for a wide variety of applications.
Characterizing Implicit Bias in Terms of Optimization Geometry
We study the implicit bias of generic optimization methods, such as mirror descent, natural gradient descent, and steepest descent with respect to different potentials and norms, when optimizing underdetermined linear regression or separable linear classification problems.
On the Blindspots of Convolutional Networks
Deep convolutional network has been the state-of-the-art approach for a wide variety of tasks over the last few years.
Fix your classifier: the marginal value of training the last weight layer
Neural networks are commonly used as models for classification for a wide variety of tasks.
The Implicit Bias of Gradient Descent on Separable Data
We examine gradient descent on unregularized logistic regression problems, with homogeneous linear predictors on linearly separable datasets.
Train longer, generalize better: closing the generalization gap in large batch training of neural networks
Following this hypothesis we conducted experiments to show empirically that the "generalization gap" stems from the relatively small number of updates rather than the batch size, and can be completely eliminated by adapting the training regime used.
Exponentially vanishing sub-optimal local minima in multilayer neural networks
We prove that, with high probability in the limit of $N\rightarrow\infty$ datapoints, the volume of differentiable regions of the empiric loss containing sub-optimal differentiable local minima is exponentially vanishing in comparison with the same volume of global minima, given standard normal input of dimension $d_{0}=\tilde{\Omega}\left(\sqrt{N}\right)$, and a more realistic number of $d_{1}=\tilde{\Omega}\left(N/d_{0}\right)$ hidden units.
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
Quantized recurrent neural networks were tested over the Penn Treebank dataset, and achieved comparable accuracy as their 32-bit counterparts using only 4-bits.
No bad local minima: Data independent training error guarantees for multilayer neural networks
We use smoothed analysis techniques to provide guarantees on the training loss of Multilayer Neural Networks (MNNs) at differentiable local minima.
Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time.
Binarized Neural Networks
We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time and when computing the parameters' gradient at train-time.
Training Binary Multilayer Neural Networks for Image Classification using Expectation Backpropagation
In this paper, we investigate the capability of BMNNs using the EBP algorithm on multiclass image classification tasks.
Expectation Backpropagation: Parameter-Free Training of Multilayer Neural Networks with Continuous or Discrete Weights
Using online EP and the central limit theorem we find an analytical approximation to the Bayes update of this posterior, as well as the resulting Bayes estimates of the weights and outputs.
A structured matrix factorization framework for large scale calcium imaging data analysis
We present a structured matrix factorization approach to analyzing calcium imaging recordings of large neuronal ensembles.
Neurons and Cognition Quantitative Methods Applications
Mean Field Bayes Backpropagation: scalable training of multilayer neural networks with binary weights
Significant success has been reported recently using deep neural networks for classification.
Neuronal Spike Generation Mechanism as an Oversampling, Noise-shaping A-to-D converter
If noise-shaping were used in neurons, it would introduce correlations in spike timing to reduce low-frequency (up to Nyquist) transmission error at the cost of high-frequency one (from Nyquist to sampling rate).
