Search Results for author:
Found 17 papers, 5 papers with code
Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neurons
When training deep neural networks, the phenomenon of $\textit{dying neurons}$ $\unicode{x2013}$units that become inactive or saturated, output zero during training$\unicode{x2013}$ has traditionally been viewed as undesirable, linked with optimization challenges, and contributing to plasticity loss in continual learning scenarios.
Unsupervised Concept Discovery Mitigates Spurious Correlations
Models prone to spurious correlations in training data often produce brittle predictions and introduce unintended biases.
How connectivity structure shapes rich and lazy learning in neural circuits
Through both empirical and theoretical analyses, we discover that high-rank initializations typically yield smaller network changes indicative of lazier learning, a finding we also confirm with experimentally-driven initial connectivity in recurrent neural networks.
Lookbehind-SAM: k steps back, 1 step forward
Sharpness-aware minimization (SAM) methods have gained increasing popularity by formulating the problem of minimizing both loss value and loss sharpness as a minimax objective.
Promoting Exploration in Memory-Augmented Adam using Critical Momenta
Adaptive gradient-based optimizers, particularly Adam, have left their mark in training large-scale deep learning models.
CrossSplit: Mitigating Label Noise Memorization through Data Splitting
We approach the problem of improving robustness of deep learning algorithms in the presence of label noise.
Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficulty
Among attempts at giving a theoretical account of the success of deep neural networks, a recent line of work has identified a so-called lazy training regime in which the network can be well approximated by its linearization around initialization.
Using Representation Expressiveness and Learnability to Evaluate Self-Supervised Learning Methods
We address the problem of evaluating the quality of self-supervised learning (SSL) models without access to supervised labels, while being agnostic to the architecture, learning algorithm or data manipulation used during training.
Learnability and Expressiveness in Self-Supervised Learning
In this work, we argue that representations induced by self-supervised learning (SSL) methods should both be expressive and learnable.
On the Regularity of Attention
Attention is a powerful component of modern neural networks across a wide variety of domains.
Implicit Regularization via Neural Feature Alignment
We approach the problem of implicit regularization in deep learning from a geometrical viewpoint.
A Mathematical Theory of Attention
Attention is a powerful component of modern neural networks across a wide variety of domains.
A Modern Take on the Bias-Variance Tradeoff in Neural Networks
The bias-variance tradeoff tells us that as model complexity increases, bias falls and variances increases, leading to a U-shaped test error curve.
Mutual Information Neural Estimation
We argue that the estimation of mutual information between high dimensional continuous random variables can be achieved by gradient descent over neural networks.
On the Spectral Bias of Neural Networks
Neural networks are known to be a class of highly expressive functions able to fit even random input-output mappings with $100\%$ accuracy.
MINE: Mutual Information Neural Estimation
We argue that the estimation of mutual information between high dimensional continuous random variables can be achieved by gradient descent over neural networks.
A3T: Adversarially Augmented Adversarial Training
Recent research showed that deep neural networks are highly sensitive to so-called adversarial perturbations, which are tiny perturbations of the input data purposely designed to fool a machine learning classifier.
