Search Results for author:
Found 22 papers, 5 papers with code
Neglected Hessian component explains mysteries in Sharpness regularization
We find that regularizing feature exploitation but not feature exploration yields performance similar to gradient penalties.
On the Foundations of Shortcut Learning
Deep-learning models can extract a rich assortment of features from data.
On student-teacher deviations in distillation: does it pay to disobey?
Knowledge distillation (KD) has been widely used to improve the test accuracy of a "student" network, by training it to mimic the soft probabilities of a trained "teacher" network.
Sharpness-Aware Minimization Improves Language Model Generalization
The allure of superhuman-level capabilities has led to considerable interest in language models like GPT-3 and T5, wherein the research has, by and large, revolved around new model architectures, training tasks, and loss objectives, along with substantial engineering efforts to scale up model capacity and dataset size.
An Empirical Study of Pre-trained Models on Out-of-distribution Generalization
Concretely, we show that larger models and larger datasets need to be simultaneously leveraged to improve OOD performance.
The Low-Rank Simplicity Bias in Deep Networks
We show empirically that our claim holds true on finite width linear and non-linear models on practical learning paradigms and show that on natural data, these are often the solutions that generalize well.
NeurIPS 2020 Competition: Predicting Generalization in Deep Learning
Understanding generalization in deep learning is arguably one of the most important questions in deep learning.
Data Augmentation via Structured Adversarial Perturbations
The advantage of adversarial augmentation is that it replaces sampling with the use of a single, calculated perturbation that maximally increases the loss.
A Unifying View on Implicit Bias in Training Linear Neural Networks
For $L$-layer linear tensor networks that are orthogonally decomposable, we show that gradient flow on separable classification finds a stationary point of the $\ell_{2/L}$ max-margin problem in a "transformed" input space defined by the network.
Sharpness-Aware Minimization for Efficiently Improving Generalization
In today's heavily overparameterized models, the value of the training loss provides few guarantees on model generalization ability.
Ranked #1 on
Image Classification
on CIFAR-100
(using extra training data)
Fine-Grained Image Classification
Learning with noisy labels
Self-Distillation Amplifies Regularization in Hilbert Space
Knowledge distillation introduced in the deep learning context is a method to transfer knowledge from one architecture to another.
Fantastic Generalization Measures and Where to Find Them
We present the first large scale study of generalization in deep networks.
A Closed-Form Learned Pooling for Deep Classification Networks
This operator can learn a strict super-set of what can be learned by average pooling or convolutions.
Semantic Redundancies in Image-Classification Datasets: The 10% You Don't Need
Large datasets have been crucial to the success of deep learning models in the recent years, which keep performing better as they are trained with more labelled data.
Predicting the Generalization Gap in Deep Networks with Margin Distributions
In this paper, we propose such a measure, and conduct extensive empirical studies on how well it can predict the generalization gap.
Large Margin Deep Networks for Classification
We present a formulation of deep learning that aims at producing a large margin classifier.
Homotopy Analysis for Tensor PCA
For the challenging problem of tensor PCA, we prove global convergence of the homotopy method in the "high noise" regime.
A Theory of Local Matching: SIFT and Beyond
Can it suggest new algorithms with reduced computational complexity or new descriptors with better accuracy for matching?
Training Recurrent Neural Networks by Diffusion
This work presents a new algorithm for training recurrent neural networks (although ideas are applicable to feedforward networks as well).
Learning with a Wasserstein Loss
In this paper we develop a loss function for multi-label learning, based on the Wasserstein distance.
The Aperture Problem for Refractive Motion
We pose and solve a generalization of the aperture problem for moving refractive elements.
A Compositional Model for Low-Dimensional Image Set Representation
Learning a low-dimensional representation of images is useful for various applications in graphics and computer vision.
