Search Results for author:
Found 52 papers, 30 papers with code
Gradient descent induces alignment between weights and the empirical NTK for deep non-linear networks
Prior works have identified that the gram matrices of the weights in trained neural networks of general architectures are proportional to the average gradient outer product of the model, in a statement known as the Neural Feature Ansatz (NFA).
Expecting The Unexpected: Towards Broad Out-Of-Distribution Detection
Our findings reveal that while these methods excel in detecting unknown classes, their performance is inconsistent when encountering other types of distribution shifts.
An Empirical Investigation of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibration
In the realm of out-of-distribution (OOD) generalization tasks, fine-tuning pre-trained models has become a prevalent strategy.
No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths
Understanding the optimization dynamics of neural networks is necessary for closing the gap between theory and practice.
Performative Prediction with Neural Networks
In this work, we instead assume that the data distribution is Lipschitz continuous with respect to the model's predictions, a more natural assumption for performative systems.
Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning
Although disentangled representations are often said to be beneficial for downstream tasks, current empirical and theoretical understanding is limited.
Empirical Study on Optimizer Selection for Out-of-Distribution Generalization
Modern deep learning systems do not generalize well when the test data distribution is slightly different to the training data distribution.
Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation
We study the problem of model selection in causal inference, specifically for the case of conditional average treatment effect (CATE) estimation under binary treatments.
Towards Out-of-Distribution Adversarial Robustness
Compared to existing methods, we obtain similar or superior worst-case adversarial robustness on attacks seen during training.
A Reproducible and Realistic Evaluation of Partial Domain Adaptation Methods
In this work, we consider the Partial Domain Adaptation (PDA) variant, where we have extra source classes not present in the target domain.
Neural Networks Efficiently Learn Low-Dimensional Representations with SGD
We further demonstrate that, SGD-trained ReLU NNs can learn a single-index target of the form $y=f(\langle\boldsymbol{u},\boldsymbol{x}\rangle) + \epsilon$ by recovering the principal direction, with a sample complexity linear in $d$ (up to log factors), where $f$ is a monotonic function with at most polynomial growth, and $\epsilon$ is the noise.
Optimal transport meets noisy label robust loss and MixUp regularization for domain adaptation
It is common in computer vision to be confronted with domain shift: images which have the same class but different acquisition conditions.
A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Games
This work studies an algorithm, which we call magnetic mirror descent, that is inspired by mirror descent and the non-Euclidean proximal gradient algorithm.
Towards efficient representation identification in supervised learning
In this work, we depart from these assumptions and ask: a) How can we get disentanglement when the auxiliary information does not provide conditional independence over the factors of variation?
Stochastic Mirror Descent: Convergence Analysis and Adaptive Variants via the Mirror Stochastic Polyak Stepsize
We investigate the convergence of stochastic mirror descent (SMD) under interpolation in relatively smooth and smooth convex optimization.
Convergence Analysis and Implicit Regularization of Feedback Alignment for Deep Linear Networks
We theoretically analyze the Feedback Alignment (FA) algorithm, an efficient alternative to backpropagation for training neural networks.
Gotta Go Fast with Score-Based Generative Models
Score-based (denoising diffusion) generative models have recently gained a lot of success in generating realistic and diverse data.
Stochastic Gradient Descent-Ascent and Consensus Optimization for Smooth Games: Convergence Analysis under Expected Co-coercivity
Two of the most prominent algorithms for solving unconstrained smooth games are the classical stochastic gradient descent-ascent (SGDA) and the recently introduced stochastic consensus optimization (SCO) [Mescheder et al., 2017].
Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization
To answer these questions, we revisit the fundamental assumptions in linear regression tasks, where invariance-based approaches were shown to provably generalize OOD.
Gotta Go Fast When Generating Data with Score-Based Models
For high-resolution images, our method leads to significantly higher quality samples than all other methods tested.
Ranked #8 on
Image Generation
on CIFAR-10
(Inception score metric)
A Study of Condition Numbers for First-Order Optimization
Since smoothness and strong convexity are not continuous, we propose a comprehensive study of existing alternative metrics which we prove to be continuous.
LEAD: Min-Max Optimization from a Physical Perspective
A central obstacle in the optimization of such games is the rotational dynamics that hinder their convergence.
In Search of Robust Measures of Generalization
A large volume of work aims to close this gap, primarily by developing bounds on generalization error, optimization error, and excess risk.
Adversarial score matching and improved sampling for image generation
Denoising Score Matching with Annealed Langevin Sampling (DSM-ALS) has recently found success in generative modeling.
Ranked #54 on
Image Generation
on CIFAR-10
Stochastic Hamiltonian Gradient Methods for Smooth Games
The success of adversarial formulations in machine learning has brought renewed motivation for smooth games.
Accelerating Smooth Games by Manipulating Spectral Shapes
Using this perspective, we propose an optimal algorithm for bilinear games.
Generalizing to unseen domains via distribution matching
In this work, we tackle such problem by focusing on domain generalization: a formalization where the data generating process at test time may yield samples from never-before-seen domains (distributions).
Ranked #61 on
Domain Generalization
on PACS
Gradient penalty from a maximum margin perspective
We present a unifying framework of expected margin maximization and show that a wide range of gradient-penalized GANs (e. g., Wasserstein, Standard, Least-Squares, and Hinge GANs) can be derived from this framework.
Ranked #131 on
Image Generation
on CIFAR-10
Linear Lower Bounds and Conditioning of Differentiable Games
In this work, we approach the question of fundamental iteration complexity by providing lower bounds to complement the linear (i. e. geometric) upper bounds observed in the literature on a wide class of problems.
A Tight and Unified Analysis of Gradient-Based Methods for a Whole Spectrum of Games
We provide new analyses of the EG's local and global convergence properties and use is to get a tighter global convergence rate for OG and CO. Our analysis covers the whole range of settings between bilinear and strongly monotone games.
Reducing the variance in online optimization by transporting past gradients
While variance reduction methods have shown that reusing past gradients can be beneficial when there is a finite number of datapoints, they do not easily extend to the online setting.
State-Reification Networks: Improving Generalization by Modeling the Distribution of Hidden Representations
Machine learning promises methods that generalize well from finite labeled data.
In Support of Over-Parametrization in Deep Reinforcement Learning: an Empirical Study
There is significant recent evidence in supervised learning that, in the over-parametrized setting, wider networks achieve better test error.
Manifold Mixup: Learning Better Representations by Interpolating Hidden States
Because the hidden states are learned, this has an important effect of encouraging the hidden states for a class to be concentrated in such a way so that interpolations within the same class or between two different classes do not intersect with the real data points from other classes.
MLSys: The New Frontier of Machine Learning Systems
Machine learning (ML) techniques are enjoying rapidly increasing adoption.
Multi-objective training of Generative Adversarial Networks with multiple discriminators
Recent literature has demonstrated promising results for training Generative Adversarial Networks by employing a set of discriminators, in contrast to the traditional game involving one generator against a single adversary.
A Modern Take on the Bias-Variance Tradeoff in Neural Networks
The bias-variance tradeoff tells us that as model complexity increases, bias falls and variances increases, leading to a U-shaped test error curve.
h-detach: Modifying the LSTM Gradient Towards Better Optimization
This problem becomes more evident in tasks where the information needed to correctly solve them exist over long time scales, because EVGP prevents important gradient components from being back-propagated adequately over a large number of steps.
Negative Momentum for Improved Game Dynamics
Games generalize the single-objective optimization paradigm by introducing different objective functions for different players.
Manifold Mixup: Better Representations by Interpolating Hidden States
Deep neural networks excel at learning the training data, but often provide incorrect and confident predictions when evaluated on slightly different test examples.
Ranked #18 on
Image Classification
on OmniBenchmark
Fortified Networks: Improving the Robustness of Deep Networks by Modeling the Manifold of Hidden Representations
Deep networks have achieved impressive results across a variety of important tasks.
Learning Generative Models with Locally Disentangled Latent Factors
One of the most successful techniques in generative models has been decomposing a complicated generation task into a series of simpler generation tasks.
Deep Learning at 15PF: Supervised and Semi-Supervised Classification for Scientific Data
This paper presents the first, 15-PetaFLOP Deep Learning system for solving scientific pattern classification problems on contemporary HPC architectures.
Improving Gibbs Sampler Scan Quality with DoGS
The pairwise influence matrix of Dobrushin has long been used as an analytical tool to bound the rate of convergence of Gibbs sampling.
Accelerated Stochastic Power Iteration
We propose a simple variant of the power iteration with an added momentum term, that achieves both the optimal sample and iteration complexity.
Learning Representations and Generative Models for 3D Point Clouds
Three-dimensional geometric data offer an excellent domain for studying representation learning and generative modeling.
YellowFin and the Art of Momentum Tuning
We revisit the momentum SGD algorithm and show that hand-tuning a single learning rate and momentum makes it competitive with Adam.
Parallel SGD: When does averaging help?
Consider a number of workers running SGD independently on the same pool of data and averaging the models every once in a while -- a common but not well understood practice.
Omnivore: An Optimizer for Multi-device Deep Learning on CPUs and GPUs
Given a specification of a convolutional neural network, our goal is to minimize the time to train this model on a cluster of commodity CPUs and GPUs.
Scan Order in Gibbs Sampling: Models in Which it Matters and Bounds on How Much
Gibbs sampling is a Markov Chain Monte Carlo sampling technique that iteratively samples variables from their conditional distributions.
Asynchrony begets Momentum, with an Application to Deep Learning
Since asynchronous methods have better hardware efficiency, this result may shed light on when asynchronous execution is more efficient for deep learning systems.
Memory Limited, Streaming PCA
Standard algorithms require $O(p^2)$ memory; meanwhile no algorithm can do better than $O(kp)$ memory, since this is what the output itself requires.
