Search Results for author:
Found 29 papers, 17 papers with code
Fine-tuning with Very Large Dropout
Although training a deep network from scratch using such dropout rates is virtually impossible, fine-tuning a large pre-trained model under such conditions is not only possible but also achieves out-of-distribution performances that exceed those of both ensembles and weight averaging methods such as model soups.
Borges and AI
Many believe that Large Language Models (LLMs) open the era of Artificial Intelligence (AI).
Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization
In this paper, we thus propose model ratatouille, a new strategy to recycle the multiple fine-tunings of the same foundation model on diverse auxiliary tasks.
Ranked #14 on
Domain Generalization
on PACS
Learning useful representations for shifting tasks and distributions
Our thesis is that such scenarios are better served by representations that are richer than those obtained with a single optimization episode.
Rich Feature Construction for the Optimization-Generalization Dilemma
On the one hand, a rich representation provides a good initialization for the optimizer.
Algorithmic Bias and Data Bias: Understanding the Relation between Distributionally Robust Optimization and Data Curation
We show that neither DRO nor curating the training set should be construed as a complete solution for bias mitigation: in the same way that there is no universally robust training set, there is no universal way to setup a DRO problem and ensure a socially acceptable set of results.
A Simple Convergence Proof of Adam and Adagrad
We provide a simple proof of convergence covering both the Adam and Adagrad adaptive optimization algorithms when applied to smooth (possibly non-convex) objective functions with bounded gradients.
Music Source Separation in the Waveform Domain
Source separation for music is the task of isolating contributions, or stems, from different instruments recorded individually and arranged together to form a song.
Ranked #3 on
Multi-task Audio Source Seperation
on MTASS
Symplectic Recurrent Neural Networks
We propose Symplectic Recurrent Neural Networks (SRNNs) as learning algorithms that capture the dynamics of physical systems from observed trajectories.
Demucs: Deep Extractor for Music Sources with extra unlabeled data remixed
We study the problem of source separation for music using deep learning with four known sources: drums, bass, vocals and other accompaniments.
Invariant Risk Minimization
We introduce Invariant Risk Minimization (IRM), a learning paradigm to estimate invariant correlations across multiple training distributions.
Beyond Folklore: A Scaling Calculus for the Design and Initialization of ReLU Networks
We propose a system for calculating a "scaling constant" for layers and weights of neural networks.
Cold Case: The Lost MNIST Digits
Although the popular MNIST dataset [LeCun et al., 1994] is derived from the NIST database [Grother and Hanaoka, 1995], the precise processing steps for this derivation have been lost to time.
Adversarial Vulnerability of Neural Networks Increases with Input Dimension
Over the past four years, neural networks have been proven vulnerable to adversarial images: targeted but imperceptible image perturbations lead to drastically different predictions.
On the Ineffectiveness of Variance Reduced Optimization for Deep Learning
The applicability of these techniques to the hard non-convex optimization problems encountered during training of modern deep neural networks is an open problem.
Controlling Covariate Shift using Balanced Normalization of Weights
We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs.
SING: Symbol-to-Instrument Neural Generator
On the generalization task of synthesizing notes for pairs of pitch and instrument not seen during training, SING produces audio with significantly improved perceptual quality compared to a state-of-the-art autoencoder based on WaveNet as measured by a Mean Opinion Score (MOS), and is about 32 times faster for training and 2, 500 times faster for inference.
WNGrad: Learn the Learning Rate in Gradient Descent
Adjusting the learning rate schedule in stochastic gradient methods is an important unresolved problem which requires tuning in practice.
First-order Adversarial Vulnerability of Neural Networks and Input Dimension
Over the past few years, neural networks were proven vulnerable to adversarial images: targeted but imperceptible image perturbations lead to drastically different predictions.
Wasserstein Generative Adversarial Networks
We introduce a new algorithm named WGAN, an alternative to traditional GAN training.
Diagonal Rescaling For Neural Networks
We define a second-order neural network stochastic gradient training algorithm whose block-diagonal structure effectively amounts to normalizing the unit activations.
Wasserstein GAN
We introduce a new algorithm named WGAN, an alternative to traditional GAN training.
Towards Principled Methods for Training Generative Adversarial Networks
The goal of this paper is not to introduce a single algorithm or method, but to make theoretical steps towards fully understanding the training dynamics of generative adversarial networks.
Optimization Methods for Large-Scale Machine Learning
This paper provides a review and commentary on the past, present, and future of numerical optimization algorithms in the context of machine learning applications.
Discovering Causal Signals in Images
Our experiments demonstrate the existence of a relation between the direction of causality and the difference between objects and their contexts, and by the same token, the existence of observable signals that reveal the causal dispositions of objects.
Unifying distillation and privileged information
Distillation (Hinton et al., 2015) and privileged information (Vapnik & Izmailov, 2015) are two techniques that enable machines to learn from other machines.
No Regret Bound for Extreme Bandits
This work is naturally framed in the extreme bandit setting, which deals with sequentially choosing which distribution from a collection to sample in order to minimize (maximize) the single best cost (reward).
Counterfactual Reasoning and Learning Systems
This work shows how to leverage causal inference to understand the behavior of complex learning systems interacting with their environment and predict the consequences of changes to the system.
The Tradeoffs of Large Scale Learning
This contribution develops a theoretical framework that takes into account the effect of approximate optimization on learning algorithms.
