Search Results for author:
Found 64 papers, 28 papers with code
From Denoising Score Matching to Langevin Sampling: A Fine-Grained Error Analysis in the Gaussian Setting
This allows us to rigorously track how the anisotropy of the data distribution (encoded by its power spectrum) interacts with key parameters of the end-to-end sampling method, including the noise amplitude, the step sizes in both score matching and diffusion, and the number of initial samples.
A Unified Perspective on the Dynamics of Deep Transformers
This representation is then exploited by the attention function, which learns dependencies between tokens and is key to the success of Transformers.
The Mathematics of Artificial Intelligence
This overview article highlights the critical role of mathematics in artificial intelligence (AI), emphasizing that mathematics provides tools to better understand and enhance AI systems.
Towards Understanding the Universality of Transformers for Next-Token Prediction
Causal Transformers are trained to predict the next token for a given context.
Transformers are Universal In-context Learners
A key aspect of our results, compared to existing findings, is that for a fixed precision, a single transformer can operate on an arbitrary (even infinite) number of tokens.
Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows
Conservation laws are well-established in the context of Euclidean gradient flow dynamics, notably for linear or ReLU neural network training.
Understanding the training of infinitely deep and wide ResNets with Conditional Optimal Transport
We study the convergence of gradient flow for the training of deep neural networks.
Enhancing Hypergradients Estimation: A Study of Preconditioning and Reparameterization
As a function of the error of the inner problem resolution, we study the error of the IFT method.
How do Transformers perform In-Context Autoregressive Learning?
More precisely, focusing on commuting orthogonal matrices $W$, we first show that a trained one-layer linear Transformer implements one step of gradient descent for the minimization of an inner objective function, when considering augmented tokens.
How Smooth Is Attention?
When the sequence length $n$ is too large for the previous bound to be tight, which we refer to as the mean-field regime, we provide an upper bound and a matching lower bound which are independent of $n$.
Structured Transforms Across Spaces with Cost-Regularized Optimal Transport
Matching a source to a target probability measure is often solved by instantiating a linear optimal transport (OT) problem, parameterized by a ground cost function that quantifies discrepancy between points.
Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flows
Besides, applying the two algorithms confirms for a number of ReLU network architectures that all known laws are recovered by the algorithm, and that there are no other independent laws.
Test like you Train in Implicit Deep Learning
Implicit deep learning has recently gained popularity with applications ranging from meta-learning to Deep Equilibrium Networks (DEQs).
Fast, Differentiable and Sparse Top-k: a Convex Analysis Perspective
In this paper, we propose new differentiable and sparse top-k operators.
Unbalanced Optimal Transport, from Theory to Numerics
Optimal Transport (OT) has recently emerged as a central tool in data sciences to compare in a geometrically faithful way point clouds and more generally probability distributions.
Do Residual Neural Networks discretize Neural Ordinary Differential Equations?
As a byproduct of our analysis, we consider the use of a memory-free discrete adjoint method to train a ResNet by recovering the activations on the fly through a backward pass of the network, and show that this method theoretically succeeds at large depth if the residual functions are Lipschitz with the input.
Smooth over-parameterized solvers for non-smooth structured optimization
Our main theoretical contribution connects gradient descent on this reformulation to a mirror descent flow with a varying Hessian metric.
Faster Unbalanced Optimal Transport: Translation invariant Sinkhorn and 1-D Frank-Wolfe
In this work, we identify the cause for this deficiency, namely the lack of a global normalization of the iterates, which equivalently corresponds to a translation of the dual OT potentials.
Global convergence of ResNets: From finite to infinite width using linear parameterization
To bridge the gap between the lazy and mean field regimes, we study Residual Networks (ResNets) in which the residual block has linear parametrization while still being nonlinear.
Randomized Stochastic Gradient Descent Ascent
RSGDA can be parameterized using optimal loop sizes that guarantee the best convergence rates known to hold for SGDA.
Sinkformers: Transformers with Doubly Stochastic Attention
We show that the row-wise stochastic attention matrices in classical Transformers get close to doubly stochastic matrices as the number of epochs increases, justifying the use of Sinkhorn normalization as an informative prior.
Smooth Bilevel Programming for Sparse Regularization
Iteratively reweighted least square (IRLS) is a popular approach to solve sparsity-enforcing regression problems in machine learning.
Linear-Time Gromov Wasserstein Distances using Low Rank Couplings and Costs
The ability to align points across two related yet incomparable point clouds (e. g. living in different spaces) plays an important role in machine learning.
Low-Rank Sinkhorn Factorization
Because matrix-vector products are pervasive in the Sinkhorn algorithm, several works have proposed to \textit{approximate} kernel matrices appearing in its iterations using low-rank factors.
Momentum Residual Neural Networks
We show on CIFAR and ImageNet that Momentum ResNets have the same accuracy as ResNets, while having a much smaller memory footprint, and show that pre-trained Momentum ResNets are promising for fine-tuning models.
Ranked #139 on
Image Classification
on CIFAR-10
Fast and accurate optimization on the orthogonal manifold without retraction
We consider the problem of minimizing a function over the manifold of orthogonal matrices.
Unsupervised Ground Metric Learning using Wasserstein Singular Vectors
Optimal Transport (OT) lifts a distance between features (the "ground metric") to a geometrically meaningful distance between samples.
Entropic Optimal Transport between Unbalanced Gaussian Measures has a Closed Form
Although optimal transport (OT) problems admit closed form solutions in a very few notable cases, e. g. in 1D or between Gaussians, these closed forms have proved extremely fecund for practitioners to define tools inspired from the OT geometry.
The Unbalanced Gromov Wasserstein Distance: Conic Formulation and Relaxation
The GW distance is however limited to the comparison of metric measure spaces endowed with a probability distribution.
Distribution-Based Invariant Deep Networks for Learning Meta-Features
On both tasks, Dida learns meta-features supporting the characterization of a (labelled) dataset.
Faster Wasserstein Distance Estimation with the Sinkhorn Divergence
We also propose and analyze an estimator based on Richardson extrapolation of the Sinkhorn divergence which enjoys improved statistical and computational efficiency guarantees, under a condition on the regularity of the approximation error, which is in particular satisfied for Gaussian densities.
Entropic Optimal Transport between (Unbalanced) Gaussian Measures has a Closed Form
Although optimal transport (OT) problems admit closed form solutions in a very few notable cases, e. g. in 1D or between Gaussians, these closed forms have proved extremely fecund for practitioners to define tools inspired from the OT geometry.
Statistics Theory Statistics Theory
Online Sinkhorn: Optimal Transport distances from sample streams
Optimal Transport (OT) distances are now routinely used as loss functions in ML tasks.
Wasserstein Control of Mirror Langevin Monte Carlo
In this paper, we consider Langevin diffusions on a Hessian-type manifold and study a discretization that is closely related to the mirror-descent scheme.
Super-efficiency of automatic differentiation for functions defined as a minimum
In most cases, the minimum has no closed-form, and an approximation is obtained via an iterative algorithm.
Degrees of freedom for off-the-grid sparse estimation
Our main contribution is a proof of a continuous counterpart to this result for the Blasso.
Ground Metric Learning on Graphs
Optimal transport (OT) distances between probability distributions are parameterized by the ground metric they use between observations.
Sinkhorn Divergences for Unbalanced Optimal Transport
Optimal transport induces the Earth Mover's (Wasserstein) distance between probability distributions, a geometric divergence that is relevant to a wide range of problems.
Geometric Losses for Distributional Learning
Building upon recent advances in entropy-regularized optimal transport, and upon Fenchel duality between measures and continuous functions , we propose a generalization of the logistic loss that incorporates a metric or cost between classes.
Universal Invariant and Equivariant Graph Neural Networks
In this paper, we consider a specific class of invariant and equivariant networks, for which we prove new universality theorems.
Stochastic Deep Networks
This allows to design discriminative networks (to classify or reduce the dimensionality of input measures), generative architectures (to synthesize measures) and recurrent pipelines (to predict measure dynamics).
Semi-dual Regularized Optimal Transport
Variational problems that involve Wasserstein distances and more generally optimal transport (OT) theory are playing an increasingly important role in data sciences.
Interpolating between Optimal Transport and MMD using Sinkhorn Divergences
Comparing probability distributions is a fundamental problem in data sciences.
Statistics Theory Statistics Theory 62
Computational Optimal Transport
Optimal transport (OT) theory can be informally described using the words of the French mathematician Gaspard Monge (1746-1818): A worker with a shovel in hand has to move a large pile of sand lying on a construction site.
Wasserstein Dictionary Learning: Optimal Transport-based unsupervised non-linear dictionary learning
Wasserstein barycenters) between dictionary atoms; such atoms are themselves synthetic histograms in the probability simplex.
Sensitivity Analysis for Mirror-Stratifiable Convex Functions
This pairing is crucial to track the strata that are identifiable by solutions of parametrized optimization problems or by iterates of optimization algorithms.
GAN and VAE from an Optimal Transport Point of View
This short article revisits some of the ideas introduced in arXiv:1701. 07875 and arXiv:1705. 07642 in a simple setup.
Learning Generative Models with Sinkhorn Divergences
The ability to compare two degenerate probability distributions (i. e. two probability distributions supported on two distinct low-dimensional manifolds living in a much higher-dimensional space) is a crucial problem arising in the estimation of generative models for high-dimensional observations such as those arising in computer vision or natural language.
Quantum Optimal Transport for Tensor Field Processing
This "quantum" formulation of OT (Q-OT) corresponds to a relaxed version of the classical Kantorovich transport problem, where the fidelity between the input PSD-valued measures is captured using the geometry of the Von-Neumann quantum entropy.
Graphics
Sparse Support Recovery with Non-smooth Loss Functions
More precisely, we focus in detail on the cases of $\ell_1$ and $\ell_\infty$ losses, and contrast them with the usual $\ell_2$ loss. While these losses are routinely used to account for either sparse ($\ell_1$ loss) or uniform ($\ell_\infty$ loss) noise models, a theoretical analysis of their performance is still lacking.
A Multi-step Inertial Forward-Backward Splitting Method for Non-convex Optimization
In this paper, we propose a multi-step inertial Forward--Backward splitting algorithm for minimizing the sum of two non-necessarily convex functions, one of which is proper lower semi-continuous while the other is differentiable with a Lipschitz continuous gradient.
Bayesian Modeling of Motion Perception using Dynamical Stochastic Textures
We use the dynamic texture model to psychophysically probe speed perception in humans using zoom-like changes in the spatial frequency content of the stimulus.
Scaling Algorithms for Unbalanced Transport Problems
This article introduces a new class of fast algorithms to approximate variational problems involving unbalanced optimal transport.
Optimization and Control 65K10
Stochastic Optimization for Large-scale Optimal Transport
We instantiate these ideas in three different setups: (i) when comparing a discrete distribution to another, we show that incremental stochastic optimization schemes can beat Sinkhorn's algorithm, the current state-of-the-art finite dimensional OT solver; (ii) when comparing a discrete distribution to a continuous density, a semi-discrete reformulation of the dual program is amenable to averaged stochastic gradient descent, leading to better performance than approximately solving the problem by discretization ; (iii) when dealing with two continuous densities, we propose a stochastic gradient descent over a reproducing kernel Hilbert space (RKHS).
Biologically Inspired Dynamic Textures for Probing Motion Perception
We study here the principled construction of a generative model specifically crafted to probe motion perception.
Unbalanced Optimal Transport: Geometry and Kantorovich Formulation
These distances are defined by two equivalent alternative formulations: (i) a "fluid dynamic" formulation defining the distance as a geodesic distance over the space of measures (ii) a static "Kantorovich" formulation where the distance is the minimum of an optimization program over pairs of couplings describing the transfer (transport, creation and destruction) of mass between two measures.
Optimization and Control
An Interpolating Distance between Optimal Transport and Fisher-Rao
This metric interpolates between the quadratic Wasserstein and the Fisher-Rao metrics and generalizes optimal transport to measures with different masses.
Analysis of PDEs
Fast Optimal Transport Averaging of Neuroimaging Data
Data are large, the geometry of the brain is complex and the between subjects variability leads to spatially or temporally non-overlapping effects of interest.
A Smoothed Dual Approach for Variational Wasserstein Problems
Variational problems that involve Wasserstein distances have been recently proposed to summarize and learn from probability measures.
Iterative Bregman Projections for Regularized Transportation Problems
This article details a general numerical framework to approximate so-lutions to linear programs related to optimal transport.
Numerical Analysis Analysis of PDEs
Local Linear Convergence of Forward--Backward under Partial Smoothness
In this paper, we consider the Forward--Backward proximal splitting algorithm to minimize the sum of two proper closed convex functions, one of which having a Lipschitz continuous gradient and the other being partly smooth relatively to an active manifold $\mathcal{M}$.
Low Complexity Regularization of Linear Inverse Problems
Inverse problems and regularization theory is a central theme in contemporary signal processing, where the goal is to reconstruct an unknown signal from partial indirect, and possibly noisy, measurements of it.
Model Consistency of Partly Smooth Regularizers
We show that a generalized "irrepresentable condition" implies stable model selection under small noise perturbations in the observations and the design matrix, when the regularization parameter is tuned proportionally to the noise level.
Regularized Discrete Optimal Transport
The resulting transportation plan can be used as a color transfer map, which is robust to mass variation across images color palettes.
