Search Results for author:
Found 44 papers, 19 papers with code
Topological Generalization Bounds for Discrete-Time Stochastic Optimization Algorithms
Our experimental results demonstrate that our new complexity measures correlate highly with generalization error in industry-standards architectures such as transformers and deep graph networks.
Differential Privacy of Noisy (S)GD under Heavy-Tailed Perturbations
Injecting heavy-tailed noise to the iterates of stochastic gradient descent (SGD) has received increasing attention over the past few years.
Generalization Bounds for Heavy-Tailed SDEs through the Fractional Fokker-Planck Equation
Understanding the generalization properties of heavy-tailed stochastic optimization algorithms has attracted increasing attention over the past years.
Tighter Generalisation Bounds via Interpolation
We also instantiate our bounds as training objectives, yielding non-trivial guarantees and practical performances.
Cyclic and Randomized Stepsizes Invoke Heavier Tails in SGD than Constant Stepsize
Our results bring a new understanding of the benefits of cyclic and randomized stepsizes compared to constant stepsize in terms of the tail behavior.
Generalization Bounds with Data-dependent Fractal Dimensions
To achieve this goal, we build up on a classical covering argument in learning theory and introduce a data-dependent fractal dimension.
Algorithmic Stability of Heavy-Tailed SGD with General Loss Functions
Very recently, new generalization bounds have been proven, indicating a non-monotonic relationship between the generalization error and heavy tails, which is more pertinent to the reported empirical observations.
Generalization Bounds for Stochastic Gradient Descent via Localized $\varepsilon$-Covers
In this paper, we propose a new covering technique localized for the trajectories of SGD.
Algorithmic Stability of Heavy-Tailed Stochastic Gradient Descent on Least Squares
Recent studies have shown that heavy tails can emerge in stochastic optimization and that the heaviness of the tails have links to the generalization error.
Chaotic Regularization and Heavy-Tailed Limits for Deterministic Gradient Descent
Recent studies have shown that gradient descent (GD) can achieve improved generalization when its dynamics exhibits a chaotic behavior.
Rate-Distortion Theoretic Generalization Bounds for Stochastic Learning Algorithms
Understanding generalization in modern machine learning settings has been one of the major challenges in statistical learning theory.
Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks
Disobeying the classical wisdom of statistical learning theory, modern deep neural networks generalize well even though they typically contain millions of parameters.
Generalization Bounds using Lower Tail Exponents in Stochastic Optimizers
Despite the ubiquitous use of stochastic optimization algorithms in machine learning, the precise impact of these algorithms and their dynamics on generalization performance in realistic non-convex settings is still poorly understood.
Fast Approximation of the Sliced-Wasserstein Distance Using Concentration of Random Projections
The Sliced-Wasserstein distance (SW) is being increasingly used in machine learning applications as an alternative to the Wasserstein distance and offers significant computational and statistical benefits.
Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms
As our main contribution, we prove that the generalization error of a stochastic optimization algorithm can be bounded based on the `complexity' of the fractal structure that underlies its invariant measure.
Heavy Tails in SGD and Compressibility of Overparametrized Neural Networks
Neural network compression techniques have become increasingly popular as they can drastically reduce the storage and computation requirements for very large networks.
Relative Positional Encoding for Transformers with Linear Complexity
Recent advances in Transformer models allow for unprecedented sequence lengths, due to linear space and time complexity.
Convergence Rates of Stochastic Gradient Descent under Infinite Noise Variance
In this paper, we provide convergence guarantees for SGD under a state-dependent and heavy-tailed noise with a potentially infinite variance, for a class of strongly convex objectives.
Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections
In this paper we focus on the so-called `implicit effect' of GNIs, which is the effect of the injected noise on the dynamics of SGD.
Self-Supervised VQ-VAE for One-Shot Music Style Transfer
While several style conversion methods tailored to musical signals have been proposed, most lack the 'one-shot' capability of classical image style transfer algorithms.
Groove2Groove: One-Shot Music Style Transfer with Supervision from Synthetic Data
Style transfer is the process of changing the style of an image, video, audio clip or musical piece so as to match the style of a given example.
Explicit Regularisation in Gaussian Noise Injections
We study the regularisation induced in neural networks by Gaussian noise injections (GNIs).
Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networks
Despite its success in a wide range of applications, characterizing the generalization properties of stochastic gradient descent (SGD) in non-convex deep learning problems is still an important challenge.
The Heavy-Tail Phenomenon in SGD
We claim that depending on the structure of the Hessian of the loss at the minimum, and the choices of the algorithm parameters $\eta$ and $b$, the SGD iterates will converge to a \emph{heavy-tailed} stationary distribution.
Synchronizing Probability Measures on Rotations via Optimal Transport
We introduce a new paradigm, $\textit{measure synchronization}$, for synchronizing graphs with measure-valued edges.
Statistical and Topological Properties of Sliced Probability Divergences
The idea of slicing divergences has been proven to be successful when comparing two probability measures in various machine learning applications including generative modeling, and consists in computing the expected value of a `base divergence' between one-dimensional random projections of the two measures.
Fractional Underdamped Langevin Dynamics: Retargeting SGD with Momentum under Heavy-Tailed Gradient Noise
Stochastic gradient descent with momentum (SGDm) is one of the most popular optimization algorithms in deep learning.
On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks
This assumption is often made for mathematical convenience, since it enables SGD to be analyzed as a stochastic differential equation (SDE) driven by a Brownian motion.
Approximate Bayesian Computation with the Sliced-Wasserstein Distance
Approximate Bayesian Computation (ABC) is a popular method for approximate inference in generative models with intractable but easy-to-sample likelihood.
Bayesian Model Selection for Identifying Markov Equivalent Causal Graphs
We believe that the Bayesian approach to causal discovery both allows the rich methodology of Bayesian inference to be used in various difficult aspects of this problem and provides a unifying framework to causal discovery research.
Supervised Symbolic Music Style Translation Using Synthetic Data
Research on style transfer and domain translation has clearly demonstrated the ability of deep learning-based algorithms to manipulate images in terms of artistic style.
First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noise
We show that the behaviors of the two systems are indeed similar for small step-sizes and we identify how the error depends on the algorithm and problem parameters.
Asymptotic Guarantees for Learning Generative Models with the Sliced-Wasserstein Distance
Minimum expected distance estimation (MEDE) algorithms have been widely used for probabilistic models with intractable likelihood functions and they have become increasingly popular due to their use in implicit generative modeling (e. g. Wasserstein generative adversarial networks, Wasserstein autoencoders).
Probabilistic Permutation Synchronization using the Riemannian Structure of the Birkhoff Polytope
We present an entirely new geometric and probabilistic approach to synchronization of correspondences across multiple sets of objects or images.
Non-Asymptotic Analysis of Fractional Langevin Monte Carlo for Non-Convex Optimization
Recent studies on diffusion-based sampling methods have shown that Langevin Monte Carlo (LMC) algorithms can be beneficial for non-convex optimization, and rigorous theoretical guarantees have been proven for both asymptotic and finite-time regimes.
Sliced-Wasserstein Flows: Nonparametric Generative Modeling via Optimal Transport and Diffusions
To the best of our knowledge, the proposed algorithm is the first nonparametric IGM algorithm with explicit theoretical guarantees.
Asynchronous Stochastic Quasi-Newton MCMC for Non-Convex Optimization
The results support our theory and show that the proposed algorithm provides a significant speedup over the recently proposed synchronous distributed L-BFGS algorithm.
Bayesian Pose Graph Optimization via Bingham Distributions and Tempered Geodesic MCMC
We introduce Tempered Geodesic Markov Chain Monte Carlo (TG-MCMC) algorithm for initializing pose graph optimization problems, arising in various scenarios such as SFM (structure from motion) or SLAM (simultaneous localization and mapping).
Fractional Langevin Monte Carlo: Exploring Levy Driven Stochastic Differential Equations for MCMC
These so called Langevin Monte Carlo (LMC) methods are based on diffusions driven by a Brownian motion, which gives rise to Gaussian proposal distributions in the resulting algorithms.
Fractional Langevin Monte Carlo: Exploring Lévy Driven Stochastic Differential Equations for Markov Chain Monte Carlo
These so called Langevin Monte Carlo (LMC) methods are based on diffusions driven by a Brownian motion, which gives rise to Gaussian proposal distributions in the resulting algorithms.
Learning the Morphology of Brain Signals Using Alpha-Stable Convolutional Sparse Coding
Neural time-series data contain a wide variety of prototypical signal waveforms (atoms) that are of significant importance in clinical and cognitive research.
Stochastic Quasi-Newton Langevin Monte Carlo
These second order methods directly approximate the inverse Hessian by using a limited history of samples and their gradients.
HAMSI: A Parallel Incremental Optimization Algorithm Using Quadratic Approximations for Solving Partially Separable Problems
We propose HAMSI (Hessian Approximated Multiple Subsets Iteration), which is a provably convergent, second order incremental algorithm for solving large-scale partially separable optimization problems.
Parallel Stochastic Gradient Markov Chain Monte Carlo for Matrix Factorisation Models
For large matrix factorisation problems, we develop a distributed Markov Chain Monte Carlo (MCMC) method based on stochastic gradient Langevin dynamics (SGLD) that we call Parallel SGLD (PSGLD).
