Search Results for author:
Found 45 papers, 20 papers with code
The Geometries of Truth Are Orthogonal Across Tasks
Large Language Models (LLMs) have demonstrated impressive generalization capabilities across various tasks, but their claim to practical relevance is still mired by concerns on their reliability.
Identifiable Multi-View Causal Discovery Without Non-Gaussianity
We propose a novel approach to linear causal discovery in the framework of multi-view Structural Equation Models (SEM).
Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging
To train this architecture, we sample random domain weights, instantiate the corresponding model, and backprop through one batch of data sampled with these domain weights.
A Unified Perspective on the Dynamics of Deep Transformers
This representation is then exploited by the attention function, which learns dependencies between tokens and is key to the success of Transformers.
MVICAD2: Multi-View Independent Component Analysis with Delays and Dilations
To address this, we propose Multi-View Independent Component Analysis with Delays and Dilations (MVICAD2), which allows sources to differ across subjects in both temporal delays and dilations.
Sparse Repellency for Shielded Generation in Text-to-image Diffusion Models
We achieve this by adding repellency terms to the diffusion SDE throughout the generation trajectory, which are triggered whenever the path is expected to land too closely to an image in the shielded reference set.
Dynamic Gradient Alignment for Online Data Mixing
DGA dynamically estimates the pre-training data mixture on which the models' gradients align as well as possible with those of the model on the specific task.
Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling
In this work, we build specialist models from large generalist training sets instead.
Theory, Analysis, and Best Practices for Sigmoid Self-Attention
Attention is a key part of the transformer architecture.
The AdEMAMix Optimizer: Better, Faster, Older
This work questions the use of a single EMA to accumulate past gradients and empirically demonstrates how this choice can be sub-optimal: a single EMA cannot simultaneously give a high weight to the immediate past, and a non-negligible weight to older gradients.
Optimization without Retraction on the Random Generalized Stiefel Manifold
Optimization over the set of matrices $X$ that satisfy $X^\top B X = I_p$, referred to as the generalized Stiefel manifold, appears in many applications involving sampled covariance matrices such as the canonical correlation analysis (CCA), independent component analysis (ICA), and the generalized eigenvalue problem (GEVP).
Enhancing Hypergradients Estimation: A Study of Preconditioning and Reparameterization
As a function of the error of the inner problem resolution, we study the error of the IFT method.
Careful with that Scalpel: Improving Gradient Surgery with an EMA
Beyond minimizing a single training loss, many deep learning estimation pipelines rely on an auxiliary objective to quantify and encourage desirable properties of the model (e. g. performance on another dataset, robustness, agreement with a prior).
Need a Small Specialized Language Model? Plan Early!
In the first scenario, we propose an effective solution based on importance sampling: we resample the pretraining set to imitate the specialization data and train a small model on it.
Ranked #1 on
Language Modelling
on The Pile
(Test perplexity metric)
How Smooth Is Attention?
When the sequence length $n$ is too large for the previous bound to be tight, which we refer to as the mean-field regime, we provide an upper bound and a matching lower bound which are independent of $n$.
MultiView Independent Component Analysis with Delays
Linear Independent Component Analysis (ICA) is a blind source separation technique that has been used in various domains to identify independent latent sources from observed signals.
Adaptive Training Distributions with Scalable Online Bilevel Optimization
Large neural networks pretrained on web-scale corpora are central to modern machine learning.
A Challenge in Reweighting Data with Bilevel Optimization
In many scenarios, one uses a large training set to train a model with the goal of performing well on a smaller testing set with a different distribution.
Learning Elastic Costs to Shape Monge Displacements
Given a source and a target probability measure supported on $\mathbb{R}^d$, the Monge problem asks to find the most efficient way to map one distribution to the other.
Test like you Train in Implicit Deep Learning
Implicit deep learning has recently gained popularity with applications ranging from meta-learning to Deep Equilibrium Networks (DEQs).
Infeasible Deterministic, Stochastic, and Variance-Reduction Algorithms for Optimization under Orthogonality Constraints
Orthogonality constraints naturally appear in many machine learning problems, from principal component analysis to robust neural network training.
A Lower Bound and a Near-Optimal Algorithm for Bilevel Empirical Risk Minimization
Bilevel optimization problems, which are problems where two optimization problems are nested, have more and more applications in machine learning.
Monge, Bregman and Occam: Interpretable Optimal Transport in High-Dimensions with Feature-Sparse Maps
Optimal transport (OT) theory focuses, among all maps $T:\mathbb{R}^d\rightarrow \mathbb{R}^d$ that can morph a probability measure onto another, on those that are the ``thriftiest'', i. e. such that the averaged cost $c(x, T(x))$ between $x$ and its image $T(x)$ be as small as possible.
Benchopt: Reproducible, efficient and collaborative optimization benchmarks
Numerical validation is at the core of machine learning research as it allows to assess the actual impact of new methods, and to confirm the agreement between theory and practice.
Do Residual Neural Networks discretize Neural Ordinary Differential Equations?
As a byproduct of our analysis, we consider the use of a memory-free discrete adjoint method to train a ResNet by recovering the activations on the fly through a backward pass of the network, and show that this method theoretically succeeds at large depth if the residual functions are Lipschitz with the input.
A framework for bilevel optimization that enables stochastic and global variance reduction algorithms
However, computing the gradient of the value function involves solving a linear system, which makes it difficult to derive unbiased stochastic estimates.
Shared Independent Component Analysis for Multi-Subject Neuroimaging
While ShICA-J is based on second-order statistics, we further propose to leverage non-Gaussianity of the components using a maximum-likelihood method, ShICA-ML, that is both more accurate and more costly.
Sinkformers: Transformers with Doubly Stochastic Attention
We show that the row-wise stochastic attention matrices in classical Transformers get close to doubly stochastic matrices as the number of epochs increases, justifying the use of Sinkhorn normalization as an informative prior.
Kernel Stein Discrepancy Descent
We investigate the properties of its Wasserstein gradient flow to approximate a target probability distribution $\pi$ on $\mathbb{R}^d$, known up to a normalization constant.
Adaptive Multi-View ICA: Estimation of noise levels for optimal inference
By contrast, we propose Adaptive multiView ICA (AVICA), a noisy ICA model where each view is a linear mixture of shared independent sources with additive noise on the sources.
Momentum Residual Neural Networks
We show on CIFAR and ImageNet that Momentum ResNets have the same accuracy as ResNets, while having a much smaller memory footprint, and show that pre-trained Momentum ResNets are promising for fine-tuning models.
Ranked #140 on
Image Classification
on CIFAR-10
Fast and accurate optimization on the orthogonal manifold without retraction
We consider the problem of minimizing a function over the manifold of orthogonal matrices.
Deep orthogonal linear networks are shallow
We consider the problem of training a deep orthogonal linear network, which consists of a product of orthogonal matrices, with no non-linearity in-between.
Spectral independent component analysis with noise modeling for M/EEG source separation
Signals are modelled as a linear mixing of independent sources corrupted by additive noise, where sources and the noise are stationary Gaussian time series.
Modeling Shared Responses in Neuroimaging Studies through MultiView ICA
Group studies involving large cohorts of subjects are important to draw general conclusions about brain functional organization.
mvlearn: Multiview Machine Learning in Python
As data are generated more and more from multiple disparate sources, multiview data sets, where each sample has features in distinct views, have ballooned in recent years.
Super-efficiency of automatic differentiation for functions defined as a minimum
In most cases, the minimum has no closed-form, and an approximation is obtained via an iterative algorithm.
Manifold-regression to predict from MEG/EEG brain signals without source modeling
We show that Wasserstein and geometric distances allow perfect out-of-sample prediction on the generative models.
Learning step sizes for unfolded sparse coding
We demonstrate that for a large class of unfolded algorithms, if the algorithm converges to the solution of the Lasso, its last layers correspond to ISTA with learned step sizes.
Beyond Pham's algorithm for joint diagonalization
The approximate joint diagonalization of a set of matrices consists in finding a basis in which these matrices are as diagonal as possible.
A Quasi-Newton algorithm on the orthogonal manifold for NMF with transform learning
Nonnegative matrix factorization (NMF) is a popular method for audio spectral unmixing.
Accelerating likelihood optimization for ICA on real signals
We study optimization methods for solving the maximum likelihood formulation of independent component analysis (ICA).
Stochastic algorithms with descent guarantees for ICA
We derive an online algorithm for the streaming setting, and an incremental algorithm for the finite sum setting, with the following benefits.
Faster ICA under orthogonal constraint
Independent Component Analysis (ICA) is a technique for unsupervised exploration of multi-channel data widely used in observational sciences.
Faster independent component analysis by preconditioning with Hessian approximations
Independent Component Analysis (ICA) is a technique for unsupervised exploration of multi-channel data that is widely used in observational sciences.
