Search Results for author:
Found 40 papers, 21 papers with code
The Elements of Differentiable Programming
Artificial intelligence has recently experienced remarkable advances, fueled by large models, vast datasets, accelerated hardware, and, last but not least, the transformative power of differentiable programming.
Implicit Diffusion: Efficient Optimization through Stochastic Sampling
We present a new algorithm to optimize distributions defined implicitly by parameterized stochastic diffusions.
How do Transformers perform In-Context Autoregressive Learning?
More precisely, focusing on commuting orthogonal matrices $W$, we first show that a trained one-layer linear Transformer implements one step of gradient descent for the minimization of an inner objective function, when considering augmented tokens.
Direct Language Model Alignment from Online AI Feedback
Moreover, responses in these datasets are often sampled from a language model distinct from the one being aligned, and since the model evolves over training, the alignment phase is inevitably off-policy.
Decoding-time Realignment of Language Models
Aligning language models with human preferences is crucial for reducing errors and biases in these models.
Routers in Vision Mixture of Experts: An Empirical Study
Routers for sparse MoEs can be further grouped into two variants: Token Choice, which matches experts to each token, and Expert Choice, which matches tokens to each expert.
Dual Gauss-Newton Directions for Deep Learning
Inspired by Gauss-Newton-like methods, we study the benefit of leveraging the structure of deep learning objectives, namely, the composition of a convex loss function and of a nonlinear network, in order to derive better direction oracles than stochastic gradients, based on the idea of partial linearization.
Fast, Differentiable and Sparse Top-k: a Convex Analysis Perspective
In this paper, we propose new differentiable and sparse top-k operators.
Sparsity-Constrained Optimal Transport
The smoothness of the objectives increases as $k$ increases, giving rise to a trade-off between convergence speed and sparsity of the optimal plan.
Learning Energy Networks with Generalized Fenchel-Young Losses
To learn the parameters of the energy function, the solution to that optimization problem is typically fed into a loss function.
Cutting Some Slack for SGD with Adaptive Polyak Stepsizes
We use this insight to develop new variants of the SPS method that are better suited to nonlinear models.
Sinkformers: Transformers with Doubly Stochastic Attention
We show that the row-wise stochastic attention matrices in classical Transformers get close to doubly stochastic matrices as the number of epochs increases, justifying the use of Sinkhorn normalization as an informative prior.
Sparse Continuous Distributions and Fenchel-Young Losses
In contrast, for finite domains, recent work on sparse alternatives to softmax (e. g., sparsemax, $\alpha$-entmax, and fusedmax), has led to distributions with varying support.
Efficient and Modular Implicit Differentiation
In this paper, we propose automatic implicit differentiation, an efficient and modular approach for implicit differentiation of optimization problems.
Implicit differentiation for fast hyperparameter selection in non-smooth convex learning
Finding the optimal hyperparameters of a model can be cast as a bilevel optimization problem, typically solved using zero-order techniques.
Self-Supervised Learning of Audio Representations from Permutations with Differentiable Ranking
Second, we show that inverting permutations is a meaningful pretext task for learning audio representations in an unsupervised fashion.
Momentum Residual Neural Networks
We show on CIFAR and ImageNet that Momentum ResNets have the same accuracy as ResNets, while having a much smaller memory footprint, and show that pre-trained Momentum ResNets are promising for fine-tuning models.
Ranked #127 on
Image Classification
on CIFAR-10
Shuffle to Learn: Self-supervised learning from permutations via differentiable ranking
In particular, we also improve music understanding by reordering spectrogram patches in the frequency space, as well as video classification by reordering frames along the time axis.
Learning with Differentiable Pertubed Optimizers
Machine learning pipelines often rely on optimizers procedures to make discrete decisions (e. g., sorting, picking closest neighbors, or shortest paths).
Differentiable Divergences Between Time Series
Soft-DTW addresses these issues, but it is not a positive definite divergence: due to the bias introduced by entropic regularization, it can be negative and it is not minimized when the time series are equal.
Implicit differentiation of Lasso-type models for hyperparameter optimization
Our approach scales to high-dimensional data by leveraging the sparsity of the solutions.
Learning with Differentiable Perturbed Optimizers
Machine learning pipelines often rely on optimization procedures to make discrete decisions (e. g., sorting, picking closest neighbors, or shortest paths).
Fast Differentiable Sorting and Ranking
While numerous works have proposed differentiable proxies to sorting and ranking, they do not achieve the $O(n \log n)$ time complexity one would expect from sorting and ranking operations.
Structured Prediction with Projection Oracles
We identify the marginal polytope, the output space's convex hull, as the best convex set on which to project.
Geometric Losses for Distributional Learning
Building upon recent advances in entropy-regularized optimal transport, and upon Fenchel duality between measures and continuous functions , we propose a generalization of the logistic loss that incorporates a metric or cost between classes.
Learning with Fenchel-Young Losses
Over the past decades, numerous loss functions have been been proposed for a variety of supervised learning tasks, including regression, classification, ranking, and more generally structured prediction.
Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithms
This paper studies Fenchel-Young losses, a generic way to construct convex loss functions from a regularization function.
Blind Source Separation with Optimal Transport Non-negative Matrix Factorization
Optimal transport as a loss for machine learning optimization problems has recently gained a lot of attention.
SparseMAP: Differentiable Sparse Structured Inference
Structured prediction requires searching over a combinatorial number of structures.
Differentiable Dynamic Programming for Structured Prediction and Attention
We showcase these instantiations on two structured prediction tasks and on structured and sparse attention for neural machine translation.
Large Scale Optimal Transport and Mapping Estimation
First, we learn an optimal transport (OT) plan, which can be thought as a one-to-many map between the two distributions.
Large-Scale Optimal Transport and Mapping Estimation
We prove two theoretical stability results of regularized OT which show that our estimations converge to the OT plan and Monge map between the underlying continuous measures.
Smooth and Sparse Optimal Transport
In this paper, we explore regularizing the primal and dual OT formulations with a strongly convex term, which corresponds to relaxing the dual and primal constraints with smooth approximations.
A Regularized Framework for Sparse and Structured Neural Attention
Modern neural networks are often augmented with an attention mechanism, which tells the network where to focus within the input.
Multi-output Polynomial Networks and Factorization Machines
On recommendation system tasks, we show how to combine our algorithm with a reduction from ordinal regression to multi-output classification and show that the resulting algorithm outperforms simple baselines in terms of ranking accuracy.
Soft-DTW: a Differentiable Loss Function for Time-Series
We propose in this paper a differentiable learning loss between time series, building upon the celebrated dynamic time warping (DTW) discrepancy.
Polynomial Networks and Factorization Machines: New Insights and Efficient Training Algorithms
Polynomial networks and factorization machines are two recently-proposed models that can efficiently use feature interactions in classification and regression tasks.
Higher-Order Factorization Machines
Factorization machines (FMs) are a supervised learning approach that can use second-order feature combinations even when the data is very high-dimensional.
API design for machine learning software: experiences from the scikit-learn project
Scikit-learn is an increasingly popular machine learning li- brary.
Scikit-learn: Machine Learning in Python
Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems.
