Search Results for author:
Found 14 papers, 7 papers with code
Fundamental limits of learning in sequence multi-index models and deep attention networks: High-dimensional asymptotics and sharp thresholds
In this manuscript, we study the learning of deep attention neural networks, defined as the composition of multiple self-attention layers, with tied and low-rank weights.
A precise asymptotic analysis of learning diffusion models: theory and insights
In this manuscript, we consider the problem of learning a flow or diffusion-based generative model parametrized by a two-layer auto-encoder, trained with online stochastic gradient descent, on a high-dimensional target density with an underlying low-dimensional manifold structure.
A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilities
This provides a sharp description of the impact of feature learning in the generalization of two-layer neural networks, beyond the random features and lazy training regimes.
High-dimensional learning of narrow neural networks
Recent years have been marked with the fast-pace diversification and increasing ubiquity of machine learning applications.
Asymptotics of Learning with Deep Structured (Random) Features
For a large class of feature maps we provide a tight asymptotic characterisation of the test error associated with learning the readout layer, in the high-dimensional limit where the input dimension, hidden layer widths, and number of training samples are proportionally large.
Asymptotics of feature learning in two-layer networks after one gradient-step
In this manuscript, we investigate the problem of how two-layer neural networks learn features from data, and improve over the kernel regime, after being trained with a single gradient descent step.
A phase transition between positional and semantic learning in a solvable model of dot-product attention
Many empirical studies have provided evidence for the emergence of algorithmic mechanisms (abilities) in the learning of language models, that lead to qualitative improvements of the model capabilities.
Analysis of learning a flow-based generative model from limited sample complexity
We study the problem of training a flow-based generative model, parametrized by a two-layer autoencoder, to sample from a high-dimensional Gaussian mixture.
Deterministic equivalent and error universality of deep random features learning
Establishing this result requires proving a deterministic equivalent for traces of the deep random features sample covariance matrices which can be of independent interest.
Bayes-optimal Learning of Deep Random Networks of Extensive-width
We consider the problem of learning a target function corresponding to a deep, extensive-width, non-linear neural network with random Gaussian weights.
Error Scaling Laws for Kernel Classification under Source and Capacity Conditions
We find that our rates tightly describe the learning curves for this class of data sets, and are also observed on real data.
Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime
In this work, we unify and extend this line of work, providing characterization of all regimes and excess error decay rates that can be observed in terms of the interplay of noise and regularization.
Learning curves of generic features maps for realistic datasets with a teacher-student model
While still solvable in a closed form, this generalization is able to capture the learning curves for a broad range of realistic data sets, thus redeeming the potential of the teacher-student framework.
Large deviations for the perceptron model and consequences for active learning
These large deviations then provide optimal achievable performance boundaries for any active learning algorithm.
