Search Results for author:
Found 23 papers, 13 papers with code
How Deep Networks Learn Sparse and Hierarchical Data: the Sparse Random Hierarchy Model
Understanding what makes high-dimensional data learnable is a fundamental question in machine learning.
A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Data
We find that the backward diffusion process acting after a time $t$ is governed by a phase transition at some threshold time, where the probability of reconstructing high-level features, like the class of an image, suddenly drops.
On the different regimes of Stochastic Gradient Descent
For small $B$ and large $\eta$, SGD corresponds to a stochastic evolution of the parameters, whose noise amplitude is governed by the ''temperature'' $T\equiv \eta/B$.
How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model
The model is a classification task where each class corresponds to a group of high-level features, chosen among several equivalent groups associated with the same class.
Dissecting the Effects of SGD Noise in Distinct Regimes of Deep Learning
They show that SGD noise can be detrimental or instead useful depending on the training regime.
How deep convolutional neural networks lose spatial information with training
Here, we (i) show empirically for various architectures that stability to image diffeomorphisms is achieved by both spatial and channel pooling, (ii) introduce a model scale-detection task which reproduces our empirical observations on spatial pooling and (iii) compute analitically how the sensitivity to diffeomorphisms and noise scales with depth due to spatial pooling.
What Can Be Learnt With Wide Convolutional Neural Networks?
Interestingly, we find that, despite their hierarchical structure, the functions generated by infinitely-wide deep CNNs are too rich to be efficiently learnable in high dimension.
Learning sparse features can lead to overfitting in neural networks
It is widely believed that the success of deep networks lies in their ability to learn a meaningful representation of the features of the data.
Failure and success of the spectral bias prediction for Kernel Ridge Regression: the case of low-dimensional data
Recently, several theories including the replica method made predictions for the generalization error of Kernel Ridge Regression.
Locality defeats the curse of dimensionality in convolutional teacher-student scenarios
Convolutional neural networks perform a local and translationally-invariant treatment of the data: quantifying which of these two aspects is central to their success remains a challenge.
How memory architecture affects learning in a simple POMDP: the two-hypothesis testing problem
Reinforcement learning is generally difficult for partially observable Markov decision processes (POMDPs), which occurs when the agent's observation is partial or noisy.
Relative stability toward diffeomorphisms indicates performance in deep nets
Understanding why deep nets can classify data in large dimensions remains a challenge.
Non-local effects reflect the jamming criticality in granular flows of frictionless particles
The jamming transition is accompanied by a rich phenomenology, such as hysteresis or non-local effects, which is still not well understood.
Soft Condensed Matter
Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
In this manuscript, we review recent results elucidating (i, ii) and the perspective they offer on the (still unexplained) curse of dimensionality paradox.
Geometric compression of invariant manifolds in neural nets
We confirm these predictions both for a one-hidden layer FC network trained on the stripe model and for a 16-layers CNN trained on MNIST, for which we also find $\beta_\text{Feature}>\beta_\text{Lazy}$.
How isotropic kernels perform on simple invariants
(ii) Next we consider support-vector binary classification and introduce the stripe model where the data label depends on a single coordinate $y(\underline{x}) = y(x_1)$, corresponding to parallel decision boundaries separating labels of different signs, and consider that there is no margin at these interfaces.
Disentangling feature and lazy training in deep neural networks
Two distinct limits for deep learning have been derived as the network width $h\rightarrow \infty$, depending on how the weights of the last layer scale with $h$.
Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm
We extract $a$ from real data by performing kernel PCA, leading to $\beta\approx0. 36$ for MNIST and $\beta\approx0. 07$ for CIFAR10, in good agreement with observations.
How collective asperity detachments nucleate slip at frictional interfaces
Sliding at a quasi-statically loaded frictional interface occurs via macroscopic slip events, which nucleate locally before propagating as rupture fronts very similar to fracture.
Disordered Systems and Neural Networks Statistical Mechanics
Scaling description of generalization with number of parameters in deep learning
At this threshold, we argue that $\|f_{N}\|$ diverges.
A jamming transition from under- to over-parametrization affects loss landscape and generalization
We argue that in fully-connected networks a phase transition delimits the over- and under-parametrized regimes where fitting can or cannot be achieved.
The jamming transition as a paradigm to understand the loss landscape of deep neural networks
In the vicinity of this transition, properties of the curvature of the minima of the loss are critical.
Comparing Dynamics: Deep Neural Networks versus Glassy Systems
We analyze numerically the training dynamics of deep neural networks (DNN) by using methods developed in statistical physics of glassy systems.
