Search Results for author: Florent Krzakala

Found 74 papers, 31 papers with code

Double Trouble in Double Descent: Bias and Variance(s) in the Lazy Regime

no code implementations ICML 2020 Stéphane d'Ascoli, Maria Refinetti, Giulio Biroli, Florent Krzakala

We demonstrate that the latter two contributions are the crux of the double descent: they lead to the overfitting peak at the interpolation threshold and to the decay of the test error upon overparametrization.

Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime

no code implementations31 May 2021 Hugo Cui, Bruno Loureiro, Florent Krzakala, Lenka Zdeborová

In this work, we unify and extend this line of work, providing characterization of all regimes and excess error decay rates that can be observed in terms of the interplay of noise and regularization.

Bayesian reconstruction of memories stored in neural networks from their connectivity

1 code implementation16 May 2021 Sebastian Goldt, Florent Krzakala, Lenka Zdeborová, Nicolas Brunel

The advent of comprehensive synaptic wiring diagrams of large neural circuits has created the field of connectomics and given rise to a number of open research questions.

Bayesian Inference

Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed

1 code implementation23 Feb 2021 Maria Refinetti, Sebastian Goldt, Florent Krzakala, Lenka Zdeborová

Here, we show theoretically that two-layer neural networks (2LNN) with only a few hidden neurons can beat the performance of kernel learning on a simple Gaussian mixture classification task.

Image Classification

Learning curves of generic features maps for realistic datasets with a teacher-student model

1 code implementation16 Feb 2021 Bruno Loureiro, Cédric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mézard, Lenka Zdeborová

While still solvable in a closed form, this generalization is able to capture the learning curves for a broad range of realistic data sets, thus redeeming the potential of the teacher-student framework.

Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment

no code implementations11 Dec 2020 Julien Launay, Iacopo Poli, Kilian Müller, Gustave Pariente, Igor Carron, Laurent Daudet, Florent Krzakala, Sylvain Gigan

We present a photonic accelerator for Direct Feedback Alignment, able to compute random projections with trillions of parameters.

Construction of optimal spectral methods in phase retrieval

1 code implementation8 Dec 2020 Antoine Maillard, Florent Krzakala, Yue M. Lu, Lenka Zdeborová

We consider the phase retrieval problem, in which the observer wishes to recover a $n$-dimensional real or complex signal $\mathbf{X}^\star$ from the (possibly noisy) observation of $|\mathbf{\Phi} \mathbf{X}^\star|$, in which $\mathbf{\Phi}$ is a matrix of size $m \times n$.

Information Theory Disordered Systems and Neural Networks Information Theory

Epidemic mitigation by statistical inference from contact tracing data

no code implementations20 Sep 2020 Antoine Baker, Indaco Biazzo, Alfredo Braunstein, Giovanni Catania, Luca Dall'Asta, Alessandro Ingrosso, Florent Krzakala, Fabio Mazza, Marc Mézard, Anna Paola Muntoni, Maria Refinetti, Stefano Sarao Mannelli, Lenka Zdeborová

We conclude that probabilistic risk estimation is capable to enhance performance of digital contact tracing and should be considered in the currently developed mobile applications.

Bayesian Inference

The Gaussian equivalence of generative models for learning with shallow neural networks

1 code implementation25 Jun 2020 Sebastian Goldt, Bruno Loureiro, Galen Reeves, Florent Krzakala, Marc Mézard, Lenka Zdeborová

Here, we go beyond this simple paradigm by studying the performance of neural networks trained on data drawn from pre-trained generative models.

Complex Dynamics in Simple Neural Networks: Understanding Gradient Flow in Phase Retrieval

no code implementations NeurIPS 2020 Stefano Sarao Mannelli, Giulio Biroli, Chiara Cammarota, Florent Krzakala, Pierfrancesco Urbani, Lenka Zdeborová

Despite the widespread use of gradient-based algorithms for optimizing high-dimensional non-convex functions, understanding their ability of finding good minima instead of being trapped in spurious ones remains to a large extent an open problem.

Reservoir Computing meets Recurrent Kernels and Structured Transforms

1 code implementation NeurIPS 2020 Jonathan Dong, Ruben Ohana, Mushegh Rafayelyan, Florent Krzakala

Reservoir Computing is a class of simple yet efficient Recurrent Neural Networks where internal weights are fixed at random and only a linear output layer is trained.

Time Series Time Series Prediction

Asymptotic Errors for Teacher-Student Convex Generalized Linear Models (or : How to Prove Kabashima's Replica Formula)

no code implementations11 Jun 2020 Cedric Gerbelot, Alia Abbara, Florent Krzakala

There has been a recent surge of interest in the study of asymptotic reconstruction performance in various cases of generalized linear estimation problems in the teacher-student setting, especially for the case of i. i. d standard normal matrices.

Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimization

no code implementations NeurIPS 2020 Benjamin Aubin, Florent Krzakala, Yue M. Lu, Lenka Zdeborová

We consider a commonly studied supervised classification of a synthetic dataset whose labels are generated by feeding a one-layer neural network with random iid inputs.

Phase retrieval in high dimensions: Statistical and computational phase transitions

1 code implementation NeurIPS 2020 Antoine Maillard, Bruno Loureiro, Florent Krzakala, Lenka Zdeborová

We consider the phase retrieval problem of reconstructing a $n$-dimensional real or complex signal $\mathbf{X}^{\star}$ from $m$ (possibly noisy) observations $Y_\mu = | \sum_{i=1}^n \Phi_{\mu i} X^{\star}_i/\sqrt{n}|$, for a large class of correlated real and complex random sensing matrices $\mathbf{\Phi}$, in a high-dimensional setting where $m, n\to\infty$ while $\alpha = m/n=\Theta(1)$.

TRAMP: Compositional Inference with TRee Approximate Message Passing

2 code implementations3 Apr 2020 Antoine Baker, Benjamin Aubin, Florent Krzakala, Lenka Zdeborová

We introduce tramp, standing for TRee Approximate Message Passing, a python package for compositional inference in high-dimensional tree-structured models.

Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime

2 code implementations2 Mar 2020 Stéphane d'Ascoli, Maria Refinetti, Giulio Biroli, Florent Krzakala

We obtain a precise asymptotic expression for the bias-variance decomposition of the test error, and show that the bias displays a phase transition at the interpolation threshold, beyond which it remains constant.

Generalisation error in learning with random features and the hidden manifold model

no code implementations ICML 2020 Federica Gerace, Bruno Loureiro, Florent Krzakala, Marc Mézard, Lenka Zdeborová

In particular, we show how to obtain analytically the so-called double descent behaviour for logistic regression with a peak at the interpolation threshold, we illustrate the superiority of orthogonal against random Gaussian projections in learning with random features, and discuss the role played by correlations in the data generated by the hidden manifold model.

Asymptotic errors for convex penalized linear regression beyond Gaussian matrices

no code implementations11 Feb 2020 Cédric Gerbelot, Alia Abbara, Florent Krzakala

We consider the problem of learning a coefficient vector $x_{0}$ in $R^{N}$ from noisy linear observations $y=Fx_{0}+w$ in $R^{M}$ in the high dimensional limit $M, N$ to infinity with $\alpha=M/N$ fixed.

Rademacher complexity and spin glasses: A link between the replica and statistical theories of learning

no code implementations5 Dec 2019 Alia Abbara, Benjamin Aubin, Florent Krzakala, Lenka Zdeborová

Statistical learning theory provides bounds of the generalization gap, using in particular the Vapnik-Chervonenkis dimension and the Rademacher complexity.

Learning Theory

Exact asymptotics for phase retrieval and compressed sensing with random generative priors

no code implementations4 Dec 2019 Benjamin Aubin, Bruno Loureiro, Antoine Baker, Florent Krzakala, Lenka Zdeborová

We consider the problem of compressed sensing and of (real-valued) phase retrieval with random measurement matrix.

Who is Afraid of Big Bad Minima? Analysis of gradient-flow in spiked matrix-tensor models

1 code implementation NeurIPS 2019 Stefano Sarao Mannelli, Giulio Biroli, Chiara Cammarota, Florent Krzakala, Lenka Zdeborová

Gradient-based algorithms are effective for many machine learning tasks, but despite ample recent effort and some progress, it often remains unclear why they work in practice in optimising high-dimensional non-convex functions and why they find good minima instead of being trapped in spurious ones. Here we present a quantitative theory explaining this behaviour in a spiked matrix-tensor model. Our framework is based on the Kac-Rice analysis of stationary points and a closed-form analysis of gradient-flow originating from statistical physics.

Modelling the influence of data structure on learning in neural networks: the hidden manifold model

1 code implementation25 Sep 2019 Sebastian Goldt, Marc Mézard, Florent Krzakala, Lenka Zdeborová

We demonstrate that learning of the hidden manifold model is amenable to an analytical treatment by proving a "Gaussian Equivalence Property" (GEP), and we use the GEP to show how the dynamics of two-layer neural networks trained using one-pass stochastic gradient descent is captured by a set of integro-differential equations that track the performance of the network at all times.

Who is Afraid of Big Bad Minima? Analysis of Gradient-Flow in a Spiked Matrix-Tensor Model

no code implementations18 Jul 2019 Stefano Sarao Mannelli, Giulio Biroli, Chiara Cammarota, Florent Krzakala, Lenka Zdeborová

Gradient-based algorithms are effective for many machine learning tasks, but despite ample recent effort and some progress, it often remains unclear why they work in practice in optimising high-dimensional non-convex functions and why they find good minima instead of being trapped in spurious ones.

Principled Training of Neural Networks with Direct Feedback Alignment

1 code implementation11 Jun 2019 Julien Launay, Iacopo Poli, Florent Krzakala

In this work, we focus on direct feedback alignment and present a set of best practices justified by observations of the alignment angles.

On the Universality of Noiseless Linear Estimation with Respect to the Measurement Matrix

1 code implementation11 Jun 2019 Alia Abbara, Antoine Baker, Florent Krzakala, Lenka Zdeborová

In a noiseless linear estimation problem, one aims to reconstruct a vector x* from the knowledge of its linear projections y=Phi x*.

The spiked matrix model with generative priors

2 code implementations NeurIPS 2019 Benjamin Aubin, Bruno Loureiro, Antoine Maillard, Florent Krzakala, Lenka Zdeborová

Here, we replace the sparsity assumption by generative modelling, and investigate the consequences on statistical and algorithmic properties.

Dimensionality Reduction

Passed & Spurious: Descent Algorithms and Local Minima in Spiked Matrix-Tensor Models

no code implementations1 Feb 2019 Stefano Sarao Mannelli, Florent Krzakala, Pierfrancesco Urbani, Lenka Zdeborová

In this work we analyse quantitatively the interplay between the loss landscape and performance of descent algorithms in a prototypical inference problem, the spiked matrix-tensor model.

Generalisation dynamics of online learning in over-parameterised neural networks

no code implementations25 Jan 2019 Sebastian Goldt, Madhu S. Advani, Andrew M. Saxe, Florent Krzakala, Lenka Zdeborová

Deep neural networks achieve stellar generalisation on a variety of problems, despite often being large enough to easily fit all their training data.

Marvels and Pitfalls of the Langevin Algorithm in Noisy High-dimensional Inference

no code implementations21 Dec 2018 Stefano Sarao Mannelli, Giulio Biroli, Chiara Cammarota, Florent Krzakala, Pierfrancesco Urbani, Lenka Zdeborová

Gradient-descent-based algorithms and their stochastic versions have widespread applications in machine learning and statistical inference.

Rank-one matrix estimation: analysis of algorithmic and information theoretic limits by the spatial coupling method

no code implementations6 Dec 2018 Jean Barbier, Mohamad Dia, Nicolas Macris, Florent Krzakala, Lenka Zdeborová

We characterize the detectability phase transitions in a large set of estimation problems, where we show that there exists a gap between what currently known polynomial algorithms (in particular spectral methods and approximate message-passing) can do and what is expected information theoretically.

Community Detection Compressive Sensing

Spectral Method for Multiplexed Phase Retrieval and Application in Optical Imaging in Complex Media

1 code implementation30 Oct 2018 Jonathan Dong, Florent Krzakala, Sylvain Gigan

We introduce a generalized version of phase retrieval called multiplexed phase retrieval.

Image and Video Processing Optics

Approximate message-passing for convex optimization with non-separable penalties

1 code implementation17 Sep 2018 Andre Manoel, Florent Krzakala, Gaël Varoquaux, Bertrand Thirion, Lenka Zdeborová

We introduce an iterative optimization scheme for convex objectives consisting of a linear loss and a non-separable penalty, based on the expectation-consistent approximation and the vector approximate message-passing (VAMP) algorithm.

Approximate Survey Propagation for Statistical Inference

no code implementations3 Jul 2018 Fabrizio Antenucci, Florent Krzakala, Pierfrancesco Urbani, Lenka Zdeborová

Approximate message passing algorithm enjoyed considerable attention in the last decade.

Fundamental limits of detection in the spiked Wigner model

no code implementations25 Jun 2018 Ahmed El Alaoui, Florent Krzakala, Michael. I. Jordan

We study the fundamental limits of detecting the presence of an additive rank-one perturbation, or spike, to a Wigner matrix.

The committee machine: Computational to statistical gaps in learning a two-layers neural network

1 code implementation NeurIPS 2018 Benjamin Aubin, Antoine Maillard, Jean Barbier, Florent Krzakala, Nicolas Macris, Lenka Zdeborová

Heuristic tools from statistical physics have been used in the past to locate the phase transitions and compute the optimal learning and generalization errors in the teacher-student scenario in multi-layer neural networks.

Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models

1 code implementation10 Aug 2017 Jean Barbier, Florent Krzakala, Nicolas Macris, Léo Miolane, Lenka Zdeborová

Non-rigorous predictions for the optimal errors existed for special cases of GLMs, e. g. for the perceptron, in the field of statistical physics based on the so-called replica method.

Streaming Bayesian inference: theoretical limits and mini-batch approximate message-passing

no code implementations2 Jun 2017 Andre Manoel, Florent Krzakala, Eric W. Tramel, Lenka Zdeborová

In statistical learning for real-world large-scale data problems, one must often resort to "streaming" algorithms which operate sequentially on small batches of data.

Bayesian Inference

A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines

no code implementations10 Feb 2017 Eric W. Tramel, Marylou Gabrié, Andre Manoel, Francesco Caltagirone, Florent Krzakala

Restricted Boltzmann machines (RBMs) are energy-based neural-networks which are commonly used as the building blocks for deep architectures neural architectures.

Denoising Latent Variable Models

Multi-Layer Generalized Linear Estimation

no code implementations24 Jan 2017 Andre Manoel, Florent Krzakala, Marc Mézard, Lenka Zdeborová

We consider the problem of reconstructing a signal from multi-layered (possibly) non-linear measurements.

Phase transitions and optimal algorithms in high-dimensional Gaussian mixture clustering

no code implementations10 Oct 2016 Thibault Lesieur, Caterina De Bacco, Jess Banks, Florent Krzakala, Cris Moore, Lenka Zdeborová

We consider the problem of Gaussian mixture clustering in the high-dimensional limit where the data consists of $m$ points in $n$ dimensions, $n, m \rightarrow \infty$ and $\alpha = m/n$ stays finite.

Scaling up Echo-State Networks with multiple light scattering

no code implementations15 Sep 2016 Jonathan Dong, Sylvain Gigan, Florent Krzakala, Gilles Wainrib

As a proof of concept, binary networks have been successfully trained to predict the chaotic Mackey-Glass time series.

Time Series

Inferring Sparsity: Compressed Sensing using Generalized Restricted Boltzmann Machines

no code implementations13 Jun 2016 Eric W. Tramel, Andre Manoel, Francesco Caltagirone, Marylou Gabrié, Florent Krzakala

In this work, we consider compressed sensing reconstruction from $M$ measurements of $K$-sparse structured signals which do not possess a writable correlation model.

Fast Randomized Semi-Supervised Clustering

no code implementations20 May 2016 Alaa Saade, Florent Krzakala, Marc Lelarge, Lenka Zdeborová

We consider the problem of clustering partially labeled data from a minimal number of randomly chosen pairwise comparisons between the items.

General Classification

Clustering from Sparse Pairwise Measurements

no code implementations25 Jan 2016 Alaa Saade, Marc Lelarge, Florent Krzakala, Lenka Zdeborová

We consider the problem of grouping items into clusters based on few random pairwise comparisons between the items.

Training Restricted Boltzmann Machine via the Thouless-Anderson-Palmer free energy

no code implementations NeurIPS 2015 Marylou Gabrie, Eric W. Tramel, Florent Krzakala

Restricted Boltzmann machines are undirected neural networks which have been shown tobe effective in many applications, including serving as initializations fortraining deep multi-layer neural networks.

Statistical physics of inference: Thresholds and algorithms

1 code implementation8 Nov 2015 Lenka Zdeborová, Florent Krzakala

Many questions of fundamental interest in todays science can be formulated as inference problems: Some partial, or noisy, observations are performed over a set of variables and the goal is to recover, or infer, the values of the variables based on the indirect information contained in the measurements.

Community Detection

MMSE of probabilistic low-rank matrix estimation: Universality with respect to the output channel

1 code implementation14 Jul 2015 Thibault Lesieur, Florent Krzakala, Lenka Zdeborová

This paper considers probabilistic estimation of a low-rank matrix from non-linear element-wise measurements of its elements.

Stochastic Block Model

Training Restricted Boltzmann Machines via the Thouless-Anderson-Palmer Free Energy

no code implementations9 Jun 2015 Marylou Gabrié, Eric W. Tramel, Florent Krzakala

Restricted Boltzmann machines are undirected neural networks which have been shown to be effective in many applications, including serving as initializations for training deep multi-layer neural networks.

Phase Transitions in Sparse PCA

1 code implementation1 Mar 2015 Thibault Lesieur, Florent Krzakala, Lenka Zdeborova

We study optimal estimation for sparse principal component analysis when the number of non-zero elements is small but on the same order as the dimension of the data.

Approximate Message Passing with Restricted Boltzmann Machine Priors

no code implementations23 Feb 2015 Eric W. Tramel, Angélique Drémeau, Florent Krzakala

Approximate Message Passing (AMP) has been shown to be an excellent statistical approach to signal inference and compressed sensing problem.

Spectral Detection in the Censored Block Model

no code implementations31 Jan 2015 Alaa Saade, Florent Krzakala, Marc Lelarge, Lenka Zdeborová

We describe two spectral algorithms for this task based on the non-backtracking and the Bethe Hessian operators.

Community Detection

Sparse Estimation with the Swept Approximated Message-Passing Algorithm

1 code implementation17 Jun 2014 Andre Manoel, Florent Krzakala, Eric W. Tramel, Lenka Zdeborová

Approximate Message Passing (AMP) has been shown to be a superior method for inference problems, such as the recovery of signals from sets of noisy, lower-dimensionality measurements, both in terms of reconstruction accuracy and in computational efficiency.

Spectral Clustering of Graphs with the Bethe Hessian

2 code implementations NeurIPS 2014 Alaa Saade, Florent Krzakala, Lenka Zdeborová

We show that this approach combines the performances of the non-backtracking operator, thus detecting clusters all the way down to the theoretical limit in the stochastic block model, with the computational, theoretical and memory advantages of real symmetric matrices.

Stochastic Block Model

Phase transitions and sample complexity in Bayes-optimal matrix factorization

no code implementations6 Feb 2014 Yoshiyuki Kabashima, Florent Krzakala, Marc Mézard, Ayaka Sakata, Lenka Zdeborová

We use the tools of statistical mechanics - the cavity and replica methods - to analyze the achievability and computational tractability of the inference problems in the setting of Bayes-optimal inference, which amounts to assuming that the two matrices have random independent elements generated from some known distribution, and this information is available to the inference algorithm.

Dictionary Learning Low-Rank Matrix Completion +1

Blind Calibration in Compressed Sensing using Message Passing Algorithms

no code implementations NeurIPS 2013 Christophe Schulke, Francesco Caltagirone, Florent Krzakala, Lenka Zdeborová

We study numerically the phase diagram of the blind calibration problem, and show that even in cases where convex relaxation is possible, our algorithm requires a smaller number of measurements and/or signals in order to perform well.

Model Selection for Degree-corrected Block Models

no code implementations17 Jul 2012 Xiaoran Yan, Cosma Rohilla Shalizi, Jacob E. Jensen, Florent Krzakala, Cristopher Moore, Lenka Zdeborova, Pan Zhang, Yaojia Zhu

We present the first principled and tractable approach to model selection between standard and degree-corrected block models, based on new large-graph asymptotics for the distribution of log-likelihood ratios under the stochastic block model, finding substantial departures from classical results for sparse graphs.

Model Selection Stochastic Block Model

Probabilistic Reconstruction in Compressed Sensing: Algorithms, Phase Diagrams, and Threshold Achieving Matrices

1 code implementation18 Jun 2012 Florent Krzakala, Marc Mézard, François Sausset, Yifan Sun, Lenka Zdeborová

We further develop the asymptotic analysis of the corresponding phase diagrams with and without measurement noise, for different distribution of signals, and discuss the best possible reconstruction performances regardless of the algorithm.

Statistical Mechanics Information Theory Information Theory

Statistical physics-based reconstruction in compressed sensing

1 code implementation20 Sep 2011 Florent Krzakala, Marc Mézard, François Sausset, Yifan Sun, Lenka Zdeborová

Compressed sensing is triggering a major evolution in signal acquisition.

Statistical Mechanics Information Theory Information Theory

Asymptotic analysis of the stochastic block model for modular networks and its algorithmic applications

no code implementations14 Sep 2011 Aurelien Decelle, Florent Krzakala, Cristopher Moore, Lenka Zdeborová

In this paper we extend our previous work on the stochastic block model, a commonly used generative model for social and biological networks, and the problem of inferring functional groups or communities from the topology of the network.

Statistical Mechanics Disordered Systems and Neural Networks Social and Information Networks Physics and Society

Phase transition in the detection of modules in sparse networks

no code implementations6 Feb 2011 Aurelien Decelle, Florent Krzakala, Cristopher Moore, Lenka Zdeborová

We present an asymptotically exact analysis of the problem of detecting communities in sparse random networks.

Cannot find the paper you are looking for? You can Submit a new open access paper.