Search Results for author:
Found 28 papers, 4 papers with code
When can transformers reason with abstract symbols?
We investigate the capabilities of transformer models on relational reasoning tasks.
Transformers learn through gradual rank increase
Our experiments support the theory and also show that phenomenon can occur in practice without the simplifying assumptions.
SGD learning on neural networks: leap complexity and saddle-to-saddle dynamics
For $d$-dimensional uniform Boolean or isotropic Gaussian data, our main conjecture states that the time complexity to learn a function $f$ with low-dimensional support is $\tilde\Theta (d^{\max(\mathrm{Leap}(f), 2)})$.
Generalization on the Unseen, Logic Reasoning and Degree Curriculum
This paper considers the learning of logical (Boolean) functions with focus on the generalization on the unseen (GOTU) setting, a strong case of out-of-distribution generalization.
On the non-universality of deep learning: quantifying the cost of symmetry
We prove limitations on what neural networks trained by noisy gradient descent (GD) can efficiently learn.
Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measures
More generally, the paper considers the learning of logical functions with gradient descent (GD) on neural networks.
An initial alignment between neural network and target is needed for gradient descent to learn
This paper introduces the notion of ``Initial Alignment'' (INAL) between a neural network at initialization and a target function.
The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networks
It is currently known how to characterize functions that neural networks can learn with SGD for two extremal parameterizations: neural networks in the linear regime, and neural networks with no structural constraints.
Binary perceptron: efficient algorithms can find solutions in a rare well-connected cluster
It was recently shown that almost all solutions in the symmetric binary perceptron are isolated, even at low constraint densities, suggesting that finding typical solutions is hard.
The staircase property: How hierarchical structure can guide deep learning
This paper identifies a structural property of data distributions that enables deep neural networks to learn hierarchically.
On the Power of Differentiable Learning versus PAC and SQ Learning
With fine enough precision relative to minibatch size, namely when $b \rho$ is small enough, SGD can go beyond SQ learning and simulate any sample-based learning algorithm and thus its learning power is equivalent to that of PAC learning; this extends prior work that achieved this result for $b=1$.
Quantifying the Benefit of Using Differentiable Learning over Tangent Kernels
Complementing this, we show that without these conditions, gradient descent can in fact learn with small error even when no kernel method, in particular using the tangent kernel, can achieve a non-trivial advantage over random guessing.
Proof of the Contiguity Conjecture and Lognormal Limit for the Symmetric Perceptron
We consider the symmetric binary perceptron model, a simple model of neural networks that has gathered significant attention in the statistical physics, information theory and probability theory communities, with recent connections made to the performance of learning algorithms in Baldassi et al. '15.
Stochastic block model entropy and broadcasting on trees with survey
The limit of the entropy in the stochastic block model (SBM) has been characterized in the sparse regime for the special case of disassortative communities [COKPZ17] and for the classical case of assortative communities but in the dense regime [DAM16].
Probability Information Theory Information Theory
On the universality of deep learning
This paper shows that deep learning, i. e., neural networks trained by SGD, can learn in polytime any function class that can be learned in polytime by some algorithm, including parities.
Maximum Multiscale Entropy and Neural Network Regularization
For different entropies and arbitrary scale transformations, it is shown that the distribution maximizing a multiscale entropy is characterized by a procedure which has an analogy to the renormalization group procedure in statistical physics.
An $\ell_p$ theory of PCA and spectral clustering
Principal Component Analysis (PCA) is a powerful tool in statistics and machine learning.
Learning Sparse Graphons and the Generalized Kesten-Stigum Threshold
The problem of learning graphons has attracted considerable attention across several scientific communities, with significant progress over the recent years in sparser regimes.
Poly-time universality and limitations of deep learning
Therefore deep learning provides a universal learning paradigm: it was known that the approximation and estimation errors could be controlled with poly-size neural nets, using ERM that is NP-hard; this new result shows that the optimization error can also be controlled with SGD in poly-time.
Chaining Meets Chain Rule: Multilevel Entropic Regularization and Training of Neural Nets
The bounds are obtained by introducing the notion of generated hierarchical coverings of neural nets and by using the technique of chaining mutual information introduced in Asadi et al. NeurIPS'18.
Provable limitations of deep learning
As the success of deep learning reaches more grounds, one would like to also envision the potential limits of deep learning.
Chaining Mutual Information and Tightening Generalization Bounds
Two important difficulties are (i) exploiting the dependencies between the hypotheses, (ii) exploiting the dependence between the algorithm's input and output.
Communication-Computation Efficient Gradient Coding
This paper develops coding techniques to reduce the running time of distributed learning tasks.
Nonbacktracking Bounds on the Influence in Independent Cascade Models
This paper develops upper and lower bounds on the influence measure in a network, more precisely, the expected number of nodes that a seed set can influence in the independent cascade model.
Community Detection and Stochastic Block Models
This monograph surveys the recent developments that establish the fundamental limits for community detection in the SBM, both with respect to information-theoretic and computational tradeoffs, and for various recovery requirements such as exact, partial and weak recovery.
Achieving the KS threshold in the general stochastic block model with linearized acyclic belief propagation
The stochastic block model (SBM) has long been studied in machine learning and network science as a canonical model for clustering and community detection.
Detection in the stochastic block model with multiple clusters: proof of the achievability conjectures, acyclic BP, and the information-computation gap
In a paper that initiated the modern study of the stochastic block model, Decelle et al., backed by Mossel et al., made the following conjecture: Denote by $k$ the number of balanced communities, $a/n$ the probability of connecting inside communities and $b/n$ across, and set $\mathrm{SNR}=(a-b)^2/(k(a+(k-1)b)$; for any $k \geq 2$, it is possible to detect communities efficiently whenever $\mathrm{SNR}>1$ (the KS threshold), whereas for $k\geq 4$, it is possible to detect communities information-theoretically for some $\mathrm{SNR}<1$.
Recovering communities in the general stochastic block model without knowing the parameters
Most recent developments on the stochastic block model (SBM) rely on the knowledge of the model parameters, or at least on the number of communities.
