Search Results for author:
Found 32 papers, 2 papers with code
Complexity Matters: Dynamics of Feature Learning in the Presence of Spurious Correlations
Existing research often posits spurious features as "easier" to learn than core features in neural network optimization, but the impact of their relative simplicity remains under-explored.
The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains
We examine how learning is affected by varying the prior distribution over Markov chains, and consider the generalization of our in-context learning of Markov chains (ICL-MC) task to $n$-grams for $n > 2$.
Pareto Frontiers in Neural Feature Learning: Data, Compute, Width, and Luck
Finally, we show that the synthetic sparse parity task can be useful as a proxy for real problems requiring axis-aligned feature learning.
Adversarial Resilience in Sequential Prediction via Abstention
We study the problem of sequential prediction in the stochastic setting with an adversary that is allowed to inject clean-label adversarial (or out-of-distribution) examples.
Learning Narrow One-Hidden-Layer ReLU Networks
We consider the well-studied problem of learning a linear combination of $k$ ReLU activations with respect to a Gaussian distribution on inputs in $d$ dimensions.
Transformers Learn Shortcuts to Automata
Algorithmic reasoning requires capabilities which are most naturally understood through recurrent models of computation, like the Turing machine.
Recurrent Convolutional Neural Networks Learn Succinct Learning Algorithms
For example, on parity problems, the NN learns as well as Gaussian elimination, an efficient algorithm that can be succinctly described.
Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit
There is mounting evidence of emergent phenomena in the capabilities of deep learning methods as we scale up datasets, model sizes, and training times.
Understanding Contrastive Learning Requires Incorporating Inductive Biases
Contrastive learning is a popular form of self-supervised learning that encourages augmentations (views) of the same input to have more similar representations compared to augmentations of different inputs.
Anti-Concentrated Confidence Bonuses for Scalable Exploration
Intrinsic rewards play a central role in handling the exploration-exploitation trade-off when designing sequential decision-making algorithms, in both foundational theory and state-of-the-art deep reinforcement learning.
Inductive Biases and Variable Creation in Self-Attention Mechanisms
Self-attention, an architectural motif designed to model long-range interactions in sequential data, has driven numerous recent breakthroughs in natural language processing and beyond.
Statistical Estimation from Dependent Data
We consider a general statistical estimation problem wherein binary labels across different observations are not independent conditioned on their feature vectors, but dependent, capturing settings where e. g. these observations are collected on a spatial domain, a temporal domain, or a social network, which induce dependencies.
Investigating the Role of Negatives in Contrastive Representation Learning
We focus on disambiguating the role of one of these parameters: the number of negative examples.
Gone Fishing: Neural Active Learning with Fisher Embeddings
There is an increasing need for effective active learning algorithms that are compatible with deep neural networks.
Acceleration via Fractal Learning Rate Schedules
In practical applications of iterative first-order optimization, the learning rate schedule remains notoriously difficult to understand and expensive to tune.
Tight Hardness Results for Training Depth-2 ReLU Networks
We are also able to obtain lower bounds on the running time in terms of the desired additive error $\epsilon$.
From Boltzmann Machines to Neural Networks and Back Again
Graphical models are powerful tools for modeling high-dimensional data, but learning graphical models in the presence of latent variables is well-known to be difficult.
Statistical-Query Lower Bounds via Functional Gradients
We give the first statistical-query lower bounds for agnostically learning any non-polynomial activation with respect to Gaussian marginals (e. g., ReLU, sigmoid, sign).
Superpolynomial Lower Bounds for Learning One-Layer Neural Networks using Gradient Descent
Our lower bounds hold for broad classes of activations including ReLU and sigmoid.
Approximation Schemes for ReLU Regression
We consider the fundamental problem of ReLU regression, where the goal is to output the best fitting ReLU with respect to square loss given access to draws from some unknown distribution.
Efficiently Learning Adversarially Robust Halfspaces with Noise
We study the problem of learning adversarially robust halfspaces in the distribution-independent setting.
Time/Accuracy Tradeoffs for Learning a ReLU with respect to Gaussian Marginals
Let $\mathsf{opt} < 1$ be the population loss of the best-fitting ReLU.
Learning Restricted Boltzmann Machines with Arbitrary External Fields
We study the problem of learning graphical models with latent variables.
Learning Mixtures of Graphs from Epidemic Cascades
When the conditions are met, i. e., when the graphs are connected with at least three edges, we give an efficient algorithm for learning the weights of both graphs with optimal sample complexity (up to log factors).
Recovering the Lowest Layer of Deep Networks with High Threshold Activations
Giving provable guarantees for learning neural networks is a core challenge of machine learning theory.
Quantifying Perceptual Distortion of Adversarial Examples
To demonstrate the value of quantifying the perceptual distortion of adversarial examples, we present and employ a unifying framework fusing different attack styles.
Learning Ising Models with Independent Failures
We give the first efficient algorithm for learning the structure of an Ising model that tolerates independent failures; that is, each entry of the observed sample is missing with some unknown probability p. Our algorithm matches the essentially optimal runtime and sample complexity bounds of recent work for learning Ising models due to Klivans and Meka (2017).
Improved Learning of One-hidden-layer Convolutional Neural Networks with Overlaps
We propose a new algorithm to learn a one-hidden-layer convolutional neural network where both the convolutional weights and the outputs weights are parameters to be learned.
Learning One Convolutional Layer with Overlapping Patches
We give the first provably efficient algorithm for learning a one hidden layer convolutional network with respect to a general class of (potentially overlapping) patches.
Learning Neural Networks with Two Nonlinear Layers in Polynomial Time
We give a polynomial-time algorithm for learning neural networks with one layer of sigmoids feeding into any Lipschitz, monotone activation function (e. g., sigmoid or ReLU).
Eigenvalue Decay Implies Polynomial-Time Learnability for Neural Networks
In this work we show that a natural distributional assumption corresponding to {\em eigenvalue decay} of the Gram matrix yields polynomial-time algorithms in the non-realizable setting for expressive classes of networks (e. g. feed-forward networks of ReLUs).
Reliably Learning the ReLU in Polynomial Time
These results are in contrast to known efficient algorithms for reliably learning linear threshold functions, where $\epsilon$ must be $\Omega(1)$ and strong assumptions are required on the marginal distribution.
