Search Results for author:
Found 29 papers, 5 papers with code
Provable Guarantees on Learning Hierarchical Generative Models with Deep CNNs
To show any positive theoretical results, one must make assumptions on the data distribution.
Universal Length Generalization with Turing Programs
Length generalization refers to the ability to extrapolate from short training sequences to long test sequences and is a challenge for current large language models.
A New Perspective on Shampoo's Preconditioner
Shampoo, a second-order optimization algorithm which uses a Kronecker product preconditioner, has recently garnered increasing attention from the machine learning community.
Transcendence: Generative Models Can Outperform The Experts That Train Them
Generative models are trained with the simple objective of imitating the conditional probability distribution induced by the data they are trained on.
The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains
We examine how learning is affected by varying the prior distribution over Markov chains, and consider the generalization of our in-context learning of Markov chains (ICL-MC) task to $n$-grams for $n > 2$.
Repeat After Me: Transformers are Better than State Space Models at Copying
Empirically, we find that transformers outperform GSSMs in terms of efficiency and generalization on synthetic tasks that require copying the context.
Auto-Regressive Next-Token Predictors are Universal Learners
Large language models display remarkable capabilities in logical and mathematical reasoning, allowing them to solve complex tasks.
Pareto Frontiers in Neural Feature Learning: Data, Compute, Width, and Luck
Finally, we show that the synthetic sparse parity task can be useful as a proxy for real problems requiring axis-aligned feature learning.
Corgi^2: A Hybrid Offline-Online Approach To Storage-Aware Data Shuffling For SGD
However, for large datasets stored in the cloud, random access to individual examples is often costly and inefficient.
Less is More: Selective Layer Finetuning with SubTuning
Finetuning a pretrained model has become a standard approach for training neural networks on novel tasks, resulting in fast convergence and improved performance.
Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit
There is mounting evidence of emergent phenomena in the capabilities of deep learning methods as we scale up datasets, model sizes, and training times.
Knowledge Distillation: Bad Models Can Be Good Role Models
We relate the notion of such samplers to knowledge distillation, where a student network imitates the outputs of a teacher on unlabeled data.
Provable Learning of Convolutional Neural Networks with Data Driven Features
Convolutional networks (CNN) are computationally hard to learn.
On the Power of Differentiable Learning versus PAC and SQ Learning
With fine enough precision relative to minibatch size, namely when $b \rho$ is small enough, SGD can go beyond SQ learning and simulate any sample-based learning algorithm and thus its learning power is equivalent to that of PAC learning; this extends prior work that achieved this result for $b=1$.
Quantifying the Benefit of Using Differentiable Learning over Tangent Kernels
Complementing this, we show that without these conditions, gradient descent can in fact learn with small error even when no kernel method, in particular using the tangent kernel, can achieve a non-trivial advantage over random guessing.
The Connection Between Approximation, Depth Separation and Learnability in Neural Networks
On the other hand, the fact that deep networks can efficiently express a target function does not mean that this target function can be learned efficiently by deep neural networks.
The Implications of Local Correlation on Learning Some Deep Functions
In fact, the proofs of such hardness results show that even weakly learning deep networks is hard.
Computational Separation Between Convolutional and Fully-Connected Networks
Convolutional neural networks (CNN) exhibit unmatched performance in a multitude of computer vision tasks.
When Hardness of Approximation Meets Hardness of Learning
A supervised learning algorithm has access to a distribution of labeled examples, and needs to return a function (hypothesis) that correctly labels the examples.
Learning Parities with Neural Networks
On the other hand, under the same distributions, these parities cannot be learned efficiently by linear methods.
Proving the Lottery Ticket Hypothesis: Pruning is All You Need
The lottery ticket hypothesis (Frankle and Carbin, 2018), states that a randomly-initialized network contains a small subnetwork such that, when trained in isolation, can compete with the performance of the original network.
Learning Boolean Circuits with Neural Networks
To separate hard from easy to learn distributions, we observe the property of local correlation: correlation between local patterns of the input and the target label.
On the Optimality of Trees Generated by ID3
Since its inception in the 1980s, ID3 has become one of the most successful and widely used algorithms for learning decision trees.
ID3 Learns Juntas for Smoothed Product Distributions
In recent years, there are many attempts to understand popular heuristics.
Decoupling Gating from Linearity
Specifically, we show a memorization result for networks of size $\tilde{\Omega}(\frac{m}{d})$, and improved generalization bounds.
Is Deeper Better only when Shallow is Good?
Using this result we prove that, at least in some distributions, the success of learning deep networks depends on whether the distribution can be well approximated by shallower networks, and we conjecture that this property holds in general.
A Provably Correct Algorithm for Deep Learning that Actually Works
We describe a layer-by-layer algorithm for training deep convolutional networks, where each step involves gradient updates for a two layer network followed by a simple clustering algorithm.
SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data
Neural networks exhibit good generalization behavior in the over-parameterized regime, where the number of network parameters exceeds the number of observations.
Decoupling "when to update" from "how to update"
Unfortunately, this approach often leads to noisy labels.
