Search Results for author:
Found 8 papers, 1 papers with code
How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks
A recent successful approach that falls under the JEPA framework is self-distillation, where an online encoder is trained to predict the output of the target encoder, sometimes using a lightweight predictor network.
Step-by-Step Diffusion: An Elementary Tutorial
We present an accessible first course on diffusion models and flow matching for machine learning, aimed at a technical audience with no diffusion experience.
A new role for circuit expansion for learning in neural networks
We find that sparse expansion of the input of a student perceptron network both increases its capacity and improves the generalization performance of the network when learning a noisy rule from a teacher perceptron when these expansions are pruned after learning.
Dimensionality compression and expansion in Deep Neural Networks
Datasets such as images, text, or movies are embedded in high-dimensional spaces.
Minnorm training: an algorithm for training over-parameterized deep neural networks
To solve this constrained optimization problem, our method employs Lagrange multipliers that act as integrators of error over training and identify `support vector'-like examples.
On the Information Bottleneck Theory of Deep Learning
The practical successes of deep neural networks have not been matched by theoretical progress that satisfyingly explains their behavior.
An equivalence between high dimensional Bayes optimal inference and M-estimation
In this work we demonstrate, when the signal distribution and the likelihood function associated with the noise are both log-concave, that optimal MMSE performance is asymptotically achievable via another M-estimation procedure.
Statistical Mechanics of High-Dimensional Inference
Our analysis uncovers fundamental limits on the accuracy of inference in high dimensions, and reveals that widely cherished inference algorithms like maximum likelihood (ML) and maximum-a posteriori (MAP) inference cannot achieve these limits.
