Search Results for author:
Found 59 papers, 28 papers with code
How Feature Learning Can Improve Neural Scaling Laws
We develop a solvable model of neural scaling laws beyond the kernel limit.
Risk and cross validation in ridge regression with correlated samples
Recent years have seen substantial advances in our understanding of high-dimensional ridge regression, but existing theories assume that training examples are independent.
Nadaraya-Watson kernel smoothing as a random energy model
We investigate the behavior of the Nadaraya-Watson kernel smoothing estimator in high dimensions using its relationship to the random energy model and to dense associative memories.
Spectral regularization for adversarially-robust representation learning
To this end, we propose a new spectral regularizer for representation learning that encourages black-box adversarial robustness in downstream classification tasks.
Convex Relaxation for Solving Large-Margin Classifiers in Hyperbolic Space
Hyperbolic spaces have increasingly been recognized for their outstanding performance in handling data with inherent hierarchical structures compared to their Euclidean counterparts.
MLPs Learn In-Context on Regression and Classification Tasks
In-context learning (ICL), the remarkable ability to solve a task from only input exemplars, is often assumed to be a unique hallmark of Transformer models.
Infinite Limits of Multi-head Transformer Dynamics
In this work, we analyze various scaling limits of the training dynamics of transformer models in the feature learning regime.
Asymptotic theory of in-context learning by linear attention
Transformers have a remarkable ability to learn and execute tasks based on examples provided within the input itself, without explicit prior training.
Scaling and renormalization in high-dimensional regression
This paper presents a succinct derivation of the training and generalization performance of a variety of high-dimensional ridge regression models using the basic tools of random matrix theory and free probability.
A Dynamical Model of Neural Scaling Laws
On a variety of tasks, the performance of neural networks predictably improves with training time, dataset size and model size across many orders of magnitude.
Grokking as the Transition from Lazy to Rich Training Dynamics
We identify sufficient statistics for the test loss of such a network, and tracking these over training reveals that grokking arises in this setting when the network first attempts to fit a kernel regression solution with its initial features, followed by late-time feature learning where a generalizing solution is identified after train loss is already low.
Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit
We provide experiments demonstrating that residual architectures including convolutional ResNets and Vision Transformers trained with this parameterization exhibit transfer of optimal hyperparameters across width and depth on CIFAR-10 and ImageNet.
Loss Dynamics of Temporal Difference Reinforcement Learning
We study how learning dynamics and plateaus depend on feature structure, learning rate, discount factor, and reward function.
Learning Curves for Noisy Heterogeneous Feature-Subsampled Ridge Ensembles
Feature bagging is a well-established ensembling method which aims to reduce prediction variance by combining predictions of many estimators trained on subsets or projections of features.
Correlative Information Maximization: A Biologically Plausible Approach to Supervised Deep Neural Networks without Weight Symmetry
Furthermore, our approach provides a natural resolution to the weight symmetry problem between forward and backward signal propagation paths, a significant critique against the plausibility of the conventional backpropagation algorithm.
Long Sequence Hopfield Memory
Sequence memory is an essential attribute of natural and artificial intelligence that enables agents to encode, store, and retrieve complex sequences of stimuli and actions.
Feature-Learning Networks Are Consistent Across Widths At Realistic Scales
We call this the bias of narrower width.
Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks
However, in the rich, feature learning regime, the fluctuations of the kernels and predictions are dynamically coupled with a variance that can be computed self-consistently.
Neural networks learn to magnify areas near decision boundaries
This holds in deep networks trained on high-dimensional image classification tasks, and even in self-supervised representation learning.
The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimes
For small training set sizes $P$, the generalization error of wide neural networks is well-approximated by the error of an infinite width neural network (NN), either in the kernel or mean-field/feature-learning regime.
Correlative Information Maximization Based Biologically Plausible Neural Networks for Correlated Source Separation
Here, we relax this limitation and propose a biologically plausible neural network that extracts correlated latent sources by exploiting information about their domains.
The Influence of Learning Rule on Representation Dynamics in Wide Neural Networks
In the lazy limit, we find that DFA and Hebb can only learn using the last layer features, while full FA can utilize earlier layers with a scale determined by the initial correlation between feedforward and feedback weight matrices.
Biologically-Plausible Determinant Maximization Neural Networks for Blind Separation of Correlated Sources
Previous work on biologically-plausible BSS algorithms assumed that observed signals are linear mixtures of statistically independent or uncorrelated sources, limiting the domain of applicability of these algorithms.
Interneurons accelerate learning dynamics in recurrent neural networks for statistical adaptation
To this end, we consider two mathematically tractable recurrent linear neural networks that statistically whiten their inputs -- one with direct recurrent connections and the other with interneurons that mediate recurrent communication.
Bandwidth Enables Generalization in Quantum Kernel Models
Quantum computers are known to provide speedups over classical state-of-the-art machine learning methods in some specialized settings.
Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks
We analyze feature learning in infinite-width neural networks trained with gradient flow through a self-consistent dynamical field theory.
Contrasting random and learned features in deep Bayesian linear regression
Moreover, we show that the leading-order correction to the kernel-limit learning curve cannot distinguish between random feature models and deep networks in which all layers are trained.
On neural network kernels and the storage capacity problem
In this short note, we reify the connection between work on the storage capacity problem in wide two-layer treelike neural networks and the rapidly-growing body of literature on kernel limits of wide neural networks.
Depth induces scale-averaging in overparameterized linear Bayesian neural networks
Inference in deep Bayesian neural networks is only fully understood in the infinite-width limit, where the posterior flexibility afforded by increased depth washes out and the posterior predictive collapses to a shallow Gaussian process.
Attention Approximates Sparse Distributed Memory
While Attention has come to be an important mechanism in deep learning, there remains limited intuition for why it works so well.
Neural Networks as Kernel Learners: The Silent Alignment Effect
Can neural networks in the rich feature learning regime learn a kernel machine with a data-dependent kernel?
Capacity of Group-invariant Linear Readouts from Equivariant Representations: How Many Objects can be Linearly Classified Under All Possible Views?
We find that the fraction of separable dichotomies is determined by the dimension of the space that is fixed by the group action.
Out-of-Distribution Generalization in Kernel Regression
Here, we study generalization in kernel regression when the training and test distributions are different using methods from statistical physics.
BIG-bench Machine Learning
Out-of-Distribution Generalization
+1
Learning Curves for SGD on Structured Features
To analyze the influence of data structure on test loss dynamics, we study an exactly solveable model of stochastic gradient descent (SGD) on mean square loss which predicts test loss when training on features with arbitrary covariance structure.
Asymptotics of representation learning in finite Bayesian neural networks
However, our theoretical understanding of how the learned hidden layer representations of finite networks differ from the fixed representations of infinite networks remains incomplete.
Exact marginal prior distributions of finite Bayesian neural networks
For deep linear networks, the prior has a simple expression in terms of the Meijer $G$-function.
Biologically plausible single-layer networks for nonnegative independent component analysis
To model how the brain performs this task, we seek a biologically plausible single-layer neural network implementation of a blind source separation algorithm.
Activation function dependence of the storage capacity of treelike neural networks
Though a wide variety of nonlinear activation functions have been proposed for use in artificial neural networks, a detailed understanding of their role in determining the expressive power of a network has not emerged.
Associative Memory in Iterated Overparameterized Sigmoid Autoencoders
Recent work showed that overparameterized autoencoders can be trained to implement associative memory via iterative maps, when the trained input-output Jacobian of the network has all of its eigenvalue norms strictly below one.
Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks
We present applications of our theory to real and synthetic datasets, and for many kernels including those that arise from training deep neural networks in the infinite-width limit.
Minimax Dynamics of Optimally Balanced Spiking Networks of Excitatory and Inhibitory Neurons
Excitation-inhibition (E-I) balance is ubiquitously observed in the cortex.
Blind Bounded Source Separation Using Neural Networks with Local Learning Rules
An important problem encountered by both natural and engineered signal processing systems is blind source separation.
Contrastive Similarity Matching for Supervised Learning
We propose a novel biologically-plausible solution to the credit assignment problem motivated by observations in the ventral visual pathway and trained deep neural networks.
Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks
We derive analytical expressions for the generalization performance of kernel regression as a function of the number of training samples using theoretical methods from Gaussian processes and statistical physics.
A Closer Look at Disentangling in $β$-VAE
In many data analysis tasks, it is beneficial to learn representations where each dimension is statistically independent and thus disentangled from the others.
Structured and Deep Similarity Matching via Structured and Deep Hebbian Networks
In single-layered and all-to-all connected neural networks, local plasticity has been shown to implement gradient-based learning on a class of cost functions that contain a term that aligns the similarity of outputs to the similarity of inputs.
Neuroscience-inspired online unsupervised learning algorithms
Although the currently popular deep learning networks achieve unprecedented performance on some tasks, the human brain still has a monopoly on general intelligence.
A Spiking Neural Network with Local Learning Rules Derived From Nonnegative Similarity Matching
The design and analysis of spiking neural network algorithms will be accelerated by the advent of new theoretical approaches.
Manifold-tiling Localized Receptive Fields are Optimal in Similarity-preserving Neural Networks
Many neurons in the brain, such as place cells in the rodent hippocampus, have localized receptive fields, i. e., they respond to a small neighborhood of stimulus space.
Efficient Principal Subspace Projection of Streaming Data Through Fast Similarity Matching
Big data problems frequently require processing datasets in a streaming fashion, either because all data are available at once but collectively are larger than available memory or because the data intrinsically arrive one data point at a time and must be processed online.
Blind nonnegative source separation using biological neural networks
Blind source separation, i. e. extraction of independent sources from a mixture, is an important problem for both artificial and natural signal processing.
Why do similarity matching objectives lead to Hebbian/anti-Hebbian networks?
Modeling self-organization of neural networks for unsupervised learning using Hebbian and anti-Hebbian plasticity has a long history in neuroscience.
Self-calibrating Neural Networks for Dimensionality Reduction
Here we propose online algorithms where the threshold is self-calibrating based on the singular values computed from the existing observations.
Optimization theory of Hebbian/anti-Hebbian networks for PCA and whitening
Here, we focus on such workhorses of signal processing as Principal Component Analysis (PCA) and whitening which maximize information transmission in the presence of noise.
A Normative Theory of Adaptive Dimensionality Reduction in Neural Networks
Here, we derive biologically plausible dimensionality reduction algorithms which adapt the number of output dimensions to the eigenspectrum of the input covariance matrix.
A Hebbian/Anti-Hebbian Network for Online Sparse Dictionary Learning Derived from Symmetric Matrix Factorization
Here, to overcome this problem, we derive sparse dictionary learning from a novel cost-function - a regularized error of the symmetric factorization of the input's similarity matrix.
A Hebbian/Anti-Hebbian Network Derived from Online Non-Negative Matrix Factorization Can Cluster and Discover Sparse Features
Despite our extensive knowledge of biophysical properties of neurons, there is no commonly accepted algorithmic theory of neuronal function.
A Hebbian/Anti-Hebbian Neural Network for Linear Subspace Learning: A Derivation from Multidimensional Scaling of Streaming Data
Such networks learn the principal subspace, in the sense of principal component analysis (PCA), by adjusting synaptic weights according to activity-dependent learning rules.
A Neuron as a Signal Processing Device
Here we propose to view a neuron as a signal processing device that represents the incoming streaming data matrix as a sparse vector of synaptic weights scaled by an outgoing sparse activity vector.
