Search Results for author:
Found 25 papers, 2 papers with code
Simplified derivations for high-dimensional convex learning problems
Statistical physics provides tools for analyzing high-dimensional problems in machine learning and theoretical neuroscience.
Diverse capability and scaling of diffusion and auto-regressive models when learning abstract rules
We evaluated their ability to generate structurally consistent samples and perform panel completion via unconditional and conditional sampling.
Robust Learning in Bayesian Parallel Branching Graph Neural Networks: The Narrow Width Limit
The infinite width limit of random neural networks is known to result in Neural Networks as Gaussian Process (NNGP) (Lee et al. [2018]), characterized by task-independent kernels.
Order parameters and phase transitions of continual learning in deep neural networks
For networks with task-specific readouts, the theory identifies a phase transition where CL performance shifts dramatically as tasks become less similar, as measured by the OPs.
Coding schemes in neural networks learning classification tasks
Our findings highlight how network properties such as scaling of weights and neuronal nonlinearity can profoundly influence the emergent representations.
Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers
Our theory shows that the predictor statistics are expressed as a sum of independent kernels, each one pairing different 'attention paths', defined as information pathways through different attention heads across layers.
Probing Biological and Artificial Neural Networks with Task-dependent Neural Manifolds
Recently, growth in our understanding of the computations performed in both biological and artificial neural networks has largely been driven by either low-level mechanistic studies or global normative approaches.
Connecting NTK and NNGP: A Unified Theoretical Framework for Wide Neural Network Learning Dynamics
This work closes the gap between the NTK and NNGP theories, providing a comprehensive framework for the learning process of deep wide neural networks and for analyzing dynamics in biological circuits.
Globally Gated Deep Linear Networks
The rich and diverse behavior of the GGDLNs suggests that they are helpful analytically tractable models of learning single and multiple tasks, in finite-width nonlinear deep networks.
A theory of learning with constrained weight-distribution
Our theory predicts that the reduction in capacity due to the constrained weight-distribution is related to the Wasserstein distance between the imposed distribution and that of the standard normal distribution.
Temporal support vectors for spiking neuronal networks
However, for dynamical systems with event based outputs, such as spiking neural networks and other continuous time threshold crossing systems, this optimality criterion is inapplicable due to the strong temporal correlations in their input and output.
Optimal quadratic binding for relational reasoning in vector symbolic neural architectures
In these processes, two different modalities, such as location and objects, events and their contextual cues, and words and their roles, need to be bound together, but little is known about the underlying neural mechanisms.
Soft-margin classification of object manifolds
A neural population responding to multiple appearances of a single object defines a manifold in the neural response space.
Macroscopic Fluctuations Emerge in Balanced Networks with Incomplete Recurrent Alignment
Finally, we define the alignment matrix as the overlap between left and right-singular vectors of the structured connectivity, and show that the singular values of the alignment matrix determine the amplitude of macroscopic variability, while its singular vectors determine the structure.
Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalization
This procedure allows us to evaluate important network properties, such as its generalization error, the role of network width and depth, the impact of the size of the training set, and the effects of weight regularization and learning stochasticity.
Predicting the Outputs of Finite Networks Trained with Noisy Gradients
A recent line of works studied wide deep neural networks (DNNs) by approximating them as Gaussian Processes (GPs).
A new role for circuit expansion for learning in neural networks
We find that sparse expansion of the input of a student perceptron network both increases its capacity and improves the generalization performance of the network when learning a noisy rule from a teacher perceptron when these expansions are pruned after learning.
Predicting the outputs of finite deep neural networks trained with noisy gradients
A recent line of works studied wide deep neural networks (DNNs) by approximating them as Gaussian Processes (GPs).
Classification and Geometry of General Perceptual Manifolds
The effects of label sparsity on the classification capacity of manifolds are elucidated, revealing a scaling relation between label sparsity and manifold radius.
Learning Data Manifolds with a Cutting Plane Method
We consider the problem of classifying data manifolds where each manifold represents invariances that are parameterized by continuous degrees of freedom.
Balanced Excitation and Inhibition are Required for High-Capacity, Noise-Robust Neuronal Selectivity
To evaluate the impact of both input and output noise, we determine the robustness of single-neuron stimulus selective responses, as well as the robustness of attractor states of networks of neurons performing memory tasks.
Optimal Architectures in a Solvable Model of Deep Networks
Deep neural networks have received a considerable attention due to the success of their training for real world machine learning applications.
Linear Readout of Object Manifolds
Objects are represented in sensory systems by continuous manifolds due to sensitivity of neuronal responses to changes in physical features such as location, orientation, and intensity.
Short-term memory in neuronal networks through dynamical compressed sensing
Prior work, in the case of gaussian input sequences and linear neuronal networks, shows that the duration of memory traces in a network cannot exceed the number of neurons (in units of the neuronal time constant), and that no network can out-perform an equivalent feedforward network.
Inferring Stimulus Selectivity from the Spatial Structure of Neural Network Dynamics
How are the spatial patterns of spontaneous and evoked population responses related?
