Search Results for author:
Found 39 papers, 17 papers with code
How connectivity structure shapes rich and lazy learning in neural circuits
Through both empirical and theoretical analyses, we discover that high-rank initializations typically yield smaller network changes indicative of lazier learning, a finding we also confirm with experimentally-driven initial connectivity in recurrent neural networks.
Amortizing intractable inference in large language models
Autoregressive large language models (LLMs) compress knowledge from their training data through next-token conditional distributions.
Leveraging Unpaired Data for Vision-Language Generative Models via Cycle Consistency
We demonstrate image generation and captioning performance on par with state-of-the-art text-to-image and image-to-text models with orders of magnitude fewer (only 3M) paired image-text data.
Discrete, compositional, and symbolic representations through attractor dynamics
Compositionality is an important feature of discrete symbolic systems, such as language and programs, as it enables them to have infinite capacity despite a finite symbol set.
Delta-AI: Local objectives for amortized inference in sparse graphical models
We present a new algorithm for amortized inference in sparse probabilistic graphical models (PGMs), which we call $\Delta$-amortized inference ($\Delta$-AI).
Synaptic Weight Distributions Depend on the Geometry of Plasticity
Overall, our work shows that the current paradigm in theoretical work on synaptic plasticity that assumes Euclidean synaptic geometry may be misguided and that it should be possible to experimentally determine the true geometry of synaptic plasticity in the brain.
Flexible Phase Dynamics for Bio-Plausible Contrastive Learning
Many learning algorithms used as normative models in neuroscience or as candidate approaches for learning on neuromorphic chips learn by contrasting one set of network states with another.
Steerable Equivariant Representation Learning
In this paper, we propose a method of learning representations that are instead equivariant to data augmentations.
Sources of Richness and Ineffability for Phenomenally Conscious States
Conscious states (states that there is something it is like to be in) seem both rich or full of detail, and ineffable or hard to fully describe or recall.
Transfer Entropy Bottleneck: Learning Sequence to Sequence Information Transfer
When presented with a data stream of two statistically dependent variables, predicting the future of one of the variables (the target stream) can benefit from information about both its history and the history of the other variable (the source stream).
Reliability of CKA as a Similarity Measure in Deep Learning
Comparing learned neural representations in neural networks is a challenging but important problem, which has been approached in different ways.
From Points to Functions: Infinite-dimensional Representations in Diffusion Models
Consequently, it is reasonable to ask if there is an intermediate time step at which the preserved information is optimal for a given downstream task.
Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficulty
Among attempts at giving a theoretical account of the success of deep neural networks, a recent line of work has identified a so-called lazy training regime in which the network can be well approximated by its linearization around initialization.
On Neural Architecture Inductive Biases for Relational Tasks
Recent work has explored how forcing relational representations to remain distinct from sensory representations, as it seems to be the case in the brain, can help artificial systems.
Is a Modular Architecture Enough?
Inspired from human cognition, machine learning systems are gradually revealing advantages of sparser and more modular architectures.
Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules
We first demonstrate that state-of-the-art biologically-plausible learning rules for training RNNs exhibit worse and more variable generalization performance compared to their machine learning counterparts that follow the true gradient more closely.
Continuous-Time Meta-Learning with Forward Mode Differentiation
Drawing inspiration from gradient-based meta-learning methods with infinitely small gradient steps, we introduce Continuous-Time Meta-Learning (COMLN), a meta-learning algorithm where adaptation follows the dynamics of a gradient vector field.
Clarifying MCMC-based training of modern EBMs : Contrastive Divergence versus Maximum Likelihood
The Energy-Based Model (EBM) framework is a very general approach to generative modeling that tries to learn and exploit probability distributions only defined though unnormalized scores.
Learning shared neural manifolds from multi-subject FMRI data
In order to understand the connection between stimuli of interest and brain activity, and analyze differences and commonalities between subjects, it becomes important to learn a meaningful embedding of the data that denoises, and reveals its intrinsic structure.
Multi-scale Feature Learning Dynamics: Insights for Double Descent
A key challenge in building theoretical foundations for deep learning is the complex optimization dynamics of neural networks, resulting from the high-dimensional interactions between the large number of network parameters.
Compositional Attention: Disentangling Search and Retrieval
Through our qualitative analysis, we demonstrate that Compositional Attention leads to dynamic specialization based on the type of retrieval needed.
Embedding Signals on Knowledge Graphs with Unbalanced Diffusion Earth Mover's Distance
We propose to compare and organize such datasets of graph signals by using an earth mover's distance (EMD) with a geodesic cost over the underlying graph.
Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning
A central goal for AI and causality is thus the joint discovery of abstract representations and causal structure.
Efficient and robust multi-task learning in the brain with modular latent primitives
For this reason, a central aspect of human learning is the ability to recycle previously acquired knowledge in a way that allows for faster, less resource-intensive acquisition of new skills.
Exploring the Geometry and Topology of Neural Network Loss Landscapes
This suggests that qualitative and quantitative examination of the loss landscape geometry could yield insights about neural network generalization performance during training.
Untangling tradeoffs between recurrence and self-attention in artificial neural networks
Attention and self-attention mechanisms, are now central to state-of-the-art deep learning on sequential tasks.
Gradient Starvation: A Learning Proclivity in Neural Networks
We identify and formalize a fundamental gradient descent phenomenon resulting in a learning proclivity in over-parameterized neural networks.
Ranked #1 on
Out-of-Distribution Generalization
on ImageNet-W
LEAD: Min-Max Optimization from a Physical Perspective
A central obstacle in the optimization of such games is the rotational dynamics that hinder their convergence.
Implicit Regularization via Neural Feature Alignment
We approach the problem of implicit regularization in deep learning from a geometrical viewpoint.
Learning to Combine Top-Down and Bottom-Up Signals in Recurrent Neural Networks with Attention over Modules
To effectively utilize the wealth of potential top-down information available, and to prevent the cacophony of intermixed signals in a bidirectional architecture, mechanisms are needed to restrict information flow.
On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools
Recurrent neural networks (RNNs) have been successfully applied to a variety of problems involving sequential data, but their optimization is sensitive to parameter initialization, architecture, and optimizer hyperparameters.
Advantages of biologically-inspired adaptive neural activation in RNNs during learning
Dynamic adaptation in single-neuron response plays a fundamental role in neural coding in biological neural networks.
Untangling tradeoffs between recurrence and self-attention in neural networks
Attention and self-attention mechanisms, are now central to state-of-the-art deep learning on sequential tasks.
Internal representation dynamics and geometry in recurrent neural networks
The efficiency of recurrent neural networks (RNNs) in dealing with sequential data has long been established.
Modelling Working Memory using Deep Recurrent Reinforcement Learning
We train these neural networks to solve the working memory task by training them with a sequence of images in supervised and reinforcement learning settings.
Recurrent neural networks learn robust representations by dynamically balancing compression and expansion
What determines the dimensionality of activity in neural circuits?
Dimensionality compression and expansion in Deep Neural Networks
Datasets such as images, text, or movies are embedded in high-dimensional spaces.
Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics
A recent strategy to circumvent the exploding and vanishing gradient problem in RNNs, and to allow the stable propagation of signals over long time scales, is to constrain recurrent connectivity matrices to be orthogonal or unitary.
An Investigation of Memory in Recurrent Neural Networks
We investigate the learned dynamical landscape of a recurrent neural network solving a simple task requiring the interaction of two memory mechanisms: long- and short-term.
