Search Results for author:
Found 53 papers, 23 papers with code
Generalizable, real-time neural decoding with hybrid state-space models
We evaluate POSSM's decoding performance and inference speed on intracortical decoding of monkey motor tasks, and show that it extends to clinical applications, namely handwriting and speech decoding in human subjects.
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention.
Sufficient conditions for offline reactivation in recurrent neural networks
During periods of quiescence, such as sleep, neural activity in many brain circuits resembles that observed during periods of task engagement.
Bidirectional Information Flow (BIF) -- A Sample Efficient Hierarchical Gaussian Process for Bayesian Optimization
Hierarchical Gaussian Process (H-GP) models divide problems into different subtasks, allowing for different models to address each part, making them well-suited for problems with inherent hierarchical structure.
In-Context Parametric Inference: Point or Distribution Estimators?
Bayesian and frequentist inference are two fundamental paradigms in statistical estimation.
Learning Versatile Optimizers on a Compute Diet
Learned optimization has emerged as a promising alternative to hand-crafted optimizers, with the potential to discover stronger learned update rules that enable faster, hyperparameter-free training of neural networks.
Multi-agent cooperation through learning-aware policy gradients
Self-interested individuals often fail to cooperate, posing a fundamental challenge for multi-agent learning.
A Complexity-Based Theory of Compositionality
Here, we propose such a definition, which we call representational compositionality, that accounts for and extends our intuitions about compositionality.
In-context learning and Occam's razor
While the No Free Lunch Theorem states that we cannot obtain theoretical guarantees for generalization without further assumptions, in practice we observe that simple models which explain the training data generalize best: a principle called Occam's razor.
Latent Representation Learning for Multimodal Brain Activity Translation
Neuroscience employs diverse neuroimaging techniques, each offering distinct insights into brain activity, from electrophysiological recordings such as EEG, which have high temporal resolution, to hemodynamic modalities such as fMRI, which have increased spatial precision.
Accelerating Training with Neuron Interaction and Nowcasting Networks
Neural network training can be accelerated when a learnable update rule is used in lieu of classic adaptive optimizers (e. g. Adam).
When can transformers compositionally generalize in-context?
Under what circumstances can transformers compositionally generalize from a subset of tasks to all possible combinations of tasks that share similar components?
Expressivity of Neural Networks with Random Weights and Learned Biases
Landmark universal function approximation results for neural networks with trained weights and biases provided the impetus for the ubiquitous use of neural networks as learning models in neuroscience and Artificial Intelligence (AI).
Does learning the right latent variables necessarily improve in-context learning?
Our study highlights the intrinsic limitations of Transformers in achieving structured ICL solutions that generalize, and shows that while inferring the right latents aids interpretability, it is not sufficient to alleviate this problem.
How connectivity structure shapes rich and lazy learning in neural circuits
Through both empirical and theoretical analyses, we discover that high-rank initializations typically yield smaller network changes indicative of lazier learning, a finding we also confirm with experimentally-driven initial connectivity in recurrent neural networks.
Amortizing intractable inference in large language models
Autoregressive large language models (LLMs) compress knowledge from their training data through next-token conditional distributions.
Leveraging Unpaired Data for Vision-Language Generative Models via Cycle Consistency
We demonstrate image generation and captioning performance on par with state-of-the-art text-to-image and image-to-text models with orders of magnitude fewer (only 3M) paired image-text data.
Discrete, compositional, and symbolic representations through attractor dynamics
Symbolic systems are powerful frameworks for modeling cognitive processes as they encapsulate the rules and relationships fundamental to many aspects of human reasoning and behavior.
Delta-AI: Local objectives for amortized inference in sparse graphical models
We present a new algorithm for amortized inference in sparse probabilistic graphical models (PGMs), which we call $\Delta$-amortized inference ($\Delta$-AI).
Synaptic Weight Distributions Depend on the Geometry of Plasticity
Overall, our work shows that the current paradigm in theoretical work on synaptic plasticity that assumes Euclidean synaptic geometry may be misguided and that it should be possible to experimentally determine the true geometry of synaptic plasticity in the brain.
Flexible Phase Dynamics for Bio-Plausible Contrastive Learning
Many learning algorithms used as normative models in neuroscience or as candidate approaches for learning on neuromorphic chips learn by contrasting one set of network states with another.
Steerable Equivariant Representation Learning
In this paper, we propose a method of learning representations that are instead equivariant to data augmentations.
Sources of Richness and Ineffability for Phenomenally Conscious States
Conscious states (states that there is something it is like to be in) seem both rich or full of detail, and ineffable or hard to fully describe or recall.
Transfer Entropy Bottleneck: Learning Sequence to Sequence Information Transfer
When presented with a data stream of two statistically dependent variables, predicting the future of one of the variables (the target stream) can benefit from information about both its history and the history of the other variable (the source stream).
Reliability of CKA as a Similarity Measure in Deep Learning
Comparing learned neural representations in neural networks is a challenging but important problem, which has been approached in different ways.
From Points to Functions: Infinite-dimensional Representations in Diffusion Models
Consequently, it is reasonable to ask if there is an intermediate time step at which the preserved information is optimal for a given downstream task.
Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficulty
Among attempts at giving a theoretical account of the success of deep neural networks, a recent line of work has identified a so-called lazy training regime in which the network can be well approximated by its linearization around initialization.
On Neural Architecture Inductive Biases for Relational Tasks
Recent work has explored how forcing relational representations to remain distinct from sensory representations, as it seems to be the case in the brain, can help artificial systems.
Is a Modular Architecture Enough?
Inspired from human cognition, machine learning systems are gradually revealing advantages of sparser and more modular architectures.
Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules
We first demonstrate that state-of-the-art biologically-plausible learning rules for training RNNs exhibit worse and more variable generalization performance compared to their machine learning counterparts that follow the true gradient more closely.
Continuous-Time Meta-Learning with Forward Mode Differentiation
Drawing inspiration from gradient-based meta-learning methods with infinitely small gradient steps, we introduce Continuous-Time Meta-Learning (COMLN), a meta-learning algorithm where adaptation follows the dynamics of a gradient vector field.
Clarifying MCMC-based training of modern EBMs : Contrastive Divergence versus Maximum Likelihood
The Energy-Based Model (EBM) framework is a very general approach to generative modeling that tries to learn and exploit probability distributions only defined though unnormalized scores.
Learning shared neural manifolds from multi-subject FMRI data
In order to understand the connection between stimuli of interest and brain activity, and analyze differences and commonalities between subjects, it becomes important to learn a meaningful embedding of the data that denoises, and reveals its intrinsic structure.
Multi-scale Feature Learning Dynamics: Insights for Double Descent
A key challenge in building theoretical foundations for deep learning is the complex optimization dynamics of neural networks, resulting from the high-dimensional interactions between the large number of network parameters.
Compositional Attention: Disentangling Search and Retrieval
Through our qualitative analysis, we demonstrate that Compositional Attention leads to dynamic specialization based on the type of retrieval needed.
Embedding Signals on Knowledge Graphs with Unbalanced Diffusion Earth Mover's Distance
We propose to compare and organize such datasets of graph signals by using an earth mover's distance (EMD) with a geodesic cost over the underlying graph.
Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning
A central goal for AI and causality is thus the joint discovery of abstract representations and causal structure.
Efficient and robust multi-task learning in the brain with modular latent primitives
For this reason, a central aspect of human learning is the ability to recycle previously acquired knowledge in a way that allows for faster, less resource-intensive acquisition of new skills.
Exploring the Geometry and Topology of Neural Network Loss Landscapes
This suggests that qualitative and quantitative examination of the loss landscape geometry could yield insights about neural network generalization performance during training.
Untangling tradeoffs between recurrence and self-attention in artificial neural networks
Attention and self-attention mechanisms, are now central to state-of-the-art deep learning on sequential tasks.
Gradient Starvation: A Learning Proclivity in Neural Networks
We identify and formalize a fundamental gradient descent phenomenon resulting in a learning proclivity in over-parameterized neural networks.
Ranked #1 on
Out-of-Distribution Generalization
on ImageNet-W
LEAD: Min-Max Optimization from a Physical Perspective
A central obstacle in the optimization of such games is the rotational dynamics that hinder their convergence.
Implicit Regularization via Neural Feature Alignment
We approach the problem of implicit regularization in deep learning from a geometrical viewpoint.
Learning to Combine Top-Down and Bottom-Up Signals in Recurrent Neural Networks with Attention over Modules
To effectively utilize the wealth of potential top-down information available, and to prevent the cacophony of intermixed signals in a bidirectional architecture, mechanisms are needed to restrict information flow.
On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools
Recurrent neural networks (RNNs) have been successfully applied to a variety of problems involving sequential data, but their optimization is sensitive to parameter initialization, architecture, and optimizer hyperparameters.
Advantages of biologically-inspired adaptive neural activation in RNNs during learning
Dynamic adaptation in single-neuron response plays a fundamental role in neural coding in biological neural networks.
Untangling tradeoffs between recurrence and self-attention in neural networks
Attention and self-attention mechanisms, are now central to state-of-the-art deep learning on sequential tasks.
Internal representation dynamics and geometry in recurrent neural networks
The efficiency of recurrent neural networks (RNNs) in dealing with sequential data has long been established.
Recurrent neural networks learn robust representations by dynamically balancing compression and expansion
What determines the dimensionality of activity in neural circuits?
Modelling Working Memory using Deep Recurrent Reinforcement Learning
We train these neural networks to solve the working memory task by training them with a sequence of images in supervised and reinforcement learning settings.
Dimensionality compression and expansion in Deep Neural Networks
Datasets such as images, text, or movies are embedded in high-dimensional spaces.
Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics
A recent strategy to circumvent the exploding and vanishing gradient problem in RNNs, and to allow the stable propagation of signals over long time scales, is to constrain recurrent connectivity matrices to be orthogonal or unitary.
An Investigation of Memory in Recurrent Neural Networks
We investigate the learned dynamical landscape of a recurrent neural network solving a simple task requiring the interaction of two memory mechanisms: long- and short-term.
