Search Results for author:
Found 23 papers, 7 papers with code
Continuous-Time Meta-Learning with Forward Mode Differentiation
Drawing inspiration from gradient-based meta-learning methods with infinitely small gradient steps, we introduce Continuous-Time Meta-Learning (COMLN), a meta-learning algorithm where adaptation follows the dynamics of a gradient vector field.
Clarifying MCMC-based training of modern EBMs : Contrastive Divergence versus Maximum Likelihood
The Energy-Based Model (EBM) framework is a very general approach to generative modeling that tries to learn and exploit probability distributions only defined though unnormalized scores.
Learning shared neural manifolds from multi-subject FMRI data
In order to understand the connection between stimuli of interest and brain activity, and analyze differences and commonalities between subjects, it becomes important to learn a meaningful embedding of the data that denoises, and reveals its intrinsic structure.
Multi-scale Feature Learning Dynamics: Insights for Double Descent
A key challenge in building theoretical foundations for deep learning is the complex optimization dynamics of neural networks, resulting from the high-dimensional interactions between the large number of network parameters.
Compositional Attention: Disentangling Search and Retrieval
Through our qualitative analysis, we demonstrate that Compositional Attention leads to dynamic specialization based on the type of retrieval needed.
Embedding Signals on Knowledge Graphs with Unbalanced Diffusion Earth Mover's Distance
We propose to compare and organize such datasets of graph signals by using an earth mover's distance (EMD) with a geodesic cost over the underlying graph.
Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning
A central goal for AI and causality is thus the joint discovery of abstract representations and causal structure.
Efficient and robust multi-task learning in the brain with modular task primitives
Using a corpus of nine different tasks, we show that a modular network endowed with task primitives allows for learning multiple tasks well while keeping parameter counts, and updates, low.
Exploring the Geometry and Topology of Neural Network Loss Landscapes
This suggests that qualitative and quantitative examination of the loss landscape geometry could yield insights about neural network generalization performance during training.
Untangling tradeoffs between recurrence and self-attention in artificial neural networks
Attention and self-attention mechanisms, are now central to state-of-the-art deep learning on sequential tasks.
Gradient Starvation: A Learning Proclivity in Neural Networks
We identify and formalize a fundamental gradient descent phenomenon resulting in a learning proclivity in over-parameterized neural networks.
LEAD: Least-Action Dynamics for Min-Max Optimization
Adversarial formulations such as generative adversarial networks (GANs) have rekindled interest in two-player min-max games.
Implicit Regularization via Neural Feature Alignment
We approach the problem of implicit regularization in deep learning from a geometrical viewpoint.
Learning to Combine Top-Down and Bottom-Up Signals in Recurrent Neural Networks with Attention over Modules
To effectively utilize the wealth of potential top-down information available, and to prevent the cacophony of intermixed signals in a bidirectional architecture, mechanisms are needed to restrict information flow.
On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools
Recurrent neural networks (RNNs) have been successfully applied to a variety of problems involving sequential data, but their optimization is sensitive to parameter initialization, architecture, and optimizer hyperparameters.
Advantages of biologically-inspired adaptive neural activation in RNNs during learning
Dynamic adaptation in single-neuron response plays a fundamental role in neural coding in biological neural networks.
Untangling tradeoffs between recurrence and self-attention in neural networks
Attention and self-attention mechanisms, are now central to state-of-the-art deep learning on sequential tasks.
Internal representation dynamics and geometry in recurrent neural networks
The efficiency of recurrent neural networks (RNNs) in dealing with sequential data has long been established.
Modelling Working Memory using Deep Recurrent Reinforcement Learning
We train these neural networks to solve the working memory task by training them with a sequence of images in supervised and reinforcement learning settings.
Recurrent neural networks learn robust representations by dynamically balancing compression and expansion
What determines the dimensionality of activity in neural circuits?
Dimensionality compression and expansion in Deep Neural Networks
Datasets such as images, text, or movies are embedded in high-dimensional spaces.
Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics
A recent strategy to circumvent the exploding and vanishing gradient problem in RNNs, and to allow the stable propagation of signals over long time scales, is to constrain recurrent connectivity matrices to be orthogonal or unitary.
An Investigation of Memory in Recurrent Neural Networks
We investigate the learned dynamical landscape of a recurrent neural network solving a simple task requiring the interaction of two memory mechanisms: long- and short-term.
