Search Results for author:
Found 15 papers, 5 papers with code
Mixture-of-Depths: Dynamically allocating compute in transformer-based language models
Our method enforces a total compute budget by capping the number of tokens ($k$) that can participate in the self-attention and MLP computations at a given layer.
Addressing Sample Inefficiency in Multi-View Representation Learning
In this work, we provide theoretical insights on the implicit bias of the BarlowTwins and VICReg loss that can explain these heuristics and guide the development of more principled recommendations.
Learning to combine top-down context and feed-forward representations under ambiguity with apical and basal dendrites
When input stimuli are ambiguous and relevant contextual information is available, the apical feedback modulates the basal signals to recover unambiguous sensory representations.
Synaptic Weight Distributions Depend on the Geometry of Plasticity
Overall, our work shows that the current paradigm in theoretical work on synaptic plasticity that assumes Euclidean synaptic geometry may be misguided and that it should be possible to experimentally determine the true geometry of synaptic plasticity in the brain.
Transfer Entropy Bottleneck: Learning Sequence to Sequence Information Transfer
When presented with a data stream of two statistically dependent variables, predicting the future of one of the variables (the target stream) can benefit from information about both its history and the history of the other variable (the source stream).
Toward Next-Generation Artificial Intelligence: Catalyzing the NeuroAI Revolution
Neuroscience has long been an essential driver of progress in artificial intelligence (AI).
Contrastive Retrospection: honing in on critical steps for rapid learning and generalization in RL
Distinct from other contemporary RL approaches to credit assignment, ConSpec takes advantage of the fact that it is easier to retrospectively identify the small set of steps that success is contingent upon (and ignoring other states) than it is to prospectively predict reward at every taken step.
The neuroconnectionist research programme
Artificial Neural Networks (ANNs) inspired by biology are beginning to be widely used to model behavioral and neural data, an approach we call neuroconnectionism.
On Neural Architecture Inductive Biases for Relational Tasks
Recent work has explored how forcing relational representations to remain distinct from sensory representations, as it seems to be the case in the brain, can help artificial systems.
Evaluating Multimodal Interactive Agents
Creating agents that can interact naturally with humans is a common goal in artificial intelligence (AI) research.
Investigating Power laws in Deep Representation Learning
Access to task relevant labels at scale is often scarce or expensive, motivating the need to learn from unlabelled datasets with self-supervised learning (SSL).
Towards Scaling Difference Target Propagation by Learning Backprop Targets
As such, it is important to explore learning algorithms that come with strong theoretical guarantees and can match the performance of backpropagation (BP) on complex tasks.
Your head is there to move you around: Goal-driven models of the primate dorsal pathway
To test this hypothesis, we trained a 3D ResNet to predict an agent's self-motion parameters from visual stimuli in a simulated environment.
The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning
We show that when we train a deep neural network architecture with two parallel pathways using a self-supervised predictive loss function, we can outperform other models in fitting mouse visual cortex.
From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligence
Machine learning has long since become a keystone technology, accelerating science and applications in a broad range of domains.
