Search Results for author: Guillaume Lajoie

Found 39 papers, 17 papers with code

How connectivity structure shapes rich and lazy learning in neural circuits

no code implementations12 Oct 2023 Yuhan Helena Liu, Aristide Baratin, Jonathan Cornford, Stefan Mihalas, Eric Shea-Brown, Guillaume Lajoie

Through both empirical and theoretical analyses, we discover that high-rank initializations typically yield smaller network changes indicative of lazier learning, a finding we also confirm with experimentally-driven initial connectivity in recurrent neural networks.

Amortizing intractable inference in large language models

1 code implementation6 Oct 2023 Edward J. Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, Nikolay Malkin

Autoregressive large language models (LLMs) compress knowledge from their training data through next-token conditional distributions.

Bayesian Inference

Leveraging Unpaired Data for Vision-Language Generative Models via Cycle Consistency

no code implementations5 Oct 2023 Tianhong Li, Sangnie Bhardwaj, Yonglong Tian, Han Zhang, Jarred Barber, Dina Katabi, Guillaume Lajoie, Huiwen Chang, Dilip Krishnan

We demonstrate image generation and captioning performance on par with state-of-the-art text-to-image and image-to-text models with orders of magnitude fewer (only 3M) paired image-text data.

Text-to-Image Generation

Discrete, compositional, and symbolic representations through attractor dynamics

no code implementations3 Oct 2023 Andrew Nam, Eric Elmoznino, Nikolay Malkin, Chen Sun, Yoshua Bengio, Guillaume Lajoie

Compositionality is an important feature of discrete symbolic systems, such as language and programs, as it enables them to have infinite capacity despite a finite symbol set.


Delta-AI: Local objectives for amortized inference in sparse graphical models

1 code implementation3 Oct 2023 Jean-Pierre Falet, Hae Beom Lee, Nikolay Malkin, Chen Sun, Dragos Secrieru, Thomas Jiralerspong, Dinghuai Zhang, Guillaume Lajoie, Yoshua Bengio

We present a new algorithm for amortized inference in sparse probabilistic graphical models (PGMs), which we call $\Delta$-amortized inference ($\Delta$-AI).

Synaptic Weight Distributions Depend on the Geometry of Plasticity

1 code implementation30 May 2023 Roman Pogodin, Jonathan Cornford, Arna Ghosh, Gauthier Gidel, Guillaume Lajoie, Blake Richards

Overall, our work shows that the current paradigm in theoretical work on synaptic plasticity that assumes Euclidean synaptic geometry may be misguided and that it should be possible to experimentally determine the true geometry of synaptic plasticity in the brain.

Flexible Phase Dynamics for Bio-Plausible Contrastive Learning

1 code implementation24 Feb 2023 Ezekiel Williams, Colin Bredenberg, Guillaume Lajoie

Many learning algorithms used as normative models in neuroscience or as candidate approaches for learning on neuromorphic chips learn by contrasting one set of network states with another.

Contrastive Learning

Steerable Equivariant Representation Learning

no code implementations22 Feb 2023 Sangnie Bhardwaj, Willie McClinton, Tongzhou Wang, Guillaume Lajoie, Chen Sun, Phillip Isola, Dilip Krishnan

In this paper, we propose a method of learning representations that are instead equivariant to data augmentations.

Image Retrieval object-detection +5

Sources of Richness and Ineffability for Phenomenally Conscious States

no code implementations13 Feb 2023 Xu Ji, Eric Elmoznino, George Deane, Axel Constant, Guillaume Dumas, Guillaume Lajoie, Jonathan Simon, Yoshua Bengio

Conscious states (states that there is something it is like to be in) seem both rich or full of detail, and ineffable or hard to fully describe or recall.


Transfer Entropy Bottleneck: Learning Sequence to Sequence Information Transfer

no code implementations29 Nov 2022 Damjan Kalajdzievski, Ximeng Mao, Pascal Fortier-Poisson, Guillaume Lajoie, Blake Richards

When presented with a data stream of two statistically dependent variables, predicting the future of one of the variables (the target stream) can benefit from information about both its history and the history of the other variable (the source stream).

Reliability of CKA as a Similarity Measure in Deep Learning

no code implementations28 Oct 2022 MohammadReza Davari, Stefan Horoi, Amine Natik, Guillaume Lajoie, Guy Wolf, Eugene Belilovsky

Comparing learned neural representations in neural networks is a challenging but important problem, which has been approached in different ways.


From Points to Functions: Infinite-dimensional Representations in Diffusion Models

1 code implementation25 Oct 2022 Sarthak Mittal, Guillaume Lajoie, Stefan Bauer, Arash Mehrjou

Consequently, it is reasonable to ask if there is an intermediate time step at which the preserved information is optimal for a given downstream task.

Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficulty

1 code implementation19 Sep 2022 Thomas George, Guillaume Lajoie, Aristide Baratin

Among attempts at giving a theoretical account of the success of deep neural networks, a recent line of work has identified a so-called lazy training regime in which the network can be well approximated by its linearization around initialization.

On Neural Architecture Inductive Biases for Relational Tasks

1 code implementation9 Jun 2022 Giancarlo Kerg, Sarthak Mittal, David Rolnick, Yoshua Bengio, Blake Richards, Guillaume Lajoie

Recent work has explored how forcing relational representations to remain distinct from sensory representations, as it seems to be the case in the brain, can help artificial systems.

Inductive Bias Out-of-Distribution Generalization

Is a Modular Architecture Enough?

1 code implementation6 Jun 2022 Sarthak Mittal, Yoshua Bengio, Guillaume Lajoie

Inspired from human cognition, machine learning systems are gradually revealing advantages of sparser and more modular architectures.

Out-of-Distribution Generalization

Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules

1 code implementation2 Jun 2022 Yuhan Helena Liu, Arna Ghosh, Blake A. Richards, Eric Shea-Brown, Guillaume Lajoie

We first demonstrate that state-of-the-art biologically-plausible learning rules for training RNNs exhibit worse and more variable generalization performance compared to their machine learning counterparts that follow the true gradient more closely.

Learning Theory

Continuous-Time Meta-Learning with Forward Mode Differentiation

no code implementations ICLR 2022 Tristan Deleu, David Kanaa, Leo Feng, Giancarlo Kerg, Yoshua Bengio, Guillaume Lajoie, Pierre-Luc Bacon

Drawing inspiration from gradient-based meta-learning methods with infinitely small gradient steps, we introduce Continuous-Time Meta-Learning (COMLN), a meta-learning algorithm where adaptation follows the dynamics of a gradient vector field.

Few-Shot Image Classification Meta-Learning

Clarifying MCMC-based training of modern EBMs : Contrastive Divergence versus Maximum Likelihood

no code implementations24 Feb 2022 Léo Gagnon, Guillaume Lajoie

The Energy-Based Model (EBM) framework is a very general approach to generative modeling that tries to learn and exploit probability distributions only defined though unnormalized scores.

Image Generation

Learning shared neural manifolds from multi-subject FMRI data

no code implementations22 Dec 2021 Jessie Huang, Erica L. Busch, Tom Wallenstein, Michal Gerasimiuk, Andrew Benz, Guillaume Lajoie, Guy Wolf, Nicholas B. Turk-Browne, Smita Krishnaswamy

In order to understand the connection between stimuli of interest and brain activity, and analyze differences and commonalities between subjects, it becomes important to learn a meaningful embedding of the data that denoises, and reveals its intrinsic structure.

Brain Computer Interface

Multi-scale Feature Learning Dynamics: Insights for Double Descent

1 code implementation6 Dec 2021 Mohammad Pezeshki, Amartya Mitra, Yoshua Bengio, Guillaume Lajoie

A key challenge in building theoretical foundations for deep learning is the complex optimization dynamics of neural networks, resulting from the high-dimensional interactions between the large number of network parameters.

Compositional Attention: Disentangling Search and Retrieval

3 code implementations ICLR 2022 Sarthak Mittal, Sharath Chandra Raparthy, Irina Rish, Yoshua Bengio, Guillaume Lajoie

Through our qualitative analysis, we demonstrate that Compositional Attention leads to dynamic specialization based on the type of retrieval needed.


Embedding Signals on Knowledge Graphs with Unbalanced Diffusion Earth Mover's Distance

no code implementations26 Jul 2021 Alexander Tong, Guillaume Huguet, Dennis Shung, Amine Natik, Manik Kuchroo, Guillaume Lajoie, Guy Wolf, Smita Krishnaswamy

We propose to compare and organize such datasets of graph signals by using an earth mover's distance (EMD) with a geodesic cost over the underlying graph.

Knowledge Graph Embedding Knowledge Graphs

Efficient and robust multi-task learning in the brain with modular latent primitives

no code implementations28 May 2021 Christian David Márton, Léo Gagnon, Guillaume Lajoie, Kanaka Rajan

For this reason, a central aspect of human learning is the ability to recycle previously acquired knowledge in a way that allows for faster, less resource-intensive acquisition of new skills.

Multi-Task Learning

Exploring the Geometry and Topology of Neural Network Loss Landscapes

no code implementations31 Jan 2021 Stefan Horoi, Jessie Huang, Bastian Rieck, Guillaume Lajoie, Guy Wolf, Smita Krishnaswamy

This suggests that qualitative and quantitative examination of the loss landscape geometry could yield insights about neural network generalization performance during training.

Dimensionality Reduction

LEAD: Min-Max Optimization from a Physical Perspective

1 code implementation26 Oct 2020 Reyhane Askari Hemmat, Amartya Mitra, Guillaume Lajoie, Ioannis Mitliagkas

A central obstacle in the optimization of such games is the rotational dynamics that hinder their convergence.

Image Generation

Learning to Combine Top-Down and Bottom-Up Signals in Recurrent Neural Networks with Attention over Modules

1 code implementation ICML 2020 Sarthak Mittal, Alex Lamb, Anirudh Goyal, Vikram Voleti, Murray Shanahan, Guillaume Lajoie, Michael Mozer, Yoshua Bengio

To effectively utilize the wealth of potential top-down information available, and to prevent the cacophony of intermixed signals in a bidirectional architecture, mechanisms are needed to restrict information flow.

Language Modelling Open-Ended Question Answering +2

On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools

no code implementations25 Jun 2020 Ryan Vogt, Maximilian Puelma Touzel, Eli Shlizerman, Guillaume Lajoie

Recurrent neural networks (RNNs) have been successfully applied to a variety of problems involving sequential data, but their optimization is sensitive to parameter initialization, architecture, and optimizer hyperparameters.

Advantages of biologically-inspired adaptive neural activation in RNNs during learning

no code implementations22 Jun 2020 Victor Geadah, Giancarlo Kerg, Stefan Horoi, Guy Wolf, Guillaume Lajoie

Dynamic adaptation in single-neuron response plays a fundamental role in neural coding in biological neural networks.

Transfer Learning

Internal representation dynamics and geometry in recurrent neural networks

no code implementations9 Jan 2020 Stefan Horoi, Guillaume Lajoie, Guy Wolf

The efficiency of recurrent neural networks (RNNs) in dealing with sequential data has long been established.


Modelling Working Memory using Deep Recurrent Reinforcement Learning

no code implementations NeurIPS Workshop Neuro_AI 2019 Pravish Sainath, Pierre Bellec, Guillaume Lajoie

We train these neural networks to solve the working memory task by training them with a sequence of images in supervised and reinforcement learning settings.

Decision Making reinforcement-learning +2

Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics

1 code implementation NeurIPS 2019 Giancarlo Kerg, Kyle Goyette, Maximilian Puelma Touzel, Gauthier Gidel, Eugene Vorontsov, Yoshua Bengio, Guillaume Lajoie

A recent strategy to circumvent the exploding and vanishing gradient problem in RNNs, and to allow the stable propagation of signals over long time scales, is to constrain recurrent connectivity matrices to be orthogonal or unitary.

An Investigation of Memory in Recurrent Neural Networks

no code implementations17 May 2019 Aude Forcione-Lambert, Guy Wolf, Guillaume Lajoie

We investigate the learned dynamical landscape of a recurrent neural network solving a simple task requiring the interaction of two memory mechanisms: long- and short-term.

Cannot find the paper you are looking for? You can Submit a new open access paper.