Search Results for author:
Found 63 papers, 33 papers with code
Theano: A Python framework for fast computation of mathematical expressions
Since its introduction, it has been one of the most used CPU and GPU mathematical compilers - especially in the machine learning community - and has shown steady performance improvements.
Masked Siamese Networks for Label-Efficient Learning
We propose Masked Siamese Networks (MSN), a self-supervised learning framework for learning image representations.
Self-Supervised Image Classification
Self-Supervised Learning
+1
Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
This paper demonstrates an approach for learning highly semantic image representations without relying on hand-crafted data-augmentations.
fastMRI: An Open Dataset and Benchmarks for Accelerated MRI
Accelerating Magnetic Resonance Imaging (MRI) by taking fewer measurements has the potential to reduce medical costs, minimize stress to patients and make MRI possible in applications where it is currently prohibitively slow or expensive.
High Fidelity Visualization of What Your Self-Supervised Representation Knows About
Discovering what is learned by neural networks remains a challenge.
Artificial Neural Networks Applied to Taxi Destination Prediction
We describe our first-place solution to the ECML/PKDD discovery challenge on taxi destination prediction.
PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning
Synthetic image datasets offer unmatched advantages for designing and evaluating deep neural networks: they make it possible to (i) render as many data samples as needed, (ii) precisely control each scene and yield granular ground truth labels (and captions), (iii) precisely control distribution shifts between training and testing to isolate variables of interest for sound experimentation.
Towards Democratizing Joint-Embedding Self-Supervised Learning
Joint Embedding Self-Supervised Learning (JE-SSL) has seen rapid developments in recent years, due to its promise to effectively leverage large unlabeled data.
Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis
Optimization algorithms that leverage gradient covariance information, such as variants of natural gradient descent (Amari, 1998), offer the prospect of yielding more effective descent directions.
Motif: Intrinsic Motivation from Artificial Intelligence Feedback
Exploring rich environments and evaluating one's actions without prior knowledge is immensely challenging.
Representation Learning: A Review and New Perspectives
The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data.
Understanding Dimensional Collapse in Contrastive Self-supervised Learning
It has been shown that non-contrastive methods suffer from a lesser collapse problem of a different nature: dimensional collapse, whereby the embedding vectors end up spanning a lower-dimensional subspace instead of the entire available embedding space.
Recombinator Networks: Learning Coarse-to-Fine Feature Aggregation
Deep neural networks with alternating convolutional, max-pooling and decimation layers are widely used in state of the art architectures for computer vision.
Disentanglement of Correlated Factors via Hausdorff Factorized Support
We show that the use of HFS consistently facilitates disentanglement and recovery of ground-truth factors across a variety of correlation settings and benchmarks, even under severe training correlations and correlation shifts, with in parts over $+60\%$ in relative improvement over existing disentanglement methods.
A Variational Inequality Perspective on Generative Adversarial Networks
Generative adversarial networks (GANs) form a generative modeling approach known for producing appealing samples, but they are notably difficult to train.
Auto-Encoding Dictionary Definitions into Consistent Word Embeddings
Monolingual dictionaries are widespread and semantically rich resources.
Do SSL Models Have Déjà Vu? A Case of Unintended Memorization in Self-supervised Learning
Self-supervised learning (SSL) algorithms can produce useful image representations by learning to associate different parts of natural images with one another.
A Closer Look at the Optimization Landscapes of Generative Adversarial Networks
Generative adversarial networks have been very successful in generative modeling, however they remain relatively challenging to train compared to standard deep neural networks.
Adversarial Example Games
We introduce Adversarial Example Games (AEG), a framework that models the crafting of adversarial examples as a min-max game between a generator of attacks and a classifier.
Generalized Denoising Auto-Encoders as Generative Models
Recent work has shown how denoising and contractive autoencoders implicitly capture the structure of the data-generating density, in the case where the corruption noise is Gaussian, the reconstruction error is the squared error, and the data is continuous-valued.
Randomized Value Functions via Multiplicative Normalizing Flows
In particular, we augment DQN and DDPG with multiplicative normalizing flows in order to track a rich approximate posterior distribution over the parameters of the value function.
Online Adversarial Attacks
Adversarial attacks expose important vulnerabilities of deep learning models, yet little attention has been paid to settings where data arrives as a stream.
WorldSense: A Synthetic Benchmark for Grounded Reasoning in Large Language Models
We propose WorldSense, a benchmark designed to assess the extent to which LLMs are consistently able to sustain tacit world models, by testing how they draw simple inferences from descriptions of simple arrangements of entities.
Implicit Regularization via Neural Feature Alignment
We approach the problem of implicit regularization in deep learning from a geometrical viewpoint.
Reducing Uncertainty in Undersampled MRI Reconstruction with Active Acquisition
The goal of MRI reconstruction is to restore a high fidelity image from partially observed measurements.
Stable Policy Optimization via Off-Policy Divergence Regularization
Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are among the most successful policy gradient approaches in deep reinforcement learning (RL).
The Z-loss: a shift and scale invariant classification loss belonging to the Spherical Family
In this paper, we introduce an alternative classification loss function, the Z-loss, which is designed to address these two issues.
An Exploration of Softmax Alternatives Belonging to the Spherical Loss Family
In particular, we focus our investigation on spherical bounds of the log-softmax loss and on two spherical log-likelihood losses, namely the log-Spherical Softmax suggested by Vincent et al. (2015) and the log-Taylor Softmax that we introduce.
Efficient Exact Gradient Update for training Deep Networks with Very Large Sparse Targets
An important class of problems involves training deep neural networks with sparse prediction targets of very high dimension D. These occur naturally in e. g. neural language models or the learning of word-embeddings, often posed as predicting the probability of next words among a vocabulary of size D (e. g. 200 000).
On the Identifiability of Quantized Factors
We introduce this novel form of identifiability, termed quantized factor identifiability, and provide a comprehensive proof of the recovery of the quantized factors.
Learning to Generate Samples from Noise through Infusion Training
In this work, we investigate a novel training procedure to learn a generative model as the transition operator of a Markov chain, such that, when applied repeatedly on an unstructured random noise sample, it will denoise it into a sample that matches the target distribution from the training set.
SVRG for Policy Evaluation with Fewer Gradient Evaluations
SVRG was later shown to work for policy evaluation, a problem in reinforcement learning in which one aims to estimate the value function of a given policy.
Revisiting Loss Modelling for Unstructured Pruning
By removing parameters from deep neural networks, unstructured pruning methods aim at cutting down memory footprint and computational cost, while maintaining prediction accuracy.
Improving Landmark Localization with Semi-Supervised Learning
First, we propose the framework of sequential multitasking and explore it here through an architecture for landmark localization where training with class labels acts as an auxiliary signal to guide the landmark localization on unlabeled data.
Ranked #41 on
Face Alignment
on 300W
Learning to Compute Word Embeddings On the Fly
Words in natural language follow a Zipfian distribution whereby some words are frequent but most are rare.
Ranked #48 on
Question Answering
on SQuAD1.1 dev
Convergent Tree Backup and Retrace with Function Approximation
Off-policy learning is key to scaling up reinforcement learning as it allows to learn about a target policy from the experience generated by a different behavior policy.
Parametric Adversarial Divergences are Good Losses for Generative Modeling
Parametric adversarial divergences, which are a generalization of the losses used to train generative adversarial networks (GANs), have often been described as being approximations of their nonparametric counterparts, such as the Jensen-Shannon divergence, which can be derived under the so-called optimal discriminator assumption.
A Cheap Linear Attention Mechanism with Fast Lookups and Fixed-Size Representations
These two limitations restrict the use of the softmax attention mechanism to relatively small-scale applications with short sequences and few lookups per sequence.
Exact gradient updates in time independent of output size for the spherical loss family
An important class of problems involves training deep neural networks with sparse prediction targets of very high dimension D. These occur naturally in e. g. neural language models or the learning of word-embeddings, often posed as predicting the probability of next words among a vocabulary of size D (e. g. 200, 000).
Hierarchical Memory Networks
In this paper, we explore a form of hierarchical memory network, which can be considered as a hybrid between hard and soft attention memory networks.
Dropout as data augmentation
Dropout is typically interpreted as bagging a large number of models sharing parameters.
Clustering is Efficient for Approximate Maximum Inner Product Search
Efficient Maximum Inner Product Search (MIPS) is an important task that has a wide applicability in recommendation systems and classification with a large number of classes.
EmoNets: Multimodal deep learning approaches for emotion recognition in video
The task of the emotion recognition in the wild (EmotiW) Challenge is to assign one of seven emotions to short video clips extracted from Hollywood style movies.
GSNs : Generative Stochastic Networks
We introduce a novel training principle for probabilistic models that is an alternative to maximum likelihood.
Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
We investigate the problem of modeling symbolic sequences of polyphonic music in a completely general piano-roll representation.
Ranked #5 on
Music Modeling
on JSB Chorales
Fast Approximate Natural Gradient Descent in a Kronecker Factored Eigenbasis
Optimization algorithms that leverage gradient covariance information, such as variants of natural gradient descent (Amari, 1998), offer the prospect of yielding more effective descent directions.
Stochastic Neural Network with Kronecker Flow
Recent advances in variational inference enable the modelling of highly structured joint distributions, but are limited in their capacity to scale to the high-dimensional setting of stochastic neural networks.
An Empirical Study of Batch Normalization and Group Normalization in Conditional Computation
Batch normalization has been widely used to improve optimization in deep neural networks.
Do sequence-to-sequence VAEs learn global features of sentences?
Using this method, we find that VAEs are prone to memorizing the first words and the sentence length, producing local features of limited usefulness.
Sharp Analysis of Smoothed Bellman Error Embedding
The \textit{Smoothed Bellman Error Embedding} algorithm~\citep{dai2018sbeed}, known as SBEED, was proposed as a provably convergent reinforcement learning algorithm with general nonlinear function approximation.
Stochastic Hamiltonian Gradient Methods for Smooth Games
The success of adversarial formulations in machine learning has brought renewed motivation for smooth games.
Conditional Networks
In this work we tackle the problem of out-of-distribution generalization through conditional computation.
Efficient Learning in Non-Stationary Linear Markov Decision Processes
We study episodic reinforcement learning in non-stationary linear (a. k. a.
Accounting for Variance in Machine Learning Benchmarks
Strong empirical evidence that one machine-learning algorithm A outperforms another one B ideally calls for multiple trials optimizing the learning pipeline over sources of variation such as data sampling, data augmentation, parameter initialization, and hyperparameters choices.
Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learning
This is a little vexing, as one would hope that the network layer at which invariance is explicitly enforced by the SSL criterion during training (the last projector layer) should be the one to use for best generalization performance downstream.
The Hidden Uniform Cluster Prior in Self-Supervised Learning
A successful paradigm in representation learning is to perform self-supervised pretraining using tasks based on mini-batch statistics (e. g., SimCLR, VICReg, SwAV, MSN).
ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotations
Equipped with ImageNet-X, we investigate 2, 200 current recognition models and study the types of mistakes as a function of model's (1) architecture, e. g. transformer vs. convolutional, (2) learning paradigm, e. g. supervised vs. self-supervised, and (3) training procedures, e. g., data augmentation.
Instance-Conditioned GAN Data Augmentation for Representation Learning
We showcase the benefits of DA_IC-GAN by plugging it out-of-the-box into the supervised training of ResNets and DeiT models on the ImageNet dataset, and achieving accuracy boosts up to between 1%p and 2%p with the highest capacity models.
A surprisingly simple technique to control the pretraining bias for better transfer: Expand or Narrow your representation
Self-Supervised Learning (SSL) models rely on a pretext task to learn representations.
Objectives Matter: Understanding the Impact of Self-Supervised Objectives on Vision Transformer Representations
Here, we aim to explain these differences by analyzing the impact of these objectives on the structure and transferability of the learned representations.
Stochastic positional embeddings improve masked image modeling
Masked Image Modeling (MIM) is a promising self-supervised learning approach that enables learning from unlabeled images.
Discovering environments with XRM
Successful out-of-distribution generalization requires environment annotations.
Self-Supervised Disentanglement by Leveraging Structure in Data Augmentations
Self-supervised representation learning often uses data augmentations to induce some invariance to "style" attributes of the data.
