Search Results for author:
Found 124 papers, 70 papers with code
Predicting masked tokens in stochastic locations improves masked image modeling
Specifically, we condition the model on stochastic masked token positions to guide the model toward learning features that are more robust to location uncertainties.
MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Features
Self-supervised learning of visual representations has been focusing on learning content features, which do not capture object motion or location, and focus on identifying and differentiating objects in images and videos.
Self-Supervised Learning with Lie Symmetries for Partial Differential Equations
Machine learning for differential equations paves the way for computationally efficient alternatives to numerical solvers, with potentially broad impacts in science and engineering.
Variance-Covariance Regularization Improves Representation Learning
Transfer learning has emerged as a key approach in the machine learning domain, enabling the application of knowledge derived from one domain to improve performance on subsequent tasks.
Introduction to Latent Variable Energy-Based Models: A Path Towards Autonomous Machine Intelligence
Current automated systems have crucial limitations that need to be addressed before artificial intelligence can reach human-like levels and bring new technological revolutions.
Reverse Engineering Self-Supervised Learning
Self-supervised learning (SSL) is a powerful tool in machine learning, but understanding the learned representations and their underlying mechanisms remains a challenge.
A Cookbook of Self-Supervised Learning
Self-supervised learning, dubbed the dark matter of intelligence, is a promising path to advance machine learning.
To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review
Information theory, and notably the information bottleneck principle, has been pivotal in shaping deep neural networks.
EMP-SSL: Towards Self-Supervised Learning in One Training Epoch
Recently, self-supervised learning (SSL) has achieved tremendous success in learning image representation.
Active Self-Supervised Learning: A Few Low-Cost Relationships Are All You Need
Third, it provides a proper active learning framework yielding low-cost solutions to annotate datasets, arguably bringing the gap between theory and practice of active learning that is based on simple-to-answer-by-non-experts queries of semantic relationships between inputs.
An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization
In this paper, we provide an information-theoretic perspective on Variance-Invariance-Covariance Regularization (VICReg) for self-supervised learning.
Augmented Language Models: a Survey
This survey reviews works in which language models (LMs) are augmented with reasoning skills and the ability to use tools.
Self-supervised learning of Split Invariant Equivariant representations
We hope that both our introduced dataset and approach will enable learning richer representations without supervision in more complex scenarios.
The SSL Interplay: Augmentations, Inductive Bias, and Generalization
Self-supervised learning (SSL) has emerged as a powerful framework to learn representations from raw data without supervision.
Blockwise Self-Supervised Learning at Scale
Current state-of-the-art deep networks are all powered by backpropagation.
Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
This paper demonstrates an approach for learning highly semantic image representations without relying on hand-crafted data-augmentations.
A Generalization of ViT/MLP-Mixer to Graphs
First, they capture long-range dependency and mitigate the issue of over-squashing as demonstrated on Long Range Graph Benchmark and TreeNeighbourMatch datasets.
Ranked #1 on
Graph Regression
on Peptides-struct
Joint Embedding Predictive Architectures Focus on Slow Features
Many common methods for learning a world model for pixel-based environments use generative architectures trained with pixel-level reconstruction objectives.
POLICE: Provably Optimal Linear Constraint Enforcement for Deep Neural Networks
In this paper we propose the first provable affine constraint enforcement method for DNNs that only requires minimal changes into a given DNN's forward-pass, that is computationally friendly, and that leaves the optimization of the DNN's parameter to be unconstrained, i. e. standard gradient-based method can be employed.
Unsupervised Learning of Structured Representations via Closed-Loop Transcription
This paper proposes an unsupervised method for learning a unified representation that serves both discriminative and generative purposes.
Toward Next-Generation Artificial Intelligence: Catalyzing the NeuroAI Revolution
Neuroscience has long been an essential driver of progress in artificial intelligence (AI).
VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignment
Extensive experiments on a wide range of vision- and vision-language downstream tasks demonstrate the effectiveness of VoLTA on fine-grained applications without compromising the coarse-grained downstream performance, often outperforming methods using significantly more caption and box annotations.
RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank
Joint-Embedding Self Supervised Learning (JE-SSL) has seen a rapid development, with the emergence of many method variations but only few principled guidelines that would help practitioners to successfully deploy them.
VICRegL: Self-Supervised Learning of Local Visual Features
Most recent self-supervised methods for learning image representations focus on either producing a global feature with invariance properties, or producing a set of local features.
Minimalistic Unsupervised Learning with the Sparse Manifold Transform
Though there remains a small performance gap between our simple constructive model and SOTA methods, the evidence points to this as a promising direction for achieving a principled and white-box approach to unsupervised learning.
Ranked #1 on
Unsupervised MNIST
on MNIST
Self-Supervised Learning
Sparse Representation-based Classification
+3
Joint Embedding Self-Supervised Learning in the Kernel Regime
The fundamental goal of self-supervised learning (SSL) is to produce useful representations of data without access to any labels for classifying the data.
Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representations
We empirically validate our findings where (i) we observe that SSL methods employing VCReg learn visual representations with greater pairwise independence than other methods, (i) we put in evidence which projector's characteristics favor pairwise independence, and show it to emerge independently from learning the projector, (ii) we use these findings to obtain nontrivial performance gains for VICReg, (iii) we demonstrate that the scope of VCReg goes beyond SSL by using it to solve Independent Component Analysis.
Light-weight probing of unsupervised representations for Reinforcement Learning
Unsupervised visual representation learning offers the opportunity to leverage large corpora of unlabeled trajectories to form useful visual representations, which can benefit the training of reinforcement learning (RL) algorithms.
What Do We Maximize in Self-Supervised Learning?
In this paper, we examine self-supervised learning methods, particularly VICReg, to provide an information-theoretical understanding of their construction.
TiCo: Transformation Invariance and Covariance Contrast for Self-Supervised Visual Representation Learning
Similar to other recent self-supervised learning methods, our method is based on maximizing the agreement among embeddings of different distorted versions of the same image, which pushes the encoder to produce transformation invariant representations.
Bag of Image Patch Embedding Behind the Success of Self-Supervised Learning
Even with 32x32 patch representation, BagSSL achieves 62% top-1 linear probing accuracy on ImageNet.
Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone
Vision-language (VL) pre-training has recently received considerable attention.
Ranked #1 on
Phrase Grounding
on Flickr30k Entities Dev
Masked Siamese ConvNets
Self-supervised learning has shown superior performances over supervised methods on various vision benchmarks.
On the duality between contrastive and non-contrastive self-supervised learning
Recent approaches in self-supervised learning of image representations can be categorized into different families of methods and, in particular, can be divided into contrastive and non-contrastive approaches.
Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods
Self-Supervised Learning (SSL) surmises that inputs and pairwise positive relationships are enough to learn meaningful representations.
Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors
Deep learning is increasingly moving towards a transfer learning paradigm whereby large foundation models are fine-tuned on downstream tasks, starting from an initialization learned on the source task.
The Effects of Regularization and Data Augmentation are Class Dependent
The optimal amount of DA or weight decay found from cross-validation leads to disastrous model performances on some classes e. g. on Imagenet with a resnet50, the "barn spider" classification test accuracy falls from $68\%$ to $46\%$ only by introducing random crop DA during training.
projUNN: efficient method for training deep networks with unitary matrices
In learning with recurrent or very deep feed-forward networks, employing unitary matrices in each layer can be very effective at maintaining long-range stability.
A Data-Augmentation Is Worth A Thousand Samples: Exact Quantification From Analytical Augmented Sample Moments
We show that for a training loss to be stable under DA sampling, the model's saliency map (gradient of the loss with respect to the model's input) must align with the smallest eigenvector of the sample variance under the considered DA augmentation, hinting at a possible explanation on why models tend to shift their focus from edges to textures.
Neural Manifold Clustering and Embedding
We argue that achieving manifold clustering with neural networks requires two essential ingredients: a domain-specific constraint that ensures the identification of the manifolds, and a learning algorithm for embedding each manifold to a linear subspace in the feature space.
Sparse Coding with Multi-Layer Decoders using Variance Regularization
Sparse coding with an $l_1$ penalty and a learned linear dictionary requires regularization of the dictionary to prevent a collapse in the $l_1$ norms of the codes.
Understanding Dimensional Collapse in Contrastive Self-supervised Learning
It has been shown that non-contrastive methods suffer from a lesser collapse problem of a different nature: dimensional collapse, whereby the embedding vectors end up spanning a lower-dimensional subspace instead of the entire available embedding space.
Learning in High Dimension Always Amounts to Extrapolation
The notion of interpolation and extrapolation is fundamental in various fields from deep learning to function approximation.
Decoupled Contrastive Learning
Further, DCL can be combined with the SOTA contrastive learning method, NNCLR, to achieve 72. 3% ImageNet-1K top-1 accuracy with 512 batch size in 400 epochs, which represents a new SOTA in contrastive learning.
Compact and Optimal Deep Learning with Recurrent Parameter Generators
We propose a drastically different approach to compact and optimal deep learning: We decouple the Degrees of freedom (DoF) and the actual number of parameters of a model, optimize a small DoF with predefined random linear constraints for a large model of arbitrary architecture, in one-stage end-to-end learning.
Ranked #96 on
Image Classification
on ObjectNet
(using extra training data)
VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning
Recent self-supervised methods for image representation learning are based on maximizing the agreement between embedding vectors from different views of the same image.
Representation Learning
Self-Supervised Image Classification
+2
MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding
We also investigate the utility of our model as an object detector on a given label set when fine-tuned in a few-shot setting.
Ranked #1 on
Visual Question Answering (VQA)
on CLEVR-Humans
Generalized Referring Expression Comprehension
Phrase Grounding
+9
Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors
Transformer networks have revolutionized NLP representation learning since they were introduced.
Barlow Twins: Self-Supervised Learning via Redundancy Reduction
This causes the embedding vectors of distorted versions of a sample to be similar, while minimizing the redundancy between the components of these vectors.
Ranked #11 on
Image Classification
on Places205
MDETR - Modulated Detection for End-to-End Multi-Modal Understanding
We also investigate the utility of our model as an object detector on a given label set when fine-tuned in a few-shot setting.
Ranked #2 on
Referring Expression Comprehension
on Talk2Car
(using extra training data)
Neural Potts Model
We propose the Neural Potts Model objective as an amortized optimization problem.
Implicit Rank-Minimizing Autoencoder
An important component of autoencoders is the method by which the information capacity of the latent representation is minimized or limited.
Inspirational Adversarial Image Generation
We design a simple optimization method to find the optimal latent parameters corresponding to the closest generation to any input inspirational image.
The role of over-parametrization in generalization of neural networks
Despite existing work on ensuring generalization of neural networks in terms of scale sensitive complexity measures, such as norms, margin and sharpness, these complexity measures do not offer an explanation of why neural networks generalize better with over-parametrization.
Unsupervised Image Matching and Object Discovery as Optimization
Learning with complete or partial supervision is powerful but relies on ever-growing human annotation efforts.
Ranked #2 on
Single-object colocalization
on Object Discovery
Learning about an exponential amount of conditional distributions
We introduce the Neural Conditioner (NC), a self-supervised machine able to learn about all the conditional distributions of a random vector $X$.
Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic
Learning a policy using only observational data is challenging because the distribution of states it induces at execution time may differ from the distribution observed during training.
A Spectral Regularizer for Unsupervised Disentanglement
A generative model with a disentangled representation allows for independent control over different aspects of the output.
GLoMo: Unsupervised Learning of Transferable Relational Graphs
We also show that the learned graphs are generic enough to be transferred to different embeddings on which the graphs have not been trained (including GloVe embeddings, ELMo embeddings, and task-specific RNN hidden units), or embedding-free units such as image pixels.
Adversarially-Trained Normalized Noisy-Feature Auto-Encoder for Text Generation
An ATNNFAE consists of an auto-encoder where the internal code is normalized on the unit sphere and corrupted by additive noise.
Learning with Reflective Likelihoods
Furthermore, the reflective likelihood objective prevents posterior collapse when used to train stochastic auto-encoders with amortized inference.
Comparing Dynamics: Deep Neural Networks versus Glassy Systems
We analyze numerically the training dynamics of deep neural networks (DNN) by using methods developed in statistical physics of glassy systems.
GLoMo: Unsupervisedly Learned Relational Graphs as Transferable Representations
We also show that the learned graphs are generic enough to be transferred to different embeddings on which the graphs have not been trained (including GloVe embeddings, ELMo embeddings, and task-specific RNN hidden unit), or embedding-free units such as image pixels.
Backpropagation for Implicit Spectral Densities
We introduce a tool that allows us to do this even when the likelihood is not explicitly set, by instead using the implicit likelihood of the model.
Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
Despite existing work on ensuring generalization of neural networks in terms of scale sensitive complexity measures, such as norms, margin and sharpness, these complexity measures do not offer an explanation of why neural networks generalize better with over-parametrization.
DeSIGN: Design Inspiration from Generative Networks
Can an algorithm create original and compelling fashion designs to serve as an inspirational assistant?
Predicting Future Instance Segmentation by Forecasting Convolutional Features
We apply the "detection head'" of Mask R-CNN on the predicted features to produce the instance segmentation of future frames.
Byte-Level Recursive Convolutional Auto-Encoder for Text
The proposed model is a multi-stage deep convolutional encoder-decoder framework using residual connections, containing up to 160 parameterized layers.
Prediction Under Uncertainty with Error Encoding Networks
In this work we introduce a new framework for performing temporal predictions in the presence of uncertainty.
A Closer Look at Spatiotemporal Convolutions for Action Recognition
In this paper we discuss several forms of spatiotemporal convolutions for video analysis and study their effects on action recognition.
Ranked #3 on
Action Recognition
on Sports-1M
Prediction Under Uncertainty with Error-Encoding Networks
In this work we introduce a new framework for performing temporal predictions in the presence of uncertainty.
A hierarchical loss and its problems when classifying non-hierarchically
We define a metric that, inter alia, can penalize failure to distinguish between a sheepdog and a skyscraper more than failure to distinguish between a sheepdog and a poodle.
Which Encoding is the Best for Text Classification in Chinese, English, Japanese and Korean?
This article offers an empirical study on the different ways of encoding Chinese, Japanese, Korean (CJK) and English languages for text classification.
Adversarially Regularized Autoencoders
This adversarially regularized autoencoder (ARAE) allows us to generate natural textual outputs as well as perform manipulations in the latent space to induce change in the output space.
Model-Based Planning with Discrete and Continuous Actions
Action planning using learned and differentiable forward models of the world is a general approach which has a number of desirable properties, including improved sample complexity over model-free RL methods, reuse of learned models across different tasks, and the ability to perform efficient gradient-based optimization in continuous action spaces.
Predicting Deeper into the Future of Semantic Segmentation
The ability to predict and therefore to anticipate the future is an important attribute of intelligence.
Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
Using unitary (instead of general) matrices in artificial neural networks (ANNs) is a promising way to solve the gradient explosion/vanishing problem, as well as to enable ANNs to learn long-term correlations in the data.
Tracking the World State with Recurrent Entity Networks
The EntNet sets a new state-of-the-art on the bAbI tasks, and is the first method to solve all the tasks in the 10k training examples setting.
Ranked #3 on
Question Answering
on bAbi
Disentangling factors of variation in deep representation using adversarial training
The only available source of supervision during the training process comes from our ability to distinguish among different observations belonging to the same category.
Geometric deep learning: going beyond Euclidean data
In many applications, such geometric data are large and complex (in the case of social networks, on the scale of billions), and are natural targets for machine learning techniques.
Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
We look at the eigenvalues of the Hessian of a loss function before and after training.
Disentangling factors of variation in deep representations using adversarial training
During training, the only available source of supervision comes from our ability to distinguish among different observations belonging to the same class.
Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
This paper proposes a new optimization algorithm called Entropy-SGD for training deep neural networks that is motivated by the local geometry of the energy landscape.
Energy-based Generative Adversarial Network
We introduce the "Energy-based Generative Adversarial Network" model (EBGAN) which views the discriminator as an energy function that attributes low energies to the regions near the data manifold and higher energies to other regions.
Very Deep Convolutional Networks for Text Classification
The dominant approach for many NLP tasks are recurrent neural networks, in particular LSTMs, and convolutional neural networks.
Ranked #17 on
Text Classification
on AG News
What is the Best Feature Learning Procedure in Hierarchical Recognition Architectures?
We also introduce a new single phase supervised learning procedure that places an L1 penalty on the output state of each layer of the network.
Recurrent Orthogonal Networks and Long-Memory Tasks
Although RNNs have been shown to be powerful tools for processing sequential data, finding architectures or optimization strategies that allow them to model very long term dependencies is still an active area of research.
Universal halting times in optimization and machine learning
Given an algorithm, which we take to be both the optimization routine and the form of the random landscape, the fluctuations of the halting time follow a distribution that, after centering and scaling, remains unchanged even when the distribution on the landscape is changed.
Super-Resolution with Deep Convolutional Sufficient Statistics
Inverse problems in image and audio, and super-resolution in particular, can be seen as high-dimensional structured prediction problems, where the goal is to characterize the conditional distribution of a high-resolution output given its low-resolution corrupted observation.
Deep multi-scale video prediction beyond mean square error
Learning to predict future images from a video sequence involves the construction of an internal representation that models the image evolution accurately, and therefore, to some degree, its content and dynamics.
Binary embeddings with structured hashed projections
We prove several theoretical results showing that projections via various structured matrices followed by nonlinear mappings accurately preserve the angular distance between input high-dimensional vectors.
Universum Prescription: Regularization using Unlabeled Data
This paper shows that simply prescribing "none of the above" labels to unlabeled data has a beneficial regularization effect to supervised learning.
Ranked #160 on
Image Classification
on CIFAR-10
Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches
We approach the problem by learning a similarity measure on small image patches using a convolutional neural network.
Very Deep Multilingual Convolutional Neural Networks for LVCSR
However, CNNs in LVCSR have not kept pace with recent advances in other domains where deeper neural networks provide superior performance.
Ranked #17 on
Speech Recognition
on Switchboard + Hub500
Character-level Convolutional Networks for Text Classification
This article offers an empirical exploration on the use of character-level convolutional networks (ConvNets) for text classification.
Ranked #16 on
Sentiment Analysis
on Yelp Fine-grained classification
Deep Convolutional Networks on Graph-Structured Data
Deep Learning's recent successes have mostly relied on Convolutional Networks, which exploit fundamental statistical properties of images, sounds and video data: the local stationarity and multi-scale compositional structure, that allows expressing long range interactions in terms of shorter, localized interactions.
Learning to Linearize Under Uncertainty
Training deep feature hierarchies to solve supervised learning tasks has achieved state of the art performance on many problems in computer vision.
Stacked What-Where Auto-encoders
The objective function includes reconstruction terms that induce the hidden states in the Deconvnet to be similar to those of the Convnet.
Unsupervised Feature Learning from Temporal Data
Current state-of-the-art classification and detection algorithms rely on supervised training.
A mathematical motivation for complex-valued convolutional networks
Courtesy of the exact correspondence, the remarkably rich and rigorous body of mathematical analysis for wavelets applies directly to (complex-valued) convnets.
Text Understanding from Scratch
This article demontrates that we can apply deep learning to text understanding from character-level inputs all the way up to abstract text concepts, using temporal convolutional networks (ConvNets).
Fast Convolutional Nets With fbfft: A GPU Performance Evaluation
We examine the performance profile of Convolutional Neural Network training on the current generation of NVIDIA Graphics Processing Units.
Audio Source Separation with Discriminative Scattering Networks
In this report we describe an ongoing line of research for solving single-channel source separation problems.
Deep learning with Elastic Averaging SGD
We empirically demonstrate that in the deep learning setting, due to the existence of many local optima, allowing more exploration can lead to the improved performance.
Explorations on high dimensional landscapes
Finding minima of a real valued non-convex function over a high dimensional space is a major challenge in science.
Unsupervised Learning of Spatiotemporally Coherent Metrics
Current state-of-the-art classification and detection algorithms rely on supervised training.
The Loss Surfaces of Multilayer Networks
We show that for large-size decoupled networks the lowest critical values of the random loss function form a layered structure and they are located in a well-defined band lower-bounded by the global minimum.
Efficient Object Localization Using Convolutional Networks
Recent state-of-the-art performance on human-body pose estimation has been achieved with Deep Convolutional Networks (ConvNets).
Ranked #42 on
Pose Estimation
on MPII Human Pose
Differentially- and non-differentially-private random decision trees
We consider supervised learning with random decision trees, where the tree construction is completely random.
MoDeep: A Deep Learning Framework Using Motion Features for Human Pose Estimation
In this work, we propose a novel and efficient method for articulated human pose estimation in videos using a convolutional network architecture, which incorporates both color and motion features.
Computing the Stereo Matching Cost with a Convolutional Neural Network
We present a method for extracting depth information from a rectified image pair.
Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation
This paper proposes a new hybrid architecture that consists of a deep Convolutional Network and a Markov Random Field.
Fast Approximation of Rotations and Hessians matrices
A new method to represent and approximate rotation matrices is introduced.
Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
We present techniques for speeding up the test-time evaluation of large convolutional networks, designed for object recognition tasks.
OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
This integrated framework is the winner of the localization task of the ImageNet Large Scale Visual Recognition Challenge 2013 (ILSVRC2013) and obtained very competitive results for the detection and classifications tasks.
Spectral Networks and Locally Connected Networks on Graphs
Convolutional Neural Networks are extremely efficient architectures in image and audio recognition tasks, thanks to their ability to exploit the local translational invariance of signal classes over their domain.
Fast Training of Convolutional Networks through FFTs
Convolutional networks are one of the most widely employed architectures in computer vision and machine learning.
Understanding Deep Architectures using a Recursive Convolutional Network
A key challenge in designing convolutional network models is sizing them appropriately.
Signal Recovery from Pooling Representations
In this work we compute lower Lipschitz bounds of $\ell_p$ pooling operators for $p=1, 2, \infty$ as well as $\ell_p$ pooling operators preceded by half-rectification layers.
Regularization of Neural Networks using DropConnect
When training with Dropout, a randomly selected subset of activations are set to zero within each layer.
Ranked #6 on
Image Classification
on MNIST
Adaptive learning rates and parallelization for stochastic, sparse, non-smooth gradients
Recent work has established an empirically successful framework for adapting learning rates for stochastic gradient descent (SGD).
Pedestrian Detection with Unsupervised Multi-Stage Feature Learning
Pedestrian detection is a problem of considerable practical interest.
No More Pesky Learning Rates
The performance of stochastic gradient descent (SGD) depends critically on how learning rates are tuned and decreased over time.
Convolutional Neural Networks Applied to House Numbers Digit Classification
We classify digits of real-world house numbers using convolutional neural networks (ConvNets).
