Search Results for author:
Found 49 papers, 27 papers with code
Proposal-based Video Completion
Video inpainting is an important technique for a wide variety of applications from video content editing to video restoration.
VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning
In this work, we show that a strong off-the-shelf frozen pretrained visual encoder, along with a well designed prediction model, can achieve state-of-the-art (SoTA) performance in forecasting and procedural planning without the need for pretraining the prediction model, nor requiring additional supervision from language or ASR.
Modeling Caption Diversity in Contrastive Vision-Language Pretraining
Contrastive Language Pretraining (CLIP) on the other hand, works by mapping an image and its caption to a single vector -- limiting how well CLIP-like models can represent the diverse ways to describe an image.
Learning and Leveraging World Models in Visual Representation Learning
Joint-Embedding Predictive Architecture (JEPA) has emerged as a promising self-supervised approach that learns by leveraging a world model.
Revisiting Feature Prediction for Learning Visual Representations from Video
This paper explores feature prediction as a stand-alone objective for unsupervised learning from video and introduces V-JEPA, a collection of vision models trained solely using a feature prediction objective, without the use of pretrained image encoders, text, negative examples, reconstruction, or other sources of supervision.
Jack of All Tasks Master of Many: Designing General-Purpose Coarse-to-Fine Vision-Language Model
In this work we introduce VistaLLM a powerful visual system that addresses coarse- and fine grained VL tasks over single and multiple input images using a unified framework.
Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model
In this work, we introduce VistaLLM, a powerful visual system that addresses coarse- and fine-grained VL tasks over single and multiple input images using a unified framework.
Discovering environments with XRM
Environment annotations are essential for the success of many out-of-distribution (OOD) generalization methods.
Stochastic positional embeddings improve masked image modeling
Masked Image Modeling (MIM) is a promising self-supervised learning approach that enables learning from unlabeled images.
DINOv2: Learning Robust Visual Features without Supervision
The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision.
Ranked #1 on
Image Retrieval
on AmsterTime
(using extra training data)
A surprisingly simple technique to control the pretraining bias for better transfer: Expand or Narrow your representation
Self-Supervised Learning (SSL) models rely on a pretext task to learn representations.
A Simple Recipe for Competitive Low-compute Self supervised Vision Models
Our main insight is that existing joint-embedding based SSL methods can be repurposed for knowledge distillation from a large self-supervised teacher to a small student model.
Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
This paper demonstrates an approach for learning highly semantic image representations without relying on hand-crafted data-augmentations.
Uniform Masking Prevails in Vision-Language Pretraining
Masked Language Modeling (MLM) has proven to be an essential component of Vision-Language (VL) pretraining.
ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotations
Equipped with ImageNet-X, we investigate 2, 200 current recognition models and study the types of mistakes as a function of model's (1) architecture, e. g. transformer vs. convolutional, (2) learning paradigm, e. g. supervised vs. self-supervised, and (3) training procedures, e. g., data augmentation.
Neural Attentive Circuits
Recent work has seen the development of general purpose neural architectures that can be trained to perform tasks across diverse data modalities.
The Hidden Uniform Cluster Prior in Self-Supervised Learning
A successful paradigm in representation learning is to perform self-supervised pretraining using tasks based on mini-batch statistics (e. g., SimCLR, VICReg, SwAV, MSN).
Cascaded Video Generation for Videos In-the-Wild
Videos can be created by first outlining a global view of the scene and then adding local details.
Masked Siamese Networks for Label-Efficient Learning
We propose Masked Siamese Networks (MSN), a self-supervised learning framework for learning image representations.
Self-Supervised Image Classification
Self-Supervised Learning
+1
BARACK: Partially Supervised Group Robustness With Guarantees
Theoretically, we provide generalization bounds for our approach in terms of the worst-group performance, which scale with respect to both the total number of training points and the number of training points with group labels.
Trade-offs of Local SGD at Scale: An Empirical Study
As datasets and models become increasingly large, distributed training has become a necessary component to allow deep neural networks to train in reasonable amounts of time.
INFERNO: Inferring Object-Centric 3D Scene Representations without Supervision
Each object representation defines a localized neural radiance field that is used to generate 2D views of the scene through a differentiable rendering process.
Ranked #6 on
Video Object Tracking
on CATER
Hierarchical Video Generation for Complex Data
Inspired by this we propose a hierarchical model for video generation which follows a coarse to fine approach.
Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples
This paper proposes a novel method of learning by predicting view assignments with support samples (PAWS).
SSW-GAN: Scalable Stage-wise Training of Video GANs
Current state-of-the-art generative models for videos have high computational requirements that impede high resolution generations beyond a few frames.
A Closer Look at Codistillation for Distributed Training
Surprisingly, we find that even at moderate batch sizes, models trained with codistillation can perform as well as models trained with synchronous data-parallel methods, despite using a much weaker synchronization mechanism.
Revisiting Loss Modelling for Unstructured Pruning
By removing parameters from deep neural networks, unstructured pruning methods aim at cutting down memory footprint and computational cost, while maintaining prediction accuracy.
Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations
We investigate a strategy for improving the efficiency of contrastive learning of visual representations by leveraging a small amount of supervised information during pre-training.
SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
We provide theoretical convergence guarantees showing that SlowMo converges to a stationary point of smooth non-convex losses.
Needles in Haystacks: On Classifying Tiny Objects in Large Images
In some important computer vision domains, such as medical or hyperspectral imaging, we care about the classification of tiny objects in large images.
Gossip-based Actor-Learner Architectures for Deep Reinforcement Learning
We show that we can run several loosely coupled GALA agents in parallel on a single GPU and achieve significantly higher hardware utilization and frame-rates than vanilla A2C at comparable power draws.
Improved Conditional VRNNs for Video Prediction
To address this issue, we propose to increase the expressiveness of the latent distributions and to use higher capacity likelihood models.
Ranked #2 on
Video Prediction
on Cityscapes 128x128
Fast Approximate Natural Gradient Descent in a Kronecker Factored Eigenbasis
Optimization algorithms that leverage gradient covariance information, such as variants of natural gradient descent (Amari, 1998), offer the prospect of yielding more effective descent directions.
Stochastic Gradient Push for Distributed Deep Learning
Distributed data-parallel algorithms aim to accelerate the training of deep neural networks by parallelizing the computation of large mini-batch gradient updates across multiple nodes.
On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length
When studying the SGD dynamics in relation to the sharpest directions in this initial phase, we find that the SGD step is large compared to the curvature and commonly fails to minimize the loss along the sharpest directions.
A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
The risks and perils of overfitting in machine learning are well known.
Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis
Optimization algorithms that leverage gradient covariance information, such as variants of natural gradient descent (Amari, 1998), offer the prospect of yielding more effective descent directions.
Three Factors Influencing Minima in SGD
In particular we find that the ratio of learning rate to batch size is a key determinant of SGD dynamics and of the width of the final minima, and that higher values of the ratio lead to wider minima and often better generalization.
Residual Connections Encourage Iterative Inference
In general, a Resnet block tends to concentrate representation learning behavior in the first few layers while higher layers perform iterative refinement of features.
A Closer Look at Memorization in Deep Networks
We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness.
A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering
In addition to presenting statistics and a description of the dataset, we perform a detailed analysis of 5 different models' predictions, and compare these with human performance.
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
We propose zoneout, a novel method for regularizing RNNs.
Theano: A Python framework for fast computation of mathematical expressions
Since its introduction, it has been one of the most used CPU and GPU mathematical compilers - especially in the machine learning community - and has shown steady performance improvements.
Recurrent Batch Normalization
We propose a reparameterization of LSTM that brings the benefits of batch normalization to recurrent neural networks.
Ranked #20 on
Language Modelling
on Text8
Dynamic Capacity Networks
The low-capacity sub-networks are applied across most of the input, but also provide a guide to select a few portions of the input on which to apply the high-capacity sub-networks.
Delving Deeper into Convolutional Networks for Learning Video Representations
We propose an approach to learn spatio-temporal features in videos from intermediate visual representations we call "percepts" using Gated-Recurrent-Unit Recurrent Networks (GRUs). Our method relies on percepts that are extracted from all level of a deep convolutional network trained on the large ImageNet dataset.
Oracle performance for visual captioning
The task of associating images and videos with a natural language description has attracted a great amount of attention recently.
Describing Videos by Exploiting Temporal Structure
In this context, we propose an approach that successfully takes into account both the local and global temporal structure of videos to produce descriptions.
FitNets: Hints for Thin Deep Nets
In this paper, we extend this idea to allow the training of a student that is deeper and thinner than the teacher, using not only the outputs but also the intermediate representations learned by the teacher as hints to improve the training process and final performance of the student.
