Search Results for author:
Found 204 papers, 83 papers with code
Neptune: The Long Orbit to Benchmarking Long Video Understanding
This paper describes a semi-automatic pipeline to generate challenging question-answer-decoy sets for understanding long videos.
Ranked #1 on
Multiple-choice
on Neptune-Full
Visual Lexicon: Rich Image Features in Language Space
We present Visual Lexicon, a novel visual language that encodes rich image information into the text space of vocabulary tokens while retaining intricate visual details that are often challenging to convey in natural language.
Language-Guided Image Tokenization for Generation
Compared to the conventional tokenizer without text conditioning, TexTok achieves average reconstruction FID improvements of 29. 2% and 48. 1% on ImageNet-256 and -512 benchmarks respectively, across varying numbers of tokens.
Grounded Video Caption Generation
We apply this approach to videos from the HowTo100M dataset, which results in a new large-scale training dataset, called HowToGround, with automatically annotated captions and spatio-temporally consistent bounding boxes with coherent natural language labels.
Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach
Web-scale visual entity recognition, the task of associating images with their corresponding entities within vast knowledge bases like Wikipedia, presents significant challenges due to the lack of clean, large-scale training data.
Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policy
3D-LOTUS++ achieves state-of-the-art performance on novel tasks of GemBench, setting a new standard for generalization in robotic manipulation.
Ranked #1 on
Robot Manipulation Generalization
on GEMBench
Towards Zero-Shot Multimodal Machine Translation
Current multimodal machine translation (MMT) systems rely on fully supervised data (i. e models are trained on sentences with their translations and accompanying images).
DataDream: Few-shot Guided Dataset Generation
While text-to-image diffusion models have been shown to achieve state-of-the-art results in image synthesis, they have yet to prove their effectiveness in downstream applications.
mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus
We additionally train two types of multilingual model to prove the benefits of mOSCAR: (1) a model trained on a subset of mOSCAR and captioning data and (2) a model train on captioning data only.
Smoke and Mirrors in Causal Downstream Tasks
Machine Learning and AI have the potential to transform data-driven scientific discovery, enabling accurate predictions for several scientific phenomena.
Learning text-to-video retrieval from image captioning
In this paper, we make use of this progress and instantiate the image experts from two types of models: a text-to-image retrieval model to provide an initial backbone, and image captioning models to provide supervision signal into unlabeled videos.
ViViDex: Learning Vision-based Dexterous Manipulation from Human Videos
To address these limitations, we propose a new framework ViViDex to improve vision-based policy learning from human videos.
MoReVQA: Exploring Modular Reasoning Models for Video Question Answering
This paper addresses the task of video question answering (videoQA) via a decomposed multi-stage, modular reasoning framework.
Ranked #5 on
Multiple-choice
on Neptune-Full
Learning Correlation Structures for Vision Transformers
We introduce a new attention mechanism, dubbed structural self-attention (StructSA), that leverages rich correlation patterns naturally emerging in key-query interactions of attention.
Ranked #5 on
Action Recognition
on Diving-48
Streaming Dense Video Captioning
An ideal model for dense video captioning -- predicting captions localized temporally in a video -- should be able to handle long input videos, predict rich, detailed textual descriptions, and be able to produce outputs before processing the entire video.
SUGAR: Pre-training 3D Visual Representations for Robotics
SUGAR employs a versatile transformer-based model to jointly address five pre-training tasks, namely cross-modal knowledge distillation for semantic learning, masked point modeling to understand geometry structures, grasping pose synthesis for object affordance, 3D instance segmentation and referring expression grounding to analyze cluttered scenes.
A Generative Approach for Wikipedia-Scale Visual Entity Recognition
In this paper, we address web-scale visual entity recognition, specifically the task of mapping a given query image to one of the 6 million existing entities in Wikipedia.
SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code
SceneCraft first models a scene graph as a blueprint, detailing the spatial relationships among assets in the scene.
Time-, Memory- and Parameter-Efficient Visual Adaptation
Here, we outperform a prior adaptor-based method which could only scale to a 1 billion parameter backbone, or fully-finetuning a smaller backbone, with the same GPU and less training time.
RAVEN: Rethinking Adversarial Video Generation with Efficient Tri-plane Networks
To capture long spatio-temporal dependencies, our approach incorporates a hybrid explicit-implicit tri-plane representation inspired by 3D-aware generative frameworks developed for three-dimensional object representation and employs a single latent code to model an entire video clip.
Pixel-Aligned Language Model
When taking locations as inputs the model performs location-conditioned captioning which generates captions for the indicated object or region.
Time- Memory- and Parameter-Efficient Visual Adaptation
Here we outperform a prior adaptor-based method which could only scale to a 1 billion parameter backbone or fully-finetuning a smaller backbone with the same GPU and less training time.
Pixel Aligned Language Models
When taking locations as inputs, the model performs location-conditioned captioning, which generates captions for the indicated object or region.
Dense Optical Tracking: Connecting the Dots
Code, data, and videos showcasing the capabilities of our approach are available in the project webpage: https://16lemoing. github. io/dot .
PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation
The ability for robots to comprehend and execute manipulation tasks based on natural language instructions is a long-term goal in robotics.
Ranked #5 on
Robot Manipulation Generalization
on GEMBench
VidChapters-7M: Video Chapters at Scale
To address this issue, we present VidChapters-7M, a dataset of 817K user-chaptered videos including 7M chapters in total.
CoVR-2: Automatic Data Construction for Composed Video Retrieval
Composed Image Retrieval (CoIR) has recently gained popularity as a task that considers both text and image queries together, to search for relevant images in a database.
Ranked #1 on
Composed Video Retrieval (CoVR)
on WebVid-CoVR
POCO: 3D Pose and Shape Estimation with Confidence
To address this, we develop POCO, a novel framework for training HPS regressors to estimate not only a 3D human body, but also their confidence, in a single feed-forward pass.
UnLoc: A Unified Framework for Video Localization Tasks
While large-scale image-text pretrained models such as CLIP have been used for multiple video-level tasks on trimmed videos, their use for temporal localization in untrimmed videos is still a relatively unexplored task.
Ranked #1 on
Action Segmentation
on COIN
Robust Visual Sim-to-Real Transfer for Robotic Manipulation
While previous work mainly evaluates DR for disembodied tasks, such as pose estimation and object detection, here we systematically explore visual domain randomization methods and benchmark them on a rich set of challenging robotic manipulation tasks.
Does Visual Pretraining Help End-to-End Reasoning?
A positive result would refute the common belief that explicit visual abstraction (e. g. object detection) is essential for compositional generalization on visual reasoning, and confirm the feasibility of a neural network "generalist" to solve visual recognition and reasoning tasks.
How can objects help action recognition?
In this paper, we investigate how we can use knowledge of objects to design better video models, namely to process fewer tokens and to improve recognition accuracy.
Waffling around for Performance: Visual Classification with Random Words and Broad Concepts
The visual classification performance of vision-language models such as CLIP has been shown to benefit from additional semantic knowledge from large language models (LLMs) such as GPT-3.
Retrieval-Enhanced Contrastive Vision-Text Models
Contrastive image-text models such as CLIP form the building blocks of many state-of-the-art systems.
Ranked #3 on
Fine-Grained Image Recognition
on OVEN
Modular Visual Question Answering via Code Generation
We present a framework that formulates visual question answering as modular code generation.
Learning Video-Conditioned Policies for Unseen Manipulation Tasks
To encourage generalization to new tasks, we avoid particular tasks during training and learn our policy from unlabelled robot trajectories and corresponding robot videos.
End-to-End Spatio-Temporal Action Localisation with Video Transformers
The most performant spatio-temporal action localisation models use external person proposals and complex external memory banks.
Ranked #1 on
Action Recognition
on AVA v2.1
(using extra training data)
gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstruction
In particular, we address reconstruction of hands and manipulated objects from monocular RGB images.
Ranked #5 on
hand-object pose
on DexYCB
Verbs in Action: Improving verb understanding in video-language models
Understanding verbs is crucial to modelling how people and objects interact with each other and the environment through space and time.
Ranked #21 on
Zero-Shot Video Question Answer
on NExT-QA
Improving Image Recognition by Retrieving from Web-Scale Image-Text Data
Retrieval augmented models are becoming increasingly popular for computer vision tasks after their recent success in NLP problems.
Ranked #1 on
Image Classification
on WebVision-1000
(using extra training data)
Exposing and Mitigating Spurious Correlations for Cross-Modal Retrieval
In this work, we introduce ODmAP@k, an object decorrelation metric that measures a model's robustness to spurious correlations in the training data.
Bridging the Gap between Model Explanations in Partially Annotated Multi-label Classification
Based on these findings, we propose to boost the attribution scores of the model trained with partial labels to make its explanation resemble that of the model trained with full labels.
AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR
(ii) We also introduce a simple curriculum scheme during training which we show is crucial to enable the model to jointly process audio and visual information effectively; and finally (iii) we show that our model achieves state of the art zero-shot results on three different AV-ASR benchmarks (How2, VisSpeech and Ego4D), while also crucially preserving decent performance on traditional audio-only speech recognition benchmarks (LibriSpeech).
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
In this work, we introduce Vid2Seq, a multi-modal single-stage dense event captioning model pretrained on narrated videos which are readily-available at scale.
Ranked #1 on
Dense Video Captioning
on ActivityNet Captions
(using extra training data)
Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation
One of the major challenges of machine translation (MT) is ambiguity, which can in some cases be resolved by accompanying context such as images.
REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory
REVEAL consists of four key components: the memory, the encoder, the retriever and the generator.
Ranked #9 on
Visual Question Answering (VQA)
on OK-VQA
Audiovisual Masked Autoencoders
Can we leverage the audiovisual information already present in video to improve self-supervised representation learning?
Ranked #1 on
Audio Classification
on EPIC-KITCHENS-100
(using extra training data)
Location-Aware Self-Supervised Transformers for Semantic Segmentation
Pixel-level labels are particularly expensive to acquire.
WALDO: Future Video Synthesis using Object Layer Decomposition and Parametric Flow Prediction
This paper presents WALDO (WArping Layer-Decomposed Objects), a novel approach to the prediction of future video frames from past ones.
AVATAR submission to the Ego4D AV Transcription Challenge
In this report, we describe our submission to the Ego4D AudioVisual (AV) Speech Transcription Challenge 2022.
Language Conditioned Spatial Relation Reasoning for 3D Object Grounding
In this work we propose a language-conditioned transformer model for grounding 3D objects and their spatial relations.
Learning Reward Functions for Robotic Manipulation by Observing Humans
Unlike prior work on leveraging human videos to teach robots, our method, Human Offline Learned Distances (HOLD) requires neither a priori data from the robot environment, nor a set of task-specific human demonstrations, nor a predefined notion of correspondence across morphologies, yet it is able to accelerate training of several manipulation tasks on a simulated robot arm compared to using only a sparse reward obtained from task completion.
A Memory Transformer Network for Incremental Learning
We study class-incremental learning, a training setup in which new classes of data are observed over time for the model to learn from.
Enforcing the consensus between Trajectory Optimization and Policy Learning for precise robot control
Reinforcement learning (RL) and trajectory optimization (TO) present strong complementary advantages.
Instruction-driven history-aware policies for robotic manipulations
In human environments, robots are expected to accomplish a variety of manipulation tasks given simple natural language instructions.
Ranked #2 on
Robot Manipulation
on RLBench
(Succ. Rate (10 tasks, 100 demos/task) metric)
Learning from Unlabeled 3D Environments for Vision-and-Language Navigation
Our resulting HM3D-AutoVLN dataset is an order of magnitude larger than existing VLN datasets in terms of navigation environments and instructions.
Ranked #1 on
Visual Navigation
on SOON Test
TL;DW? Summarizing Instructional Videos with Task Relevance & Cross-Modal Saliency
In this work, we focus on summarizing instructional videos, an under-explored area of video summarization.
AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction
We show that such aligned SDFs better focus on reconstructing shape details and improve reconstruction accuracy both for hands and objects.
Ranked #9 on
hand-object pose
on DexYCB
Beyond Transfer Learning: Co-finetuning for Action Localisation
Transfer learning is the predominant paradigm for training deep networks on small target datasets.
M&M Mix: A Multimodal Multiview Transformer Ensemble
This report describes the approach behind our winning solution to the 2022 Epic-Kitchens Action Recognition Challenge.
Ranked #2 on
Action Recognition
on EPIC-KITCHENS-100
(using extra training data)
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models
Manual annotation of question and answers for videos, however, is tedious and prohibits scalability.
Ranked #1 on
Zero-Shot Video Question Answer
on TVQA
AVATAR: Unconstrained Audiovisual Speech Recognition
Audio-visual automatic speech recognition (AV-ASR) is an extension of ASR that incorporates visual cues, often from the movements of a speaker's mouth.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Learning to Answer Visual Questions from Web Videos
We use our method to generate the WebVidVQA3M dataset from the WebVid dataset, i. e., videos with alt-text annotations, and show its benefits for training VideoQA models.
Weakly-supervised segmentation of referring expressions
Visual grounding localizes regions (boxes or segments) in the image corresponding to given referring expressions.
Assembly Planning from Observations under Physical Constraints
This paper addresses the problem of copying an unknown assembly of primitives with known shape and appearance using information extracted from a single photograph by an off-the-shelf procedure for object detection and pose estimation.
Learning Audio-Video Modalities from Image Captions
To close this gap we propose a new video mining pipeline which involves transferring captions from image captioning datasets to video clips with no additional manual effort.
Ranked #6 on
Zero-shot Text to Audio Retrieval
on AudioCaps
TubeDETR: Spatio-Temporal Video Grounding with Transformers
We consider the problem of localizing a spatio-temporal tube in a video corresponding to a given text query.
Ranked #2 on
Spatio-Temporal Video Grounding
on VidSTG
The Right Spin: Learning Object Motion from Rotation-Compensated Flow Fields
In this work, we argue that the coupling of camera rotation and camera translation can create complex motion fields that are difficult for a deep network to untangle directly.
Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigation
To balance the complexity of large action space reasoning and fine-grained language grounding, we dynamically combine a fine-scale encoding over local observations and a coarse-scale encoding on a global map via graph transformers.
Ranked #5 on
Visual Navigation
on SOON Test
Learning with Neighbor Consistency for Noisy Labels
Recent advances in deep learning have relied on large, labelled datasets to train high-capacity models.
End-to-end Generative Pretraining for Multimodal Video Captioning
Recent video and language pretraining frameworks lack the ability to generate sentences.
Ranked #15 on
Video Captioning
on MSR-VTT
(using extra training data)
Multiview Transformers for Video Recognition
Video understanding requires reasoning at multiple spatiotemporal resolutions -- from short fine-grained motions to events taking place over longer durations.
Ranked #5 on
Action Classification
on MiT
(using extra training data)
Masking Modalities for Cross-modal Video Retrieval
Our proposal is to pre-train a video encoder using all the available video modalities as supervision, namely, appearance, sound, and transcribed speech.
History Aware Multimodal Transformer for Vision-and-Language Navigation
Vision-and-language navigation (VLN) aims to build autonomous visual agents that follow instructions and navigate in real scenes.
Ranked #3 on
Vision and Language Navigation
on RxR
Differentiable Rendering with Perturbed Optimizers
Notably, images depend both on the properties of observed scenes and on the process of image formation.
Variational Perturbations for Visual Feature Attribution
Explaining a complex black-box system in a post-hoc manner is important to understand its predictions.
Airbert: In-domain Pretraining for Vision-and-Language Navigation
Given the scarcity of domain-specific training data and the high diversity of image and language inputs, the generalization of VLN agents to unseen environments remains challenging.
Ranked #3 on
Vision and Language Navigation
on VLN Challenge
Towards unconstrained joint hand-object reconstruction from RGB videos
Our work aims to obtain 3D reconstruction of hands and manipulated objects from monocular videos.
Ranked #5 on
hand-object pose
on HO-3D v2
CCVS: Context-aware Controllable Video Synthesis
The prediction model is doubly autoregressive, in the latent space of an autoencoder for forecasting, and in image space for updating contextual information, which is also used to enforce spatio-temporal consistency through a learnable optical flow module.
Ranked #8 on
Video Generation
on BAIR Robot Pushing
Goal-Conditioned Reinforcement Learning with Imagined Subgoals
Goal-conditioned reinforcement learning endows an agent with a large variety of skills, but it often struggles to solve tasks that require more temporally extended reasoning.
Attention Bottlenecks for Multimodal Fusion
Humans perceive the world by concurrently processing and fusing high-dimensional inputs from multiple modalities such as vision and audio.
Ranked #2 on
Action Classification
on Kinetics-Sounds
HDMapGen: A Hierarchical Graph Generative Model of High Definition Maps
To address this issue, we introduce a new challenging task to generate HD maps.
Residual Reinforcement Learning from Demonstrations
Residual reinforcement learning (RL) has been proposed as a way to solve challenging robotic tasks by adapting control actions from a conventional feedback controller to maximize a reward signal.
Large-Scale Unsupervised Object Discovery
Extensive experiments on COCO and OpenImages show that, in the single-object discovery setting where a single prominent object is sought in each image, the proposed LOD (Large-scale Object Discovery) approach is on par with, or better than the state of the art for medium-scale datasets (up to 120K images), and over 37% better than the only other algorithms capable of scaling up to 1. 7M images.
Episodic Transformer for Vision-and-Language Navigation
We demonstrate that encoding the history with a transformer is critical to solve compositional tasks, and that pretraining and joint training with synthetic instructions further improve the performance.
Segmenter: Transformer for Semantic Segmentation
In this paper we introduce Segmenter, a transformer model for semantic segmentation.
Ranked #15 on
Semantic Segmentation
on PASCAL Context
Class-Balanced Distillation for Long-Tailed Visual Recognition
An effective and simple approach to long-tailed visual recognition is to learn feature representations and a classifier separately, with instance and class-balanced sampling, respectively.
Ranked #13 on
Long-tail Learning
on iNaturalist 2018
Local Metrics for Multi-Object Tracking
This paper introduces temporally local metrics for Multi-Object Tracking.
Composable Augmentation Encoding for Video Representation Learning
We focus on contrastive methods for self-supervised video representation learning.
Improving robustness against common corruptions with frequency biased models
CNNs perform remarkably well when the training and test distributions are i. i. d, but unseen image corruptions can cause a surprisingly large drop in performance.
ViViT: A Video Vision Transformer
We present pure-transformer based models for video classification, drawing upon the recent success of such models in image classification.
Ranked #8 on
Action Classification
on MiT
(Top 5 Accuracy metric, using extra
training data)
Unified Graph Structured Models for Video Understanding
Accurate video understanding involves reasoning about the relationships between actors, objects and their environment, often over long temporal intervals.
Learning Temporal Dynamics from Cycles in Narrated Video
Learning to model how the world changes as time elapses has proven a challenging problem for the computer vision community.
Image Matching with Scale Adjustment
In this paper we address the problem of matching two images with two different resolutions: a high-resolution image and a low-resolution one.
Look Before you Speak: Visually Contextualized Utterances
Leveraging recent advances in multimodal learning, our model consists of a novel co-attentional multimodal video transformer, and when trained on both textual and visual context, outperforms baselines that use textual inputs alone.
Just Ask: Learning to Answer Questions from Millions of Narrated Videos
In this work, we propose to avoid manual annotation and generate a large-scale training dataset for video question answering making use of automatic cross-modal supervision.
Ranked #1 on
Video Question Answering
on VideoQA
Learning Obstacle Representations for Neural Motion Planning
Motion planning and obstacle avoidance is a key challenge in robotics applications.
Robotics
TNT: Target-driveN Trajectory Prediction
Our key insight is that for prediction within a moderate time horizon, the future modes can be effectively captured by a set of target states.
The End-of-End-to-End: A Video Understanding Pentathlon Challenge (2020)
This report summarizes the results of the first edition of the challenge together with the findings of the participants.
Learning Video Representations from Textual Web Supervision
Based on this observation, we propose to use text as a method for learning video representations.
Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos
Despite the recent advances in video classification, progress in spatio-temporal action recognition has lagged behind.
Multi-modal Transformer for Video Retrieval
In this paper, we present a multi-modal transformer to jointly encode the different modalities in video, which allows each of them to attend to the others.
Ranked #1 on
Zero-Shot Video Retrieval
on MSR-VTT
(text-to-video Mean Rank metric, using extra
training data)
Unsupervised Learning of Video Representations via Dense Trajectory Clustering
This paper addresses the task of unsupervised learning of representations for action recognition in videos.
Consistency Guided Scene Flow Estimation
To handle inherent modeling error in the consistency loss (e. g. Lambertian assumptions) and for better generalization, we further introduce a learned, output refinement network, which takes the initial predictions, the loss, and the gradient as input, and efficiently predicts a correlated output update.
TAO: A Large-Scale Benchmark for Tracking Any Object
To this end, we ask annotators to label objects that move at any point in the video, and give names to them post factum.
What Makes for Good Views for Contrastive Learning?
Contrastive learning between multiple views of the data has recently achieved state of the art performance in the field of self-supervised representation learning.
Ranked #2 on
Contrastive Learning
on imagenet-1k
VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation
Behavior prediction in dynamic, multi-agent systems is an important problem in the context of self-driving cars, due to the complex representations and interactions of road components, including moving agents (e. g. pedestrians and vehicles) and road context information (e. g. lanes, traffic lights).
Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstruction
Modeling hand-object manipulations is essential for understanding how humans interact with their environment.
Ranked #9 on
hand-object pose
on HO-3D v2
Learning visual policies for building 3D shape categories
We then show the success of our visual policies for building arches from different primitives.
Memory-Efficient Incremental Learning Through Feature Adaptation
We assume that the model is updated incrementally for new classes as new data becomes available sequentially. This requires adapting the previously stored feature vectors to the updated feature space without having access to the corresponding original training images.
Speech2Action: Cross-modal Supervision for Action Recognition
We train a BERT-based Speech2Action classifier on over a thousand movie screenplays, to predict action labels from transcribed speech segments.
Selecting Relevant Features from a Multi-domain Representation for Few-shot Classification
Popular approaches for few-shot classification consist of first learning a generic data representation based on a large annotated dataset, before adapting the representation to new classes given only a few labeled samples.
Ranked #4 on
Few-Shot Image Classification
on Meta-Dataset Rank
Beyond the Camera: Neural Networks in World Coordinates
Eye movement and strategic placement of the visual field onto the retina, gives animals increased resolution of the scene and suppresses distracting information.
Radioactive data: tracing through training
The mark is robust to strong variations such as different architectures or optimization methods.
Optimized Generic Feature Learning for Few-shot Classification across Domains
To learn models or features that generalize across tasks and domains is one of the grand goals of machine learning.
Synthetic Humans for Action Recognition from Unseen Viewpoints
Although synthetic training data has been shown to be beneficial for tasks such as human pose estimation, its use for RGB human action recognition is relatively unexplored.
Learning to Track Any Object
Moreover, at test time the same network can be applied to detection and tracking, resulting in a unified approach for the two tasks.
Graph convolutional networks for learning with few clean and many noisy labels
In this work we consider the problem of learning a classifier from noisy labels when a few clean labeled examples are given.
White-box vs Black-box: Bayes Optimal Strategies for Membership Inference
Membership inference determines, given a sample and trained parameters of a machine learning model, whether the sample was part of the training set.
Learning to combine primitive skills: A step towards versatile robotic manipulation
Manipulation tasks such as preparing a meal or assembling furniture remain highly challenging for robotics and vision.
Moulding Humans: Non-parametric 3D Human Shape Estimation from Single Images
In this paper, we tackle the problem of 3D human shape estimation from single RGB images.
Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera
We present GLNet, a self-supervised framework for learning depth, optical flow, camera pose and intrinsic parameters from monocular video - addressing the difficulty of acquiring realistic ground-truth for such tasks.
Learning Video Representations using Contrastive Bidirectional Transformer
This paper proposes a self-supervised learning approach for video features that results in significantly improved performance on downstream tasks (such as video classification, captioning and segmentation) compared to existing methods.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
A Study on Action Detection in the Wild
In this work we study the problem of action detection in a highly-imbalanced dataset.
Learning joint reconstruction of hands and manipulated objects
Previous work has made significant progress towards reconstruction of hand poses and object shapes in isolation.
Ranked #7 on
hand-object pose
on DexYCB
VideoBERT: A Joint Model for Video and Language Representation Learning
Self-supervised learning has become increasingly important to leverage the abundance of unlabeled data available on platforms like YouTube.
Ranked #1 on
Action Classification
on YouCook2
Diversity with Cooperation: Ensemble Methods for Few-Shot Classification
Few-shot classification consists of learning a predictive model that is able to effectively adapt to a new class, given only a few annotated samples.
Learning to Augment Synthetic Images for Sim2Real Policy Transfer
Policies learned in simulators, however, do not transfer well to real scenes given the domain gap between real and synthetic data.
AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection
The dataset contains temporally labeled face tracks in video, where each face instance is labeled as speaking or not, and whether the speech is audible.
Active Speaker Detection
Audio-Visual Active Speaker Detection
+4
Adaptive Density Estimation for Generative Models
We show that our model significantly improves over existing hybrid models: offering GAN-like samples, IS and FID scores that are competitive with fully adversarial models, and improved likelihood scores.
Detecting unseen visual relations using analogies
We seek to detect visual relations in images of the form of triplets t = (subject, predicate, object), such as "person riding dog", where training examples of the individual entities are available but their combinations are unseen at training.
A Structured Model For Action Detection
A dominant paradigm for learning-based approaches in computer vision is training generic models, such as ResNet for image recognition, or I3D for video understanding, on large datasets and allowing them to discover the optimal representation for the problem at hand.
Modulated Policy Hierarchies
To achieve this, we study different modulation signals and exploration for hierarchical controllers.
Coverage and Quality Driven Training of Generative Image Models
First, we propose a model that extends variational autoencoders by using deterministic invertible transformation layers to map samples from the decoder to the image space.
Déjà Vu: an empirical evaluation of the memorization properties of ConvNets
Convolutional neural networks memorize part of their training data, which is why strategies such as data augmentation and drop-out are employed to mitigate overfitting.
On the Importance of Visual Context for Data Augmentation in Scene Understanding
In this work, we consider object detection, semantic and instance segmentation and augment the training images by blending objects in existing scenes, using instance segmentation annotations.
Actor-Centric Relation Network
A visualization of the learned relation features confirms that our approach is able to attend to the relevant relations for each action.
Ranked #15 on
Action Recognition
on AVA v2.1
How good is my GAN?
Generative adversarial networks (GANs) are one of the most popular methods for generating images today.
End-to-End Incremental Learning
Although deep learning approaches have stood out in recent years due to their state-of-the-art results, they continue to suffer from catastrophic forgetting, a dramatic decrease in overall performance when training with new classes added incrementally.
Ranked #2 on
Incremental Learning
on ImageNet100 - 10 steps
(# M Params metric)
Modeling Visual Context is Key to Augmenting Object Detection Datasets
For this approach to be successful, we show that modeling appropriately the visual context surrounding objects is crucial to place them in the right environment.
A flexible model for training action localization with varying levels of supervision
Our model is based on discriminative clustering and integrates different types of supervision as constraints on the optimization.
Modeling Spatio-Temporal Human Track Structure for Action Localization
In order to localize actions in time, we propose a recurrent localization network (RecLNet) designed to model the temporal structure of actions on the level of person tracks.
Spreading vectors for similarity search
Discretizing multi-dimensional data distributions is a fundamental step of modern indexing methods.
PoTion: Pose MoTion Representation for Action Recognition
We use the human joints as these keypoints and term our Pose moTion representation PoTion.
Ranked #1 on
Skeleton Based Action Recognition
on J-HMDB
Unsupervised Learning of Artistic Styles with Archetypal Style Analysis
In this paper, we introduce an unsupervised learning approach to automatically discover, summarize, and manipulate artistic styles from large collections of paintings.
Actor and Observer: Joint Modeling of First and Third-Person Videos
Several theories in cognitive neuroscience suggest that when people interact with the world, or simulate interactions, they do so from a first-person egocentric perspective, and seamlessly transfer knowledge between third-person (observer) and first-person (actor).
Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos
In this paper we describe the egocentric aspect of the dataset and present annotations for Charades-Ego with 68, 536 activity instances in 68. 8 hours of first and third-person video, making it one of the largest and most diverse egocentric datasets available.
BodyNet: Volumetric Inference of 3D Human Body Shapes
Human shape estimation is an important task for video editing, animation and fashion industry.
Ranked #3 on
3D Human Pose Estimation
on Surreal
(using extra training data)
LCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Images
We propose an end-to-end architecture for joint 2D and 3D human pose estimation in natural images.
3D Human Pose Estimation
3D Multi-Person Pose Estimation (absolute)
+1
Image-based Synthesis for Deep 3D Human Pose Estimation
Here, we propose a solution to generate a large set of photorealistic synthetic images of humans with 3D pose annotations.
Learning to Segment Moving Objects
We formulate this as a learning problem and design our framework with three cues: (i) independent object motion between a pair of frames, which complements object recognition, (ii) object appearance, which helps to correct errors in motion estimation, and (iii) temporal consistency, which imposes additional constraints on the segmentation.