Search Results for author:
Found 78 papers, 49 papers with code
Learning Actionness via Long-range Temporal Order Verification
The annotation is particularly difficult for temporal action localization where large parts of the video present no action, or background.
Large-scale Pre-training for Grounded Video Caption Generation
We perform extensive ablations that demonstrate the importance of pre-training using our automatically annotated HowToGround1M dataset followed by fine-tuning on the manually annotated iGround dataset and validate the key technical contributions of our model.
6D Object Pose Tracking in Internet Videos for Robotic Manipulation
Third, we thoroughly evaluate and ablate our 6D pose estimation method on YCB-V and HOPE-Video datasets as well as a new dataset of instructional videos manually annotated with approximate 6D object trajectories.
Learning to engineer protein flexibility
Third, we introduce a method for fine-tuning a protein inverse folding model to steer it toward desired flexibility in specified regions.
ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions
The goal of this work is to generate step-by-step visual instructions in the form of a sequence of images, given an input image that provides the scene context and the sequence of textual instructions.
Generative Timelines for Instructed Visual Assembly
The objective of this work is to manipulate visual timelines (e. g. a video) through natural language instructions, making complex timeline editing tasks accessible to non-expert or potentially even disabled users.
Grounded Video Caption Generation
We apply this approach to videos from the HowTo100M dataset, which results in a new large-scale training dataset, called HowToGround, with automatically annotated captions and spatio-temporally consistent bounding boxes with coherent natural language labels.
Training on test proteins improves fitness, structure, and function prediction
Data scarcity and distribution shifts often hinder the ability of machine learning models to generalize when applied to proteins and other biological data.
MassSpecGym: A benchmark for the discovery and identification of molecules
To address this problem, we propose MassSpecGym -- the first comprehensive benchmark for the discovery and identification of molecules from MS/MS data.
De novo molecule generation from MS/MS spectrum
De novo molecule generation from MS/MS spectrum (bonus chemical formulae)
+4
Revealing data leakage in protein interaction benchmarks
To overcome the data leakage, we recommend constructing data splits based on 3D structural similarity of protein-protein interfaces and suggest corresponding algorithms.
POP-3D: Open-Vocabulary 3D Occupancy Prediction from Images
We describe an approach to predict open-vocabulary 3D semantic voxel occupancy map from input 2D images with the objective of enabling 3D grounding, segmentation and retrieval of free-form language queries.
3D Semantic Occupancy Prediction
3D Semantic Segmentation
+3
GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videos
We address the task of generating temporally consistent and physically plausible images of actions and object state transformations.
NewMove: Customizing text-to-video models with novel motions
To avoid overfitting to the new custom motion, we introduce an approach for regularization over videos.
FocalPose++: Focal Length and Object Pose Estimation via Render and Compare
We introduce FocalPose++, a neural render-and-compare method for jointly estimating the camera-object 6D pose and camera focal length given a single RGB input image depicting a known object.
Learning to design protein-protein interactions with enhanced generalization
Discovering mutations enhancing protein-protein interactions (PPIs) is critical for advancing biomedical research and developing improved therapeutics.
VidChapters-7M: Video Chapters at Scale
To address this issue, we present VidChapters-7M, a dataset of 817K user-chaptered videos including 7M chapters in total.
Meta-Personalizing Vision-Language Models to Find Named Instances in Video
Large-scale vision-language models (VLM) have shown impressive results for language-guided search applications.
Language-Guided Music Recommendation for Video via Prompt Analogies
A key challenge of this problem setting is that existing music video datasets provide the needed (video, music) training pairs, but lack text descriptions of the music.
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
In this work, we introduce Vid2Seq, a multi-modal single-stage dense event captioning model pretrained on narrated videos which are readily-available at scale.
Ranked #1 on
Dense Video Captioning
on ActivityNet Captions
(using extra training data)
MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare
Second, we introduce a novel approach for coarse pose estimation which leverages a network trained to classify whether the pose error between a synthetic rendering and an observed image of the same object can be corrected by the refiner.
Multi-Task Learning of Object State Changes from Uncurated Videos
We aim to learn to temporally localize object state changes and the corresponding state-modifying actions by observing people interacting with objects in long uncurated web videos.
Benchmarking Learning Efficiency in Deep Reservoir Computing
In this paper, we introduce a benchmark of increasingly difficult tasks together with a data efficiency metric to measure how quickly machine learning models learn from training data.
Imitrob: Imitation Learning Dataset for Training and Evaluating 6D Object Pose Estimators
This paper introduces a dataset for training and evaluating methods for 6D pose estimation of hand-held tools in task demonstrations captured by a standard RGB camera.
Learning Object Manipulation Skills from Video via Approximate Differentiable Physics
We evaluate our approach on a 3D reconstruction task that consists of 54 video demonstrations sourced from 9 actions such as pull something from right to left or put something in front of something.
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models
Manual annotation of question and answers for videos, however, is tedious and prohibits scalability.
Ranked #1 on
Zero-Shot Video Question Answer
on TVQA
Learning to Answer Visual Questions from Web Videos
We use our method to generate the WebVidVQA3M dataset from the WebVid dataset, i. e., videos with alt-text annotations, and show its benefits for training VideoQA models.
Focal Length and Object Pose Estimation via Render and Compare
We introduce FocalPose, a neural render-and-compare method for jointly estimating the camera-object 6D pose and camera focal length given a single RGB input image depicting a known object.
TubeDETR: Spatio-Temporal Video Grounding with Transformers
We consider the problem of localizing a spatio-temporal tube in a video corresponding to a given text query.
Ranked #3 on
Spatio-Temporal Video Grounding
on VidSTG
Look for the Change: Learning Object States and State-Modifying Actions from Untrimmed Web Videos
In this paper, we seek to temporally localize object states (e. g. "empty" and "full" cup) together with the corresponding state-modifying actions ("pouring coffee") in long uncurated videos with minimal supervision.
Drive&Segment: Unsupervised Semantic Segmentation of Urban Scenes via Cross-modal Distillation
This work investigates learning pixel-wise semantic image segmentation in urban scenes without any manual annotation, just from the raw non-curated data collected by cars which, equipped with cameras and LiDAR sensors, drive around a city.
Estimating 3D Motion and Forces of Human-Object Interactions from Internet Videos
First, we introduce an approach to jointly estimate the motion and the actuation forces of the person on the manipulated object by modeling contacts and the dynamics of the interactions.
Weakly Supervised Human-Object Interaction Detection in Video via Contrastive Spatiotemporal Regions
Our task poses unique challenges as a system does not know what types of human-object interactions are present in a video or the actual spatiotemporal location of the human and the object.
Reconstructing and grounding narrated instructional videos in 3D
Contrary to the standard scenario of instance-level 3D reconstruction, where identical objects or scenes are present in all views, objects in different instructional videos may have large appearance variations given varying conditions and versions of the same product.
Single-view robot pose and joint angle estimation via render & compare
We introduce RoboPose, a method to estimate the joint angles and the 6D camera-to-robot pose of a known articulated robot from a single RGB image.
Ranked #3 on
Robot Pose Estimation
on DREAM-dataset
Visualizing computation in large-scale cellular automata
Emergent processes in complex systems such as cellular automata can perform computations of increasing complexity, and could possibly lead to artificial evolution.
Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
We also extend our method to the video domain, improving the state of the art on the VATEX dataset.
Just Ask: Learning to Answer Questions from Millions of Narrated Videos
In this work, we propose to avoid manual annotation and generate a large-scale training dataset for video question answering making use of automatic cross-modal supervision.
Ranked #1 on
Video Question Answering
on VideoQA
Learning Object Manipulation Skills via Approximate State Estimation from Real Videos
We evaluate our method on simple single- and two-object actions from the Something-Something dataset.
CosyPose: Consistent multi-view multi-object 6D pose estimation
Second, we develop a robust method for matching individual 6D object pose hypotheses across different input images in order to jointly estimate camera viewpoints and 6D poses of all objects in a single consistent scene.
RareAct: A video dataset of unusual interactions
This paper introduces a manually annotated video dataset of unusual actions, namely RareAct, including actions such as "blend phone", "cut keyboard" and "microwave shoes".
Occlusion resistant learning of intuitive physics from videos
In this work we propose a probabilistic formulation of learning intuitive physics in 3D scenes with significant inter-object occlusions.
Efficient Neighbourhood Consensus Networks via Submanifold Sparse Convolutions
In this work we target the problem of estimating accurately localised correspondences between a pair of images.
End-to-End Learning of Visual Representations from Uncurated Instructional Videos
Annotating videos is cumbersome, expensive and not scalable.
Ranked #3 on
Action Recognition
on RareAct
Evolving Structures in Complex Systems
In this paper we propose an approach for measuring growth of complexity of emerging patterns in complex systems such as cellular automata.
Is This The Right Place? Geometric-Semantic Pose Verification for Indoor Visual Localization
The pose with the largest geometric consistency with the query image, e. g., in the form of an inlier count, is then selected in a second stage.
Learning to combine primitive skills: A step towards versatile robotic manipulation
Manipulation tasks such as preparing a meal or assembling furniture remain highly challenging for robotics and vision.
Finding Moments in Video Collections Using Natural Language
We evaluate our approach on two recently proposed datasets for temporal localization of moments in video with natural language (DiDeMo and Charades-STA) extended to our video corpus moment retrieval setting.
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
In this work, we propose instead to learn such embeddings from video data with readily available natural language annotations in the form of automatically transcribed narrations.
Ranked #4 on
Temporal Action Localization
on CrossTask
Action Localization
Long Video Retrieval (Background Removed)
+3
D2-Net: A Trainable CNN for Joint Detection and Description of Local Features
In this work we address the problem of finding reliable pixel-level correspondences under difficult imaging conditions.
Ranked #8 on
Image Matching
on IMC PhotoTourism
Monte-Carlo Tree Search for Efficient Visually Guided Rearrangement Planning
We address the problem of visually guided rearrangement planning with many movable objects, i. e., finding a sequence of actions to move a set of objects from an initial arrangement to a desired one, while relying on visual inputs coming from an RGB camera.
Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video
First, we introduce an approach to jointly estimate the motion and the actuation forces of the person on the manipulated object by modeling contacts and the dynamics of their interactions.
Cross-task weakly supervised learning from instructional videos
In this paper we investigate learning visual models for the steps of ordinary tasks using weak supervision via instructional narrations and an ordered list of steps instead of strong supervision via temporal annotations.
Ranked #5 on
Temporal Action Localization
on CrossTask
Detecting unseen visual relations using analogies
We seek to detect visual relations in images of the form of triplets t = (subject, predicate, object), such as "person riding dog", where training examples of the individual entities are available but their combinations are unseen at training.
Neighbourhood Consensus Networks
Second, we demonstrate that the model can be trained effectively from weak supervision in the form of matching and non-matching image pairs without the need for costly manual annotation of point to point correspondences.
Ranked #2 on
Semantic correspondence
on PF-PASCAL
(PCK (weak) metric)
Localizing Moments in Video with Temporal Language
To benchmark whether our model, and other recent video localization models, can effectively reason about temporal language, we collect the novel TEMPOral reasoning in video and language (TEMPO) dataset.
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
We evaluate our method on the task of video retrieval and report results for the MPII Movie Description and MSR-VTT datasets.
Ranked #33 on
Video Retrieval
on LSMDC
(using extra training data)
InLoc: Indoor Visual Localization with Dense Matching and View Synthesis
We seek to predict the 6 degree-of-freedom (6DoF) pose of a query photograph with respect to a large indoor 3D map.
End-to-end weakly-supervised semantic alignment
We tackle the task of semantic alignment where the goal is to compute dense semantic correspondence aligning two images depicting objects of the same category.
Localizing Moments in Video with Natural Language
A key obstacle to training our MCN model is that current video datasets do not include pairs of localized video segments and referring expressions, or text descriptions which uniquely identify a corresponding moment.
Weakly-supervised learning of visual relations
This paper introduces a novel approach for modeling visual relations between pairs of objects.
Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions
Visual localization enables autonomous vehicles to navigate in their surroundings and augmented reality applications to link virtual to real worlds.
Learning from Video and Text via Large-Scale Discriminative Clustering
Discriminative clustering has been successfully applied to a number of weakly-supervised learning tasks.
Ranked #35 on
Video Retrieval
on LSMDC
Are Large-Scale 3D Models Really Necessary for Accurate Visual Localization?
3D structure-based methods employ 3D models of the scene to estimate the full 6DOF pose of a camera very accurately.
Learnable pooling with Context Gating for video classification
In particular, we evaluate our method on the large-scale multi-modal Youtube-8M v2 dataset and outperform all other methods in the Youtube 8M Large-Scale Video Understanding challenge.
ActionVLAD: Learning spatio-temporal aggregation for action classification
In this work, we introduce a new video representation for action classification that aggregates local convolutional features across the entire spatio-temporal extent of the video.
Ranked #8 on
Long-video Activity Recognition
on Breakfast
Convolutional neural network architecture for geometric matching
We address the problem of determining correspondences between two images in agreement with a geometric model such as an affine or thin-plate spline transformation, and estimating its parameters.
NetVLAD: CNN architecture for weakly supervised place recognition
We tackle the problem of large scale visual place recognition, where the task is to quickly and accurately recognize the location of a given query photograph.
Ranked #3 on
Visual Place Recognition
on Mid-Atlantic Ridge
Unsupervised Learning from Narrated Instruction Videos
Third, we experimentally demonstrate that the proposed method can automatically discover, in an unsupervised manner, the main steps to achieve the task and locate the steps in the input videos.
Ranked #7 on
Temporal Action Localization
on CrossTask
24/7 Place Recognition by View Synthesis
We address the problem of large-scale visual place recognition for situations where the scene undergoes a major change in appearance, for example, due to illumination (day/night), change of seasons, aging, or structural modifications over time such as buildings built or destroyed.
Is Object Localization for Free? - Weakly-Supervised Learning With Convolutional Neural Networks
Successful visual object recognition methods typically rely on training datasets containing lots of richly annotated images.
On Pairwise Costs for Network Flow Multi-Object Tracking
Multi-object tracking has been recently approached with the min-cost network flow optimization techniques.
Weakly Supervised Action Labeling in Videos Under Ordering Constraints
We are given a set of video clips, each one annotated with an {\em ordered} list of actions, such as "walk" then "sit" then "answer phone" extracted from, for example, the associated text script.
Learning and Transferring Mid-Level Image Representations using Convolutional Neural Networks
We show that despite differences in image statistics and tasks in the two datasets, the transferred representation leads to significantly improved results for object and action classification, outperforming the current state of the art on Pascal VOC 2007 and 2012 datasets.
Seeing 3D Chairs: Exemplar Part-based 2D-3D Alignment using a Large Dataset of CAD Models
This paper poses object category detection in images as a type of 2D-to-3D alignment problem, utilizing the large quantities of 3D CAD models that have been made publicly available online.
Learning and Calibrating Per-Location Classifiers for Visual Place Recognition
The aim of this work is to localize a query photograph by finding other images depicting the same place in a large geotagged image database.
Visual Place Recognition with Repetitive Structures
Even more importantly, they violate the feature independence assumed in the bag-of-visual-words representation which often leads to over-counting evidence and significant degradation of retrieval performance.
Learning person-object interactions for action recognition in still images
First, we replace the standard quantized local HOG/SIFT features with stronger discriminatively trained body part and object detectors.
Segmenting Scenes by Matching Image Composites
In contrast to recent work in semantic alignment of scenes, we allow an input image to be explained by partial matches of similar scenes.
