Search Results for author:
Found 34 papers, 16 papers with code
Scaling 4D Representations
Scaling has not yet been convincingly demonstrated for pure self-supervised learning from video.
Motion Prompting: Controlling Video Generation with Motion Trajectories
Motion control is crucial for generating expressive and compelling video content; however, most existing video generation models rely mainly on text prompts for control, which struggle to capture the nuances of dynamic actions and temporal compositions.
Moving Off-the-Grid: Scene-Grounded Video Representations
By using a combination of cross-attention and positional embeddings we disentangle the representation structure and image structure.
Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation
To train the policy, we use an order of magnitude less robot interaction data compared to what the video prediction model was trained on.
TAPVid-3D: A Benchmark for Tracking Any Point in 3D
We introduce a new benchmark, TAPVid-3D, for evaluating the task of long-range Tracking Any Point in 3D (TAP-3D).
BootsTAP: Bootstrapped Training for Tracking-Any-Point
To endow models with greater understanding of physics and motion, it is useful to enable them to perceive how solid surfaces move and deform in real scenes.
Ranked #1 on
Point Tracking
on TAP-Vid-RGB-Stacking
Learning from One Continuous Video Stream
We introduce a framework for online learning from a single continuous video stream -- the way people and animals learn, without mini-batches, data augmentation or shuffling.
RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitation
For robots to be useful outside labs and specialized factories we need a way to teach them new useful behaviors quickly.
TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement
We present a novel model for Tracking Any Point (TAP) that effectively tracks any queried point on any physical surface throughout a video sequence.
Ranked #1 on
Visual Tracking
on Kinetics
Perception Test: A Diagnostic Benchmark for Multimodal Video Models
We propose a novel multimodal video benchmark - the Perception Test - to evaluate the perception and reasoning skills of pre-trained multimodal models (e. g. Flamingo, SeViLA, or GPT-4).
Ranked #1 on
Point Tracking
on Perception Test
TAP-Vid: A Benchmark for Tracking Any Point in a Video
Generic motion understanding from video involves not only tracking objects, but also perceiving how their surfaces deform and move.
Perception Test: A Diagnostic Benchmark for Multimodal Models
We propose a novel multimodal benchmark – the Perception Test – that aims to extensively evaluate perception and reasoning skills of multimodal models.
Input-level Inductive Biases for 3D Reconstruction
Much of the recent progress in 3D vision has been driven by the development of specialized architectures that incorporate geometrical inductive biases.
Perceiver IO: A General Architecture for Structured Inputs & Outputs
A central goal of machine learning is the development of systems that can solve many problems in as many data domains as possible.
Ranked #1 on
Optical Flow Estimation
on KITTI 2015
(Average End-Point Error metric)
Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view.
CrossTransformers: spatially-aware few-shot transfer
In this work, we illustrate how the neural network representations which underpin modern vision systems are subject to supervision collapse, whereby they lose any information that is not necessary for performing the training task, including information that may be necessary for transfer to new tasks or domains.
Bootstrap your own latent: A new approach to self-supervised Learning
From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view.
Ranked #2 on
Self-Supervised Person Re-Identification
on SYSU-30k
Sim2real transfer learning for 3D human pose estimation: motion to the rescue
In this paper, we show that standard neural-network approaches, which perform poorly when trained on synthetic RGB images, can perform well when the data is pre-processed to extract cues about the person's motion, notably as optical flow and the motion of 2D keypoints.
Data-Efficient Image Recognition with Contrastive Predictive Coding
Human observers can learn to recognize new categories of images from a handful of examples, yet doing so with artificial ones remains an open challenge.
Ranked #6 on
Contrastive Learning
on imagenet-1k
Exploiting temporal context for 3D human pose estimation in the wild
We present a bundle-adjustment-based algorithm for recovering accurate 3D human pose and meshes from monocular videos.
Ranked #1 on
Monocular 3D Human Pose Estimation
on Human3.6M
(Use Video Sequence metric)
Structured agents for physical construction
Our results show that agents which use structured representations (e. g., objects and scene graphs) and structured policies (e. g., object-centric actions) outperform those which use less structured representations, and generalize better beyond their training when asked to reason about larger scenes.
Video Action Transformer Network
We introduce the Action Transformer model for recognizing and localizing human actions in video clips.
Ranked #6 on
Action Recognition
on AVA v2.1
The Visual QA Devil in the Details: The Impact of Early Fusion and Batch Norm on CLEVR
Visual QA is a pivotal challenge for higher-level reasoning, requiring understanding language, vision, and relationships between many objects in a scene.
Learning Visual Question Answering by Bootstrapping Hard Attention
Attention mechanisms in biological perception are thought to select subsets of perceptual information for more sophisticated processing which would be prohibitive to perform on all sensory inputs.
Ranked #8 on
Visual Question Answering (VQA)
on CLEVR
A Better Baseline for AVA
We introduce a simple baseline for action localization on the AVA dataset.
Ranked #12 on
Action Recognition
on AVA v2.1
Kickstarting Deep Reinforcement Learning
Our method places no constraints on the architecture of the teacher or student agents, and it regulates itself to allow the students to surpass their teachers in performance.
Multi-task Self-Supervised Visual Learning
We investigate methods for combining multiple self-supervised tasks--i. e., supervised tasks where data can be collected without manual labeling--in order to train a single visual representation.
Ranked #45 on
Self-Supervised Image Classification
on ImageNet
(Top 5 Accuracy metric)
An Uncertain Future: Forecasting from Static Images using Variational Autoencoders
We show that our method is able to successfully predict events in a wide variety of scenes and can produce multiple different predictions when the future is ambiguous.
Tutorial on Variational Autoencoders
In just three years, Variational Autoencoders (VAEs) have emerged as one of the most popular approaches to unsupervised learning of complicated distributions.
Data-dependent Initializations of Convolutional Neural Networks
Convolutional Neural Networks spread through computer vision like a wildfire, impacting almost all visual tasks imaginable.
Unsupervised Visual Representation Learning by Context Prediction
This work explores the use of spatial context as a source of free and plentiful supervisory signal for training a rich visual representation.
Mid-level Elements for Object Detection
Building on the success of recent discriminative mid-level elements, we propose a surprisingly simple approach for object detection which performs comparable to the current state-of-the-art approaches on PASCAL VOC comp-3 detection challenge (no external data).
Mid-level Visual Element Discovery as Discriminative Mode Seeking
We also propose the Purity-Coverage plot as a principled way of experimentally analyzing and evaluating different visual discovery approaches, and compare our method against prior work on the Paris Street View dataset.
