Search Results for author:
Found 44 papers, 28 papers with code
Diffusion Autoencoders are Scalable Image Tokenizers
Our key insight is that a single learning objective, diffusion L2 loss, can be used for training scalable image tokenizers.
LLMs can see and hear without any training
We present MILS: Multimodal Iterative LLM Solver, a surprisingly simple, training-free approach, to imbue multimodal capabilities into your favorite LLM.
MotiF: Making Text Count in Image Animation with Motion Focal Loss
To overcome this limitation, we introduce MotiF, a simple yet effective approach that directs the model's learning to the regions with more motion, thereby improving the text alignment and motion generation.
Human Action Anticipation: A Survey
Predicting future human behavior is an increasingly popular topic in computer vision, driven by the interest in applications such as autonomous vehicles, digital assistants and human-robot interactions.
Movie Gen: A Cast of Media Foundation Models
Our models set a new state-of-the-art on multiple tasks: text-to-video synthesis, video personalization, video editing, video-to-audio generation, and text-to-audio generation.
The Llama 3 Herd of Models
This paper presents a new set of foundation models, called Llama 3.
Ranked #3 on
Question Answering
on PeerQA
SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos
We propose a novel self-supervised embedding to learn how actions sound from narrated in-the-wild egocentric videos.
InstanceDiffusion: Instance-level Control for Image Generation
Text-to-image diffusion models produce high quality images but do not offer control over individual instances in the image.
Ranked #3 on
Conditional Text-to-Image Synthesis
on COCO-MIG
Generating Illustrated Instructions
We introduce the new task of generating Illustrated Instructions, i. e., visual instructions customized to a user's needs.
Motion-Conditioned Image Animation for Video Editing
We introduce MoCA, a Motion-Conditioned Image Animation approach for video editing.
Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning
We present Emu Video, a text-to-video generation model that factorizes the generation into two steps: first generating an image conditioned on the text, and then generating a video conditioned on the text and the generated image.
VideoCutLER: Surprisingly Simple Unsupervised Video Instance Segmentation
Existing approaches to unsupervised video instance segmentation typically rely on motion estimates and experience difficulties tracking small or divergent motions.
ImageBind: One Embedding Space To Bind Them All
We show that all combinations of paired data are not necessary to train such a joint embedding, and only image-paired data is sufficient to bind the modalities together.
Ranked #2 on
Zero-shot Classification (unified classes)
on LLVIP
The effectiveness of MAE pre-pretraining for billion-scale pretraining
While MAE has only been shown to scale with the size of models, we find that it scales with the size of the training dataset as well.
Ranked #1 on
Few-Shot Image Classification
on ImageNet - 10-shot
(using extra training data)
Learning to Substitute Ingredients in Recipes
Recipe personalization through ingredient substitution has the potential to help people meet their dietary needs and preferences, avoid potential allergens, and ease culinary exploration in everyone's kitchen.
Cut and Learn for Unsupervised Object Detection and Instance Segmentation
We propose Cut-and-LEaRn (CutLER), a simple approach for training unsupervised object detection and segmentation models.
Ranked #1 on
Unsupervised Instance Segmentation
on UVO
What You Say Is What You Show: Visual Narration Detection in Instructional Videos
Narrated ''how-to'' videos have emerged as a promising data source for a wide range of learning problems, from learning visual representations to training robot policies.
HierVL: Learning Hierarchical Video-Language Embeddings
Video-language embeddings are a promising avenue for injecting semantics into visual representations, but existing methods capture only short-term associations between seconds-long video clips and their accompanying text.
Ranked #3 on
Long Term Action Anticipation
on Ego4D
Learning Video Representations from Large Language Models
We introduce LaViLa, a new approach to learning video-language representations by leveraging Large Language Models (LLMs).
Ranked #1 on
Action Recognition
on Charades-Ego
OmniMAE: Single Model Masked Pretraining on Images and Videos
Furthermore, this model can be learned by dropping 90% of the image and 95% of the video patches, enabling extremely fast training of huge model architectures.
Omnivore: A Single Model for Many Visual Modalities
Prior work has studied different visual modalities in isolation and developed separate architectures for recognition of images, videos, and 3D data.
Ranked #1 on
Scene Recognition
on SUN-RGBD
(using extra training data)
Detecting Twenty-thousand Classes using Image-level Supervision
For the first time, we train a detector with all the twenty-one-thousand classes of the ImageNet dataset and show that it generalizes to new datasets without finetuning.
Ranked #2 on
Open Vocabulary Object Detection
on OpenImages-v4
Cross-Domain Few-Shot Object Detection
image-classification
+2
Mask2Former for Video Instance Segmentation
We find Mask2Former also achieves state-of-the-art performance on video instance segmentation without modifying the architecture, the loss or even the training pipeline.
Masked-attention Mask Transformer for Universal Image Segmentation
While only the semantics of each task differ, current research focuses on designing specialized architectures for each task.
Ranked #3 on
Semantic Segmentation
on Mapillary val
Ego4D: Around the World in 3,000 Hours of Egocentric Video
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite.
An End-to-End Transformer Model for 3D Object Detection
We propose 3DETR, an end-to-end Transformer based object detection model for 3D point clouds.
Ranked #22 on
3D Object Detection
on SUN-RGBD val
Anticipative Video Transformer
We propose Anticipative Video Transformer (AVT), an end-to-end attention-based video modeling architecture that attends to the previously observed video in order to anticipate future actions.
Ranked #2 on
Action Anticipation
on EPIC-KITCHENS-100 (test)
(using extra training data)
3D Spatial Recognition without Spatially Labeled 3D
We introduce WyPR, a Weakly-supervised framework for Point cloud Recognition, requiring only scene-level class tags as supervision.
Physical Reasoning Using Dynamics-Aware Models
A common approach to solving physical reasoning tasks is to train a value learner on example tasks.
Ranked #1 on
Visual Reasoning
on PHYRE-1B-Within
Self-Supervised Pretraining of 3D Features on any Point-Cloud
Pretraining on large labeled datasets is a prerequisite to achieve good performance in many computer vision tasks like 2D object recognition, video classification etc.
Forward Prediction for Physical Reasoning
Physical reasoning requires forward prediction: the ability to forecast what will happen next given some initial world state.
Ranked #2 on
Visual Reasoning
on PHYRE-1B-Within
Video Understanding as Machine Translation
With the advent of large-scale multimodal video datasets, especially sequences with audio or transcribed speech, there has been a growing interest in self-supervised learning of video representations.
CATER: A diagnostic dataset for Compositional Actions & TEmporal Reasoning
In this work, we build a video dataset with fully observable and controllable object and scene bias, and which truly requires spatiotemporal understanding in order to be solved.
Are we asking the right questions in MovieQA?
Joint vision and language tasks like visual question answering are fascinating because they explore high-level understanding, but at the same time, can be more prone to language biases.
MetaPix: Few-Shot Video Retargeting
We address the task of unsupervised retargeting of human actions from one video to another.
CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning
In this work, we build a video dataset with fully observable and controllable object and scene bias, and which truly requires spatiotemporal understanding in order to be solved.
DistInit: Learning Video Representations Without a Single Labeled Video
In this work, we propose an alternative approach to learning video representations that require no semantically labeled videos and instead leverages the years of effort in collecting and labeling large and clean still-image datasets.
Ranked #72 on
Action Recognition
on HMDB-51
(using extra training data)
Video Action Transformer Network
We introduce the Action Transformer model for recognizing and localizing human actions in video clips.
Ranked #6 on
Action Recognition
on AVA v2.1
A Better Baseline for AVA
We introduce a simple baseline for action localization on the AVA dataset.
Ranked #12 on
Action Recognition
on AVA v2.1
Binge Watching: Scaling Affordance Learning from Sitcoms
In this paper, we tackle the challenge of creating one of the biggest dataset for learning affordances.
Detect-and-Track: Efficient Pose Estimation in Videos
This paper addresses the problem of estimating and tracking human body keypoints in complex, multi-person video.
Ranked #7 on
Keypoint Detection
on COCO test-challenge
Attentional Pooling for Action Recognition
We introduce a simple yet surprisingly powerful model to incorporate attention in action recognition and human object interaction tasks.
Ranked #7 on
Human-Object Interaction Detection
on HICO
ActionVLAD: Learning spatio-temporal aggregation for action classification
In this work, we introduce a new video representation for action classification that aggregates local convolutional features across the entire spatio-temporal extent of the video.
Ranked #8 on
Long-video Activity Recognition
on Breakfast
Learning a Predictable and Generative Vector Representation for Objects
The network consists of two components: (a) an autoencoder that ensures the representation is generative; and (b) a convolutional network that ensures the representation is predictable.
