Search Results for author:
Found 41 papers, 20 papers with code
Siamese Vision Transformers are Scalable Audio-visual Learners
Our framework uses a single shared vision transformer backbone to process audio and visual inputs, improving its parameter efficiency, reducing the GPU memory footprint, and allowing us to scale our method to larger datasets and model sizes.
Augmented Reality Demonstrations for Scalable Robot Imitation Learning
Our framework facilitates scalable and diverse demonstration collection for real-world tasks.
DAM: Dynamic Adapter Merging for Continual Video QA Learning
Our DAM model outperforms prior state-of-the-art continual learning approaches by 9. 1% while exhibiting 1. 9% less forgetting on 6 VidQA datasets spanning various domains.
Video ReCap: Recursive Captioning of Hour-Long Videos
We utilize a curriculum learning training scheme to learn the hierarchical structure of videos, starting from clip-level captions describing atomic actions, then focusing on segment-level descriptions, and concluding with generating summaries for hour-long videos.
Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences
However, current MLLM benchmarks are predominantly designed to evaluate reasoning based on static information about a single image, and the ability of modern MLLMs to extrapolate from image sequences, which is essential for understanding our ever-changing world, has been less investigated.
A Simple LLM Framework for Long-Range Video Question-Answering
Furthermore, we show that a specialized prompt that asks the LLM first to summarize the noisy short-term visual captions and then answer a given input question leads to a significant LVQA performance boost.
Ranked #1 on
Zero-Shot Video Question Answer
on NExT-GQA
RGNet: A Unified Clip Retrieval and Grounding Network for Long Videos
Adapting existing short video (5-30 seconds) grounding methods to this problem yields poor performance.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge.
Unified Coarse-to-Fine Alignment for Video-Text Retrieval
Specifically, our model captures the cross-modal similarity information at different granularity levels.
Ranked #11 on
Video Retrieval
on MSR-VTT
LoCoNet: Long-Short Context Network for Active Speaker Detection
These two contexts are complementary to each other and can help infer the active speaker.
Efficient Movie Scene Detection using State-Space Transformers
Given a sequence of frames divided into movie shots (uninterrupted periods where the camera position does not change), the S4A block first applies self-attention to capture short-range intra-shot dependencies.
Vision Transformers are Parameter-Efficient Audio-Visual Learners
To do so, we propose a latent audio-visual hybrid (LAVISH) adapter that adapts pretrained ViTs to audio-visual tasks by injecting a small number of trainable parameters into every layer of a frozen ViT.
Ranked #4 on
Audio-visual Question Answering
on MUSIC-AVQA
VindLU: A Recipe for Effective Video-and-Language Pretraining
Furthermore, our model also obtains state-of-the-art video question-answering results on ActivityNet-QA, MSRVTT-QA, MSRVTT-MC and TVQA.
Ranked #2 on
Video Retrieval
on Condensed Movies
(using extra training data)
SimpleClick: Interactive Image Segmentation with Simple Vision Transformers
Although this design is simple and has been proven effective, it has not yet been explored for interactive image segmentation.
Ranked #2 on
Interactive Segmentation
on SBD
MuMUR : Multilingual Multimodal Universal Retrieval
In this paper, we propose a framework MuMUR, that utilizes knowledge transfer from a multilingual model to boost the performance of multi-modal (image and video) retrieval.
Object State Change Classification in Egocentric Videos using the Divided Space-Time Attention Mechanism
This report describes our submission called "TarHeels" for the Ego4D: Object State Change Classification Challenge.
Learning to Retrieve Videos by Asking Questions
To overcome this limitation, we propose a novel framework for Video Retrieval using Dialog (ViReD), which enables the user to interact with an AI agent via multiple rounds of dialog, where the user refines retrieved results by answering questions generated by an AI agent.
ECLIPSE: Efficient Long-range Video Retrieval using Sight and Sound
We introduce an audiovisual method for long-range text-to-video retrieval.
Long Movie Clip Classification with State-Space Video Models
Most modern video recognition models are designed to operate on short video clips (e. g., 5-10s in length).
Ranked #4 on
Video Classification
on Breakfast
TALLFormer: Temporal Action Localization with a Long-memory Transformer
To address these issues, we propose TALLFormer, a memory-efficient and end-to-end trainable Temporal Action Localization Transformer with Long-term memory.
Learning To Recognize Procedural Activities with Distant Supervision
In this paper we consider the problem of classifying fine-grained, multi-step activities (e. g., cooking different recipes, making disparate home improvements, creating various forms of arts and crafts) from long videos spanning up to several minutes.
Ranked #3 on
Video Classification
on Breakfast
Long-Short Temporal Contrastive Learning of Video Transformers
Our approach, named Long-Short Temporal Contrastive Learning (LSTCL), enables video transformers to learn an effective clip-level representation by predicting temporal context captured from a longer temporal extent.
Is Space-Time Attention All You Need for Video Understanding?
We present a convolution-free approach to video classification built exclusively on self-attention over space and time.
Ranked #1 on
Video Question Answering
on Howto100M-QA
VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs
We present \textsc{Vx2Text}, a framework for text generation from multimodal inputs consisting of video plus text, speech, or audio.
COBE: Contextualized Object Embeddings from Narrated Instructional Video
A fully-supervised approach to recognizing object states and their contexts in the real-world is unfortunately marred by the long-tailed, open-ended distribution of the data, which would effectively require massive amounts of annotations to capture the appearance of objects in all their different forms.
Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation
We introduce a method for simultaneously classifying, segmenting and tracking object instances in a video sequence.
Learning Temporal Pose Estimation from Sparsely-Labeled Videos
To reduce the need for dense annotations, we propose a PoseWarper network that leverages training videos with sparse annotations (every k frames) to learn to perform dense temporal pose propagation and estimation.
Ranked #2 on
Multi-Person Pose Estimation
on PoseTrack2018
(using extra training data)
Attentive Action and Context Factorization
This is a challenging task due to the subtlety of human actions in video and the co-occurrence of contextual elements.
Learning Discriminative Motion Features Through Detection
Our network learns to spatially sample features from Frame B in order to maximize pose detection accuracy in Frame A.
Object Detection in Video with Spatiotemporal Sampling Networks
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable convolutions across time for object detection in videos.
Egocentric Basketball Motion Planning from a Single First-Person Image
We present a model that uses a single first-person image to generate an egocentric basketball motion sequence in the form of a 12D camera configuration trajectory, which encodes a player's 3D location and 3D head orientation throughout the sequence.
Using Cross-Model EgoSupervision to Learn Cooperative Basketball Intention
We present a first-person method for cooperative basketball intention prediction: we predict with whom the camera wearer will cooperate in the near future from unlabeled first-person images.
Am I a Baller? Basketball Performance Assessment from First-Person Videos
Finally, we use this feature to learn a basketball assessment model from pairs of labeled first-person basketball videos, for which a basketball expert indicates, which of the two players is better.
Unsupervised Learning of Important Objects from First-Person Videos
In this work, we show that we can detect important objects in first-person images without the supervision by the camera wearer or even third-person labelers.
Convolutional Random Walk Networks for Semantic Image Segmentation
It combines these two objectives via a novel random walk layer that enforces consistent spatial grouping in the deep layers of the network.
Local Perturb-and-MAP for Structured Prediction
In this work, we present a new Local Perturb-and-MAP (locPMAP) framework that replaces the global optimization with a local optimization by exploiting our observed connection between locPMAP and the pseudolikelihood of the original CRF model.
First Person Action-Object Detection with EgoNet
Unlike traditional third-person cameras mounted on robots, a first-person camera, captures a person's visual sensorimotor object interactions from up close.
Semantic Segmentation with Boundary Neural Fields
To overcome these problems, we introduce a Boundary Neural Field (BNF), which is a global energy model integrating FCN predictions with boundary cues.
Exploiting Egocentric Object Prior for 3D Saliency Detection
We empirically show that this representation can accurately characterize the egocentric object prior by testing it on an egocentric RGBD dataset for three tasks: the 3D saliency detection, future saliency prediction, and interaction classification.
High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision
We can view this process as a "Low-for-High" scheme, where low-level boundaries aid high-level vision tasks.
DeepEdge: A Multi-Scale Bifurcated Deep Network for Top-Down Contour Detection
This section of the network is applied to four different scales of the image input.
