Search Results for author:
Found 33 papers, 14 papers with code
Revealing Single Frame Bias for Video-and-Language Learning
Training an effective video-and-language model intuitively requires multiple frames as model inputs.
Ranked #5 on
Video Retrieval
on SSv2-template retrieval
(using extra training data)
End-to-End Visual Editing with a Generatively Pre-Trained Artist
We consider the targeted image editing problem: blending a region in a source image with a driver image that specifies the desired change.
LoopITR: Combining Dual and Cross Encoder Architectures for Image-Text Retrieval
In this work, we propose LoopITR, which combines them in the same network for joint learning.
CommerceMM: Large-Scale Commerce MultiModal Representation Learning with Omni Retrieval
We introduce CommerceMM - a multimodal model capable of providing a diverse and granular understanding of commerce topics associated to the given piece of content (image, text, image+text), and having the capability to generalize to a wide range of tasks, including Multimodal Categorization, Image-Text Retrieval, Query-to-Product Retrieval, Image-to-Product Retrieval, etc.
MTVR: Multilingual Moment Retrieval in Videos
We introduce mTVR, a large-scale multilingual video moment retrieval dataset, containing 218K English and Chinese queries from 21. 8K TV show video clips.
QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
Each video in the dataset is annotated with: (1) a human-written free-form NL query, (2) relevant moments in the video w. r. t.
Ranked #11 on
Highlight Detection
on QVHighlights
Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
Experiments on text-to-video retrieval and video question answering on six datasets demonstrate that ClipBERT outperforms (or is on par with) existing methods that exploit full-length videos, suggesting that end-to-end learning with just a few sparsely sampled clips is often more accurate than using densely extracted offline features from full-length videos, proving the proverbial less-is-more principle.
Ranked #24 on
Visual Question Answering (VQA)
on MSRVTT-QA
(using extra training data)
What is More Likely to Happen Next? Video-and-Language Future Event Prediction
Given a video with aligned dialogue, people can often infer what is more likely to happen next.
MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning
Generating multi-sentence descriptions for videos is one of the most challenging captioning tasks due to its high requirements for not only visual relevance but also discourse-based coherence across the sentences in the paragraph.
Ranked #5 on
Video Captioning
on ActivityNet Captions
TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval
The queries are also labeled with query types that indicate whether each of them is more related to video or subtitle or both, allowing for in-depth analysis of the dataset and the methods that built on top of it.
Ranked #2 on
Video Retrieval
on TVR
IMP: Instance Mask Projection for High Accuracy Semantic Segmentation of Things
In addition, the instance mask projection operator works well on other (non-clothing) datasets, providing an improvement of 3 points in mIOU on Thing classes of Cityscapes, a self-driving dataset, on top of a state-of-the-art approach.
TVQA+: Spatio-Temporal Grounding for Video Question Answering
We present the task of Spatio-Temporal Video Question Answering, which requires intelligent systems to simultaneously retrieve relevant moments and detect referenced visual concepts (people and objects) to answer natural language questions about videos.
Ranked #6 on
Video Question Answering
on TVQA
Multi-Target Embodied Question Answering
To address this, we propose a modular architecture composed of a program generator, a controller, a navigator, and a VQA module.
Dance Dance Generation: Motion Transfer for Internet Videos
This work presents computational methods for transferring body movements from one person to another with videos collected in the wild.
TVQA: Localized, Compositional Video Question Answering
Recent years have witnessed an increasing interest in image-based question-answering (QA) tasks.
Ranked #4 on
Video Question Answering
on SUTD-TrafficQA
Image2GIF: Generating Cinemagraphs using Recurrent Deep Q-Networks
Given a still photograph, one can imagine how dynamic objects might move against a static background.
MAttNet: Modular Attention Network for Referring Expression Comprehension
In this paper, we address referring expression comprehension: localizing an image region described by a natural language expression.
Generalized Referring Expression Segmentation
Referring Expression
+1
Visual to Sound: Generating Natural Sound for Videos in the Wild
As two of the five traditional human senses (sight, hearing, taste, smell, and touch), vision and sound are basic sources through which humans understand the world.
Hierarchically-Attentive RNN for Album Summarization and Storytelling
For this task, we make use of the Visual Storytelling dataset and a model composed of three hierarchically-attentive Recurrent Neural Nets (RNNs) to: encode the album photos, select representative (summary) photos, and compose the story.
Ranked #15 on
Visual Storytelling
on VIST
(BLEU-3 metric)
A Joint Speaker-Listener-Reinforcer Model for Referring Expressions
The speaker generates referring expressions, the listener comprehends referring expressions, and the reinforcer introduces a reward function to guide sampling of more discriminative expressions.
Combining Multiple Cues for Visual Madlibs Question Answering
This paper presents an approach for answering fill-in-the-blank multiple choice questions from the Visual Madlibs dataset.
Learning Temporal Transformations From Time-Lapse Videos
Based on life-long observations of physical, chemical, and biologic phenomena in the natural world, humans can often easily picture in their minds what an object will look like in the future.
When was that made?
Toward this goal, we utilize features from existing deep networks and also fine-tune new networks for temporal estimation.
Solving Visual Madlibs with Multiple Cues
This paper focuses on answering fill-in-the-blank style multiple choice questions from the Visual Madlibs dataset.
Modeling Context in Referring Expressions
Humans refer to objects in their environments all the time, especially in dialogue with other people.
Where to Buy It: Matching Street Clothing Photos in Online Shops
In this paper, we define a new task, Exact Street to Shop, where our goal is to match a real-world example of a garment item to the same item in an online shop.
Visual Madlibs: Fill in the Blank Description Generation and Question Answering
In this paper, we introduce a new dataset consisting of 360, 001 focused natural language descriptions for 10, 738 images.
Temporal Perception and Prediction in Ego-Centric Video
Given a video of an activity, can we predict what will happen next?
Visual Madlibs: Fill in the blank Image Generation and Question Answering
In this paper, we introduce a new dataset consisting of 360, 001 focused natural language descriptions for 10, 738 images.
TreeTalk: Composition and Compression of Trees for Image Descriptions
We present a new tree based approach to composing expressive image descriptions that makes use of naturally occuring web images with captions.
Studying Relationships between Human Gaze, Description, and Computer Vision
We posit that user behavior during natural viewing of images contains an abundance of information about the content of images as well as information related to user intent and user defined content importance.
Two-person interaction detection using body-pose features and multiple instance learning
Human activity recognition has potential to impact a wide range of applications from surveillance to human computer interfaces to content based video retrieval.
Im2Text: Describing Images Using 1 Million Captioned Photographs
We develop and demonstrate automatic image description methods using a large captioned photo collection.
