Search Results for author:
Found 33 papers, 21 papers with code
FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Tasks
In the fashion domain, there exists a variety of vision-and-language (V+L) tasks, including cross-modal retrieval, text-guided image retrieval, multi-modal classification, and image captioning.
RoPAWS: Robust Semi-supervised Representation Learning from Uncurated Data
Semi-supervised learning aims to train a model using limited labels.
Que2Engage: Embedding-based Retrieval for Relevant and Engaging Products at Facebook Marketplace
In this paper we presents Que2Engage, a search EBR system built towards bridging the gap between retrieval and ranking for end-to-end optimizations.
CiT: Curation in Training for Effective Vision-Language Data
Large vision-language models are generally applicable to many downstream tasks, but come at an exorbitant training cost that only large institutions can afford.
Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generation
Inspired by this, we introduce a novel task, text-guided video completion (TVC), which requests the model to generate a video from partial frames guided by an instruction.
Ranked #3 on
Video Prediction
on BAIR Robot Pushing
FaD-VLP: Fashion Vision-and-Language Pre-training towards Unified Retrieval and Captioning
Additionally, these works have mainly been restricted to multimodal understanding tasks.
FashionViL: Fashion-Focused Vision-and-Language Representation Learning
We thus propose a Multi-View Contrastive Learning task for pulling closer the visual representation of one image to the compositional multimodal representation of another image+text.
GEB+: A Benchmark for Generic Event Boundary Captioning, Grounding and Retrieval
In this paper, we introduce a new dataset called Kinetic-GEB+.
Ranked #1 on
Text to Video Retrieval
on Kinetics-GEB+
(text-to-video R@1 metric)
LoopITR: Combining Dual and Cross Encoder Architectures for Image-Text Retrieval
In this work, we propose LoopITR, which combines them in the same network for joint learning.
CommerceMM: Large-Scale Commerce MultiModal Representation Learning with Omni Retrieval
We introduce CommerceMM - a multimodal model capable of providing a diverse and granular understanding of commerce topics associated to the given piece of content (image, text, image+text), and having the capability to generalize to a wide range of tasks, including Multimodal Categorization, Image-Text Retrieval, Query-to-Product Retrieval, Image-to-Product Retrieval, etc.
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation
Most existing video-and-language (VidL) research focuses on a single dataset, or multiple datasets of a single task.
Connecting What to Say With Where to Look by Modeling Human Attention Traces
Learning the grounding of each word is challenging, due to noise in the human-provided traces and the presence of words that cannot be meaningfully visually grounded.
What is More Likely to Happen Next? Video-and-Language Future Event Prediction
Given a video with aligned dialogue, people can often infer what is more likely to happen next.
Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
To reveal the secrets behind the scene of these powerful models, we present VALUE (Vision-And-Language Understanding Evaluation), a set of meticulously designed probing tasks (e. g., Visual Coreference Resolution, Visual Relation Detection, Linguistic Probing Tasks) generalizable to standard pre-trained V+L models, aiming to decipher the inner workings of multimodal pre-training (e. g., the implicit knowledge garnered in individual attention heads, the inherent cross-modal alignment learned through contextualized multimodal embeddings).
HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
We present HERO, a novel framework for large-scale video+language omni-representation learning.
Ranked #1 on
Video Question Answering
on Howto100M-QA
BachGAN: High-Resolution Image Synthesis from Salient Object Layout
We propose a new task towards more practical application for image generation - high-quality image synthesis from salient object layout.
VIOLIN: A Large-Scale Dataset for Video-and-Language Inference
We introduce a new task, Video-and-Language Inference, for joint multimodal understanding of video and text.
TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval
The queries are also labeled with query types that indicate whether each of them is more related to video or subtitle or both, allowing for in-depth analysis of the dataset and the methods that built on top of it.
Ranked #2 on
Video Retrieval
on TVR
UNITER: UNiversal Image-TExt Representation Learning
Different from previous work that applies joint random masking to both modalities, we use conditional masking on pre-training tasks (i. e., masked language/region modeling is conditioned on full observation of image/text).
Ranked #2 on
Visual Question Answering
on VCR (Q-A) test
UNITER: Learning UNiversal Image-TExt Representations
Joint image-text embedding is the bedrock for most Vision-and-Language (V+L) tasks, where multimodality inputs are jointly processed for visual and textual understanding.
TVQA+: Spatio-Temporal Grounding for Video Question Answering
We present the task of Spatio-Temporal Video Question Answering, which requires intelligent systems to simultaneously retrieve relevant moments and detect referenced visual concepts (people and objects) to answer natural language questions about videos.
Ranked #4 on
Video Question Answering
on TVQA
Multi-Target Embodied Question Answering
To address this, we propose a modular architecture composed of a program generator, a controller, a navigator, and a VQA module.
Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout
Next, we apply semi-supervised learning (via back-translation) on these dropped-out environments to generate new paths and instructions.
Ranked #1 on
Vision-Language Navigation
on Room2Room
TVQA: Localized, Compositional Video Question Answering
Recent years have witnessed an increasing interest in image-based question-answering (QA) tasks.
Ranked #3 on
Video Question Answering
on SUTD-TrafficQA
MAttNet: Modular Attention Network for Referring Expression Comprehension
In this paper, we address referring expression comprehension: localizing an image region described by a natural language expression.
Ranked #12 on
Referring Expression Segmentation
on RefCOCO+ testA
A unified framework for manifold landmarking
Active manifold learning aims to select and label representative landmarks on a manifold from a given set of samples to improve semi-supervised manifold learning.
Hierarchically-Attentive RNN for Album Summarization and Storytelling
For this task, we make use of the Visual Storytelling dataset and a model composed of three hierarchically-attentive Recurrent Neural Nets (RNNs) to: encode the album photos, select representative (summary) photos, and compose the story.
A Joint Speaker-Listener-Reinforcer Model for Referring Expressions
The speaker generates referring expressions, the listener comprehends referring expressions, and the reinforcer introduces a reward function to guide sampling of more discriminative expressions.
Detailed Garment Recovery from a Single-View Image
Most recent garment capturing techniques rely on acquiring multiple views of clothing, which may not always be readily available, especially in the case of pre-existing photographs from the web.
Modeling Context in Referring Expressions
Humans refer to objects in their environments all the time, especially in dialogue with other people.
Visual Madlibs: Fill in the Blank Description Generation and Question Answering
In this paper, we introduce a new dataset consisting of 360, 001 focused natural language descriptions for 10, 738 images.
Visual Madlibs: Fill in the blank Image Generation and Question Answering
In this paper, we introduce a new dataset consisting of 360, 001 focused natural language descriptions for 10, 738 images.
