Search Results for author:
Found 25 papers, 18 papers with code
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
We present Unified-IO 2, the first autoregressive multimodal model that is capable of understanding and generating image, text, audio, and action.
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks
We propose Unified-IO, a model that performs a large variety of AI tasks spanning classical computer vision tasks, including pose estimation, object detection, depth estimation and image generation, vision-and-language tasks such as region captioning and referring expression, to natural language processing tasks such as question answering and paraphrasing.
Ranked #1 on
Object Segmentation
on GRIT
ASC me to Do Anything: Multi-task Training for Embodied AI
We propose Atomic Skill Completion (ASC), an approach for multi-task training for Embodied AI, where a set of atomic skills shared across multiple tasks are composed together to perform the tasks.
MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound
Given a video, we replace snippets of text and audio with a MASK token; the model learns by choosing the correct masked-out snippet.
Ranked #6 on
Action Classification
on Kinetics-600
(using extra training data)
Container: Context Aggregation Networks
Convolutional neural networks (CNNs) are ubiquitous in computer vision, with a myriad of effective and efficient variations.
A Simple Long-Tailed Recognition Baseline via Vision-Language Model
Recent advances in large-scale contrastive visual-language pretraining shed light on a new pathway for visual recognition.
Ranked #4 on
Long-tail Learning
on Places-LT
(using extra training data)
Container: Context Aggregation Network
Convolutional neural networks (CNNs) are ubiquitous in computer vision, with a myriad of effective and efficient variations.
Ranked #465 on
Image Classification
on ImageNet
Multi-Modal Answer Validation for Knowledge-Based VQA
Instead of searching for the answer in a vast collection of often irrelevant facts as most existing approaches do, MAVEx aims to learn how to extract relevant knowledge from noisy sources, which knowledge source to trust for each answer candidate, and how to validate the candidate using that source.
Transferable Feature Learning on Graphs Across Visual Domains
In this paper, we propose a novel Transferable Feature Learning approach on Graphs (TFLG) for unsupervised adversarial domain adaptation, which jointly incorporates sample-level and class-level structure information across two domains.
X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers
X-LXMERT's image generation capabilities rival state of the art generative models while its question answering and captioning abilities remains comparable to LXMERT.
Dialog without Dialog Data: Learning Visual Dialog Agents from VQA Data
Can we develop visually grounded dialog agents that can efficiently adapt to new tasks without forgetting how to talk to people?
Spatially Aware Multimodal Transformers for TextVQA
Further, each head in our multi-head self-attention layer focuses on a different subset of relations.
12-in-1: Multi-Task Vision and Language Representation Learning
Much of vision-and-language research focuses on a small but diverse set of independent tasks and supporting datasets often studied in isolation; however, the visually-grounded language understanding skills required for success at these tasks overlap significantly.
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language.
Ranked #5 on
Referring Expression Comprehension
on Talk2Car
Emergence of Compositional Language with Deep Generational Transmission
In this paper, we introduce these cultural evolutionary dynamics into language emergence by periodically replacing agents in a population to create a knowledge gap, implicitly inducing cultural transmission of language.
Self-Monitoring Navigation Agent via Auxiliary Progress Estimation
The Vision-and-Language Navigation (VLN) task entails an agent following navigational instruction in photo-realistic unknown environments.
Ranked #115 on
Vision and Language Navigation
on VLN Challenge
Natural Language Visual Grounding
Vision and Language Navigation
+2
Visual Curiosity: Learning to Ask Questions to Learn Visual Recognition
Our question generation policy generalizes to new environments and a new pair of eyes, i. e., new visual system.
Graph R-CNN for Scene Graph Generation
We propose a novel scene graph generation model called Graph R-CNN, that is both effective and efficient at detecting objects and their relations in images.
Ranked #12 on
Scene Graph Generation
on Visual Genome
Neural Baby Talk
We introduce a novel framework for image captioning that can produce natural language explicitly grounded in entities that object detectors find in the image.
Best of Both Worlds: Transferring Knowledge from Discriminative Learning to a Generative Visual Dialog Model
In contrast, discriminative dialog models (D) that are trained to rank a list of candidate human responses outperform their generative counterparts; in terms of automatic metrics, diversity, and informativeness of the responses.
Ranked #8 on
Visual Dialog
on VisDial v0.9 val
ParlAI: A Dialog Research Software Platform
We introduce ParlAI (pronounced "par-lay"), an open-source software platform for dialog research implemented in Python, available at http://parl. ai.
Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning
The model decides whether to attend to the image and where, in order to extract meaningful information for sequential word generation.
Hierarchical Question-Image Co-Attention for Visual Question Answering
In addition, our model reasons about the question (and consequently the image via the co-attention mechanism) in a hierarchical fashion via a novel 1-dimensional convolution neural networks (CNN).
Ranked #3 on
Visual Question Answering (VQA)
on VQA v1 test-std
Human Action Segmentation With Hierarchical Supervoxel Consistency
Detailed analysis of human action, such as action classification, detection and localization has received increasing attention from the community; datasets like JHMDB have made it plausible to conduct studies analyzing the impact that such deeper information has on the greater action understanding problem.
VQA: Visual Question Answering
Given an image and a natural language question about the image, the task is to provide an accurate natural language answer.
