Search Results for author:
Found 53 papers, 18 papers with code
Retouchdown: Releasing Touchdown on StreetLearn as a Public Resource for Language Grounding Tasks in Street View
These have been added to the StreetLearn dataset and can be obtained via the same process as used previously for StreetLearn.
Multi-Level Gazetteer-Free Geocoding
We present a multi-level geocoding model (MLG) that learns to associate texts to geographic coordinates.
MURAL: Multimodal, Multitask Representations Across Languages
Both image-caption pairs and translation pairs provide the means to learn deep representations of and connections between languages.
Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation
With extensive experimentation and human evaluation on a range of model configurations (LLM, VQA, and T2I), we empirically demonstrate that DSG addresses the challenges noted above.
Gaussian Process Probes (GPP) for Uncertainty-Aware Probing
Our experiments show it can (1) probe a model's representations of concepts even with a very small number of examples, (2) accurately measure both epistemic uncertainty (how confident the probe is) and aleatory uncertainty (how fuzzy the concepts are to the model), and (3) detect out of distribution data using those uncertainty measures as well as classic methods do.
CoBIT: A Contrastive Bi-directional Image-Text Generation Model
The field of vision and language has witnessed a proliferation of pre-trained foundation models.
Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting
Through extensive human evaluation on EditBench, we find that object-masking during training leads to across-the-board improvements in text-image alignment -- such that Imagen Editor is preferred over DALL-E 2 and Stable Diffusion -- and, as a cohort, these models are better at object-rendering than text-rendering, and handle material/color/size attributes better than count/shape attributes.
Underspecification in Scene Description-to-Depiction Tasks
Questions regarding implicitness, ambiguity and underspecification are crucial for understanding the task validity and ethical concerns of multimodal image+text systems, yet have received little attention to date.
A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learning
Recent studies in Vision-and-Language Navigation (VLN) train RL agents to execute natural-language navigation instructions in photorealistic environments, as a step towards robots that can follow human instructions.
Ranked #1 on
Vision and Language Navigation
on RxR
(using extra training data)
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
We present the Pathways Autoregressive Text-to-Image (Parti) model, which generates high-fidelity photorealistic images and supports content-rich synthesis involving complex compositions and world knowledge.
Ranked #1 on
Text-to-Image Generation
on LAION COCO
Simple and Effective Synthesis of Indoor 3D Scenes
We study the problem of synthesizing immersive 3D indoor scenes from one or more images.
Less is More: Generating Grounded Navigation Instructions from Landmarks
We study the automatic generation of navigation instructions from 360-degree images captured on indoor routes.
Vector-quantized Image Modeling with Improved VQGAN
Motivated by this success, we explore a Vector-quantized Image Modeling (VIM) approach that involves pretraining a Transformer to predict rasterized image tokens autoregressively.
MURAL: Multimodal, Multitask Retrieval Across Languages
Both image-caption pairs and translation pairs provide the means to learn deep representations of and connections between languages.
Ranked #1 on
Semantic Image-Text Similarity
on CxC
Pathdreamer: A World Model for Indoor Navigation
People navigating in unfamiliar buildings take advantage of myriad visual, spatial and semantic cues to efficiently achieve their navigation goals.
Talk, Don't Write: A Study of Direct Speech-Based Image Retrieval
Speech-based image retrieval has been studied as a proxy for joint representation learning, usually without emphasis on retrieval itself.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
PanGEA: The Panoramic Graph Environment Annotation Toolkit
PanGEA, the Panoramic Graph Environment Annotation toolkit, is a lightweight toolkit for collecting speech and text annotations in photo-realistic 3D environments.
On the Evaluation of Vision-and-Language Navigation Instructions
Vision-and-Language Navigation wayfinding agents can be enhanced by exploiting automatically generated navigation instructions.
Cross-Modal Contrastive Learning for Text-to-Image Generation
The quality of XMC-GAN's output is a major step up from previous models, as we show on three challenging datasets.
Ranked #26 on
Text-to-Image Generation
on COCO
(using extra training data)
Text-to-Image Generation Grounded by Fine-Grained User Attention
Localized Narratives is a dataset with detailed natural language descriptions of images paired with mouse traces that provide a sparse, fine-grained visual grounding for phrases.
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding
We introduce Room-Across-Room (RxR), a new Vision-and-Language Navigation (VLN) dataset.
Ranked #5 on
Vision and Language Navigation
on RxR
Spatial Language Representation with Multi-Level Geocoding
We present a multi-level geocoding model (MLG) that learns to associate texts to geographic locations.
Text Classification with Few Examples using Controlled Generalization
Training data for text classification is often limited in practice, especially for applications with many output classes or involving many related classification problems.
Mapping Natural Language Instructions to Mobile UI Action Sequences
We present a new problem: grounding natural language instructions to mobile user interface actions, and create three new datasets for it.
Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO
By supporting multi-modal retrieval training and evaluation, image captioning datasets have spurred remarkable progress on representation learning.
Retouchdown: Adding Touchdown to StreetLearn as a Shareable Resource for Language Grounding Tasks in Street View
These have been added to the StreetLearn dataset and can be obtained via the same process as used previously for StreetLearn.
Ranked #7 on
Vision and Language Navigation
on Touchdown Dataset
Extending Machine Language Models toward Human-Level Language Understanding
We take language to be a part of a system for understanding and communicating about situations.
Learning Dense Representations for Entity Retrieval
We show that it is feasible to perform entity linking by training a dual encoder (two-tower) model that encodes mentions and entities in the same dense vector space, where candidate entities are retrieved by approximate nearest neighbor search.
Large-scale representation learning from visually grounded untranscribed speech
Systems that can associate images with their spoken audio captions are an important step towards visually grounded language learning.
PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification
Most existing work on adversarial data generation focuses on English.
Transferable Representation Learning in Vision-and-Language Navigation
Vision-and-Language Navigation (VLN) tasks such as Room-to-Room (R2R) require machine agents to interpret natural language instructions and learn to act in visually realistic environments to achieve navigation goals.
Ranked #115 on
Vision and Language Navigation
on VLN Challenge
General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping
We address fundamental flaws in previously used metrics and show how Dynamic Time Warping (DTW), a long known method of measuring similarity between two time series, can be used for evaluation of navigation agents.
Multi-modal Discriminative Model for Vision-and-Language Navigation
Vision-and-Language Navigation (VLN) is a natural language grounding task where agents have to interpret natural language instructions in the context of visual scenes in a dynamic environment to achieve prescribed navigation goals.
Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation
We also show that the existing paths in the dataset are not ideal for evaluating instruction following because they are direct-to-goal shortest paths.
PAWS: Paraphrase Adversaries from Word Scrambling
Existing paraphrase identification datasets lack sentence pairs that have high lexical overlap without being paraphrases.
A task in a suit and a tie: paraphrase generation with semantic augmentation
Paraphrasing is rooted in semantics.
Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns
Coreference resolution is an important task for natural language understanding, and the resolution of ambiguous pronouns a longstanding challenge.
A Fast, Compact, Accurate Model for Language Identification of Codemixed Text
We address fine-grained multilingual language identification: providing a language code for every token in a sentence, including codemixed text containing multiple languages.
Learning To Split and Rephrase From Wikipedia Edit History
Split and rephrase is the task of breaking down a sentence into shorter ones that together convey the same meaning.
Fill it up: Exploiting partial dependency annotations in a minimum spanning tree parser
Unsupervised models of dependency parsing typically require large amounts of clean, unlabeled data plus gold-standard part-of-speech tags.
A framework for (under)specifying dependency syntax without overloading annotators
We introduce a framework for lightweight dependency syntax annotation.
