Search Results for author:
Found 33 papers, 19 papers with code
Prompt Expansion for Adaptive Text-to-Image Generation
Text-to-image generation models are powerful but difficult to use.
Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation
With extensive experimentation and human evaluation on a range of model configurations (LLM, VQA, and T2I), we empirically demonstrate that DSG addresses the challenges noted above.
Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting
Through extensive human evaluation on EditBench, we find that object-masking during training leads to across-the-board improvements in text-image alignment -- such that Imagen Editor is preferred over DALL-E 2 and Stable Diffusion -- and, as a cohort, these models are better at object-rendering than text-rendering, and handle material/color/size attributes better than count/shape attributes.
Iterative Vision-and-Language Navigation
We present Iterative Vision-and-Language Navigation (IVLN), a paradigm for evaluating language-guided agents navigating in a persistent environment over time.
A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learning
Recent studies in Vision-and-Language Navigation (VLN) train RL agents to execute natural-language navigation instructions in photorealistic environments, as a step towards robots that can follow human instructions.
Ranked #1 on
Vision and Language Navigation
on RxR
(using extra training data)
Simple and Effective Synthesis of Indoor 3D Scenes
We study the problem of synthesizing immersive 3D indoor scenes from one or more images.
Less is More: Generating Grounded Navigation Instructions from Landmarks
We study the automatic generation of navigation instructions from 360-degree images captured on indoor routes.
Pathdreamer: A World Model for Indoor Navigation
People navigating in unfamiliar buildings take advantage of myriad visual, spatial and semantic cues to efficiently achieve their navigation goals.
PanGEA: The Panoramic Graph Environment Annotation Toolkit
PanGEA, the Panoramic Graph Environment Annotation toolkit, is a lightweight toolkit for collecting speech and text annotations in photo-realistic 3D environments.
On the Evaluation of Vision-and-Language Navigation Instructions
Vision-and-Language Navigation wayfinding agents can be enhanced by exploiting automatically generated navigation instructions.
Where Are You? Localization from Embodied Dialog
In this paper, we focus on the LED task -- providing a strong baseline model with detailed ablations characterizing both dataset biases and the importance of various modeling choices.
Sim-to-Real Transfer for Vision-and-Language Navigation
We study the challenging problem of releasing a robot in a previously unseen environment, and having it follow unconstrained natural language navigation instructions.
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding
We introduce Room-Across-Room (RxR), a new Vision-and-Language Navigation (VLN) dataset.
Ranked #5 on
Vision and Language Navigation
on RxR
Spatially Aware Multimodal Transformers for TextVQA
Further, each head in our multi-head self-attention layer focuses on a different subset of relations.
Extended Abstract: Improving Vision-and-Language Navigation with Image-Text Pairs from the Web
Following a navigation instruction such as 'Walk down the stairs and stop near the sofa' requires an agent to ground scene elements referenced via language (e. g.'stairs') to visual content in the environment (pixels corresponding to 'stairs').
Improving Vision-and-Language Navigation with Image-Text Pairs from the Web
Following a navigation instruction such as 'Walk down the stairs and stop at the brown sofa' requires embodied AI agents to ground scene elements referenced via language (e. g. 'stairs') to visual content in the environment (pixels corresponding to 'stairs').
Ranked #6 on
Vision and Language Navigation
on VLN Challenge
Chasing Ghosts: Instruction Following as Bayesian State Tracking
Our experiments show that our approach outperforms a strong LingUNet baseline when predicting the goal location on the map.
REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments
One of the long-term challenges of robotics is to enable robots to interact with humans in the visual world via natural language, as humans are visual animals that communicate through language.
nocaps: novel object captioning at scale
To encourage the development of image captioning models that can learn visual concepts from alternative data sources, such as object detection datasets, we present the first large-scale benchmark for this task.
Disfluency Detection using Auto-Correlational Neural Networks
In recent years, the natural language processing community has moved away from task-specific feature engineering, i. e., researchers discovering ad-hoc feature representations for various tasks, in favor of general-purpose methods that learn the input representation by themselves.
On Evaluation of Embodied Navigation Agents
Skillful mobile operation in three-dimensional environments is a primary topic of study in Artificial Intelligence.
Face-Cap: Image Captioning using Facial Expression Analysis
In this work, we present two variants of our Face-Cap model, which embed facial expression features in different ways, to generate image captions.
Connecting Language and Vision to Actions
A long-term goal of AI research is to build intelligent agents that can see the rich visual environment around us, communicate this understanding in natural language to humans and other agents, and act in a physical or embodied environment.
Predicting accuracy on large datasets from smaller pilot data
Because obtaining training data is often the most difficult part of an NLP or ML project, we develop methods for predicting how much data is required to achieve a desired test accuracy by extrapolating results from models trained on a small pilot training dataset.
Partially-Supervised Image Captioning
To address this problem, we teach image captioning models new visual concepts from labeled images and object detection datasets.
Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments
This is significant because a robot interpreting a natural-language navigation instruction on the basis of what it sees is carrying out a vision and language process that is similar to Visual Question Answering.
Ranked #10 on
Visual Navigation
on R2R
Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge
This paper presents a state-of-the-art model for visual question answering (VQA), which won the first place in the 2017 VQA Challenge.
Ranked #30 on
Visual Question Answering (VQA)
on VQA v2 test-std
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Top-down visual attention mechanisms have been used extensively in image captioning and visual question answering (VQA) to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning.
Ranked #29 on
Visual Question Answering (VQA)
on VQA v2 test-std
Guided Open Vocabulary Image Captioning with Constrained Beam Search
Existing image captioning models do not generalize well to out-of-domain images containing novel scenes or objects.
SPICE: Semantic Propositional Image Caption Evaluation
There is considerable interest in the task of automatically generating image captions.
On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimization
Some recent works in machine learning and computer vision involve the solution of a bi-level optimization problem.
Discriminative Hierarchical Rank Pooling for Activity Recognition
We present hierarchical rank pooling, a video sequence encoding method for activity recognition.
