Search Results for author:
Found 24 papers, 19 papers with code
VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generation
To support diverse tasks and complex instructions, we employ a curriculum learning strategy: first aligning the MLLM and video diffusion model with large-scale instructional image editing data, followed by end-to-end fine-tuning on high-quality multitask video data.
Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheel
In this paper, we introduce a Self-Refining Data Flywheel (SRDF) that generates high-quality and large-scale navigational instruction-trajectory pairs by iteratively refining the data pool through the collaboration between two models, the instruction generator and the navigator, without any human-in-the-loop annotation.
SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Experts
The academic field of learning instruction-guided visual navigation can be generally categorized into high-level category-specific search and low-level language-guided navigation, depending on the granularity of language instruction, in which the former emphasizes the exploration process, while the latter concentrates on following detailed textual commands.
Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splats
Unlike previous feed-forward models that are limited to processing 1~4 input images and can only reconstruct a small portion of a large scene, Long-LRM reconstructs the entire scene in a single feed-forward step.
Progressive Autoregressive Video Diffusion Models
Current frontier video diffusion models have demonstrated remarkable results at generating high-quality videos.
NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Models
Capitalizing on the remarkable advancements in Large Language Models (LLMs), there is a burgeoning initiative to harness LLMs for instruction following robotic navigation.
Augmented Commonsense Knowledge for Remote Object Grounding
To address enhancing representation, we propose an augmented commonsense knowledge model (ACK) to leverage commonsense information as a spatio-temporal knowledge graph for improving agent navigation.
NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation
Vision-and-language navigation (VLN) stands as a key research problem of Embodied AI, aiming at enabling agents to navigate in unseen environments following linguistic instructions.
LRM: Large Reconstruction Model for Single Image to 3D
We propose the first Large Reconstruction Model (LRM) that predicts the 3D model of an object from a single input image within just 5 seconds.
Scaling Data Generation in Vision-and-Language Navigation
Recent research in language-guided visual navigation has demonstrated a significant demand for the diversity of traversable environments and the quantity of supervision for training generalizable agents.
Learning Navigational Visual Representations with Semantic Map Supervision
Being able to perceive the semantics and the spatial structure of the environment is essential for visual navigation of a household robot.
NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models
Trained with an unprecedented scale of data, large language models (LLMs) like ChatGPT and GPT-4 exhibit the emergence of significant reasoning abilities from model scaling.
Bi-directional Training for Composed Image Retrieval via Text Prompt Learning
Composed image retrieval searches for a target image based on a multi-modal user query comprised of a reference image and modification text describing the desired changes.
Ranked #7 on
Image Retrieval
on Fashion IQ
HOP+: History-enhanced and Order-aware Pre-training for Vision-and-Language Navigation
To address these problems, we present a history-enhanced and order-aware pre-training with the complementing fine-tuning paradigm (HOP+) for VLN.
1st Place Solutions for RxR-Habitat Vision-and-Language Navigation Competition (CVPR 2022)
Our model consists of three modules: the candidate waypoints predictor (CWP), the history enhanced planner and the tryout controller.
HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigation
Pre-training has been adopted in a few of recent works for Vision-and-Language Navigation (VLN).
Ranked #5 on
Visual Navigation
on R2R
Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigation
To bridge the discrete-to-continuous gap, we propose a predictor to generate a set of candidate waypoints during navigation, so that agents designed with high-level actions can be transferred to and trained in continuous environments.
VLN BERT: A Recurrent Vision-and-Language BERT for Navigation
In this paper we propose a recurrent BERT model that is time-aware for use in VLN.
Learning structure-aware semantic segmentation with image-level supervision
Within this pipeline, the class activation map (CAM) is obtained and further processed to serve as a pseudo label to train the semantic segmentation model in a fully-supervised manner.
The Road to Know-Where: An Object-and-Room Informed Sequential BERT for Indoor Vision-Language Navigation
Vision-and-Language Navigation (VLN) requires an agent to find a path to a remote location on the basis of natural-language instructions and a set of photo-realistic panoramas.
A Recurrent Vision-and-Language BERT for Navigation
In this paper we propose a recurrent BERT model that is time-aware for use in VLN.
Ranked #8 on
Visual Navigation
on R2R
Language and Visual Entity Relationship Graph for Agent Navigation
From both the textual and visual perspectives, we find that the relationships among the scene, its objects, and directional clues are essential for the agent to interpret complex instructions and correctly perceive the environment.
Sub-Instruction Aware Vision-and-Language Navigation
Vision-and-language navigation requires an agent to navigate through a real 3D environment following natural language instructions.
