Search Results for author:
Found 28 papers, 18 papers with code
Cut Your Losses in Large-Vocabulary Language Models
We implement a custom kernel that performs the matrix multiplications and the log-sum-exp reduction over the vocabulary in flash memory, making global memory consumption for the cross-entropy computation negligible.
Does Spatial Cognition Emerge in Frontier Models?
Results suggest that contemporary frontier models fall short of the spatial intelligence of animals, performing near chance level on a number of classic tests of animal cognition.
Emergence of Maps in the Memories of Blind Navigation Agents
A positive answer to this question would (a) explain the surprising phenomenon in recent literature of ostensibly map-free neural-networks achieving strong performance, and (b) strengthen the evidence of mapping as a fundamental mechanism for navigation by intelligent embodied agents, whether they be biological or artificial.
PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNav
We find that BC$\rightarrow$RL on human demonstrations outperforms BC$\rightarrow$RL on SP and FE trajectories, even when controlled for same BC-pretraining success on train, and even on a subset of val episodes where BC-pretraining success favors the SP or FE policies.
VER: Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied Rearrangement
Specifically, the Pick skill involves a robot picking an object from a table.
Is Mapping Necessary for Realistic PointGoal Navigation?
However, for PointNav in a realistic setting (RGB-D and actuation noise, no GPS+Compass), this is an open question; one we tackle in this paper.
Realistic PointGoal Navigation via Auxiliary Losses and Information Bottleneck
Under this setting, the agent incurs a penalty for using this privileged information, encouraging the agent to only leverage this information when it is crucial to learning.
Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI
When compared to existing photorealistic 3D datasets such as Replica, MP3D, Gibson, and ScanNet, images rendered from HM3D have 20 - 85% higher visual fidelity w. r. t.
Megaverse: Simulating Embodied Agents at One Million Experiences per Second
We present Megaverse, a new 3D simulation platform for reinforcement learning and embodied AI research.
Habitat 2.0: Training Home Assistants to Rearrange their Habitat
We introduce Habitat 2. 0 (H2. 0), a simulation platform for training virtual robots in interactive 3D environments and complex physics-enabled scenarios.
Auxiliary Tasks and Exploration Enable ObjectNav
We instead re-enable a generic learned agent by adding auxiliary learning tasks and an exploration reward.
Ranked #2 on
Robot Navigation
on Habitat 2020 Object Nav test-std
Large Batch Simulation for Deep Reinforcement Learning
We accelerate deep reinforcement learning-based training in visually complex 3D environments by two orders of magnitude over prior work, realizing end-to-end training speeds of over 19, 000 frames of experience per second on a single GPU and up to 72, 000 frames per second on a single eight-GPU machine.
Auxiliary Tasks and Exploration Enable ObjectGoal Navigation
We instead re-enable a generic learned agent by adding auxiliary learning tasks and an exploration reward.
THDA: Treasure Hunt Data Augmentation for Semantic Navigation
We show that this is a natural consequence of optimizing for the task metric (which in fact penalizes exploration), is enabled by powerful observation encoders, and is possible due to the finite set of training environment configurations.
How to Train PointGoal Navigation Agents on a (Sample and Compute) Budget
PointGoal navigation has seen significant recent interest and progress, spurred on by the Habitat platform and associated challenge.
Seeing the Un-Scene: Learning Amodal Semantic Maps for Room Navigation
We also demonstrate that reducing the task of room navigation to point navigation improves the performance further.
Auxiliary Tasks Speed Up Learning PointGoal Navigation
PointGoal Navigation is an embodied task that requires agents to navigate to a specified point in an unseen environment.
ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects
In particular, the agent is initialized at a random location and pose in an environment and asked to find an instance of an object category, e. g., find a chair, by navigating to it.
Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments – Extended Abstract
We develop a language-guided navigation task set in a continuous 3D environment where agents must execute low-level actions to follow natural language navigation directions.
Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments
We develop a language-guided navigation task set in a continuous 3D environment where agents must execute low-level actions to follow natural language navigation directions.
Analyzing Visual Representations in Embodied Navigation Tasks
Recent advances in deep reinforcement learning require a large amount of training data and generally result in representations that are often over specialized to the target task.
Insights on Visual Representations for Embodied Navigation Tasks
Surprisingly, we find that slight differences in task have no measurable effect on the visual representation for both SqueezeNet and ResNet architectures.
Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance?
Second, we investigate the sim2real predictivity of Habitat-Sim for PointGoal navigation.
DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames
We leverage this scaling to train an agent for 2. 5 Billion steps of experience (the equivalent of 80 years of human experience) -- over 6 months of GPU-time training in under 3 days of wall-clock time with 64 GPUs.
Ranked #1 on
PointGoal Navigation
on Gibson PointGoal Navigation
The Replica Dataset: A Digital Replica of Indoor Spaces
We introduce Replica, a dataset of 18 highly photo-realistic 3D indoor scene reconstructions at room and building scale.
Embodied Question Answering in Photorealistic Environments with Point Cloud Perception
To help bridge the gap between internet vision-style problems and the goal of vision for embodied perception we instantiate a large-scale navigation task -- Embodied Question Answering [1] in photo-realistic environments (Matterport 3D).
Habitat: A Platform for Embodied AI Research
We present Habitat, a platform for research in embodied artificial intelligence (AI).
Ranked #2 on
PointGoal Navigation
on Gibson PointGoal Navigation
Exploiting 2D Floorplan for Building-scale Panorama RGBD Alignment
To the best of our knowledge, we present the first effective system that utilizes a 2D floorplan image for building-scale 3D pointcloud alignment.
