Search Results for author:
Found 48 papers, 27 papers with code
Linking WordNet to 3D Shapes
We describe a project to link the Princeton WordNet to 3D representations of real objects and scenes.
Evaluating 3D Shape Analysis Methods for Robustness to Rotation Invariance
Our study, on a synthetic dataset of 3D scenes where objects instances occur in different orientations, reveals that deep learning-based rotation invariant methods are effective for relatively easy settings with easy-to-distinguish pairs.
Reduce, Reuse, Recycle: Modular Multi-Object Navigation
We systematically investigate the inherent modularity of this task by dividing our approach to contain four modules: (a) an object detection module trained to identify objects from RGB images, (b) a map building module to build a semantic map of the observed objects, (c) an exploration module enabling the agent to explore its surroundings, and finally (d) a navigation module to move to identified target objects.
OPDMulti: Openable Part Detection for Multiple Objects
We then address this more challenging scenario with OPDFormer: a part-aware transformer architecture.
Emergence of Maps in the Memories of Blind Navigation Agents
A positive answer to this question would (a) explain the surprising phenomenon in recent literature of ostensibly map-free neural-networks achieving strong performance, and (b) strengthen the evidence of mapping as a fundamental mechanism for navigation by intelligent embodied agents, whether they be biological or artificial.
Retrospectives on the Embodied AI Workshop
We present a retrospective on the state of Embodied AI research.
Habitat-Matterport 3D Semantics Dataset
The scale, quality, and diversity of object annotations far exceed those of prior datasets.
Articulated 3D Human-Object Interactions from RGB Videos: An Empirical Analysis of Approaches and Challenges
Human-object interactions with articulated objects are common in everyday life.
OPD: Single-view 3D Openable Part Detection
The input is a single image of an object, and as output we detect what parts of the object can open, and the motion parameters describing the articulation of each openable part.
Learning Body-Aware 3D Shape Generative Models
The body-shape-conditioned models produce chairs which will be comfortable for a person with the given body shape; the pose-conditioned models produce chairs which accommodate the given sitting pose.
Roominoes: Generating Novel 3D Floor Plans From Existing 3D Rooms
We then discuss different strategies for solving the problem, and design two representative pipelines: one uses available 2D floor plans to guide selection and deformation of 3D rooms; the other learns to retrieve a set of compatible 3D rooms and combine them into novel layouts.
Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agents
We show that the emergent communication can be grounded to the agent observations and the spatial structure of the 3D environment.
Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI
When compared to existing photorealistic 3D datasets such as Replica, MP3D, Gibson, and ScanNet, images rendered from HM3D have 20 - 85% higher visual fidelity w. r. t.
D3D-HOI: Dynamic 3D Human-Object Interactions from Videos
We evaluate this approach on our dataset, demonstrating that human-object relations can significantly reduce the ambiguity of articulated object reconstructions from challenging real-world videos.
Habitat 2.0: Training Home Assistants to Rearrange their Habitat
We introduce Habitat 2. 0 (H2. 0), a simulation platform for training virtual robots in interactive 3D environments and complex physics-enabled scenarios.
Mirror3D: Depth Refinement for Mirror Surfaces
Despite recent progress in depth sensing and 3D reconstruction, mirror surfaces are a significant source of errors.
Plan2Scene: Converting Floorplans to 3D Scenes
We address the task of converting a floorplan and a set of associated photos of a residence into a textured 3D mesh model, a task which we call Plan2Scene.
Ranked #1 on
Plan2Scene
on Rent3D++
Large Batch Simulation for Deep Reinforcement Learning
We accelerate deep reinforcement learning-based training in visually complex 3D environments by two orders of magnitude over prior work, realizing end-to-end training speeds of over 19, 000 frames of experience per second on a single GPU and up to 72, 000 frames per second on a single eight-GPU machine.
LayoutGMN: Neural Graph Matching for Structural Layout Similarity
In particular, retrieval results by our network better match human judgement of structural layout similarity compared to both IoUs and other baselines including a state-of-the-art method based on graph neural networks and image convolution.
MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigation
We propose the multiON task, which requires navigation to an episode-specific sequence of objects in a realistic environment.
Rearrangement: A Challenge for Embodied AI
In the rearrangement task, the goal is to bring a given physical environment into a specified state.
MCMI: Multi-Cycle Image Translation with Mutual Information Constraints
We present a mutual information-based framework for unsupervised image-to-image translation.
ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects
In particular, the agent is initialized at a random location and pose in an environment and asked to find an instance of an object category, e. g., find a chair, by navigating to it.
Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance?
Second, we investigate the sim2real predictivity of Habitat-Sim for PointGoal navigation.
DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames
We leverage this scaling to train an agent for 2. 5 Billion steps of experience (the equivalent of 80 years of human experience) -- over 6 months of GPU-time training in under 3 days of wall-clock time with 64 GPUs.
Ranked #1 on
PointGoal Navigation
on Gibson PointGoal Navigation
Relational Graph Learning for Crowd Navigation
We present a relational graph learning approach for robotic crowd navigation using model-based deep reinforcement learning that plans actions by looking into the future.
The Replica Dataset: A Digital Replica of Indoor Spaces
We introduce Replica, a dataset of 18 highly photo-realistic 3D indoor scene reconstructions at room and building scale.
Habitat: A Platform for Embodied AI Research
We present Habitat, a platform for research in embodied artificial intelligence (AI).
Ranked #2 on
PointGoal Navigation
on Gibson PointGoal Navigation
Hierarchy Denoising Recursive Autoencoders for 3D Scene Layout Prediction
Indoor scenes exhibit rich hierarchical structure in 3D object layouts.
Scan2CAD: Learning CAD Model Alignment in RGB-D Scans
For a 3D reconstruction of an indoor scene, our method takes as input a set of CAD models, and predicts a 9DoF pose that aligns each model to the underlying scan geometry.
Ranked #1 on
3D Reconstruction
on Scan2CAD
On Evaluation of Embodied Navigation Agents
Skillful mobile operation in three-dimensional environments is a primary topic of study in Artificial Intelligence.
Im2Pano3D: Extrapolating 360° Structure and Semantics Beyond the Field of View
We present Im2Pano3D, a convolutional neural network that generates a dense prediction of 3D structure and a probability distribution of semantic labels for a full 360 panoramic view of an indoor scene when given only a partial observation ( <=50%) in the form of an RGB-D image.
Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings
To this end, we first learn joint embeddings of freeform text descriptions and colored 3D shapes.
Im2Pano3D: Extrapolating 360 Structure and Semantics Beyond the Field of View
We present Im2Pano3D, a convolutional neural network that generates a dense prediction of 3D structure and a probability distribution of semantic labels for a full 360 panoramic view of an indoor scene when given only a partial observation (<= 50%) in the form of an RGB-D image.
MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments
We present MINOS, a simulator designed to support the development of multisensory models for goal-directed navigation in complex indoor environments.
Large-Scale 3D Shape Reconstruction and Segmentation from ShapeNet Core55
We introduce a large-scale 3D shape understanding benchmark using data and annotation from ShapeNet 3D object database.
Matterport3D: Learning from RGB-D Data in Indoor Environments
Access to large, diverse RGB-D datasets is critical for training RGB-D scene understanding algorithms.
Learning Where to Look: Data-Driven Viewpoint Set Selection for 3D Scenes
We provide a search algorithm that generates a sampling of likely candidate views according to the example distribution, and a set selection algorithm that chooses a subset of the candidates that jointly cover the example distribution.
SceneSeer: 3D Scene Design with Natural Language
We present SceneSeer: an interactive text to 3D scene generation system that allows a user to design 3D scenes using natural language.
SceneSuggest: Context-driven 3D Scene Design
We present SceneSuggest: an interactive 3D scene design system providing context-driven suggestions for 3D model retrieval and placement.
Graphics Human-Computer Interaction
ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
A key requirement for leveraging supervised deep learning methods is the availability of large, labeled datasets.
Ranked #9 on
Semantic Segmentation
on ScanNetV2
Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks
One of the bottlenecks in training for better representations is the amount of available per-pixel ground truth data that is required for core scene understanding tasks such as semantic segmentation, normal prediction, and object edge detection.
Semantic Scene Completion from a Single Depth Image
This paper focuses on semantic scene completion, a task for producing a complete 3D voxel representation of volumetric occupancy and semantic labels for a scene from a single-view depth map observation.
Ranked #11 on
3D Semantic Scene Completion
on SemanticKITTI
ShapeNet: An Information-Rich 3D Model Repository
We present ShapeNet: a richly-annotated, large-scale repository of shapes represented by 3D CAD models of objects.
Text to 3D Scene Generation with Rich Lexical Grounding
The ability to map descriptions of scenes to 3D geometric representations has many applications in areas such as art, education, and robotics.
