Search Results for author:
Found 59 papers, 23 papers with code
HSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generation
We present HSM, a hierarchical framework for indoor scene generation with dense object arrangements across spatial scales.
NuiScene: Exploring Efficient Generation of Unbounded Outdoor Scenes
In this paper, we explore the task of generating expansive outdoor scenes, ranging from castles to high-rises.
SceneEval: Evaluating Semantic Coherence in Text-Conditioned 3D Indoor Scene Synthesis
SceneEval includes metrics for both explicit user requirements, such as the presence of specific objects and their attributes described in the input text, and implicit expectations, like the absence of object collisions, providing a comprehensive assessment of scene quality.
Learning Object Placement Programs for Indoor Scene Synthesis with Iterative Self Training
Empirical observations show that current systems tend to produce incomplete next object location distributions.
Enhancing DNA Foundation Models to Address Masking Inefficiencies
Masked language modelling (MLM) as a pretraining objective has been widely adopted in genomic sequence modelling.
Semantic Mapping in Indoor Embodied AI -- A Comprehensive Survey and Future Directions
Among many skills that the agents need to possess, building and maintaining a semantic map of the environment is most crucial in long-horizon tasks.
ViGiL3D: A Linguistically Diverse Dataset for 3D Visual Grounding
3D visual grounding (3DVG) involves localizing entities in a 3D scene referred to by natural language text.
Diorama: Unleashing Zero-shot Single-view 3D Scene Modeling
Reconstructing structured 3D scenes from RGB images using CAD objects unlocks efficient and compact scene representations that maintain compositionality and interactability.
Zero-shot Object-Centric Instruction Following: Integrating Foundation Models with Traditional Navigation
Large scale scenes such as multifloor homes can be robustly and efficiently mapped with a 3D graph of landmarks estimated jointly with robot poses in a factor graph, a technique commonly used in commercial robots such as drones and robot vacuums.
SINGAPO: Single Image Controlled Generation of Articulated Parts in Objects
To capture the ambiguity in part shape and motion posed by a single view of the object, we design a diffusion model that learns the plausible variations of objects in terms of geometry and kinematics.
S2O: Static to Openable Enhancement for Articulated 3D Objects
Despite much progress in large 3D datasets there are currently few interactive 3D object datasets, and their scale is limited due to the manual effort required in their construction.
An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion
We introduce a new approach for generating realistic 3D models with UV maps through a representation termed "Object Images."
BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversity
We propose three benchmark experiments to demonstrate the impact of the multi-modal data types on the classification and clustering accuracy.
Duoduo CLIP: Efficient 3D Understanding with Multi-View Images
Our model also achieves better performance in more fine-grained text to shape retrieval, demonstrating better text-and-shape alignment than point cloud based models.
CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale
Measuring biodiversity is crucial for understanding ecosystem health.
When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models
Hence, with this paper, we aim to chart a course for future research that explores and expands the capabilities of 3D-LLMs in understanding and interacting with the complex 3D world.
Text-to-3D Shape Generation
Recent years have seen an explosion of work and interest in text-to-3D shape generation.
Generalizing Single-View 3D Shape Retrieval to Occlusions and Unseen Objects
Single-view 3D shape retrieval is a challenging task that is increasingly important with the growth of available 3D data.
BarcodeBERT: Transformers for Biodiversity Analysis
We compared the performance of BarcodeBERT on taxonomic identification tasks against a spectrum of machine learning approaches including supervised training of classical neural architectures and fine-tuning of general DNA foundation models.
Multi3DRefer: Grounding Text Description to Multiple 3D Objects
We introduce the task of localizing a flexible number of objects in real-world 3D scenes using natural language descriptions.
A Step Towards Worldwide Biodiversity Assessment: The BIOSCAN-1M Insect Dataset
In an effort to catalog insect biodiversity, we propose a new large dataset of hand-labelled insect images, the BIOSCAN-Insect Dataset.
Ranked #1 on
Classification
on BIOSCAN_1M_Insect Dataset
Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation
Surprisingly, we observe that agents trained on just 122 scenes from our dataset outperform agents trained on 10, 000 scenes from the ProcTHOR-10K dataset in terms of zero-shot generalization in real-world scanned environments.
Evaluating 3D Shape Analysis Methods for Robustness to Rotation Invariance
Our study, on a synthetic dataset of 3D scenes where objects instances occur in different orientations, reveals that deep learning-based rotation invariant methods are effective for relatively easy settings with easy-to-distinguish pairs.
Exploiting Proximity-Aware Tasks for Embodied Social Navigation
In this paper, we propose an end-to-end architecture that exploits Proximity-Aware Tasks (referred as to Risk and Proximity Compass) to inject into a reinforcement learning navigation policy the ability to infer common-sense social behaviors.
UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Grounding
Performing 3D dense captioning and visual grounding requires a common and shared understanding of the underlying multimodal relationships.
Retrospectives on the Embodied AI Workshop
We present a retrospective on the state of Embodied AI research.
Understanding Pure CLIP Guidance for Voxel Grid NeRF Models
We illustrate how different image-based augmentations prevent the adversarial generation problem, and how the generated results are impacted.
Articulated 3D Human-Object Interactions from RGB Videos: An Empirical Analysis of Approaches and Challenges
Human-object interactions with articulated objects are common in everyday life.
OPD: Single-view 3D Openable Part Detection
The input is a single image of an object, and as output we detect what parts of the object can open, and the motion parameters describing the articulation of each openable part.
TriCoLo: Trimodal Contrastive Loss for Text to Shape Retrieval
Text-to-shape retrieval is an increasingly relevant problem with the growth of 3D shape data.
Roominoes: Generating Novel 3D Floor Plans From Existing 3D Rooms
We then discuss different strategies for solving the problem, and design two representative pipelines: one uses available 2D floor plans to guide selection and deformation of 3D rooms; the other learns to retrieve a set of compatible 3D rooms and combine them into novel layouts.
D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Grounding
Our D3Net unifies dense captioning and visual grounding in 3D in a self-critical manner.
Ranked #8 on
3D dense captioning
on Nr3D
Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agents
We show that the emergent communication can be grounded to the agent observations and the spatial structure of the 3D environment.
Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environments
Prior work supervises the agent with actions based on the shortest path from the agent's location to the goal, but such goal-oriented supervision is often not in alignment with the instruction.
Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI
When compared to existing photorealistic 3D datasets such as Replica, MP3D, Gibson, and ScanNet, images rendered from HM3D have 20 - 85% higher visual fidelity w. r. t.
Mirror3D: Depth Refinement for Mirror Surfaces
Despite recent progress in depth sensing and 3D reconstruction, mirror surfaces are a significant source of errors.
Plan2Scene: Converting Floorplans to 3D Scenes
We address the task of converting a floorplan and a set of associated photos of a residence into a textured 3D mesh model, a task which we call Plan2Scene.
Ranked #1 on
Plan2Scene
on Rent3D++
MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigation
We propose the multiON task, which requires navigation to an episode-specific sequence of objects in a realistic environment.
Scan2Cap: Context-aware Dense Captioning in RGB-D Scans
We introduce the task of dense captioning in 3D scans from commodity RGB-D sensors.
Ranked #9 on
3D dense captioning
on ScanRefer Dataset
Rearrangement: A Challenge for Embodied AI
In the rearrangement task, the goal is to bring a given physical environment into a specified state.
SAPIEN: A SimulAted Part-based Interactive ENvironment
To achieve this task, a simulated environment with physically realistic simulation, sufficient articulated objects, and transferability to the real robot is indispensable.
ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language
We introduce the task of 3D object localization in RGB-D scans using natural language descriptions.
Mimic and Rephrase: Reflective Listening in Open-Ended Dialogue
Reflective listening{--}demonstrating that you have heard your conversational partner{--}is key to effective communication.
PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding
We present PartNet: a consistent, large-scale dataset of 3D objects annotated with fine-grained, instance-level, and hierarchical 3D part information.
Ranked #3 on
3D Instance Segmentation
on PartNet
Scan2CAD: Learning CAD Model Alignment in RGB-D Scans
For a 3D reconstruction of an indoor scene, our method takes as input a set of CAD models, and predicts a 9DoF pose that aligns each model to the underlying scan geometry.
Ranked #1 on
3D Reconstruction
on Scan2CAD
Im2Pano3D: Extrapolating 360° Structure and Semantics Beyond the Field of View
We present Im2Pano3D, a convolutional neural network that generates a dense prediction of 3D structure and a probability distribution of semantic labels for a full 360 panoramic view of an indoor scene when given only a partial observation ( <=50%) in the form of an RGB-D image.
Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings
To this end, we first learn joint embeddings of freeform text descriptions and colored 3D shapes.
Im2Pano3D: Extrapolating 360 Structure and Semantics Beyond the Field of View
We present Im2Pano3D, a convolutional neural network that generates a dense prediction of 3D structure and a probability distribution of semantic labels for a full 360 panoramic view of an indoor scene when given only a partial observation (<= 50%) in the form of an RGB-D image.
MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments
We present MINOS, a simulator designed to support the development of multisensory models for goal-directed navigation in complex indoor environments.
Learning Where to Look: Data-Driven Viewpoint Set Selection for 3D Scenes
We provide a search algorithm that generates a sampling of likely candidate views according to the example distribution, and a set selection algorithm that chooses a subset of the candidates that jointly cover the example distribution.
SceneSeer: 3D Scene Design with Natural Language
We present SceneSeer: an interactive text to 3D scene generation system that allows a user to design 3D scenes using natural language.
SceneSuggest: Context-driven 3D Scene Design
We present SceneSuggest: an interactive 3D scene design system providing context-driven suggestions for 3D model retrieval and placement.
Graphics Human-Computer Interaction
ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
A key requirement for leveraging supervised deep learning methods is the availability of large, labeled datasets.
Ranked #11 on
Semantic Segmentation
on ScanNetV2
Semantic Scene Completion from a Single Depth Image
This paper focuses on semantic scene completion, a task for producing a complete 3D voxel representation of volumetric occupancy and semantic labels for a scene from a single-view depth map observation.
Ranked #2 on
3D Semantic Scene Completion
on KITTI-360
Evaluating the word-expert approach for Named-Entity Disambiguation
Named Entity Disambiguation (NED) is the task of linking a named-entity mention to an instance in a knowledge-base, typically Wikipedia.
ShapeNet: An Information-Rich 3D Model Repository
We present ShapeNet: a richly-annotated, large-scale repository of shapes represented by 3D CAD models of objects.
SUTime: A library for recognizing and normalizing time expressions
We describe SUTIME, a temporal tagger for recognizing and normalizing temporal expressions in English text.
A Cross-Lingual Dictionary for English Wikipedia Concepts
We present a resource for automatically associating strings of text with English Wikipedia concepts.
