Search Results for author:
Found 68 papers, 34 papers with code
Visual Chronicles: Using Multimodal LLMs to Analyze Massive Collections of Images
We present a system using Multimodal LLMs (MLLMs) to analyze a large database with tens of millions of images captured at different times, with the aim of discovering patterns in temporal changes.
SplatTalk: 3D VQA with Gaussian Splatting
Language-guided 3D scene understanding is important for advancing applications in robotics, AR/VR, and human-computer interaction, enabling models to comprehend and interact with 3D environments through natural language.
3D Open-Vocabulary Panoptic Segmentation with 2D-3D Vision-Language Distillation
3D panoptic segmentation is a challenging perception task, especially in autonomous driving.
Gaussian3Diff: 3D Gaussian Diffusion for 3D Full Head Synthesis and Editing
We present a novel framework for generating photorealistic 3D human head and subsequently manipulating and reposing them with remarkable flexibility.
TidyBot: Personalized Robot Assistance with Large Language Models
For a robot to personalize physical assistance effectively, it must learn user preferences that can be generally reapplied to future scenarios.
Clutter Detection and Removal in 3D Scenes with View-Consistent Inpainting
Once clutter is removed, we inpaint geometry and texture in the resulting holes by merging inpainted RGB-D images.
Learning Object-Centric Neural Scattering Functions for Free-Viewpoint Relighting and Scene Composition
We propose Object-Centric Neural Scattering Functions (OSFs) for learning to reconstruct object appearance from only images.
Nerflets: Local Radiance Fields for Efficient Structure-Aware 3D Scene Representation from 2D Supervision
We address efficient and structure-aware 3D scene representation from images.
Polynomial Neural Fields for Subband Decomposition and Manipulation
We use this framework to design Fourier PNFs, which match state-of-the-art performance in signal representation tasks that use neural fields.
OpenScene: 3D Scene Understanding with Open Vocabularies
Traditional 3D scene understanding approaches rely on labeled 3D datasets to train a model for a single task with supervision.
Ranked #7 on
3D Open-Vocabulary Instance Segmentation
on Replica
3D Open-Vocabulary Instance Segmentation
3D Semantic Segmentation
+1
FutureHuman3D: Forecasting Complex Long-Term 3D Human Behavior from Video Observations
Thus, we design our method to only require 2D RGB data at inference time while being able to generate 3D human motion sequences.
MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures
Neural Radiance Fields (NeRFs) have demonstrated amazing ability to synthesize images of 3D scenes from novel views.
Ranked #2 on
Novel View Synthesis
on LLFF
Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation
Our model builds a panoptic radiance field representation of any scene from just color images.
Learning Pneumatic Non-Prehensile Manipulation with a Mobile Blower
We investigate pneumatic non-prehensile manipulation (i. e., blowing) as a means of efficiently moving scattered objects into a target receptacle.
Neural Dual Contouring
We introduce neural dual contouring (NDC), a new data-driven approach to mesh reconstruction based on dual contouring (DC).
Revisiting 3D Object Detection From an Egocentric Perspective
Given the insight that SDE would benefit from more accurate geometry descriptions, we propose to represent objects as amodal contours, specifically amodal star-shaped polygons, and devise a simple model, StarPoly, to predict such contours.
Urban Radiance Fields
The goal of this work is to perform 3D reconstruction and novel view synthesis from data captured by scanning platforms commonly deployed for world mapping in urban outdoor environments (e. g., Street View).
Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations
In this work, we propose the Scene Representation Transformer (SRT), a method which processes posed or unposed RGB images of a new area, infers a "set-latent scene representation", and synthesises novel views, all in a single feed-forward pass.
Learning 3D Semantic Segmentation with only 2D Image Supervision
With the recent growth of urban mapping and autonomous driving efforts, there has been an explosion of raw 3D data collected from terrestrial platforms with lidar scanners and color cameras.
Ranked #9 on
LIDAR Semantic Segmentation
on nuScenes
Multiresolution Deep Implicit Functions for 3D Shape Representation
To the best of our knowledge, MDIF is the first deep implicit function model that can at the same time (1) represent different levels of detail and allow progressive decoding; (2) support both encoder-decoder inference and decoder-only latent optimization, and fulfill multiple applications; (3) perform detailed decoder-only shape completion.
Contrastive Multimodal Fusion with TupleInfoNCE
Another approach is to concatenate all the modalities into a tuple and then contrast positive and negative tuple correspondences.
Ranked #81 on
Semantic Segmentation
on NYU Depth v2
Spatial Intention Maps for Multi-Agent Mobile Manipulation
The ability to communicate intention enables decentralized multi-agent robots to collaborate while performing physical tasks.
IBRNet: Learning Multi-View Image-Based Rendering
Unlike neural scene representation work that optimizes per-scene functions for rendering, we learn a generic view interpolation function that generalizes to novel scenes.
P4Contrast: Contrastive Learning with Pairs of Point-Pixel Pairs for RGB-D Scene Understanding
Self-supervised representation learning is a critical problem in computer vision, as it provides a way to pretrain feature extractors on large unlabeled datasets that can be used as an initialization for more efficient and effective training on downstream tasks.
Object-Centric Neural Scene Rendering
We present a method for composing photorealistic scenes from captured images of objects.
Robust Neural Routing Through Space Partitions for Camera Relocalization in Dynamic Indoor Environments
Localizing the camera in a known indoor environment is a key building block for scene mapping, robot navigation, AR, etc.
Forecasting Characteristic 3D Poses of Human Actions
To predict characteristic poses, we propose a probabilistic approach that models the possible multi-modality in the distribution of likely characteristic poses.
Learning to Infer Semantic Parameters for 3D Shape Editing
Many applications in 3D shape design and augmentation require the ability to make specific edits to an object's semantic parameters (e. g., the pose of a person's arm or the length of an airplane's wing) while preserving as much existing details as possible.
Multi-Frame to Single-Frame: Knowledge Distillation for 3D Object Detection
A common dilemma in 3D object detection for autonomous driving is that high-quality, dense point clouds are only available during training, but not testing.
Virtual Multi-view Fusion for 3D Semantic Segmentation
Features from multiple per view predictions are finally fused on 3D mesh vertices to predict mesh semantic segmentation labels.
Ranked #15 on
Semantic Segmentation
on ScanNet
(test mIoU metric)
An LSTM Approach to Temporal 3D Object Detection in LiDAR Point Clouds
We use a U-Net style 3D sparse convolution network to extract features for each frame's LiDAR point-cloud.
Pillar-based Object Detection for Autonomous Driving
We present a simple and flexible object detection framework optimized for autonomous driving.
Complete & Label: A Domain Adaptation Approach to Semantic Segmentation of LiDAR Point Clouds
We study an unsupervised domain adaptation problem for the semantic labeling of 3D point clouds, with a particular focus on domain discrepancies induced by different LiDAR sensors.
Spatial Action Maps for Mobile Manipulation
Typical end-to-end formulations for learning robotic navigation involve predicting a small set of steering command actions (e. g., step forward, turn left, turn right, etc.)
DOPS: Learning to Detect 3D Objects and Predict their 3D Shapes
In contrast, we propose a general-purpose method that works on both indoor and outdoor scenes.
Local Implicit Grid Representations for 3D Scenes
Then, we use the decoder as a component in a shape optimization that solves for a set of latent codes on a regular grid of overlapping crops such that an interpolation of the decoded local shapes matches a partial or noisy observation.
Adversarial Texture Optimization from RGB-D Scans
In this work, we present a novel approach for color texture generation using a conditional adversarial loss obtained from weakly-supervised views.
Local Deep Implicit Functions for 3D Shape
The goal of this project is to learn a 3D shape representation that enables accurate surface reconstruction, compact storage, efficient computation, consistency for similar shapes, generalization across diverse shape categories, and inference from depth camera observations.
Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrations
A key aspect of our grasping model is that it uses "action-view" based rendering to simulate future states with respect to different possible actions.
Rescan: Inductive Instance Segmentation for Indoor RGBD Scans
In depth-sensing applications ranging from home robotics to AR/VR, it will be common to acquire 3D scans of interior spaces repeatedly at sparse time intervals (e. g., as part of regular daily use).
Neural Illumination: Lighting Prediction for Indoor Environments
This paper addresses the task of estimating the light arriving from all directions to a 3D point observed at a selected pixel in an RGB image.
Learning Shape Templates with Structured Implicit Functions
To allow for widely varying geometry and topology, we choose an implicit surface representation based on composition of local shape elements.
FrameNet: Learning Local Canonical Frames of 3D Surfaces from a Single RGB Image
In this work, we introduce the novel problem of identifying dense canonical 3D coordinate frames from a single RGB image.
TossingBot: Learning to Throw Arbitrary Objects with Residual Physics
In this work, we propose an end-to-end formulation that jointly learns to infer control parameters for grasping and throwing motion primitives from visual observations (images of arbitrary objects in a bin) through trial and error.
TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes
We introduce, TextureNet, a neural network architecture designed to extract features from high-resolution signals associated with 3D surface meshes (e. g., color texture maps).
Ranked #26 on
Semantic Segmentation
on ScanNet
(test mIoU metric)
Structure-Aware Shape Synthesis
We propose a new procedure to guide training of a data-driven shape generative model using a structure-aware loss function.
ActiveStereoNet: End-to-End Self-Supervised Learning for Active Stereo Systems
In this paper we present ActiveStereoNet, the first deep learning solution for active stereo systems.
Im2Pano3D: Extrapolating 360° Structure and Semantics Beyond the Field of View
We present Im2Pano3D, a convolutional neural network that generates a dense prediction of 3D structure and a probability distribution of semantic labels for a full 360 panoramic view of an indoor scene when given only a partial observation ( <=50%) in the form of an RGB-D image.
Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning
Skilled robotic manipulation benefits from complex synergies between non-prehensile (e. g. pushing) and prehensile (e. g. grasping) actions: pushing can help rearrange cluttered objects to make space for arms and fingers; likewise, grasping can help displace objects to make pushing movements more precise and collision-free.
Deep Depth Completion of a Single RGB-D Image
The goal of our work is to complete the depth channel of an RGB-D image.
PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction
We introduce a novel RGB-D patch descriptor designed for detecting coplanar surfaces in SLAM reconstruction.
Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings
To this end, we first learn joint embeddings of freeform text descriptions and colored 3D shapes.
Im2Pano3D: Extrapolating 360 Structure and Semantics Beyond the Field of View
We present Im2Pano3D, a convolutional neural network that generates a dense prediction of 3D structure and a probability distribution of semantic labels for a full 360 panoramic view of an indoor scene when given only a partial observation (<= 50%) in the form of an RGB-D image.
MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments
We present MINOS, a simulator designed to support the development of multisensory models for goal-directed navigation in complex indoor environments.
Large-Scale 3D Shape Reconstruction and Segmentation from ShapeNet Core55
We introduce a large-scale 3D shape understanding benchmark using data and annotation from ShapeNet 3D object database.
Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching
Since product images are readily available for a wide range of objects (e. g., from the web), the system works out-of-the-box for novel objects without requiring any additional training data.
Matterport3D: Learning from RGB-D Data in Indoor Environments
Access to large, diverse RGB-D datasets is critical for training RGB-D scene understanding algorithms.
Interactive 3D Modeling with a Generative Adversarial Network
This paper proposes the idea of using a generative adversarial network (GAN) to assist a novice user in designing real-world shapes with a simple interface.
Dilated Residual Networks
Convolutional networks for image classification progressively reduce resolution until the image is represented by tiny feature maps in which the spatial structure of the scene is no longer discernible.
Learning Where to Look: Data-Driven Viewpoint Set Selection for 3D Scenes
We provide a search algorithm that generates a sampling of likely candidate views according to the example distribution, and a set selection algorithm that chooses a subset of the candidates that jointly cover the example distribution.
ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
A key requirement for leveraging supervised deep learning methods is the availability of large, labeled datasets.
Ranked #11 on
Semantic Segmentation
on ScanNetV2
Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks
One of the bottlenecks in training for better representations is the amount of available per-pixel ground truth data that is required for core scene understanding tasks such as semantic segmentation, normal prediction, and object edge detection.
Semantic Scene Completion from a Single Depth Image
This paper focuses on semantic scene completion, a task for producing a complete 3D voxel representation of volumetric occupancy and semantic labels for a scene from a single-view depth map observation.
Ranked #2 on
3D Semantic Scene Completion
on KITTI-360
Fine-To-Coarse Global Registration of RGB-D Scans
RGB-D scanning of indoor environments is important for many applications, including real estate, interior design, and virtual reality.
3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions
To amass training data for our model, we propose a self-supervised feature learning method that leverages the millions of correspondence labels found in existing RGB-D reconstructions.
Ranked #2 on
3D Reconstruction
on Scan2CAD
ShapeNet: An Information-Rich 3D Model Repository
We present ShapeNet: a richly-annotated, large-scale repository of shapes represented by 3D CAD models of objects.
LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
While there has been remarkable progress in the performance of visual recognition algorithms, the state-of-the-art models tend to be exceptionally data-hungry.
Semantic Alignment of LiDAR Data at City Scale
This paper describes an automatic algorithm for global alignment of LiDAR data collected with Google Street View cars in urban environments.
