Search Results for author:
Found 67 papers, 32 papers with code
3D ShapeNets: A Deep Representation for Volumetric Shapes
Our model, 3D ShapeNets, learns the distribution of complex 3D shapes across different object categories and arbitrary poses from raw CAD data, and discovers hierarchical compositional part representations automatically.
Ranked #35 on
3D Point Cloud Classification
on ModelNet40
(Mean Accuracy metric)
SUN RGB-D: A RGB-D Scene Understanding Benchmark Suite
Although RGB-D sensors have enabled major breakthroughs for several vision tasks, such as 3D reconstruction, we have not attained the same level of success in high-level scene understanding.
LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
While there has been remarkable progress in the performance of visual recognition algorithms, the state-of-the-art models tend to be exceptionally data-hungry.
Robot In a Room: Toward Perfect Object Recognition in Closed Environments
By constraining a robot to stay in a limited territory, we can ensure that the robot has seen most objects before and the speed of introducing a new object is slow.
Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images
We focus on the task of amodal 3D object detection in RGB-D images, which aims to produce a 3D bounding box of an object in metric form at its full extent.
Ranked #6 on
3D Object Detection
on SUN-RGBD val
(Inference Speed (s) metric)
ShapeNet: An Information-Rich 3D Model Repository
We present ShapeNet: a richly-annotated, large-scale repository of shapes represented by 3D CAD models of objects.
3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions
To amass training data for our model, we propose a self-supervised feature learning method that leverages the millions of correspondence labels found in existing RGB-D reconstructions.
Ranked #2 on
3D Reconstruction
on Scan2CAD
Multi-view Self-supervised Deep Learning for 6D Pose Estimation in the Amazon Picking Challenge
The approach was part of the MIT-Princeton Team system that took 3rd- and 4th- place in the stowing and picking tasks, respectively at APC 2016.
Semantic Scene Completion from a Single Depth Image
This paper focuses on semantic scene completion, a task for producing a complete 3D voxel representation of volumetric occupancy and semantic labels for a scene from a single-view depth map observation.
Ranked #2 on
3D Semantic Scene Completion
on KITTI-360
Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks
One of the bottlenecks in training for better representations is the amount of available per-pixel ground truth data that is required for core scene understanding tasks such as semantic segmentation, normal prediction, and object edge detection.
Matterport3D: Learning from RGB-D Data in Indoor Environments
Access to large, diverse RGB-D datasets is critical for training RGB-D scene understanding algorithms.
Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching
Since product images are readily available for a wide range of objects (e. g., from the web), the system works out-of-the-box for novel objects without requiring any additional training data.
Im2Pano3D: Extrapolating 360 Structure and Semantics Beyond the Field of View
We present Im2Pano3D, a convolutional neural network that generates a dense prediction of 3D structure and a probability distribution of semantic labels for a full 360 panoramic view of an indoor scene when given only a partial observation (<= 50%) in the form of an RGB-D image.
Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning
Skilled robotic manipulation benefits from complex synergies between non-prehensile (e. g. pushing) and prehensile (e. g. grasping) actions: pushing can help rearrange cluttered objects to make space for arms and fingers; likewise, grasping can help displace objects to make pushing movements more precise and collision-free.
Im2Pano3D: Extrapolating 360° Structure and Semantics Beyond the Field of View
We present Im2Pano3D, a convolutional neural network that generates a dense prediction of 3D structure and a probability distribution of semantic labels for a full 360 panoramic view of an indoor scene when given only a partial observation ( <=50%) in the form of an RGB-D image.
Neural Graph Matching Networks for Fewshot 3D Action Recognition
We propose Neural Graph Matching (NGM) Networks, a novel framework that can learn to recognize a previous unseen 3D action class with only a few examples.
Ranked #1 on
Skeleton Based Action Recognition
on CAD-120
Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation
The goal of this paper is to estimate the 6D pose and dimensions of unseen object instances in an RGB-D image.
Ranked #2 on
6D Pose Estimation using RGBD
on CAMERA25
TossingBot: Learning to Throw Arbitrary Objects with Residual Physics
In this work, we propose an end-to-end formulation that jointly learns to infer control parameters for grasping and throwing motion primitives from visual observations (images of arbitrary objects in a bin) through trial and error.
Neural Illumination: Lighting Prediction for Indoor Environments
This paper addresses the task of estimating the light arriving from all directions to a 3D point observed at a selected pixel in an RGB image.
ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation
To address these challenges, we present ClearGrasp -- a deep learning approach for estimating accurate 3D geometry of transparent objects from a single RGB-D image for robotic manipulation.
Ranked #1 on
Semantic Segmentation
on Cleargrasp (Novel)
Visual Hide and Seek
We train embodied agents to play Visual Hide and Seek where a prey must navigate in a simulated environment in order to avoid capture from a predator.
Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly
This formulation enables the model to acquire a broader understanding of how shapes and surfaces fit together for assembly -- allowing it to generalize to new objects and kits.
Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrations
A key aspect of our grasping model is that it uses "action-view" based rendering to simulate future states with respect to different possible actions.
Category-Level Articulated Object Pose Estimation
We develop a deep network based on PointNet++ that predicts ANCSH from a single depth point cloud, including part segmentation, normalized coordinates, and joint parameters in the canonical object space.
Spatial Action Maps for Mobile Manipulation
Typical end-to-end formulations for learning robotic navigation involve predicting a small set of steering command actions (e. g., step forward, turn left, turn right, etc.)
Multitask Learning Strengthens Adversarial Robustness
Although deep networks achieve strong accuracy on a range of computer vision benchmarks, they remain vulnerable to adversarial attacks, where imperceptible input perturbations fool the network.
Learning 3D Dynamic Scene Representations for Robot Manipulation
3D scene representation for robot manipulation should capture three key object properties: permanency -- objects that become occluded over time continue to exist; amodal completeness -- objects have 3D occupancy, even if only partial observations are available; spatiotemporal continuity -- the movement of each object is continuous over space and time.
Learning a Decentralized Multi-arm Motion Planner
In this paper, we tackle this problem with multi-agent reinforcement learning, where a decentralized policy is trained to control one robot arm in the multi-arm system to reach its target end-effector pose given observations of its workspace state and target end-effector pose.
Fit2Form: 3D Generative Model for Robot Gripper Form Design
We propose Fit2Form, a 3D generative design framework that generates pairs of finger shapes to maximize design objectives (i. e., grasp success, stability, and robustness) for target grasp objects.
AdaGrasp: Learning an Adaptive Gripper-Aware Grasping Policy
We propose AdaGrasp, a method to learn a single grasping policy that generalizes to novel grippers.
Robotics
Perspectives on Sim2Real Transfer for Robotics: A Summary of the R:SS 2020 Workshop
This report presents the debates, posters, and discussions of the Sim2Real workshop held in conjunction with the 2020 edition of the "Robotics: Science and System" conference.
SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigation
We introduce SSCNav, an algorithm that explicitly models scene priors using a confidence-aware semantic scene completion module to complete the scene and guide the agent's navigation planning.
Spatial Intention Maps for Multi-Agent Mobile Manipulation
The ability to communicate intention enables decentralized multi-agent robots to collaborate while performing physical tasks.
GarmentNets: Category-Level Pose Estimation for Garments via Canonical Space Shape Completion
By mapping the observed partial surface to the canonical space and completing it in this space, the output representation describes the garment's full configuration using a complete 3D mesh with the per-vertex canonical coordinate label.
Act the Part: Learning Interaction Strategies for Articulated Object Part Discovery
People often use physical intuition when manipulating articulated objects, irrespective of object semantics.
Visual Perspective Taking for Opponent Behavior Modeling
We suggest that visual behavior modeling and perspective taking skills will play a critical role in the ability of physical robots to fully integrate into real-world multi-agent activities.
Leveraging SE(3) Equivariance for Self-supervised Category-Level Object Pose Estimation from Point Clouds
To reduce the huge amount of pose annotations needed for category-level learning, we propose for the first time a self-supervised learning framework to estimate category-level 6D object pose from single 3D point clouds.
Learning to See before Learning to Act: Visual Pre-training for Manipulation
With just a small amount of robotic experience, we can further fine-tune the affordance model to achieve better results.
UMPNet: Universal Manipulation Policy Network for Articulated Objects
We introduce the Universal Manipulation Policy Network (UMPNet) -- a single image-based policy network that infers closed-loop action sequences for manipulating arbitrary articulated objects.
Scene Editing as Teleoperation: A Case Study in 6DoF Kit Assembly
To make teleoperation accessible to non-expert users, we propose the framework "Scene Editing as Teleoperation" (SEaT), where the key idea is to transform the traditional "robot-centric" interface into a "scene-centric" interface -- instead of controlling the robot, users focus on specifying the task's goal by manipulating digital twins of the real-world objects.
Leveraging SE(3) Equivariance for Self-Supervised Category-Level Object Pose Estimation
Category-level object pose estimation aims to find 6D object poses of previously unseen object instances from known categories without access to object CAD models.
TANDEM: Learning Joint Exploration and Decision Making with Tactile Sensors
Inspired by the human ability to perform complex manipulation in the complete absence of vision (like retrieving an object from a pocket), the robotic manipulation field is motivated to develop new methods for tactile-based object interaction.
CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation
To better evaluate L-ZSON, we introduce the Pasture benchmark, which considers finding uncommon objects, objects described by spatial and appearance attributes, and hidden objects described relative to visible objects.
Continuous Scene Representations for Embodied AI
Our method captures feature relationships between objects, composes them into a graph structure on-the-fly, and situates an embodied agent within the representation.
Learning Pneumatic Non-Prehensile Manipulation with a Mobile Blower
We investigate pneumatic non-prehensile manipulation (i. e., blowing) as a means of efficiently moving scattered objects into a target receptacle.
BusyBot: Learning to Interact, Reason, and Plan in a BusyBoard Environment
We introduce BusyBoard, a toy-inspired robot learning environment that leverages a diverse set of articulated objects and inter-object functional relations to provide rich visual feedback for robot interactions.
Structure from Action: Learning Interactions for Articulated Object 3D Structure Discovery
We introduce Structure from Action (SfA), a framework to discover 3D part geometry and joint parameters of unseen articulated objects via a sequence of inferred interactions.
Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models
We study open-world 3D scene understanding, a family of tasks that require agents to reason about their 3D environment with an open-set vocabulary and out-of-domain visual inputs - a critical skill for robots to operate in the unstructured 3D world.
Patching open-vocabulary models by interpolating weights
We study model patching, where the goal is to improve accuracy on specific tasks without degrading accuracy on tasks where performance is already adequate.
TANDEM3D: Active Tactile Exploration for 3D Object Recognition
In this work, we propose TANDEM3D, a method that applies a co-training framework for exploration and decision making to 3D object recognition with tactile signals.
Tracking and Reconstructing Hand Object Interactions from Point Cloud Sequences in the Wild
We for the first time propose a point cloud based hand joint tracking network, HandTrackNet, to estimate the inter-frame hand joint motion.
ASPiRe:Adaptive Skill Priors for Reinforcement Learning
We introduce ASPiRe (Adaptive Skill Prior for RL), a new approach that leverages prior experience to accelerate reinforcement learning.
CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learning
On a real robot setup, CACTI enables efficient training of a single policy that can perform 10 manipulation tasks involving kitchen objects, and is robust to varying layouts of distractors.
Decision Making for Human-in-the-loop Robotic Agents via Uncertainty-Aware Reinforcement Learning
In a Human-in-the-Loop paradigm, a robotic agent is able to act mostly autonomously in solving a task, but can request help from an external expert when needed.
DataComp: In search of the next generation of multimodal datasets
Multimodal datasets are a critical component in recent breakthroughs such as Stable Diffusion and GPT-4, yet their design does not receive the same research attention as model architectures or training algorithms.
TidyBot: Personalized Robot Assistance with Large Language Models
For a robot to personalize physical assistance effectively, it must learn user preferences that can be generally reapplied to future scenarios.
REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction
The ability to detect and analyze failed executions automatically is crucial for an explainable and robust robotic system.
Rearrangement Planning for General Part Assembly
Most successes in autonomous robotic assembly have been restricted to single target or category.
RoCo: Dialectic Multi-Robot Collaboration with Large Language Models
We propose a novel approach to multi-robot collaboration that harnesses the power of pre-trained large language models (LLMs) for both high-level communication and low-level path planning.
XSkill: Cross Embodiment Skill Discovery
Human demonstration videos are a widely available data source for robot learning and an intuitive user interface for expressing desired behavior.
RIC: Rotate-Inpaint-Complete for Generalizable Scene Reconstruction
General scene reconstruction refers to the task of estimating the full 3D geometry and texture of a scene containing previously unseen objects.
MD-Splatting: Learning Metric Deformation from 4D Gaussians in Highly Deformable Scenes
MD-Splatting builds on recent advances in Gaussian splatting, a method that learns the properties of a large number of Gaussians for state-of-the-art and fast novel view synthesis.
Tactile-based Object Retrieval From Granular Media
We also introduce a training curriculum that enables learning these behaviors in simulation, followed by zero-shot transfer to real hardware.
Dynamics-Guided Diffusion Model for Robot Manipulator Design
The design objective constructed from the target and predicted interaction profiles provides a gradient to guide the refinement of finger geometry for the task.
Language models scale reliably with over-training and on downstream tasks
We fit scaling laws that extrapolate in both the number of model parameters and the ratio of training tokens to parameters.
ContactHandover: Contact-Guided Robot-to-Human Object Handover
We propose ContactHandover, a robot to human handover system that consists of two phases: a contact-guided grasping phase and an object delivery phase.
