Search Results for author:
Found 77 papers, 35 papers with code
TidyBot++: An Open-Source Holonomic Mobile Manipulator for Robot Learning
In our experiments, we use this interface to collect data and show that the resulting learned policies can successfully perform a variety of common household mobile manipulation tasks.
Differentiable Robot Rendering
Vision foundation models trained on massive amounts of visual data have shown unprecedented reasoning and planning skills in open-world settings.
Semantically Controllable Augmentations for Generalizable Robot Learning
By demonstrating the effectiveness of image-text generative models in diverse real-world robotic applications, our generative augmentation framework provides a scalable and efficient path for boosting generalization in robot learning at no extra human cost.
Flow as the Cross-Domain Manipulation Interface
By leveraging real-world human videos and simulated robot play data, we bypass the challenges of teleoperating physical robots in the real world, resulting in a scalable system for diverse tasks.
EquiBot: SIM(3)-Equivariant Diffusion Policy for Generalizable and Data Efficient Learning
Building effective imitation learning methods that enable robots to learn from limited data and still generalize across diverse real-world environments is a long-standing problem in robot learning.
ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data
Audio signals provide rich information for the robot interaction and object properties through contact.
Dreamitate: Real-World Visuomotor Policy Learning via Video Generation
A key challenge in manipulation is learning a policy that can robustly generalize to diverse visual environments.
DataComp-LM: In search of the next generation of training sets for language models
We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models.
Real2Code: Reconstruct Articulated Objects via Code Generation
By leveraging pre-trained vision and language models, our approach scales elegantly with the number of articulated parts, and generalizes from synthetic training data to real world objects in unstructured environments.
DoughNet: A Visual Predictive Model for Topological Manipulation of Deformable Objects
Manipulation of elastoplastic objects like dough often involves topological changes such as splitting and merging.
ContactHandover: Contact-Guided Robot-to-Human Object Handover
We propose ContactHandover, a robot to human handover system that consists of two phases: a contact-guided grasping phase and an object delivery phase.
Language models scale reliably with over-training and on downstream tasks
Second, we relate the perplexity of a language model to its downstream task performance by proposing a power law.
Dynamics-Guided Diffusion Model for Robot Manipulator Design
The design objective constructed from the target and predicted interaction profiles provides a gradient to guide the refinement of finger geometry for the task.
Tactile-based Object Retrieval From Granular Media
We also introduce a training curriculum that enables learning these behaviors in simulation, followed by zero-shot transfer to real hardware.
DeformGS: Scene Flow in Highly Deformable Scenes for Deformable Object Manipulation
Just as object pose estimation and tracking have aided robots for rigid manipulation, dense 3D tracking (scene flow) of highly deformable objects will enable new applications in robotics while aiding existing approaches, such as imitation learning or creating digital twins with real2sim transfer.
RIC: Rotate-Inpaint-Complete for Generalizable Scene Reconstruction
General scene reconstruction refers to the task of estimating the full 3D geometry and texture of a scene containing previously unseen objects.
XSkill: Cross Embodiment Skill Discovery
Human demonstration videos are a widely available data source for robot learning and an intuitive user interface for expressing desired behavior.
RoCo: Dialectic Multi-Robot Collaboration with Large Language Models
We propose a novel approach to multi-robot collaboration that harnesses the power of pre-trained large language models (LLMs) for both high-level communication and low-level path planning.
Rearrangement Planning for General Part Assembly
Most successes in autonomous robotic assembly have been restricted to single target or category.
REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction
The ability to detect and analyze failed executions automatically is crucial for an explainable and robust robotic system.
TidyBot: Personalized Robot Assistance with Large Language Models
For a robot to personalize physical assistance effectively, it must learn user preferences that can be generally reapplied to future scenarios.
DataComp: In search of the next generation of multimodal datasets
Multimodal datasets are a critical component in recent breakthroughs such as Stable Diffusion and GPT-4, yet their design does not receive the same research attention as model architectures or training algorithms.
Decision Making for Human-in-the-loop Robotic Agents via Uncertainty-Aware Reinforcement Learning
In a Human-in-the-Loop paradigm, a robotic agent is able to act mostly autonomously in solving a task, but can request help from an external expert when needed.
CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learning
On a real robot setup, CACTI enables efficient training of a single policy that can perform 10 manipulation tasks involving kitchen objects, and is robust to varying layouts of distractors.
ASPiRe:Adaptive Skill Priors for Reinforcement Learning
We introduce ASPiRe (Adaptive Skill Prior for RL), a new approach that leverages prior experience to accelerate reinforcement learning.
Tracking and Reconstructing Hand Object Interactions from Point Cloud Sequences in the Wild
We for the first time propose a point cloud based hand joint tracking network, HandTrackNet, to estimate the inter-frame hand joint motion.
TANDEM3D: Active Tactile Exploration for 3D Object Recognition
In this work, we propose TANDEM3D, a method that applies a co-training framework for exploration and decision making to 3D object recognition with tactile signals.
Patching open-vocabulary models by interpolating weights
We study model patching, where the goal is to improve accuracy on specific tasks without degrading accuracy on tasks where performance is already adequate.
Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models
We study open-world 3D scene understanding, a family of tasks that require agents to reason about their 3D environment with an open-set vocabulary and out-of-domain visual inputs - a critical skill for robots to operate in the unstructured 3D world.
Structure from Action: Learning Interactions for Articulated Object 3D Structure Discovery
We introduce Structure from Action (SfA), a framework to discover 3D part geometry and joint parameters of unseen articulated objects via a sequence of inferred interactions.
BusyBot: Learning to Interact, Reason, and Plan in a BusyBoard Environment
We introduce BusyBoard, a toy-inspired robot learning environment that leverages a diverse set of articulated objects and inter-object functional relations to provide rich visual feedback for robot interactions.
Learning Pneumatic Non-Prehensile Manipulation with a Mobile Blower
We investigate pneumatic non-prehensile manipulation (i. e., blowing) as a means of efficiently moving scattered objects into a target receptacle.
Continuous Scene Representations for Embodied AI
Our method captures feature relationships between objects, composes them into a graph structure on-the-fly, and situates an embodied agent within the representation.
CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation
To better evaluate L-ZSON, we introduce the Pasture benchmark, which considers finding uncommon objects, objects described by spatial and appearance attributes, and hidden objects described relative to visible objects.
TANDEM: Learning Joint Exploration and Decision Making with Tactile Sensors
Inspired by the human ability to perform complex manipulation in the complete absence of vision (like retrieving an object from a pocket), the robotic manipulation field is motivated to develop new methods for tactile-based object interaction.
Leveraging SE(3) Equivariance for Self-Supervised Category-Level Object Pose Estimation
Category-level object pose estimation aims to find 6D object poses of previously unseen object instances from known categories without access to object CAD models.
Scene Editing as Teleoperation: A Case Study in 6DoF Kit Assembly
To make teleoperation accessible to non-expert users, we propose the framework "Scene Editing as Teleoperation" (SEaT), where the key idea is to transform the traditional "robot-centric" interface into a "scene-centric" interface -- instead of controlling the robot, users focus on specifying the task's goal by manipulating digital twins of the real-world objects.
UMPNet: Universal Manipulation Policy Network for Articulated Objects
We introduce the Universal Manipulation Policy Network (UMPNet) -- a single image-based policy network that infers closed-loop action sequences for manipulating arbitrary articulated objects.
Learning to See before Learning to Act: Visual Pre-training for Manipulation
With just a small amount of robotic experience, we can further fine-tune the affordance model to achieve better results.
Leveraging SE(3) Equivariance for Self-supervised Category-Level Object Pose Estimation from Point Clouds
To reduce the huge amount of pose annotations needed for category-level learning, we propose for the first time a self-supervised learning framework to estimate category-level 6D object pose from single 3D point clouds.
Visual Perspective Taking for Opponent Behavior Modeling
We suggest that visual behavior modeling and perspective taking skills will play a critical role in the ability of physical robots to fully integrate into real-world multi-agent activities.
Act the Part: Learning Interaction Strategies for Articulated Object Part Discovery
People often use physical intuition when manipulating articulated objects, irrespective of object semantics.
GarmentNets: Category-Level Pose Estimation for Garments via Canonical Space Shape Completion
By mapping the observed partial surface to the canonical space and completing it in this space, the output representation describes the garment's full configuration using a complete 3D mesh with the per-vertex canonical coordinate label.
Spatial Intention Maps for Multi-Agent Mobile Manipulation
The ability to communicate intention enables decentralized multi-agent robots to collaborate while performing physical tasks.
SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigation
We introduce SSCNav, an algorithm that explicitly models scene priors using a confidence-aware semantic scene completion module to complete the scene and guide the agent's navigation planning.
Perspectives on Sim2Real Transfer for Robotics: A Summary of the R:SS 2020 Workshop
This report presents the debates, posters, and discussions of the Sim2Real workshop held in conjunction with the 2020 edition of the "Robotics: Science and System" conference.
AdaGrasp: Learning an Adaptive Gripper-Aware Grasping Policy
We propose AdaGrasp, a method to learn a single grasping policy that generalizes to novel grippers.
Robotics
Fit2Form: 3D Generative Model for Robot Gripper Form Design
We propose Fit2Form, a 3D generative design framework that generates pairs of finger shapes to maximize design objectives (i. e., grasp success, stability, and robustness) for target grasp objects.
Learning a Decentralized Multi-arm Motion Planner
In this paper, we tackle this problem with multi-agent reinforcement learning, where a decentralized policy is trained to control one robot arm in the multi-arm system to reach its target end-effector pose given observations of its workspace state and target end-effector pose.
Learning 3D Dynamic Scene Representations for Robot Manipulation
3D scene representation for robot manipulation should capture three key object properties: permanency -- objects that become occluded over time continue to exist; amodal completeness -- objects have 3D occupancy, even if only partial observations are available; spatiotemporal continuity -- the movement of each object is continuous over space and time.
Multitask Learning Strengthens Adversarial Robustness
Although deep networks achieve strong accuracy on a range of computer vision benchmarks, they remain vulnerable to adversarial attacks, where imperceptible input perturbations fool the network.
Spatial Action Maps for Mobile Manipulation
Typical end-to-end formulations for learning robotic navigation involve predicting a small set of steering command actions (e. g., step forward, turn left, turn right, etc.)
Category-Level Articulated Object Pose Estimation
We develop a deep network based on PointNet++ that predicts ANCSH from a single depth point cloud, including part segmentation, normalized coordinates, and joint parameters in the canonical object space.
Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrations
A key aspect of our grasping model is that it uses "action-view" based rendering to simulate future states with respect to different possible actions.
Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly
This formulation enables the model to acquire a broader understanding of how shapes and surfaces fit together for assembly -- allowing it to generalize to new objects and kits.
Visual Hide and Seek
We train embodied agents to play Visual Hide and Seek where a prey must navigate in a simulated environment in order to avoid capture from a predator.
ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation
To address these challenges, we present ClearGrasp -- a deep learning approach for estimating accurate 3D geometry of transparent objects from a single RGB-D image for robotic manipulation.
Ranked #1 on
Semantic Segmentation
on Cleargrasp (Novel)
Neural Illumination: Lighting Prediction for Indoor Environments
This paper addresses the task of estimating the light arriving from all directions to a 3D point observed at a selected pixel in an RGB image.
TossingBot: Learning to Throw Arbitrary Objects with Residual Physics
In this work, we propose an end-to-end formulation that jointly learns to infer control parameters for grasping and throwing motion primitives from visual observations (images of arbitrary objects in a bin) through trial and error.
Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation
The goal of this paper is to estimate the 6D pose and dimensions of unseen object instances in an RGB-D image.
Ranked #2 on
6D Pose Estimation using RGBD
on CAMERA25
Neural Graph Matching Networks for Fewshot 3D Action Recognition
We propose Neural Graph Matching (NGM) Networks, a novel framework that can learn to recognize a previous unseen 3D action class with only a few examples.
Ranked #1 on
Skeleton Based Action Recognition
on CAD-120
Im2Pano3D: Extrapolating 360° Structure and Semantics Beyond the Field of View
We present Im2Pano3D, a convolutional neural network that generates a dense prediction of 3D structure and a probability distribution of semantic labels for a full 360 panoramic view of an indoor scene when given only a partial observation ( <=50%) in the form of an RGB-D image.
Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning
Skilled robotic manipulation benefits from complex synergies between non-prehensile (e. g. pushing) and prehensile (e. g. grasping) actions: pushing can help rearrange cluttered objects to make space for arms and fingers; likewise, grasping can help displace objects to make pushing movements more precise and collision-free.
Im2Pano3D: Extrapolating 360 Structure and Semantics Beyond the Field of View
We present Im2Pano3D, a convolutional neural network that generates a dense prediction of 3D structure and a probability distribution of semantic labels for a full 360 panoramic view of an indoor scene when given only a partial observation (<= 50%) in the form of an RGB-D image.
Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching
Since product images are readily available for a wide range of objects (e. g., from the web), the system works out-of-the-box for novel objects without requiring any additional training data.
Matterport3D: Learning from RGB-D Data in Indoor Environments
Access to large, diverse RGB-D datasets is critical for training RGB-D scene understanding algorithms.
Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks
One of the bottlenecks in training for better representations is the amount of available per-pixel ground truth data that is required for core scene understanding tasks such as semantic segmentation, normal prediction, and object edge detection.
Semantic Scene Completion from a Single Depth Image
This paper focuses on semantic scene completion, a task for producing a complete 3D voxel representation of volumetric occupancy and semantic labels for a scene from a single-view depth map observation.
Ranked #2 on
3D Semantic Scene Completion
on KITTI-360
Multi-view Self-supervised Deep Learning for 6D Pose Estimation in the Amazon Picking Challenge
The approach was part of the MIT-Princeton Team system that took 3rd- and 4th- place in the stowing and picking tasks, respectively at APC 2016.
3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions
To amass training data for our model, we propose a self-supervised feature learning method that leverages the millions of correspondence labels found in existing RGB-D reconstructions.
Ranked #2 on
3D Reconstruction
on Scan2CAD
ShapeNet: An Information-Rich 3D Model Repository
We present ShapeNet: a richly-annotated, large-scale repository of shapes represented by 3D CAD models of objects.
Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images
We focus on the task of amodal 3D object detection in RGB-D images, which aims to produce a 3D bounding box of an object in metric form at its full extent.
Ranked #6 on
3D Object Detection
on SUN-RGBD val
(Inference Speed (s) metric)
Robot In a Room: Toward Perfect Object Recognition in Closed Environments
By constraining a robot to stay in a limited territory, we can ensure that the robot has seen most objects before and the speed of introducing a new object is slow.
LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
While there has been remarkable progress in the performance of visual recognition algorithms, the state-of-the-art models tend to be exceptionally data-hungry.
SUN RGB-D: A RGB-D Scene Understanding Benchmark Suite
Although RGB-D sensors have enabled major breakthroughs for several vision tasks, such as 3D reconstruction, we have not attained the same level of success in high-level scene understanding.
3D ShapeNets: A Deep Representation for Volumetric Shapes
Our model, 3D ShapeNets, learns the distribution of complex 3D shapes across different object categories and arbitrary poses from raw CAD data, and discovers hierarchical compositional part representations automatically.
Ranked #34 on
3D Point Cloud Classification
on ModelNet40
(Mean Accuracy metric)
