Search Results for author:
Found 46 papers, 16 papers with code
Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes
Reinforcement learning (RL), particularly its combination with deep neural networks referred to as deep RL (DRL), has shown tremendous promise across a wide range of applications, suggesting its potential for enabling the development of sophisticated robotic behaviors.
Natural Language Can Help Bridge the Sim2Real Gap
We demonstrate that training the image encoder to predict the language description or the distance between descriptions of a sim or real image serves as a useful, data-efficient pretraining step that helps learn a domain-invariant image representation.
BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation
We introduce the BEHAVIOR Vision Suite (BVS), a set of tools and assets to generate fully customized synthetic data for systematic evaluation of computer vision models, based on the newly developed embodied AI benchmark, BEHAVIOR-1K.
ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projection
In this work, we aim to enable robots to learn bimanual manipulation behaviors from human video demonstrations and fine-tune them through interaction.
BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation
We present BEHAVIOR-1K, a comprehensive simulation benchmark for human-centered robotics.
TeleMoMa: A Modular and Versatile Teleoperation System for Mobile Manipulation
This problem is more severe in mobile manipulation, where collecting demonstrations is harder than in stationary manipulation due to the lack of available and easy-to-use teleoperation interfaces.
Model-Based Runtime Monitoring with Interactive Imitation Learning
Unlike prior work that cannot foresee future failures or requires failure experiences for training, our method learns a latent-space dynamics model and a failure classifier, enabling our method to simulate future action outcomes and detect out-of-distribution and high-risk states preemptively.
Mini-BEHAVIOR: A Procedurally Generated Benchmark for Long-horizon Decision-Making in Embodied AI
We present Mini-BEHAVIOR, a novel benchmark for embodied AI that challenges agents to use reasoning and decision-making skills to solve complex activities that resemble everyday human challenges.
Principles and Guidelines for Evaluating Social Robot Navigation Algorithms
A major challenge to deploying robots widely is navigation in human-populated environments, commonly referred to as social robot navigation.
Task-Driven Graph Attention for Hierarchical Relational Object Navigation
Embodied AI agents in large scenes often need to navigate to find objects.
Modeling Dynamic Environments with Scene Graph Memory
We evaluate our method in the Dynamic House Simulator, a new benchmark that creates diverse dynamic graphs following the semantic patterns typically seen at homes, and show that NEP can be trained to predict the locations of objects in a variety of environments with diverse object movement dynamics, outperforming baselines both in terms of new scene adaptability and overall accuracy.
ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding
It achieves a new SOTA of 50. 6% (top-1) on Objaverse-LVIS and 84. 7% (top-1) on ModelNet40 in zero-shot classification.
Ranked #8 on
3D Point Cloud Classification
on ScanObjectNN
(using extra training data)
Causal Policy Gradient for Whole-Body Mobile Manipulation
Current approaches often segregate tasks into navigation without manipulation and stationary manipulation without locomotion by manually matching parts of the action space to MoMa sub-objectives (e. g. learning base actions for locomotion objectives and learning arm actions for manipulation).
Procedure-Aware Pretraining for Instructional Video Understanding
This graph can then be used to generate pseudo labels to train a video representation that encodes the procedural knowledge in a more accessible form to generalize to multiple procedure understanding tasks.
Hierarchical Point Attention for Indoor 3D Object Detection
By plugging our proposed modules into the state-of-the-art transformer-based 3D detectors, we improve the previous best results on both benchmarks, with more significant improvements on smaller objects.
ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding
Then, ULIP learns a 3D representation space aligned with the common image-text space, using a small number of automatically synthesized triplets.
Ranked #3 on
Training-free 3D Point Cloud Classification
on ModelNet40
(using extra training data)
Retrospectives on the Embodied AI Workshop
We present a retrospective on the state of Embodied AI research.
MaskViT: Masked Visual Pre-Training for Video Prediction
This work shows that we can create good video prediction models by pre-training transformers via masked visual modeling.
BEHAVIOR in Habitat 2.0: Simulator-Independent Logical Task Description for Benchmarking Embodied AI Agents
Robots excel in performing repetitive and precision-sensitive tasks in controlled environments such as warehouses and factories, but have not been yet extended to embodied AI agents providing assistance in household tasks.
Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation
Doing this is challenging for two reasons: on the data side, current interfaces make collecting high-quality human demonstrations difficult, and on the learning side, policies trained on limited data can suffer from covariate shift when deployed.
What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
Based on the study, we derive a series of lessons including the sensitivity to different algorithmic design choices, the dependence on the quality of the demonstrations, and the variability based on the stopping criteria due to the different objectives in training and evaluation.
iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks
We evaluate the new capabilities of iGibson 2. 0 to enable robot learning of novel tasks, in the hope of demonstrating the potential of this new simulator to support new research in embodied AI.
BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments
We introduce BEHAVIOR, a benchmark for embodied AI with 100 activities in simulation, spanning a range of everyday household chores such as cleaning, maintenance, and food preparation.
LASER: Learning a Latent Action Space for Efficient Reinforcement Learning
Additionally, similar tasks or instances of the same task family impose latent manifold constraints on the most effective action space: the task family can be best solved with actions in a manifold of the entire action space of the robot.
Human-in-the-Loop Imitation Learning using Remote Teleoperation
We develop a simple and effective algorithm to train the policy iteratively on new data collected by the system that encourages the policy to learn how to traverse bottlenecks through the interventions.
Learning Multi-Arm Manipulation Through Collaborative Teleoperation
To address these challenges, we present Multi-Arm RoboTurk (MART), a multi-user data collection platform that allows multiple remote users to simultaneously teleoperate a set of robotic arms and collect demonstrations for multi-arm tasks.
Semantic and Geometric Modeling with Neural Message Passing in 3D Scene Graphs for Hierarchical Mechanical Search
We propose to use a 3D scene graph representation to capture the hierarchical, semantic, and geometric aspects of this problem.
IGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes
We present iGibson 1. 0, a novel simulation environment to develop robotic solutions for interactive tasks in large-scale realistic scenes.
Robot Navigation in Constrained Pedestrian Environments using Reinforcement Learning
Navigating fluently around pedestrians is a necessary capability for mobile robots deployed in human environments, such as buildings and homes.
robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
robosuite is a simulation framework for robot learning powered by the MuJoCo physics engine.
ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation
To validate our method, we apply ReLMoGen to two types of tasks: 1) Interactive Navigation tasks, navigation problems where interactions with the environment are required to reach the destination, and 2) Mobile Manipulation tasks, manipulation tasks that require moving the robot base.
Visuomotor Mechanical Search: Learning to Retrieve Target Objects in Clutter
When searching for objects in cluttered environments, it is often necessary to perform complex interactions in order to move occluding objects out of the way and fully reveal the object of interest and make it graspable.
Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations
In the second stage of GTI, we collect a small set of rollouts from the unconditioned stochastic policy of the first stage, and train a goal-directed agent to generalize to novel start and goal configurations.
JRMOT: A Real-Time 3D Multi-Object Tracker and a New Large-Scale Dataset
In this work we present JRMOT, a novel 3D MOT system that integrates information from RGB images and 3D point clouds to achieve real-time, state-of-the-art tracking performance.
Ranked #9 on
Multiple Object Tracking
on KITTI Tracking test
Leveraging Pretrained Image Classifiers for Language-Based Segmentation
Current semantic segmentation models cannot easily generalize to new object classes unseen during train time: they require additional annotated images and retraining.
Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environments
We present Interactive Gibson Benchmark, the first comprehensive benchmark for training and evaluating Interactive Navigation: robot navigation strategies where physical interaction with objects is allowed and even encouraged to accomplish a task.
JRDB: A Dataset and Benchmark of Egocentric Robot Visual Perception of Humans in Built Environments
We present JRDB, a novel egocentric dataset collected from our social mobile manipulator JackRabbot.
6-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypoints
We present 6-PACK, a deep learning approach to category-level 6D object pose tracking on RGB-D data.
Ranked #1 on
6D Pose Estimation using RGBD
on REAL275
(Rerr metric)
Regression Planning Networks
Recent learning-to-plan methods have shown promising results on planning directly from observation space.
Improving Social Awareness Through DANTE: A Deep Affinity Network for Clustering Conversational Interactants
We propose a data-driven approach to detect conversational groups by identifying spatial arrangements typical of these focused social encounters.
Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks
This problem is compounded by the presence of social interactions between humans and their physical interactions with the scene.
Ranked #17 on
Trajectory Prediction
on ETH/UCY
Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks
This paper studies the effect of different action spaces in deep RL and advocates for Variable Impedance Control in End-effector Space (VICES) as an advantageous action space for constrained and contact-rich tasks.
Deep Local Trajectory Replanning and Control for Robot Navigation
We present a navigation system that combines ideas from hierarchical planning and machine learning.
Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter
In this paper, we formalize Mechanical Search and study a version where distractor objects are heaped over the target object in a bin.
Robotics
DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion
A key technical challenge in performing 6D object pose estimation from RGB-D image is to fully leverage the two complementary data sources.
Ranked #4 on
6D Pose Estimation
on LineMOD
The RBO Dataset of Articulated Objects and Interactions
Each interaction with an object is annotated with the ground truth poses of its rigid parts and the kinematic state obtained by a motion capture system.
