Search Results for author: Dieter Fox

Found 126 papers, 53 papers with code

Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects

8 code implementations27 Sep 2018 Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox, Stan Birchfield

Using synthetic data generated in this manner, we introduce a one-shot deep neural network that is able to perform competitively against a state-of-the-art network trained on a combination of real and synthetic data.

Robotics

BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects

1 code implementation CVPR 2023 Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas Muller, Alex Evans, Dieter Fox, Jan Kautz, Stan Birchfield

We present a near real-time method for 6-DoF tracking of an unknown object from a monocular RGBD video sequence, while simultaneously performing neural 3D reconstruction of the object.

3D Object Tracking 3D Reconstruction +5

PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes

11 code implementations1 Nov 2017 Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, Dieter Fox

We conduct extensive experiments on our YCB-Video dataset and the OccludedLINEMOD dataset to show that PoseCNN is highly robust to occlusions, can handle symmetric objects, and provide accurate pose estimation using only color images as input.

6D Pose Estimation 6D Pose Estimation using RGB +2

CLIPort: What and Where Pathways for Robotic Manipulation

1 code implementation24 Sep 2021 Mohit Shridhar, Lucas Manuelli, Dieter Fox

We even learn one multi-task policy for 10 simulated and 9 real-world tasks that is better or comparable to single-task policies.

Imitation Learning Robotic Grasping

DynamicFusion: Reconstruction and Tracking of Non-Rigid Scenes in Real-Time

1 code implementation CVPR 2015 Richard A. Newcombe, Dieter Fox, Steven M. Seitz

We present the first dense SLAM system capable of reconstructing non-rigidly deforming scenes in real-time, by fusing together RGBD scans captured from commodity sensors.

ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks

7 code implementations CVPR 2020 Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, Dieter Fox

We present ALFRED (Action Learning From Realistic Environments and Directives), a benchmark for learning a mapping from natural language instructions and egocentric vision to sequences of actions for household tasks.

Natural Language Visual Grounding

Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation

1 code implementation12 Sep 2022 Mohit Shridhar, Lucas Manuelli, Dieter Fox

With this formulation, we train a single multi-task Transformer for 18 RLBench tasks (with 249 variations) and 7 real-world tasks (with 18 variations) from just a few demonstrations per task.

Robot Manipulation

Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes

1 code implementation25 Mar 2021 Martin Sundermeyer, Arsalan Mousavian, Rudolph Triebel, Dieter Fox

Our novel grasp representation treats 3D points of the recorded point cloud as potential grasp contacts.

Grasp Generation Robotic Grasping

DeepIM: Deep Iterative Matching for 6D Pose Estimation

2 code implementations ECCV 2018 Yi Li, Gu Wang, Xiangyang Ji, Yu Xiang, Dieter Fox

Estimating the 6D pose of objects from images is an important problem in various applications such as robot manipulation and virtual reality.

6D Pose Estimation 6D Pose Estimation using RGB +1

ACRONYM: A Large-Scale Grasp Dataset Based on Simulation

2 code implementations18 Nov 2020 Clemens Eppner, Arsalan Mousavian, Dieter Fox

We introduce ACRONYM, a dataset for robot grasp planning based on physics simulation.

Self-supervised 6D Object Pose Estimation for Robot Manipulation

3 code implementations23 Sep 2019 Xinke Deng, Yu Xiang, Arsalan Mousavian, Clemens Eppner, Timothy Bretl, Dieter Fox

In this way, our system is able to continuously collect data and improve its pose estimation modules.

Robotics

Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation

1 code implementation30 Jul 2020 Yu Xiang, Christopher Xie, Arsalan Mousavian, Dieter Fox

In this work, we propose a new method for unseen object instance segmentation by learning RGB-D feature embeddings from synthetic data.

Clustering Metric Learning +4

Predicting Stable Configurations for Semantic Placement of Novel Objects

1 code implementation26 Aug 2021 Chris Paxton, Chris Xie, Tucker Hermans, Dieter Fox

We further demonstrate the ability of our planner to generate and execute diverse manipulation plans through a set of real-world experiments with a variety of objects.

Motion Planning valid

RVT: Robotic View Transformer for 3D Object Manipulation

1 code implementation26 Jun 2023 Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, Dieter Fox

In simulations, we find that a single RVT model works well across 18 RLBench tasks with 249 task variations, achieving 26% higher relative success than the existing state-of-the-art method (PerAct).

Object Robot Manipulation

Camera-to-Robot Pose Estimation from a Single Image

2 code implementations21 Nov 2019 Timothy E. Lee, Jonathan Tremblay, Thang To, Jia Cheng, Terry Mosier, Oliver Kroemer, Dieter Fox, Stan Birchfield

We show experimental results for three different camera sensors, demonstrating that our approach is able to achieve accuracy with a single frame that is better than that of classic off-line hand-eye calibration using multiple frames.

Robotics

PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Tracking

1 code implementation22 May 2019 Xinke Deng, Arsalan Mousavian, Yu Xiang, Fei Xia, Timothy Bretl, Dieter Fox

In this work, we formulate the 6D object pose tracking problem in the Rao-Blackwellized particle filtering framework, where the 3D rotation and the 3D translation of an object are decoupled.

6D Pose Estimation 6D Pose Estimation using RGB +3

Motion Policy Networks

1 code implementation21 Oct 2022 Adam Fishman, Adithyavairan Murali, Clemens Eppner, Bryan Peele, Byron Boots, Dieter Fox

Collision-free motion generation in unknown environments is a core building block for robot manipulation.

Motion Planning Robot Manipulation

Manipulation Trajectory Optimization with Online Grasp Synthesis and Selection

1 code implementation22 Nov 2019 Lirui Wang, Yu Xiang, Dieter Fox

In robot manipulation, planning the motion of a robot manipulator to grasp an object is a fundamental problem.

Robotics

Re3 : Real-Time Recurrent Regression Networks for Visual Tracking of Generic Objects

10 code implementations17 May 2017 Daniel Gordon, Ali Farhadi, Dieter Fox

Robust object tracking requires knowledge and understanding of the object being tracked: its appearance, its motion, and how it changes over time.

Object Object Tracking +2

Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds

1 code implementation2 Oct 2020 Lirui Wang, Yu Xiang, Wei Yang, Arsalan Mousavian, Dieter Fox

We demonstrate that our learned policy can be integrated into a tabletop 6D grasping system and a human-robot handover system to improve the grasping performance of unseen objects.

Imitation Learning Motion Planning +2

Causal Discovery in Physical Systems from Videos

1 code implementation NeurIPS 2020 Yunzhu Li, Antonio Torralba, Animashree Anandkumar, Dieter Fox, Animesh Garg

We assume access to different configurations and environmental conditions, i. e., data from unknown interventions on the underlying system; thus, we can hope to discover the correct underlying causal graph without explicit interventions.

Causal Discovery counterfactual

Watching the World Go By: Representation Learning from Unlabeled Videos

1 code implementation18 Mar 2020 Daniel Gordon, Kiana Ehsani, Dieter Fox, Ali Farhadi

Recent single image unsupervised representation learning techniques show remarkable success on a variety of tasks.

Data Augmentation Representation Learning

SPNets: Differentiable Fluid Dynamics for Deep Neural Networks

1 code implementation15 Jun 2018 Connor Schenck, Dieter Fox

In this paper we introduce Smooth Particle Networks (SPNets), a framework for integrating fluid dynamics with deep networks.

Robotics

Online Replanning in Belief Space for Partially Observable Task and Motion Problems

1 code implementation11 Nov 2019 Caelan Reed Garrett, Chris Paxton, Tomás Lozano-Pérez, Leslie Pack Kaelbling, Dieter Fox

To solve multi-step manipulation tasks in the real world, an autonomous robot must take actions to observe its environment and react to unexpected observations.

Continuous Control

RGB-D Local Implicit Function for Depth Completion of Transparent Objects

1 code implementation CVPR 2021 Luyang Zhu, Arsalan Mousavian, Yu Xiang, Hammad Mazhar, Jozef van Eenbergen, Shoubhik Debnath, Dieter Fox

Key to our approach is a local implicit neural representation built on ray-voxel pairs that allows our method to generalize to unseen objects and achieve fast inference speed.

Depth Completion Depth Estimation +1

RMPflow: A Computational Graph for Automatic Motion Policy Generation

1 code implementation16 Nov 2018 Ching-An Cheng, Mustafa Mukadam, Jan Issac, Stan Birchfield, Dieter Fox, Byron Boots, Nathan Ratliff

We develop a novel policy synthesis algorithm, RMPflow, based on geometrically consistent transformations of Riemannian Motion Policies (RMPs).

Robotics Systems and Control

Break and Make: Interactive Structural Understanding Using LEGO Bricks

2 code implementations27 Jul 2022 Aaron Walsman, Muru Zhang, Klemen Kotar, Karthik Desingh, Ali Farhadi, Dieter Fox

We pair this simulator with a new dataset of fan-made LEGO creations that have been uploaded to the internet in order to provide complex scenes containing over a thousand unique brick shapes.

Object Rearrangement Using Learned Implicit Collision Functions

1 code implementation21 Nov 2020 Michael Danielczuk, Arsalan Mousavian, Clemens Eppner, Dieter Fox

The learned model outperforms both traditional pipelines and learned ablations by 9. 8% in accuracy on a dataset of simulated collision queries and is 75x faster than the best-performing baseline.

Object

BayesSimIG: Scalable Parameter Inference for Adaptive Domain Randomization with IsaacGym

1 code implementation9 Jul 2021 Rika Antonova, Fabio Ramos, Rafael Possas, Dieter Fox

This paper outlines BayesSimIG: a library that provides an implementation of BayesSim integrated with the recently released NVIDIA IsaacGym.

Reinforcement Learning (RL)

SORNet: Spatial Object-Centric Representations for Sequential Manipulation

1 code implementation8 Sep 2021 Wentao Yuan, Chris Paxton, Karthik Desingh, Dieter Fox

Sequential manipulation tasks require a robot to perceive the state of an environment and plan a sequence of actions leading to a desired goal state.

Object Relation Classification +1

A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution

1 code implementation12 Jul 2021 Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, Yoav Artzi

Natural language provides an accessible and expressive interface to specify long-term tasks for robotic agents.

RICE: Refining Instance Masks in Cluttered Environments with Graph Neural Networks

1 code implementation29 Jun 2021 Christopher Xie, Arsalan Mousavian, Yu Xiang, Dieter Fox

We postulate that a network architecture that encodes relations between objects at a high-level can be beneficial.

BayesSim: adaptive domain randomization via probabilistic inference for robotics simulators

1 code implementation4 Jun 2019 Fabio Ramos, Rafael Carvalhaes Possas, Dieter Fox

We introduce BayesSim, a framework for robotics simulations allowing a full Bayesian treatment for the parameters of the simulator.

Motion Planning

Multimodal Trajectory Prediction via Topological Invariance for Navigation at Uncontrolled Intersections

1 code implementation8 Nov 2020 Junha Roh, Christoforos Mavrogiannis, Rishabh Madan, Dieter Fox, Siddhartha S. Srinivasa

Our key insight is that the geometric structure of the intersection and the incentive of agents to move efficiently and avoid collisions (rationality) reduces the space of likely behaviors, effectively relaxing the problem of trajectory prediction.

Trajectory Prediction

THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation

1 code implementation13 Feb 2024 Wilbert Pumacay, Ishika Singh, Jiafei Duan, Ranjay Krishna, Jesse Thomason, Dieter Fox

To realize effective large-scale, real-world robotic applications, we must evaluate how well our robot policies adapt to changes in environmental conditions.

Robot Manipulation Generalization

iCaps: Iterative Category-level Object Pose and Shape Estimation

1 code implementation31 Dec 2021 Xinke Deng, Junyi Geng, Timothy Bretl, Yu Xiang, Dieter Fox

The auto-encoder can be used in a particle filter framework to estimate and track 6D poses of objects in a category.

Object

Euclideanizing Flows: Diffeomorphic Reduction for Learning Stable Dynamical Systems

2 code implementations L4DC 2020 Muhammad Asif Rana, Anqi Li, Dieter Fox, Byron Boots, Fabio Ramos, Nathan Ratliff

The complex motions are encoded as rollouts of a stable dynamical system, which, under a change of coordinates defined by a diffeomorphism, is equivalent to a simple, hand-specified dynamical system.

Density Estimation

Value Iteration in Continuous Actions, States and Time

1 code implementation10 May 2021 Michael Lutter, Shie Mannor, Jan Peters, Dieter Fox, Animesh Garg

This algorithm enables dynamic programming for continuous states and actions with a known dynamics model.

Robust Value Iteration for Continuous Control Tasks

1 code implementation25 May 2021 Michael Lutter, Shie Mannor, Jan Peters, Dieter Fox, Animesh Garg

The adversarial perturbations encourage a optimal policy that is robust to changes in the dynamics.

Continuous Control reinforcement-learning +1

Continuous-Time Fitted Value Iteration for Robust Policies

1 code implementation5 Oct 2021 Michael Lutter, Boris Belousov, Shie Mannor, Dieter Fox, Animesh Garg, Jan Peters

Especially for continuous control, solving this differential equation and its extension the Hamilton-Jacobi-Isaacs equation, is important as it yields the optimal policy that achieves the maximum reward on a give task.

Continuous Control

Gaussian Processes for Data-Efficient Learning in Robotics and Control

1 code implementation10 Feb 2015 Marc Peter Deisenroth, Dieter Fox, Carl Edward Rasmussen

Autonomous learning has been a promising direction in control and robotics for more than a decade since data-driven learning allows to reduce the amount of engineering knowledge, which is otherwise required.

Gaussian Processes Reinforcement Learning (RL)

ContactGrasp: Functional Multi-finger Grasp Synthesis from Contact

4 code implementations7 Apr 2019 Samarth Brahmbhatt, Ankur Handa, James Hays, Dieter Fox

Using a dataset of contact demonstrations from humans grasping diverse household objects, we synthesize functional grasps for three hand models and two functional intents.

Object

DA-RNN: Semantic Mapping with Data Associated Recurrent Neural Networks

1 code implementation9 Mar 2017 Yu Xiang, Dieter Fox

3D scene understanding is important for robots to interact with the 3D world in a meaningful way.

Scene Understanding

Factory: Fast Contact for Robotic Assembly

1 code implementation7 May 2022 Yashraj Narang, Kier Storey, Iretiayo Akinola, Miles Macklin, Philipp Reist, Lukasz Wawrzyniak, Yunrong Guo, Adam Moravanszky, Gavriel State, Michelle Lu, Ankur Handa, Dieter Fox

We aim for Factory to open the doors to using simulation for robotic assembly, as well as many other contact-rich applications in robotics.

Simulating Action Dynamics with Neural Process Networks

no code implementations ICLR 2018 Antoine Bosselut, Omer Levy, Ari Holtzman, Corin Ennis, Dieter Fox, Yejin Choi

Understanding procedural language requires anticipating the causal effects of actions, even when they are not explicitly stated.

Dynamic High Resolution Deformable Articulated Tracking

no code implementations21 Nov 2017 Aaron Walsman, Weilin Wan, Tanner Schmidt, Dieter Fox

The last several years have seen significant progress in using depth cameras for tracking articulated objects such as human bodies, hands, and robotic manipulators.

Pose Estimation Vocal Bursts Intensity Prediction

SE3-Pose-Nets: Structured Deep Dynamics Models for Visuomotor Planning and Control

no code implementations2 Oct 2017 Arunkumar Byravan, Felix Leeb, Franziska Meier, Dieter Fox

In this work, we present an approach to deep visuomotor control using structured deep dynamics models.

Perceiving and Reasoning About Liquids Using Fully Convolutional Networks

no code implementations5 Mar 2017 Conor Schenck, Dieter Fox

That is, a robot asks the questions What in the visual data stream is liquid?

See the Glass Half Full: Reasoning about Liquid Containers, their Volume and Content

no code implementations ICCV 2017 Roozbeh Mottaghi, Connor Schenck, Dieter Fox, Ali Farhadi

Doing so requires estimating the volume of the cup, approximating the amount of water in the pitcher, and predicting the behavior of water when we tilt the pitcher.

Reasoning About Liquids via Closed-Loop Simulation

no code implementations5 Mar 2017 Connor Schenck, Dieter Fox

In this paper, we show how to close the loop between liquid simulation and real-time perception.

Liquid Simulation

SE3-Nets: Learning Rigid Body Motion using Deep Neural Networks

no code implementations8 Jun 2016 Arunkumar Byravan, Dieter Fox

We introduce SE3-Nets, which are deep neural networks designed to model and learn rigid body motion from raw point cloud data.

Visual Closed-Loop Control for Pouring Liquids

no code implementations9 Oct 2016 Connor Schenck, Dieter Fox

We propose both a model-based and a model-free method utilizing deep learning for estimating the volume of liquid in a container.

Towards Learning to Perceive and Reason About Liquids

no code implementations2 Aug 2016 Connor Schenck, Dieter Fox

Recent advances in AI and robotics have claimed many incredible results with deep learning, yet no work to date has applied deep learning to the problem of liquid perception and reasoning.

Detection and Tracking of Liquids with Fully Convolutional Networks

no code implementations20 Jun 2016 Connor Schenck, Dieter Fox

In this paper, we apply fully-convolutional deep neural networks to the tasks of detecting and tracking liquids.

Image Segmentation Semantic Segmentation

Building Hierarchies of Concepts via Crowdsourcing

no code implementations27 Apr 2015 Yuyin Sun, Adish Singla, Dieter Fox, Andreas Krause

Hierarchies of concepts are useful in many applications from navigation to organization of objects.

Multi-Task Policy Search

no code implementations2 Jul 2013 Marc Peter Deisenroth, Peter Englert, Jan Peters, Dieter Fox

Learning policies that generalize across multiple tasks is an important and challenging research topic in reinforcement learning and robotics.

Imitation Learning reinforcement-learning +1

Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience

no code implementations12 Oct 2018 Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, Dieter Fox

In doing so, we are able to change the distribution of simulations to improve the policy transfer by matching the policy behavior in simulation and the real world.

Early Fusion for Goal Directed Robotic Vision

no code implementations21 Nov 2018 Aaron Walsman, Yonatan Bisk, Saadia Gabriel, Dipendra Misra, Yoav Artzi, Yejin Choi, Dieter Fox

Building perceptual systems for robotics which perform well under tight computational budgets requires novel architectures which rethink the traditional computer vision pipeline.

Imitation Learning Retrieval

Kernel Descriptors for Visual Recognition

no code implementations NeurIPS 2010 Liefeng Bo, Xiaofeng Ren, Dieter Fox

We highlight the kernel view of orientation histograms, and show that they are equivalent to a certain type of match kernels over image patches.

Attribute Image Classification +1

What Should I Do Now? Marrying Reinforcement Learning and Symbolic Planning

no code implementations6 Jan 2019 Daniel Gordon, Dieter Fox, Ali Farhadi

In this work we propose Hierarchical Planning and Reinforcement Learning (HIP-RL), a method for merging the benefits and capabilities of Symbolic Planning with the learning abilities of Deep Reinforcement Learning.

Question Answering reinforcement-learning +1

Multipath Sparse Coding Using Hierarchical Matching Pursuit

no code implementations CVPR 2013 Liefeng Bo, Xiaofeng Ren, Dieter Fox

Complex real-world signals, such as images, contain discriminative structures that differ in many aspects including scale, invariance, and data channel.

Image Classification

Prospection: Interpretable Plans From Language By Predicting the Future

no code implementations20 Mar 2019 Chris Paxton, Yonatan Bisk, Jesse Thomason, Arunkumar Byravan, Dieter Fox

High-level human instructions often correspond to behaviors with multiple implicit steps.

GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning

no code implementations12 Oct 2018 Jacky Liang, Viktor Makoviychuk, Ankur Handa, Nuttapong Chentanez, Miles Macklin, Dieter Fox

Most Deep Reinforcement Learning (Deep RL) algorithms require a prohibitively large number of training samples for learning complex tasks.

Robotics

The Best of Both Modes: Separately Leveraging RGB and Depth for Unseen Object Instance Segmentation

no code implementations30 Jul 2019 Christopher Xie, Yu Xiang, Arsalan Mousavian, Dieter Fox

We show that our method, trained on this dataset, can produce sharp and accurate masks, outperforming state-of-the-art methods on unseen object instance segmentation.

Object Segmentation +2

Part Segmentation for Highly Accurate Deformable Tracking in Occlusions via Fully Convolutional Neural Networks

no code implementations5 Aug 2019 Weilin Wan, Aaron Walsman, Dieter Fox

While recent work has shown direct estimation techniques can be quite powerful, geometric tracking methods using point clouds can provide a very high level of 3D accuracy which is necessary for many robotic applications.

Data Augmentation Pose Estimation

Riemannian Motion Policy Fusion through Learnable Lyapunov Function Reshaping

no code implementations7 Oct 2019 Mustafa Mukadam, Ching-An Cheng, Dieter Fox, Byron Boots, Nathan Ratliff

RMPfusion supplements RMPflow with weight functions that can hierarchically reshape the Lyapunov functions of the subtask RMPs according to the current configuration of the robot and environment.

Imitation Learning

DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System

no code implementations7 Oct 2019 Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff, Dieter Fox

Teleoperation offers the possibility of imparting robotic systems with sophisticated reasoning skills, intuition, and creativity to perform tasks.

Conditional Driving from Natural Language Instructions

no code implementations16 Oct 2019 Junha Roh, Chris Paxton, Andrzej Pronobis, Ali Farhadi, Dieter Fox

Widespread adoption of self-driving cars will depend not only on their safety but largely on their ability to interact with human users.

Imitation Learning Self-Driving Cars

Motion-Nets: 6D Tracking of Unknown Objects in Unseen Environments using RGB

no code implementations30 Oct 2019 Felix Leeb, Arunkumar Byravan, Dieter Fox

In this work, we bridge the gap between recent pose estimation and tracking work to develop a powerful method for robots to track objects in their surroundings.

Pose Estimation Translation

IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data

no code implementations13 Nov 2019 Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Li Fei-Fei, Animesh Garg, Dieter Fox

For simple short-horizon manipulation tasks with modest variation in task instances, offline learning from a small set of demonstrations can produce controllers that successfully solve the task.

Robot Manipulation

Motion Reasoning for Goal-Based Imitation Learning

no code implementations13 Nov 2019 De-An Huang, Yu-Wei Chao, Chris Paxton, Xinke Deng, Li Fei-Fei, Juan Carlos Niebles, Animesh Garg, Dieter Fox

We further show that by using the automatically inferred goal from the video demonstration, our robot is able to reproduce the same task in a real kitchen environment.

Imitation Learning Motion Planning +1

Contextual Reinforcement Learning of Visuo-tactile Multi-fingered Grasping Policies

no code implementations21 Nov 2019 Visak Kumar, Tucker Herman, Dieter Fox, Stan Birchfield, Jonathan Tremblay

We propose a Grasping Objects Approach for Tactile (GOAT) robotic hands, learning to overcome the reality gap problem.

Robotics

A Billion Ways to Grasp: An Evaluation of Grasp Sampling Schemes on a Dense, Physics-based Grasp Data Set

no code implementations11 Dec 2019 Clemens Eppner, Arsalan Mousavian, Dieter Fox

With the increasing speed and quality of physics simulations, generating large-scale grasping data sets that feed learning algorithms is becoming more and more popular.

Information Theoretic Model Predictive Q-Learning

no code implementations31 Dec 2019 Mohak Bhardwaj, Ankur Handa, Dieter Fox, Byron Boots

Model-free Reinforcement Learning (RL) works well when experience can be collected cheaply and model-based RL is effective when system dynamics can be modeled accurately.

Decision Making Model Predictive Control +3

The Limits and Potentials of Deep Learning for Robotics

no code implementations18 Apr 2018 Niko Sünderhauf, Oliver Brock, Walter Scheirer, Raia Hadsell, Dieter Fox, Jürgen Leitner, Ben Upcroft, Pieter Abbeel, Wolfram Burgard, Michael Milford, Peter Corke

In this paper we discuss a number of robotics-specific learning, reasoning, and embodiment challenges for deep learning.

Robotics

Transferable Task Execution from Pixels through Deep Planning Domain Learning

no code implementations8 Mar 2020 Kei Kase, Chris Paxton, Hammad Mazhar, Tetsuya OGATA, Dieter Fox

On the other hand, symbolic planning methods such as STRIPS have long been able to solve new problems given only a domain definition and a symbolic goal, but these approaches often struggle on the real world robotic tasks due to the challenges of grounding these symbols from sensor data in a partially-observable world.

Human Grasp Classification for Reactive Human-to-Robot Handovers

no code implementations12 Mar 2020 Wei Yang, Chris Paxton, Maya Cakmak, Dieter Fox

In this paper, we propose an approach for human-to-robot handovers in which the robot meets the human halfway, by classifying the human's grasp of the object and quickly planning a trajectory accordingly to take the object from the human's hand according to their intent.

Classification General Classification +1

Guided Uncertainty-Aware Policy Optimization: Combining Learning and Model-Based Strategies for Sample-Efficient Policy Learning

no code implementations21 May 2020 Michelle A. Lee, Carlos Florensa, Jonathan Tremblay, Nathan Ratliff, Animesh Garg, Fabio Ramos, Dieter Fox

Traditional robotic approaches rely on an accurate model of the environment, a detailed description of how to perform the task, and a robust perception system to keep track of the current state.

Stein Variational Model Predictive Control

no code implementations15 Nov 2020 Alexander Lambert, Adam Fishman, Dieter Fox, Byron Boots, Fabio Ramos

By casting MPC as a Bayesian inference problem, we employ variational methods for posterior computation, naturally encoding the complexity and multi-modality of the decision making problem.

Bayesian Inference Decision Making +2

Reactive Long Horizon Task Execution via Visual Skill and Precondition Models

no code implementations17 Nov 2020 Shohin Mukherjee, Chris Paxton, Arsalan Mousavian, Adam Fishman, Maxim Likhachev, Dieter Fox

Zero-shot execution of unseen robotic tasks is important to allowing robots to perform a wide variety of tasks in human environments, but collecting the amounts of data necessary to train end-to-end policies in the real-world is often infeasible.

Reactive Human-to-Robot Handovers of Arbitrary Objects

no code implementations17 Nov 2020 Wei Yang, Chris Paxton, Arsalan Mousavian, Yu-Wei Chao, Maya Cakmak, Dieter Fox

We demonstrate the generalizability, usability, and robustness of our approach on a novel benchmark set of 26 diverse household objects, a user study with naive users (N=6) handing over a subset of 15 objects, and a systematic evaluation examining different ways of handing objects.

Grasp Generation Motion Planning

Perspectives on Sim2Real Transfer for Robotics: A Summary of the R:SS 2020 Workshop

no code implementations7 Dec 2020 Sebastian Höfer, Kostas Bekris, Ankur Handa, Juan Camilo Gamboa, Florian Golemo, Melissa Mozifian, Chris Atkeson, Dieter Fox, Ken Goldberg, John Leonard, C. Karen Liu, Jan Peters, Shuran Song, Peter Welinder, Martha White

This report presents the debates, posters, and discussions of the Sim2Real workshop held in conjunction with the 2020 edition of the "Robotics: Science and System" conference.

Towards Coordinated Robot Motions: End-to-End Learning of Motion Policies on Transform Trees

no code implementations24 Dec 2020 M. Asif Rana, Anqi Li, Dieter Fox, Sonia Chernova, Byron Boots, Nathan Ratliff

The policy structure provides the user an interface to 1) specifying the spaces that are directly relevant to the completion of the tasks, and 2) designing policies for certain tasks that do not need to be learned.

NeRP: Neural Rearrangement Planning for Unknown Objects

no code implementations2 Jun 2021 Ahmed H. Qureshi, Arsalan Mousavian, Chris Paxton, Michael C. Yip, Dieter Fox

We propose NeRP (Neural Rearrangement Planning), a deep learning based approach for multi-step neural object rearrangement planning which works with never-before-seen objects, that is trained on simulation data, and generalizes to the real world.

LanguageRefer: Spatial-Language Model for 3D Visual Grounding

no code implementations7 Jul 2021 Junha Roh, Karthik Desingh, Ali Farhadi, Dieter Fox

Specifically, given a reconstructed 3D scene in the form of point clouds with 3D bounding boxes of potential object candidates, and a language utterance referring to a target object in the scene, our model successfully identifies the target object from a set of potential candidates.

Language Modelling Object +1

Generalizing Successor Features to continuous domains for Multi-task Learning

no code implementations29 Sep 2021 Melissa Mozifian, Dieter Fox, David Meger, Fabio Ramos, Animesh Garg

In this paper, we consider the problem of continuous control for various robot manipulation tasks with an explicit representation that promotes skill reuse while learning multiple tasks, related through the reward function.

Continuous Control Decision Making +3

StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects

no code implementations19 Oct 2021 Weiyu Liu, Chris Paxton, Tucker Hermans, Dieter Fox

Geometric organization of objects into semantically meaningful arrangements pervades the built world.

Object

Learning Perceptual Concepts by Bootstrapping from Human Queries

no code implementations9 Nov 2021 Andreea Bobu, Chris Paxton, Wei Yang, Balakumar Sundaralingam, Yu-Wei Chao, Maya Cakmak, Dieter Fox

Second, we treat this low-dimensional concept as an automatic labeler to synthesize a large-scale high-dimensional data set with the simulator.

Motion Planning Object

IFOR: Iterative Flow Minimization for Robotic Object Rearrangement

no code implementations CVPR 2022 Ankit Goyal, Arsalan Mousavian, Chris Paxton, Yu-Wei Chao, Brian Okorn, Jia Deng, Dieter Fox

Accurate object rearrangement from vision is a crucial problem for a wide variety of real-world robotics applications in unstructured environments.

Object Optical Flow Estimation

Neural Motion Fields: Encoding Grasp Trajectories as Implicit Value Functions

no code implementations29 Jun 2022 Yun-Chun Chen, Adithyavairavan Murali, Balakumar Sundaralingam, Wei Yang, Animesh Garg, Dieter Fox

The pipeline of current robotic pick-and-place methods typically consists of several stages: grasp pose detection, finding inverse kinematic solutions for the detected poses, planning a collision-free trajectory, and then executing the open-loop trajectory to the grasp pose with a low-level tracking controller.

Object

ProgPrompt: Generating Situated Robot Task Plans using Large Language Models

no code implementations22 Sep 2022 Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, Animesh Garg

To ameliorate that effort, large language models (LLMs) can be used to score potential next actions during task planning, and even generate action sequences directly, given an instruction in natural language with no additional domain information.

DexTransfer: Real World Multi-fingered Dexterous Grasping with Minimal Human Demonstrations

no code implementations28 Sep 2022 Zoey Qiuyu Chen, Karl Van Wyk, Yu-Wei Chao, Wei Yang, Arsalan Mousavian, Abhishek Gupta, Dieter Fox

The policy learned from our dataset can generalize well on unseen object poses in both simulation and the real world

Object

One-Shot Neural Fields for 3D Object Understanding

no code implementations21 Oct 2022 Valts Blukis, Taeyeop Lee, Jonathan Tremblay, Bowen Wen, In So Kweon, Kuk-Jin Yoon, Dieter Fox, Stan Birchfield

At test-time, we build the representation from a single RGB input image observing the scene from only one viewpoint.

3D Reconstruction Object +2

Sequence-Based Plan Feasibility Prediction for Efficient Task and Motion Planning

no code implementations3 Nov 2022 Zhutian Yang, Caelan Reed Garrett, Tomás Lozano-Pérez, Leslie Kaelbling, Dieter Fox

The core of our algorithm is PIGINet, a novel Transformer-based learning method that takes in a task plan, the goal, and the initial state, and predicts the probability of finding motion trajectories associated with the task plan.

Motion Planning Task and Motion Planning +1

MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare

no code implementations13 Dec 2022 Yann Labbé, Lucas Manuelli, Arsalan Mousavian, Stephen Tyree, Stan Birchfield, Jonathan Tremblay, Justin Carpentier, Mathieu Aubry, Dieter Fox, Josef Sivic

Second, we introduce a novel approach for coarse pose estimation which leverages a network trained to classify whether the pose error between a synthetic rendering and an observed image of the same object can be corrected by the refiner.

6D Pose Estimation Object

A Joint Model of Language and Perception for Grounded Attribute Learning

no code implementations27 Jun 2012 Cynthia Matuszek, Nicholas FitzGerald, Luke Zettlemoyer, Liefeng Bo, Dieter Fox

As robots become more ubiquitous and capable, it becomes ever more important to enable untrained users to easily interact with them.

Attribute Language Modelling

Learning Human-to-Robot Handovers from Point Clouds

no code implementations CVPR 2023 Sammy Christen, Wei Yang, Claudia Pérez-D'Arpino, Otmar Hilliges, Dieter Fox, Yu-Wei Chao

We propose the first framework to learn control policies for vision-based human-to-robot handovers, a critical task for human-robot interaction.

Partial-View Object View Synthesis via Filtered Inversion

no code implementations3 Apr 2023 Fan-Yun Sun, Jonathan Tremblay, Valts Blukis, Kevin Lin, Danfei Xu, Boris Ivanovic, Peter Karkus, Stan Birchfield, Dieter Fox, Ruohan Zhang, Yunzhu Li, Jiajun Wu, Marco Pavone, Nick Haber

At inference, given one or more views of a novel real-world object, FINV first finds a set of latent codes for the object by inverting the generative model from multiple initial seeds.

Object

CabiNet: Scaling Neural Collision Detection for Object Rearrangement with Procedural Scene Generation

no code implementations18 Apr 2023 Adithyavairavan Murali, Arsalan Mousavian, Clemens Eppner, Adam Fishman, Dieter Fox

CabiNet is a collision model that accepts object and scene point clouds, captured from a single-view depth observation, and predicts collisions for SE(3) object poses in the scene.

Navigate Object +1

Imitating Task and Motion Planning with Visuomotor Transformers

no code implementations25 May 2023 Murtaza Dalal, Ajay Mandlekar, Caelan Garrett, Ankur Handa, Ruslan Salakhutdinov, Dieter Fox

In this work, we show that the combination of large-scale datasets generated by TAMP supervisors and flexible Transformer models to fit them is a powerful paradigm for robot manipulation.

Imitation Learning Motion Planning +2

AR2-D2:Training a Robot Without a Robot

no code implementations23 Jun 2023 Jiafei Duan, Yi Ru Wang, Mohit Shridhar, Dieter Fox, Ranjay Krishna

By contrast, we introduce AR2-D2: a system for collecting demonstrations which (1) does not require people with specialized training, (2) does not require any real robots during data collection, and therefore, (3) enables manipulation of diverse objects with a real robot.

Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangement

no code implementations10 Jul 2023 Anthony Simeonov, Ankit Goyal, Lucas Manuelli, Lin Yen-Chen, Alina Sarmiento, Alberto Rodriguez, Pulkit Agrawal, Dieter Fox

We propose a system for rearranging objects in a scene to achieve a desired object-scene placing relationship, such as a book inserted in an open slot of a bookshelf.

AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System

no code implementations10 Jul 2023 Yuzhe Qin, Wei Yang, Binghao Huang, Karl Van Wyk, Hao Su, Xiaolong Wang, Yu-Wei Chao, Dieter Fox

For real-world experiments, AnyTeleop can outperform a previous system that was designed for a specific robot hardware with a higher success rate, using the same robot.

Imitation Learning

NEWTON: Are Large Language Models Capable of Physical Reasoning?

no code implementations10 Oct 2023 Yi Ru Wang, Jiafei Duan, Dieter Fox, Siddhartha Srinivasa

To address this gap, we introduce NEWTON, a repository and benchmark for evaluating the physics reasoning skills of LLMs.

Attribute Common Sense Reasoning

Human-in-the-Loop Task and Motion Planning for Imitation Learning

no code implementations24 Oct 2023 Ajay Mandlekar, Caelan Garrett, Danfei Xu, Dieter Fox

Finally, we collected 2. 1K demos with HITL-TAMP across 12 contact-rich, long-horizon tasks and show that the system often produces near-perfect agents.

Imitation Learning Motion Planning +1

MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations

no code implementations26 Oct 2023 Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, Dieter Fox

Imitation learning from a large set of human demonstrations has proved to be an effective paradigm for building capable robot agents.

Imitation Learning

M2T2: Multi-Task Masked Transformer for Object-centric Pick and Place

no code implementations2 Nov 2023 Wentao Yuan, Adithyavairavan Murali, Arsalan Mousavian, Dieter Fox

With the advent of large language models and large-scale robotic datasets, there has been tremendous progress in high-level decision-making for object manipulation.

Decision Making valid

STOW: Discrete-Frame Segmentation and Tracking of Unseen Objects for Warehouse Picking Robots

no code implementations4 Nov 2023 Yi Li, Muru Zhang, Markus Grotz, Kaichun Mo, Dieter Fox

Segmentation and tracking of unseen object instances in discrete frames pose a significant challenge in dynamic industrial robotic contexts, such as distribution warehouses.

Cannot find the paper you are looking for? You can Submit a new open access paper.