Search Results for author: Lerrel Pinto

Found 62 papers, 31 papers with code

Hierarchically Decoupled Morphological Transfer

no code implementations ICML 2020 Donald Hejna, Lerrel Pinto, Pieter Abbeel

Learning long-range behaviors on complex high-dimensional agents is a fundamental problem in robot learning.

Behavior Generation with Latent Actions

1 code implementation5 Mar 2024 Seungjae Lee, Yibin Wang, Haritheja Etukuru, H. Jin Kim, Nur Muhammad Mahi Shafiullah, Lerrel Pinto

Unlike language or image generation, decision making requires modeling actions - continuous-valued vectors that are multimodal in their distribution, potentially drawn from uncurated sources, where generation errors can compound in sequential prediction.

Autonomous Driving Decision Making +2

Hierarchical State Space Models for Continuous Sequence-to-Sequence Modeling

1 code implementation15 Feb 2024 Raunaq Bhirangi, Chenyu Wang, Venkatesh Pattabiraman, Carmel Majidi, Abhinav Gupta, Tess Hellebrekers, Lerrel Pinto

Reasoning from sequences of raw sensory data is a ubiquitous problem across fields ranging from medical devices to robotics.

OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics

1 code implementation22 Jan 2024 Peiqi Liu, Yaswanth Orru, Jay Vakil, Chris Paxton, Nur Muhammad Mahi Shafiullah, Lerrel Pinto

The results demonstrate that OK-Robot achieves a 58. 5% success rate in open-ended pick-and-drop tasks, representing a new state-of-the-art in Open Vocabulary Mobile Manipulation (OVMM) with nearly 1. 8x the performance of prior work.

object-detection Object Detection

diff History for Neural Language Agents

1 code implementation12 Dec 2023 Ulyana Piterbarg, Lerrel Pinto, Rob Fergus

On NetHack, an unsolved video game that requires long-horizon reasoning for decision-making, LMs tuned with diff history match state-of-the-art performance for neural agents while needing 1800x fewer training examples compared to prior work.

Decision Making NetHack +1

On Bringing Robots Home

1 code implementation27 Nov 2023 Nur Muhammad Mahi Shafiullah, Anant Rai, Haritheja Etukuru, Yiqian Liu, Ishan Misra, Soumith Chintala, Lerrel Pinto

We use the Stick to collect 13 hours of data in 22 homes of New York City, and train Home Pretrained Representations (HPR).

Improving Long-Horizon Imitation Through Instruction Prediction

1 code implementation21 Jun 2023 Joey Hejna, Pieter Abbeel, Lerrel Pinto

Complex, long-horizon planning and its combinatorial nature pose steep challenges for learning-based agents.

Train Offline, Test Online: A Real Robot Learning Benchmark

1 code implementation1 Jun 2023 Gaoyue Zhou, Victoria Dean, Mohan Kumar Srirama, Aravind Rajeswaran, Jyothish Pari, Kyle Hatch, Aryan Jain, Tianhe Yu, Pieter Abbeel, Lerrel Pinto, Chelsea Finn, Abhinav Gupta

Three challenges limit the progress of robot learning research: robots are expensive (few labs can participate), everyone uses different robots (findings do not generalize across labs), and we lack internet-scale robotics data.

Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play

no code implementations21 Mar 2023 Irmak Guzey, Ben Evans, Soumith Chintala, Lerrel Pinto

In the first phase, we collect 2. 5 hours of play data, which is used to train self-supervised tactile encoders.

Representation Learning

Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations

1 code implementation2 Mar 2023 Siddhant Haldar, Jyothish Pari, Anant Rai, Lerrel Pinto

Given a weak base-policy trained by offline imitation of demonstrations, FISH computes rewards that correspond to the "match" between the robot's behavior and the demonstrations.

Imitation Learning

From Play to Policy: Conditional Behavior Generation from Uncurated Robot Data

no code implementations18 Oct 2022 Zichen Jeff Cui, Yibin Wang, Nur Muhammad Mahi Shafiullah, Lerrel Pinto

While large-scale sequence modeling from offline data has led to impressive performance gains in natural language and image generation, directly translating such ideas to robotics has been challenging.

Image Generation

Holo-Dex: Teaching Dexterity with Immersive Mixed Reality

no code implementations12 Oct 2022 Sridhar Pandian Arunachalam, Irmak Güzey, Soumith Chintala, Lerrel Pinto

A fundamental challenge in teaching robots is to provide an effective interface for human teachers to demonstrate useful skills to a robot.

Mixed Reality

CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory

2 code implementations11 Oct 2022 Nur Muhammad Mahi Shafiullah, Chris Paxton, Lerrel Pinto, Soumith Chintala, Arthur Szlam

We propose CLIP-Fields, an implicit scene model that can be used for a variety of tasks, such as segmentation, instance identification, semantic search over space, and view localization.

Segmentation Semantic Segmentation +1

That Sounds Right: Auditory Self-Supervision for Dynamic Robot Manipulation

1 code implementation3 Oct 2022 Abitha Thankaraj, Lerrel Pinto

Learning to produce contact-rich, dynamic behaviors from raw sensory data has been a longstanding challenge in robotics.

Robot Manipulation Self-Supervised Learning

Human Decision Makings on Curriculum Reinforcement Learning with Difficulty Adjustment

no code implementations4 Aug 2022 Yilei Zeng, Jiali Duan, Yang Li, Emilio Ferrara, Lerrel Pinto, C. -C. Jay Kuo, Stefanos Nikolaidis

In this work, we guide the curriculum reinforcement learning results towards a preferred performance level that is neither too hard nor too easy via learning from the human decision process.

reinforcement-learning Reinforcement Learning (RL)

Watch and Match: Supercharging Imitation with Regularized Optimal Transport

no code implementations30 Jun 2022 Siddhant Haldar, Vaibhav Mathur, Denis Yarats, Lerrel Pinto

Our experiments on 20 visual control tasks across the DeepMind Control Suite, the OpenAI Robotics Suite, and the Meta-World Benchmark demonstrate an average of 7. 8X faster imitation to reach 90% of expert performance compared to prior state-of-the-art methods.

Imitation Learning

Behavior Transformers: Cloning $k$ modes with one stone

2 code implementations22 Jun 2022 Nur Muhammad Mahi Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, Lerrel Pinto

In this work, we present Behavior Transformer (BeT), a new technique to model unlabeled demonstration data with multiple modes.

Object Detection Offline RL

Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulation

no code implementations24 Mar 2022 Sridhar Pandian Arunachalam, Sneha Silwal, Ben Evans, Lerrel Pinto

Optimizing behaviors for dexterous manipulation has been a longstanding challenge in robotics, with a variety of methods from model-based control to model-free reinforcement learning having been previously explored in literature.

Imitation Learning

One After Another: Learning Incremental Skills for a Changing World

1 code implementation ICLR 2022 Nur Muhammad Shafiullah, Lerrel Pinto

In this work, we propose a new framework for skill discovery, where skills are learned one after another in an incremental fashion.

Context is Everything: Implicit Identification for Dynamics Adaptation

no code implementations10 Mar 2022 Ben Evans, Abitha Thankaraj, Lerrel Pinto

Understanding environment dynamics is necessary for robots to act safely and optimally in the world.

The Surprising Effectiveness of Representation Learning for Visual Imitation

1 code implementation2 Dec 2021 Jyothish Pari, Nur Muhammad Shafiullah, Sridhar Pandian Arunachalam, Lerrel Pinto

One reason such complexities arise is because standard visual imitation frameworks try to solve two coupled problems at once: learning a succinct but good representation from the diverse visual data, while simultaneously learning to associate the demonstrated actions with such representations.

Imitation Learning Representation Learning +1

URLB: Unsupervised Reinforcement Learning Benchmark

1 code implementation28 Oct 2021 Michael Laskin, Denis Yarats, Hao liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, Pieter Abbeel

Deep Reinforcement Learning (RL) has emerged as a powerful paradigm to solve a range of complex yet specific control tasks.

Continuous Control reinforcement-learning +2

Improving Long-Horizon Imitation Through Language Prediction

no code implementations29 Sep 2021 Donald Joseph Hejna III, Pieter Abbeel, Lerrel Pinto

Complex, long-horizon planning and its combinatorial nature pose steep challenges for learning-based agents.

Playful Interactions for Representation Learning

no code implementations19 Jul 2021 Sarah Young, Jyothish Pari, Pieter Abbeel, Lerrel Pinto

In this work, we propose to use playful interactions in a self-supervised manner to learn visual representations for downstream tasks.

Imitation Learning Representation Learning

GEM: Group Enhanced Model for Learning Dynamical Control Systems

no code implementations7 Apr 2021 Philippe Hansen-Estruch, Wenling Shang, Lerrel Pinto, Pieter Abbeel, Stas Tiomkin

In this work, we take advantage of these structures to build effective dynamical models that are amenable to sample-based learning.

Continuous Control Model-based Reinforcement Learning

Simultaneous Navigation and Construction Benchmarking Environments

1 code implementation31 Mar 2021 Wenyu Han, Chen Feng, Haoran Wu, Alexander Gao, Armand Jordana, Dong Liu, Lerrel Pinto, Ludovic Righetti

We need intelligent robots for mobile construction, the process of navigating in an environment and modifying its structure according to a geometric design.

Benchmarking Reinforcement Learning (RL) +2

Task-Agnostic Morphology Evolution

1 code implementation ICLR 2021 Donald J. Hejna III, Pieter Abbeel, Lerrel Pinto

Deep reinforcement learning primarily focuses on learning behavior, usually overlooking the fact that an agent's function is largely determined by form.

Reinforcement Learning with Prototypical Representations

1 code implementation22 Feb 2021 Denis Yarats, Rob Fergus, Alessandro Lazaric, Lerrel Pinto

Unfortunately, in RL, representation learning is confounded with the exploratory experience of the agent -- learning a useful representation requires diverse data, while effective exploration is only possible with coherent representations.

Continuous Control reinforcement-learning +3

Mobile Construction Benchmark

no code implementations1 Jan 2021 Wenyu Han, Chen Feng, Haoran Wu, Alexander Gao, Armand Jordana, Dongdong Liu, Lerrel Pinto, Ludovic Righetti

We need intelligent robots to perform mobile construction, the process of moving in an environment and modifying its geometry according to a design plan.

Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation

no code implementations14 Dec 2020 Albert Zhan, Ruihan Zhao, Lerrel Pinto, Pieter Abbeel, Michael Laskin

We present Contrastive Pre-training and Data Augmentation for Efficient Robotic Learning (CoDER), a method that utilizes data augmentation and unsupervised learning to achieve sample-efficient training of real-robot arm policies from sparse rewards.

Data Augmentation reinforcement-learning +2

Visual Imitation Made Easy

no code implementations11 Aug 2020 Sarah Young, Dhiraj Gandhi, Shubham Tulsiani, Abhinav Gupta, Pieter Abbeel, Lerrel Pinto

We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector.

Imitation Learning

Self-Supervised Policy Adaptation during Deployment

2 code implementations ICLR 2021 Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyà, Pieter Abbeel, Alexei A. Efros, Lerrel Pinto, Xiaolong Wang

A natural solution would be to keep training after deployment in the new environment, but this cannot be done if the new environment offers no reward signal.

Swoosh! Rattle! Thump! -- Actions that Sound

no code implementations3 Jul 2020 Dhiraj Gandhi, Abhinav Gupta, Lerrel Pinto

In this work, we perform the first large-scale study of the interactions between sound and robotic action.

Automatic Curriculum Learning through Value Disagreement

1 code implementation NeurIPS 2020 Yunzhi Zhang, Pieter Abbeel, Lerrel Pinto

Our key insight is that if we can sample goals at the frontier of the set of goals that an agent is able to reach, it will provide a significantly stronger learning signal compared to randomly sampled goals.

Reinforcement Learning (RL)

Reinforcement Learning with Augmented Data

2 code implementations NeurIPS 2020 Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, Aravind Srinivas

To this end, we present Reinforcement Learning with Augmented Data (RAD), a simple plug-and-play module that can enhance most RL algorithms.

Data Augmentation OpenAI Gym +2

State-Only Imitation Learning for Dexterous Manipulation

no code implementations7 Apr 2020 Ilija Radosavovic, Xiaolong Wang, Lerrel Pinto, Jitendra Malik

To tackle this setting, we train an inverse dynamics model and use it to predict actions for state-only demonstrations.

Imitation Learning

Learning Predictive Representations for Deformable Objects Using Contrastive Estimation

1 code implementation11 Mar 2020 Wilson Yan, Ashwin Vangipuram, Pieter Abbeel, Lerrel Pinto

Using visual model-based learning for deformable object manipulation is challenging due to difficulties in learning plannable visual representations along with complex dynamic models.

Deformable Object Manipulation

Hierarchically Decoupled Imitation for Morphological Transfer

1 code implementation3 Mar 2020 Donald J. Hejna III, Pieter Abbeel, Lerrel Pinto

Learning long-range behaviors on complex high-dimensional agents is a fundamental problem in robot learning.

Generalized Hindsight for Reinforcement Learning

no code implementations NeurIPS 2020 Alexander C. Li, Lerrel Pinto, Pieter Abbeel

Compared to standard relabeling techniques, Generalized Hindsight provides a substantially more efficient reuse of samples, which we empirically demonstrate on a suite of multi-task navigation and manipulation tasks.

reinforcement-learning Reinforcement Learning (RL)

Discovering Motor Programs by Recomposing Demonstrations

no code implementations ICLR 2020 Tanmay Shankar, Shubham Tulsiani, Lerrel Pinto, Abhinav Gupta

In this paper, we present an approach to learn recomposable motor primitives across large-scale and diverse manipulation demonstrations.

Hierarchical Reinforcement Learning

Learning to Manipulate Deformable Objects without Demonstrations

2 code implementations29 Oct 2019 Yilin Wu, Wilson Yan, Thanard Kurutach, Lerrel Pinto, Pieter Abbeel

Second, instead of jointly learning both the pick and the place locations, we only explicitly learn the placing policy conditioned on random pick points.

Deformable Object Manipulation Object +1

Swoosh! Rattle! Thump! - Actions that Sound

no code implementations25 Sep 2019 Dhiraj Gandhi, Abhinav Gupta, Lerrel Pinto

In this work, we perform the first large-scale study of the interactions between sound and robotic action.


Environment Probing Interaction Policies

1 code implementation ICLR 2019 Wenxuan Zhou, Lerrel Pinto, Abhinav Gupta

A key challenge in reinforcement learning (RL) is environment generalization: a policy trained to solve a task in one environment often fails to solve the same task in a slightly different test environment.

Reinforcement Learning (RL)

Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation

no code implementations16 Oct 2018 Pratyusha Sharma, Lekha Mohan, Lerrel Pinto, Abhinav Gupta

In order to make progress and capture the space of manipulation, we would need to collect a large-scale dataset of diverse tasks such as pouring, opening bottles, stacking objects etc.

Trajectory Prediction

Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias

no code implementations NeurIPS 2018 Abhinav Gupta, Adithyavairavan Murali, Dhiraj Gandhi, Lerrel Pinto

The models trained with our home dataset showed a marked improvement of 43. 7% over a baseline model trained with data collected in lab.

Robotic Grasping

Asymmetric Actor Critic for Image-Based Robot Learning

no code implementations18 Oct 2017 Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, Pieter Abbeel

While several recent works have shown promising results in transferring policies trained in simulation to the real world, they often do not fully utilize the advantage of working with a simulator.

Decision Making Reinforcement Learning (RL)

Predictive-State Decoders: Encoding the Future into Recurrent Networks

no code implementations NeurIPS 2017 Arun Venkatraman, Nicholas Rhinehart, Wen Sun, Lerrel Pinto, Martial Hebert, Byron Boots, Kris M. Kitani, J. Andrew Bagnell

We seek to combine the advantages of RNNs and PSRs by augmenting existing state-of-the-art recurrent neural networks with Predictive-State Decoders (PSDs), which add supervision to the network's internal state representation to target predicting future observations.

Imitation Learning

CASSL: Curriculum Accelerated Self-Supervised Learning

no code implementations4 Aug 2017 Adithyavairavan Murali, Lerrel Pinto, Dhiraj Gandhi, Abhinav Gupta

Recent self-supervised learning approaches focus on using a few thousand data points to learn policies for high-level, low-dimensional action spaces.

Self-Supervised Learning

Robust Adversarial Reinforcement Learning

7 code implementations ICML 2017 Lerrel Pinto, James Davidson, Rahul Sukthankar, Abhinav Gupta

Deep neural networks coupled with fast simulation and improved computation have led to recent successes in the field of reinforcement learning (RL).

Friction reinforcement-learning +1

Supervision via Competition: Robot Adversaries for Learning Tasks

1 code implementation5 Oct 2016 Lerrel Pinto, James Davidson, Abhinav Gupta

Due to large number of experiences required for training, most of these approaches use a self-supervised paradigm: using sensors to measure success/failure.

Learning to Push by Grasping: Using multiple tasks for effective learning

no code implementations28 Sep 2016 Lerrel Pinto, Abhinav Gupta

The argument of the difficulty in scalability to multiple tasks is well founded, since training these tasks often require hundreds or thousands of examples.

Multi-Task Learning

The Curious Robot: Learning Visual Representations via Physical Interactions

no code implementations5 Apr 2016 Lerrel Pinto, Dhiraj Gandhi, Yuanfeng Han, Yong-Lae Park, Abhinav Gupta

We argue that biological agents use physical interactions with the world to learn visual representations unlike current vision systems which just use passive observations (images and videos downloaded from web).

Image Classification Representation Learning +1

Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours

no code implementations23 Sep 2015 Lerrel Pinto, Abhinav Gupta

Our experiments clearly show the benefit of using large-scale datasets (and multi-stage training) for the task of grasping.

Binary Classification

Cannot find the paper you are looking for? You can Submit a new open access paper.