Search Results for author: Deepak Pathak

Found 53 papers, 30 papers with code

Beyond Games: Bringing Exploration to Robots in Real-world

no code implementations ICLR 2019 Deepak Pathak, Dhiraj Gandhi, Abhinav Gupta

But most importantly, we are able to implement an exploration policy on a robot which learns to interact with objects completely from scratch just using data collected via the differentiable exploration module.

Efficient Exploration

Learning General-Purpose Controllers via Locally Communicating Sensorimotor Modules

no code implementations ICML 2020 Wenlong Huang, Igor Mordatch, Deepak Pathak

We observe a wide variety of drastically diverse locomotion styles across morphologies as well as centralized coordination emerging via message passing between decentralized modules purely from the reinforcement learning objective.

reinforcement-learning

Human-to-Robot Imitation in the Wild

no code implementations19 Jul 2022 Shikhar Bahl, Abhinav Gupta, Deepak Pathak

We approach the problem of learning by watching humans in the wild.

Adapting Rapid Motor Adaptation for Bipedal Robots

no code implementations30 May 2022 Ashish Kumar, Zhongyu Li, Jun Zeng, Deepak Pathak, Koushil Sreenath, Jitendra Malik

In this work, we leverage recent advances in rapid adaptation for locomotion control, and extend them to work on bipedal robots.

Topologically-Aware Deformation Fields for Single-View 3D Reconstruction

1 code implementation CVPR 2022 Shivam Duggal, Deepak Pathak

The 3D shapes are generated implicitly as deformations to a category-specific signed distance field and are learned in an unsupervised manner solely from unaligned image collections and their poses without any 3D supervision.

3D Reconstruction Single-View 3D Reconstruction

Generating Fast and Slow: Scene Decomposition via Reconstruction

no code implementations21 Mar 2022 Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, Katerina Fragkiadaki

We show the proposed curriculum suffices to break the reconstruction-segmentation trade-off, and slow inference greatly improves segmentation in out-of-distribution scenes.

Object Discovery Scene Segmentation

Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube

no code implementations21 Feb 2022 Aravind Sivakumar, Kenneth Shaw, Deepak Pathak

Human hands and robot hands differ in shape, size, and joint structure, and performing this translation from a single uncalibrated camera is a highly underconstrained problem.

REvolveR: Continuous Evolutionary Models for Robot-to-robot Policy Transfer

1 code implementation10 Feb 2022 Xingyu Liu, Deepak Pathak, Kris M. Kitani

We interpolate between the source robot and the target robot by finding a continuous evolutionary change of robot parameters.

Imitation Learning

Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents

1 code implementation18 Jan 2022 Wenlong Huang, Pieter Abbeel, Deepak Pathak, Igor Mordatch

However, the plans produced naively by LLMs often cannot map precisely to admissible actions.

The CLEAR Benchmark: Continual LEArning on Real-World Imagery

1 code implementation17 Jan 2022 Zhiqiu Lin, Jia Shi, Deepak Pathak, Deva Ramanan

The major strength of CLEAR over prior CL benchmarks is the smooth temporal evolution of visual concepts with real-world imagery, including both high-quality labeled data along with abundant unlabeled samples per time period for continual semi-supervised learning.

Continual Learning Image Classification +2

Functional Regularization for Reinforcement Learning via Learned Fourier Features

1 code implementation NeurIPS 2021 Alexander C. Li, Deepak Pathak

We propose a simple architecture for deep reinforcement learning by embedding inputs into a learned Fourier basis and show that it improves the sample efficiency of both state-based and image-based RL.

reinforcement-learning

Coupling Vision and Proprioception for Navigation of Legged Robots

no code implementations CVPR 2022 Zipeng Fu, Ashish Kumar, Ananye Agarwal, Haozhi Qi, Jitendra Malik, Deepak Pathak

A safety advisor module adds sensed unexpected obstacles to the occupancy map and environment-determined speed limits to the velocity command generator.

Legged Robots

Interesting Object, Curious Agent: Learning Task-Agnostic Exploration

2 code implementations NeurIPS 2021 Simone Parisi, Victoria Dean, Deepak Pathak, Abhinav Gupta

In this setup, the agent first learns to explore across many environments without any extrinsic goal in a task-agnostic manner.

Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learning

no code implementations4 Nov 2021 Wenlong Huang, Igor Mordatch, Pieter Abbeel, Deepak Pathak

We show that a single generalist policy can perform in-hand manipulation of over 100 geometrically-diverse real-world objects and generalize to new objects with unseen shape or size.

Multi-Task Learning reinforcement-learning

Minimizing Energy Consumption Leads to the Emergence of Gaits in Legged Robots

no code implementations25 Oct 2021 Zipeng Fu, Ashish Kumar, Jitendra Malik, Deepak Pathak

We demonstrate that learning to minimize energy consumption plays a key role in the emergence of natural locomotion gaits at different speeds in real quadruped robots.

Legged Robots

Discovering and Achieving Goals via World Models

1 code implementation NeurIPS 2021 Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, Deepak Pathak

How can artificial agents learn to solve many diverse tasks in complex visual environments in the absence of any supervision?

Zero-Shot Reward Specification via Grounded Natural Language

no code implementations29 Sep 2021 Parsa Mahmoudieh, Sayna Ebrahimi, Deepak Pathak, Trevor Darrell

Reward signals in reinforcement learning can be expensive signals in many tasks and often require access to direct state.

reinforcement-learning

How to Adapt Your Large-Scale Vision-and-Language Model

no code implementations29 Sep 2021 Konwoo Kim, Michael Laskin, Igor Mordatch, Deepak Pathak

Finally, we provide an empirical analysis and recommend general recipes for efficient transfer learning of vision and language models.

Image Classification Language Modelling +1

Hierarchical Neural Dynamic Policies

no code implementations12 Jul 2021 Shikhar Bahl, Abhinav Gupta, Deepak Pathak

We tackle the problem of generalization to unseen configurations for dynamic tasks in the real world while learning from high-dimensional image input.

RMA: Rapid Motor Adaptation for Legged Robots

no code implementations8 Jul 2021 Ashish Kumar, Zipeng Fu, Deepak Pathak, Jitendra Malik

Successful real-world deployment of legged robots would require them to adapt in real-time to unseen scenarios like changing terrains, changing payloads, wear and tear.

Legged Robots

Unsupervised Learning of Visual 3D Keypoints for Control

1 code implementation14 Jun 2021 Boyuan Chen, Pieter Abbeel, Deepak Pathak

Prior works show that structured latent space such as visual keypoints often outperforms unstructured representations for robotic control.

Discovering and Achieving Goals with World Models

no code implementations ICML Workshop URL 2021 Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, Deepak Pathak

How can an artificial agent learn to solve a wide range of tasks in a complex visual environment in the absence of external supervision?

Auto-Tuned Sim-to-Real Transfer

1 code implementation15 Apr 2021 Yuqing Du, Olivia Watkins, Trevor Darrell, Pieter Abbeel, Deepak Pathak

Policies trained in simulation often fail when transferred to the real world due to the `reality gap' where the simulator is unable to accurately capture the dynamics and visual properties of the real world.

FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation

1 code implementation15 Dec 2020 Tarun Kalluri, Deepak Pathak, Manmohan Chandraker, Du Tran

A majority of methods for video frame interpolation compute bidirectional optical flow between adjacent frames of a video, followed by a suitable warping algorithm to generate the output frames.

Action Recognition Optical Flow Estimation +1

Neural Dynamic Policies for End-to-End Sensorimotor Learning

no code implementations NeurIPS 2020 Shikhar Bahl, Mustafa Mukadam, Abhinav Gupta, Deepak Pathak

We show that NDPs outperform the prior state-of-the-art in terms of either efficiency or performance across several robotic control tasks for both imitation and reinforcement learning setups.

Imitation Learning reinforcement-learning

One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control

2 code implementations ICML 2020 Wenlong Huang, Igor Mordatch, Deepak Pathak

We observe that a wide variety of drastically diverse locomotion styles across morphologies as well as centralized coordination emerges via message passing between decentralized modules purely from the reinforcement learning objective.

reinforcement-learning

Locally Masked Convolution for Autoregressive Models

1 code implementation22 Jun 2020 Ajay Jain, Pieter Abbeel, Deepak Pathak

For tasks such as image completion, these models are unable to use much of the observed context.

Anomaly Detection Density Estimation +2

Planning to Explore via Self-Supervised World Models

3 code implementations12 May 2020 Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, Deepak Pathak

Reinforcement learning allows solving complex tasks, however, the learning tends to be task-specific and the sample efficiency remains a challenge.

reinforcement-learning

Exploring Exploration: Comparing Children with RL Agents in Unified Environments

1 code implementation6 May 2020 Eliza Kosoy, Jasmine Collins, David M. Chan, Sandy Huang, Deepak Pathak, Pulkit Agrawal, John Canny, Alison Gopnik, Jessica B. Hamrick

Research in developmental psychology consistently shows that children explore the world thoroughly and efficiently and that this exploration allows them to learn.

DeepMind

Sparse Graphical Memory for Robust Planning

1 code implementation NeurIPS 2020 Scott Emmons, Ajay Jain, Michael Laskin, Thanard Kurutach, Pieter Abbeel, Deepak Pathak

To operate effectively in the real world, agents should be able to act from high-dimensional raw sensory input such as images and achieve diverse goals across long time-horizons.

Imitation Learning Visual Navigation

Third-Person Visual Imitation Learning via Decoupled Hierarchical Controller

1 code implementation NeurIPS 2019 Pratyusha Sharma, Deepak Pathak, Abhinav Gupta

We study a generalized setup for learning from demonstration to build an agent that can manipulate novel objects in unseen scenarios by looking at only a single video of human demonstration from a third-person perspective.

Imitation Learning

Weakly-Supervised Trajectory Segmentation for Learning Reusable Skills

no code implementations25 Sep 2019 Parsa Mahmoudieh, Trevor Darrell, Deepak Pathak

Instead of direct manual supervision which is tedious and prone to bias, in this work, our goal is to extract reusable skills from a collection of human demonstrations collected directly for several end-tasks.

Multiple Instance Learning

Compositional GAN (Extended Abstract): Learning Image-Conditional Binary Composition

no code implementations ICLR Workshop DeepGenStruct 2019 Samaneh Azadi, Deepak Pathak, Sayna Ebrahimi, Trevor Darrell

Generative Adversarial Networks (GANs) can produce images of surprising complexity and realism but are generally structured to sample from a single latent source ignoring the explicit spatial interaction between multiple entities that could be present in a scene.

Large-Scale Study of Curiosity-Driven Learning

4 code implementations ICLR 2019 Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, Alexei A. Efros

However, annotating each environment with hand-designed, dense rewards is not scalable, motivating the need for developing reward functions that are intrinsic to the agent.

Atari Games SNES Games

Compositional GAN: Learning Image-Conditional Binary Composition

1 code implementation19 Jul 2018 Samaneh Azadi, Deepak Pathak, Sayna Ebrahimi, Trevor Darrell

Generative Adversarial Networks (GANs) can produce images of remarkable complexity and realism but are generally structured to sample from a single latent source ignoring the explicit spatial interaction between multiple entities that could be present in a scene.

Learning Instance Segmentation by Interaction

1 code implementation21 Jun 2018 Deepak Pathak, Yide Shentu, Dian Chen, Pulkit Agrawal, Trevor Darrell, Sergey Levine, Jitendra Malik

The agent uses its current segmentation model to infer pixels that constitute objects and refines the segmentation model by interacting with these pixels.

Instance Segmentation Semantic Segmentation

Learning Features by Watching Objects Move

1 code implementation CVPR 2017 Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, Bharath Hariharan

Given the extensive evidence that motion plays a key role in the development of the human visual system, we hope that this straightforward approach to unsupervised learning will be more effective than cleverly designed 'pretext' tasks studied in the literature.

object-detection Object Detection +1

Context Encoders: Feature Learning by Inpainting

11 code implementations CVPR 2016 Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, Alexei A. Efros

In order to succeed at this task, context encoders need to both understand the content of the entire image, as well as produce a plausible hypothesis for the missing part(s).

Constrained Convolutional Neural Networks for Weakly Supervised Segmentation

1 code implementation ICCV 2015 Deepak Pathak, Philipp Krähenbühl, Trevor Darrell

We propose Constrained CNN (CCNN), a method which uses a novel loss function to optimize for any set of linear constraints on the output space (i. e. predicted label distribution) of a CNN.

Semantic Segmentation TAG +1

Detector Discovery in the Wild: Joint Multiple Instance and Representation Learning

no code implementations CVPR 2015 Judy Hoffman, Deepak Pathak, Trevor Darrell, Kate Saenko

We develop methods for detector learning which exploit joint training over both weak and strong labels and which transfer learned perceptual representations from strongly-labeled auxiliary tasks.

Multiple Instance Learning Representation Learning +1

Cannot find the paper you are looking for? You can Submit a new open access paper.