no code implementations • 7 Nov 2024 • Yecheng Jason Ma, Joey Hejna, Ayzaan Wahid, Chuyuan Fu, Dhruv Shah, Jacky Liang, Zhuo Xu, Sean Kirmani, Peng Xu, Danny Driess, Ted Xiao, Jonathan Tompson, Osbert Bastani, Dinesh Jayaraman, Wenhao Yu, Tingnan Zhang, Dorsa Sadigh, Fei Xia
Instead, GVL poses value estimation as a temporal ordering problem over shuffled video frames; this seemingly more challenging task encourages VLMs to more fully exploit their underlying semantic and temporal grounding capabilities to differentiate frames based on their perceived task progress, consequently producing significantly better value predictions.
no code implementations • 4 Nov 2024 • William Liang, Sam Wang, Hung-Ju Wang, Osbert Bastani, Dinesh Jayaraman, Yecheng Jason Ma
We validate Eurekaverse's effectiveness in the domain of quadrupedal parkour learning, in which a quadruped robot must traverse through a variety of obstacle courses.
no code implementations • 30 Oct 2024 • Edward S. Hu, Kwangjun Ahn, Qinghua Liu, Haoran Xu, Manan Tomar, Ada Langford, Dinesh Jayaraman, Alex Lamb, John Langford
We introduce the "Belief State Transformer", a next-token predictor that takes both a prefix and suffix as inputs, with a novel objective of predicting both the next token for the prefix and the previous token for the suffix.
no code implementations • 3 Oct 2024 • Long Le, Jason Xie, William Liang, Hung-Ju Wang, Yue Yang, Yecheng Jason Ma, Kyle Vedder, Arjun Krishna, Dinesh Jayaraman, Eric Eaton
Interactive 3D simulated objects are crucial in AR/VR, animations, and robotics, driving immersive experiences and advanced automation.
1 code implementation • 17 Sep 2024 • Jake Welde, Nishanth Rao, Pratik Kunapuli, Dinesh Jayaraman, Vijay Kumar
Next, we prove that symmetry in the underlying dynamics and running costs leads to an MDP homomorphism, a mapping that allows a policy trained on a lower-dimensional "quotient" MDP to be lifted to an optimal tracking controller for the original system.
no code implementations • 4 Jun 2024 • Yecheng Jason Ma, William Liang, Hung-Ju Wang, Sam Wang, Yuke Zhu, Linxi Fan, Osbert Bastani, Dinesh Jayaraman
Transferring policies learned in simulation to the real world is a promising strategy for acquiring robot skills at scale.
no code implementations • 24 May 2024 • Jianing Qian, Anastasios Panagopoulos, Dinesh Jayaraman
Generic re-usable pre-trained image representation encoders have become a standard component of methods for many computer vision tasks.
no code implementations • 23 May 2024 • Edward S. Hu, James Springer, Oleh Rybkin, Dinesh Jayaraman
We consider such sensory scaffolding setups for training artificial agents.
no code implementations • 20 Apr 2024 • Junyao Shi, Jianing Qian, Yecheng Jason Ma, Dinesh Jayaraman
There have recently been large advances both in pre-training visual representations for robotic control and segmenting unknown category objects in general images.
no code implementations • 1 Mar 2024 • Chuan Wen, Dinesh Jayaraman, Yang Gao
Spatial relationships between objects represent key scene information for humans to understand and interact with the world.
no code implementations • 7 Dec 2023 • Weilin Wan, Yiming Huang, Shutong Wu, Taku Komura, Wenping Wang, Dinesh Jayaraman, Lingjie Liu
In this study, we introduce a learning-based method for generating high-quality human motion sequences from text descriptions (e. g., ``A person walks forward").
no code implementations • 28 Nov 2023 • Weilin Wan, Zhiyang Dou, Taku Komura, Wenping Wang, Dinesh Jayaraman, Lingjie Liu
Controllable human motion synthesis is essential for applications in AR/VR, gaming and embodied AI.
1 code implementation • 19 Oct 2023 • Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, Anima Anandkumar
The generality of Eureka also enables a new gradient-free in-context learning approach to reinforcement learning from human feedback (RLHF), readily incorporating human inputs to improve the quality and the safety of the generated rewards without model updating.
no code implementations • 12 Oct 2023 • Zichen Zhang, Yunshuang Li, Osbert Bastani, Abhishek Gupta, Dinesh Jayaraman, Yecheng Jason Ma, Luca Weihs
Learning long-horizon manipulation tasks, however, is a long-standing challenge, and demands decomposing the overarching task into several manageable subtasks to facilitate policy learning and generalization to unseen tasks.
no code implementations • 9 Oct 2023 • Kaustubh Sridhar, Souradeep Dutta, Dinesh Jayaraman, James Weimer, Insup Lee
Imitation learning considerably simplifies policy synthesis compared to alternative approaches by exploiting access to expert demonstrations.
1 code implementation • 1 Jun 2023 • Yecheng Jason Ma, William Liang, Vaidehi Som, Vikash Kumar, Amy Zhang, Osbert Bastani, Dinesh Jayaraman
We present Language-Image Value learning (LIV), a unified objective for vision-language representation and reward learning from action-free videos with text annotations.
no code implementations • 22 May 2023 • Yecheng Jason Ma, Kausik Sivakumar, Jason Yan, Osbert Bastani, Dinesh Jayaraman
Standard model-based reinforcement learning (MBRL) approaches fit a transition model of the environment to all past experience, but this wastes model capacity on data that is irrelevant for policy improvement.
Model-based Reinforcement Learning reinforcement-learning +1
1 code implementation • 17 May 2023 • Kyle Vedder, Neehar Peri, Nathaniel Chodosh, Ishan Khatri, Eric Eaton, Dinesh Jayaraman, Yang Liu, Deva Ramanan, James Hays
Scene flow estimation is the task of describing the 3D motion field between temporally successive point clouds.
Ranked #2 on Self-supervised Scene Flow Estimation on Argoverse 2 (using extra training data)
1 code implementation • 23 Mar 2023 • Edward S. Hu, Richard Chang, Oleh Rybkin, Dinesh Jayaraman
We address this question within the goal-conditioned reinforcement learning paradigm, by identifying how the agent should set its goals at training time to maximize exploration.
1 code implementation • 17 Dec 2022 • Kun Huang, Edward S. Hu, Dinesh Jayaraman
Physical interactions can often help reveal information that is not readily apparent.
no code implementations • 28 Oct 2022 • Sriram Narayanan, Dinesh Jayaraman, Manmohan Chandraker
We address key challenges in long-horizon embodied exploration and navigation by proposing a new object transport task and a novel modular framework for temporally extended navigation.
1 code implementation • 30 Sep 2022 • Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, Amy Zhang
Given the inherent cost and scarcity of in-domain, task-specific robot data, learning from large, diverse, offline human videos has emerged as a promising path towards acquiring a generally useful visual representation for control; however, how these human videos can be used for general-purpose reward learning remains an open question.
no code implementations • 25 Sep 2022 • Elijah S. Lee, Giuseppe Loianno, Dinesh Jayaraman, Vijay Kumar
Previous studies in the perimeter defense game have largely focused on the fully observable setting where the true player states are known to all players.
no code implementations • 22 Jun 2022 • Chuan Wen, Jianing Qian, Jierui Lin, Jiaye Teng, Dinesh Jayaraman, Yang Gao
Across applications spanning supervised classification and sequential control, deep learning has been reported to find "shortcut" solutions that fail catastrophically under minor changes in the data distribution.
1 code implementation • 7 Jun 2022 • Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman, Osbert Bastani
Offline goal-conditioned reinforcement learning (GCRL) promises general-purpose skill learning in the form of reaching diverse goals from purely offline datasets.
2 code implementations • 4 Feb 2022 • Yecheng Jason Ma, Andrew Shen, Dinesh Jayaraman, Osbert Bastani
We propose State Matching Offline DIstribution Correction Estimation (SMODICE), a novel and versatile regression-based offline imitation learning (IL) algorithm derived via state-occupancy matching.
no code implementations • 19 Jan 2022 • Ashwin De Silva, Rahul Ramesh, Lyle Ungar, Marshall Hussain Shuler, Noah J. Cowan, Michael Platt, Chen Li, Leyla Isik, Seung-Eon Roh, Adam Charles, Archana Venkataraman, Brian Caffo, Javier J. How, Justus M Kebschull, John W. Krakauer, Maxim Bichuch, Kaleab Alemayehu Kinfu, Eva Yezerets, Dinesh Jayaraman, Jong M. Shin, Soledad Villar, Ian Phillips, Carey E. Priebe, Thomas Hartung, Michael I. Miller, Jayanta Dey, Ningyuan, Huang, Eric Eaton, Ralph Etienne-Cummings, Elizabeth L. Ogburn, Randal Burns, Onyema Osuagwu, Brett Mensh, Alysson R. Muotri, Julia Brown, Chris White, Weiwei Yang, Andrei A. Rusu, Timothy Verstynen, Konrad P. Kording, Pratik Chaudhari, Joshua T. Vogelstein
We conjecture that certain sequences of tasks are not retrospectively learnable (in which the data distribution is fixed), but are prospectively learnable (in which distributions may be dynamic), suggesting that prospective learning is more difficult in kind than retrospective learning.
1 code implementation • 14 Dec 2021 • Yecheng Jason Ma, Andrew Shen, Osbert Bastani, Dinesh Jayaraman
Further, CAP adaptively tunes this penalty during training using true cost feedback from the environment.
no code implementations • 29 Sep 2021 • Chuan Wen, Jianing Qian, Jierui Lin, Dinesh Jayaraman, Yang Gao
When operating in partially observed settings, it is important for a control policy to fuse information from a history of observations.
no code implementations • ICLR 2022 • Edward S. Hu, Kun Huang, Oleh Rybkin, Dinesh Jayaraman
Training visual control policies from scratch on a new robot typically requires generating large amounts of robot-specific data.
1 code implementation • ICCV 2021 • Nikos Kolotouros, Georgios Pavlakos, Dinesh Jayaraman, Kostas Daniilidis
This paper focuses on the problem of 3D human reconstruction from 2D evidence.
Ranked #2 on Multi-Hypotheses 3D Human Pose Estimation on AH36M
1 code implementation • 19 Jul 2021 • Edward S. Hu, Kun Huang, Oleh Rybkin, Dinesh Jayaraman
Training visual control policies from scratch on a new robot typically requires generating large amounts of robot-specific data.
1 code implementation • NeurIPS 2021 • Yecheng Jason Ma, Dinesh Jayaraman, Osbert Bastani
We prove that CODAC learns a conservative return distribution -- in particular, for finite MDPs, CODAC converges to an uniform lower bound on the quantiles of the return distribution; our proof relies on a novel analysis of the distributional Bellman operator.
no code implementations • 11 Jun 2021 • Chuan Wen, Jierui Lin, Jianing Qian, Yang Gao, Dinesh Jayaraman
Imitation learning trains control policies by mimicking pre-recorded expert demonstrations.
1 code implementation • 2 Apr 2021 • Jingxi Xu, Bruce Lee, Nikolai Matni, Dinesh Jayaraman
The difficulty of optimal control problems has classically been characterized in terms of system properties such as minimum eigenvalues of controllability/observability gramians.
1 code implementation • ICCV 2021 • Yecheng Jason Ma, Jeevana Priya Inala, Dinesh Jayaraman, Osbert Bastani
We propose Likelihood-Based Diverse Sampling (LDS), a method for improving the quality and the diversity of trajectory samples from a pre-trained flow model.
no code implementations • NeurIPS 2020 • Chuan Wen, Jierui Lin, Trevor Darrell, Dinesh Jayaraman, Yang Gao
Imitation learning trains policies to map from input observations to the actions that an expert would choose.
no code implementations • 18 Oct 2020 • Neha Das, Sarah Bechtle, Todor Davchev, Dinesh Jayaraman, Akshara Rai, Franziska Meier
Scaling model-based inverse reinforcement learning (IRL) to real robotic manipulation tasks with unknown dynamics remains an open problem.
1 code implementation • ICML 2020 • Jesse Zhang, Brian Cheung, Chelsea Finn, Sergey Levine, Dinesh Jayaraman
Reinforcement learning (RL) in real-world safety-critical target settings like urban driving is hazardous, imperiling the RL agent, other agents, and the environment.
1 code implementation • NeurIPS 2020 • Karl Pertsch, Oleh Rybkin, Frederik Ebert, Chelsea Finn, Dinesh Jayaraman, Sergey Levine
In this work we propose a framework for visual prediction and planning that is able to overcome both of these limitations.
1 code implementation • 29 May 2020 • Mike Lambeta, Po-Wei Chou, Stephen Tian, Brian Yang, Benjamin Maloon, Victoria Rose Most, Dave Stroud, Raymond Santos, Ahmad Byagowi, Gregg Kammerer, Dinesh Jayaraman, Roberto Calandra
Despite decades of research, general purpose in-hand manipulation remains one of the unsolved challenges of robotics.
1 code implementation • 7 Jan 2020 • Santhosh K. Ramakrishnan, Dinesh Jayaraman, Kristen Grauman
Embodied computer vision considers perception for robots in novel, unstructured environments.
no code implementations • 31 Dec 2019 • Brian Yang, Dinesh Jayaraman, Glen Berseth, Alexei Efros, Sergey Levine
Existing approaches for visuomotor robotic control typically require characterizing the robot in advance by calibrating the camera or performing system identification.
1 code implementation • ICLR 2021 • Glen Berseth, Daniel Geng, Coline Devin, Nicholas Rhinehart, Chelsea Finn, Dinesh Jayaraman, Sergey Levine
Every living organism struggles against disruptive environmental forces to carve out and maintain an orderly niche.
no code implementations • 25 Sep 2019 • Oleh Rybkin, Karl Pertsch, Frederik Ebert, Dinesh Jayaraman, Chelsea Finn, Sergey Levine
Prior work on video generation largely focuses on prediction models that only observe frames from the beginning of the video.
no code implementations • 25 Sep 2019 • Glen Berseth, Daniel Geng, Coline Devin, Dinesh Jayaraman, Chelsea Finn, Sergey Levine
All living organisms struggle against the forces of nature to carve out niches where they can maintain relative stasis.
Unsupervised Pre-training Unsupervised Reinforcement Learning
no code implementations • 25 Sep 2019 • Jesse Zhang, Brian Cheung, Chelsea Finn, Dinesh Jayaraman, Sergey Levine
We study the problem of safe adaptation: given a model trained on a variety of past experiences for some task, can this model learn to perform that task in a new situation while avoiding catastrophic failure?
1 code implementation • Science Robotics 2019 • Santhosh K. Ramakrishnan, Dinesh Jayaraman, Kristen Grauman
Standard computer vision systems assume access to intelligently captured inputs (e. g., photos from a human photographer), yet autonomously capturing good observations is a major challenge in itself.
2 code implementations • NeurIPS 2019 • Pim de Haan, Dinesh Jayaraman, Sergey Levine
Such discriminative models are non-causal: the training procedure is unaware of the causal structure of the interaction between the expert and the environment.
no code implementations • 17 May 2019 • Brian Yang, Jesse Zhang, Vitchyr Pong, Sergey Levine, Dinesh Jayaraman
We envision REPLAB as a framework for reproducible research across manipulation tasks, and as a step in this direction, we define a template for a grasping benchmark consisting of a task definition, evaluation protocol, performance measures, and a dataset of 92k grasp attempts.
no code implementations • 11 Mar 2019 • Stephen Tian, Frederik Ebert, Dinesh Jayaraman, Mayur Mudigonda, Chelsea Finn, Roberto Calandra, Sergey Levine
Touch sensing is widely acknowledged to be important for dexterous robotic manipulation, but exploiting tactile sensing for continuous, non-prehensile manipulation is challenging.
no code implementations • ICLR 2019 • Dinesh Jayaraman, Frederik Ebert, Alexei A. Efros, Sergey Levine
Prediction is arguably one of the most basic functions of an intelligent system.
no code implementations • 28 May 2018 • Roberto Calandra, Andrew Owens, Dinesh Jayaraman, Justin Lin, Wenzhen Yuan, Jitendra Malik, Edward H. Adelson, Sergey Levine
This model -- a deep, multimodal convolutional network -- predicts the outcome of a candidate grasp adjustment, and then executes a grasp by iteratively selecting the most promising actions.
2 code implementations • CVPR 2018 • Dinesh Jayaraman, Kristen Grauman
It is common to implicitly assume access to intelligently captured inputs (e. g., photos from a human photographer), yet autonomously capturing good observations is itself a major challenge.
no code implementations • ECCV 2018 • Dinesh Jayaraman, Ruohan Gao, Kristen Grauman
We introduce an unsupervised feature learning approach that embeds 3D shape information into a single-view image representation.
no code implementations • 7 Dec 2016 • Yu-Chuan Su, Dinesh Jayaraman, Kristen Grauman
AutoCam leverages NFOV web video to discriminatively identify space-time "glimpses" of interest at each time instant, and then uses dynamic programming to select optimal human-like camera trajectories.
no code implementations • 1 Dec 2016 • Ruohan Gao, Dinesh Jayaraman, Kristen Grauman
Compared to existing temporal coherence methods, our idea has the advantage of lightweight preprocessing of the unlabeled video (no tracking required) while still being able to extract object-level regions from which to learn invariances.
no code implementations • 30 Apr 2016 • Dinesh Jayaraman, Kristen Grauman
To verify this hypothesis, we attempt to induce this capacity in our active recognition pipeline, by simultaneously learning to forecast the effects of the agent's motions on its internal representation of the environment conditional on all past views.
no code implementations • CVPR 2016 • Dinesh Jayaraman, Kristen Grauman
While this standard approach captures the fact that high-level visual signals change slowly over time, it fails to capture *how* the visual content changes.
1 code implementation • ICCV 2015 • Dinesh Jayaraman, Kristen Grauman
Understanding how images of objects and scenes behave in response to specific ego-motions is a crucial aspect of proper visual development, yet existing visual learning methods are conspicuously disconnected from the physical source of their images.
no code implementations • NeurIPS 2014 • Dinesh Jayaraman, Kristen Grauman
In principle, zero-shot learning makes it possible to train an object recognition model simply by specifying the category's attributes.
no code implementations • 15 Sep 2014 • Dinesh Jayaraman, Kristen Grauman
In principle, zero-shot learning makes it possible to train a recognition model simply by specifying the category's attributes.
no code implementations • CVPR 2014 • Dinesh Jayaraman, Fei Sha, Kristen Grauman
Existing methods to learn visual attributes are prone to learning the wrong thing---namely, properties that are correlated with the attribute of interest among training samples.