Search Results for author: Dinesh Jayaraman

Found 63 papers, 24 papers with code

Vision Language Models are In-Context Value Learners

no code implementations7 Nov 2024 Yecheng Jason Ma, Joey Hejna, Ayzaan Wahid, Chuyuan Fu, Dhruv Shah, Jacky Liang, Zhuo Xu, Sean Kirmani, Peng Xu, Danny Driess, Ted Xiao, Jonathan Tompson, Osbert Bastani, Dinesh Jayaraman, Wenhao Yu, Tingnan Zhang, Dorsa Sadigh, Fei Xia

Instead, GVL poses value estimation as a temporal ordering problem over shuffled video frames; this seemingly more challenging task encourages VLMs to more fully exploit their underlying semantic and temporal grounding capabilities to differentiate frames based on their perceived task progress, consequently producing significantly better value predictions.

In-Context Learning World Knowledge

Eurekaverse: Environment Curriculum Generation via Large Language Models

no code implementations4 Nov 2024 William Liang, Sam Wang, Hung-Ju Wang, Osbert Bastani, Dinesh Jayaraman, Yecheng Jason Ma

We validate Eurekaverse's effectiveness in the domain of quadrupedal parkour learning, in which a quadruped robot must traverse through a variety of obstacle courses.

Code Generation

Learning to Achieve Goals with Belief State Transformers

no code implementations30 Oct 2024 Edward S. Hu, Kwangjun Ahn, Qinghua Liu, Haoran Xu, Manan Tomar, Ada Langford, Dinesh Jayaraman, Alex Lamb, John Langford

We introduce the "Belief State Transformer", a next-token predictor that takes both a prefix and suffix as inputs, with a novel objective of predicting both the next token for the prefix and the previous token for the suffix.

Articulate-Anything: Automatic Modeling of Articulated Objects via a Vision-Language Foundation Model

no code implementations3 Oct 2024 Long Le, Jason Xie, William Liang, Hung-Ju Wang, Yue Yang, Yecheng Jason Ma, Kyle Vedder, Arjun Krishna, Dinesh Jayaraman, Eric Eaton

Interactive 3D simulated objects are crucial in AR/VR, animations, and robotics, driving immersive experiences and advanced automation.

Leveraging Symmetry to Accelerate Learning of Trajectory Tracking Controllers for Free-Flying Robotic Systems

1 code implementation17 Sep 2024 Jake Welde, Nishanth Rao, Pratik Kunapuli, Dinesh Jayaraman, Vijay Kumar

Next, we prove that symmetry in the underlying dynamics and running costs leads to an MDP homomorphism, a mapping that allows a policy trained on a lower-dimensional "quotient" MDP to be lifted to an optimal tracking controller for the original system.

Reinforcement Learning (RL)

DrEureka: Language Model Guided Sim-To-Real Transfer

no code implementations4 Jun 2024 Yecheng Jason Ma, William Liang, Hung-Ju Wang, Sam Wang, Yuke Zhu, Linxi Fan, Osbert Bastani, Dinesh Jayaraman

Transferring policies learned in simulation to the real world is a promising strategy for acquiring robot skills at scale.

Language Modelling

Recasting Generic Pretrained Vision Transformers As Object-Centric Scene Encoders For Manipulation Policies

no code implementations24 May 2024 Jianing Qian, Anastasios Panagopoulos, Dinesh Jayaraman

Generic re-usable pre-trained image representation encoders have become a standard component of methods for many computer vision tasks.

Composing Pre-Trained Object-Centric Representations for Robotics From "What" and "Where" Foundation Models

no code implementations20 Apr 2024 Junyao Shi, Jianing Qian, Yecheng Jason Ma, Dinesh Jayaraman

There have recently been large advances both in pre-training visual representations for robotic control and segmenting unknown category objects in general images.

Object Systematic Generalization

Can Transformers Capture Spatial Relations between Objects?

no code implementations1 Mar 2024 Chuan Wen, Dinesh Jayaraman, Yang Gao

Spatial relationships between objects represent key scene information for humans to understand and interact with the world.

Relation

DiffusionPhase: Motion Diffusion in Frequency Domain

no code implementations7 Dec 2023 Weilin Wan, Yiming Huang, Shutong Wu, Taku Komura, Wenping Wang, Dinesh Jayaraman, Lingjie Liu

In this study, we introduce a learning-based method for generating high-quality human motion sequences from text descriptions (e. g., ``A person walks forward").

Diversity Motion Generation

TLControl: Trajectory and Language Control for Human Motion Synthesis

no code implementations28 Nov 2023 Weilin Wan, Zhiyang Dou, Taku Komura, Wenping Wang, Dinesh Jayaraman, Lingjie Liu

Controllable human motion synthesis is essential for applications in AR/VR, gaming and embodied AI.

Motion Synthesis

Eureka: Human-Level Reward Design via Coding Large Language Models

1 code implementation19 Oct 2023 Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, Anima Anandkumar

The generality of Eureka also enables a new gradient-free in-context learning approach to reinforcement learning from human feedback (RLHF), readily incorporating human inputs to improve the quality and the safety of the generated rewards without model updating.

Decision Making In-Context Learning +1

Universal Visual Decomposer: Long-Horizon Manipulation Made Easy

no code implementations12 Oct 2023 Zichen Zhang, Yunshuang Li, Osbert Bastani, Abhishek Gupta, Dinesh Jayaraman, Yecheng Jason Ma, Luca Weihs

Learning long-horizon manipulation tasks, however, is a long-standing challenge, and demands decomposing the overarching task into several manageable subtasks to facilitate policy learning and generalization to unseen tasks.

reinforcement-learning

Memory-Consistent Neural Networks for Imitation Learning

no code implementations9 Oct 2023 Kaustubh Sridhar, Souradeep Dutta, Dinesh Jayaraman, James Weimer, Insup Lee

Imitation learning considerably simplifies policy synthesis compared to alternative approaches by exploiting access to expert demonstrations.

Imitation Learning

LIV: Language-Image Representations and Rewards for Robotic Control

1 code implementation1 Jun 2023 Yecheng Jason Ma, William Liang, Vaidehi Som, Vikash Kumar, Amy Zhang, Osbert Bastani, Dinesh Jayaraman

We present Language-Image Value learning (LIV), a unified objective for vision-language representation and reward learning from action-free videos with text annotations.

Contrastive Learning Imitation Learning

TOM: Learning Policy-Aware Models for Model-Based Reinforcement Learning via Transition Occupancy Matching

no code implementations22 May 2023 Yecheng Jason Ma, Kausik Sivakumar, Jason Yan, Osbert Bastani, Dinesh Jayaraman

Standard model-based reinforcement learning (MBRL) approaches fit a transition model of the environment to all past experience, but this wastes model capacity on data that is irrelevant for policy improvement.

Model-based Reinforcement Learning reinforcement-learning +1

ZeroFlow: Scalable Scene Flow via Distillation

1 code implementation17 May 2023 Kyle Vedder, Neehar Peri, Nathaniel Chodosh, Ishan Khatri, Eric Eaton, Dinesh Jayaraman, Yang Liu, Deva Ramanan, James Hays

Scene flow estimation is the task of describing the 3D motion field between temporally successive point clouds.

Ranked #2 on Self-supervised Scene Flow Estimation on Argoverse 2 (using extra training data)

Self-supervised Scene Flow Estimation

Planning Goals for Exploration

1 code implementation23 Mar 2023 Edward S. Hu, Richard Chang, Oleh Rybkin, Dinesh Jayaraman

We address this question within the goal-conditioned reinforcement learning paradigm, by identifying how the agent should set its goals at training time to maximize exploration.

Training Robots to Evaluate Robots: Example-Based Interactive Reward Functions for Policy Learning

1 code implementation17 Dec 2022 Kun Huang, Edward S. Hu, Dinesh Jayaraman

Physical interactions can often help reveal information that is not readily apparent.

Long-HOT: A Modular Hierarchical Approach for Long-Horizon Object Transport

no code implementations28 Oct 2022 Sriram Narayanan, Dinesh Jayaraman, Manmohan Chandraker

We address key challenges in long-horizon embodied exploration and navigation by proposing a new object transport task and a novel modular framework for temporally extended navigation.

Motion Planning

VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training

1 code implementation30 Sep 2022 Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, Amy Zhang

Given the inherent cost and scarcity of in-domain, task-specific robot data, learning from large, diverse, offline human videos has emerged as a promising path towards acquiring a generally useful visual representation for control; however, how these human videos can be used for general-purpose reward learning remains an open question.

Offline RL Open-Ended Question Answering +2

Vision-based Perimeter Defense via Multiview Pose Estimation

no code implementations25 Sep 2022 Elijah S. Lee, Giuseppe Loianno, Dinesh Jayaraman, Vijay Kumar

Previous studies in the perimeter defense game have largely focused on the fully observable setting where the true player states are known to all players.

Pose Estimation

Fighting Fire with Fire: Avoiding DNN Shortcuts through Priming

no code implementations22 Jun 2022 Chuan Wen, Jianing Qian, Jierui Lin, Jiaye Teng, Dinesh Jayaraman, Yang Gao

Across applications spanning supervised classification and sequential control, deep learning has been reported to find "shortcut" solutions that fail catastrophically under minor changes in the data distribution.

Autonomous Driving Classification +6

How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via $f$-Advantage Regression

1 code implementation7 Jun 2022 Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman, Osbert Bastani

Offline goal-conditioned reinforcement learning (GCRL) promises general-purpose skill learning in the form of reaching diverse goals from purely offline datasets.

regression reinforcement-learning +1

Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching

2 code implementations4 Feb 2022 Yecheng Jason Ma, Andrew Shen, Dinesh Jayaraman, Osbert Bastani

We propose State Matching Offline DIstribution Correction Estimation (SMODICE), a novel and versatile regression-based offline imitation learning (IL) algorithm derived via state-occupancy matching.

Imitation Learning Reinforcement Learning (RL)

Fight fire with fire: countering bad shortcuts in imitation learning with good shortcuts

no code implementations29 Sep 2021 Chuan Wen, Jianing Qian, Jierui Lin, Dinesh Jayaraman, Yang Gao

When operating in partially observed settings, it is important for a control policy to fuse information from a history of observations.

Autonomous Driving continuous-control +2

Transferable Visual Control Policies Through Robot-Awareness

no code implementations ICLR 2022 Edward S. Hu, Kun Huang, Oleh Rybkin, Dinesh Jayaraman

Training visual control policies from scratch on a new robot typically requires generating large amounts of robot-specific data.

Know Thyself: Transferable Visual Control Policies Through Robot-Awareness

1 code implementation19 Jul 2021 Edward S. Hu, Kun Huang, Oleh Rybkin, Dinesh Jayaraman

Training visual control policies from scratch on a new robot typically requires generating large amounts of robot-specific data.

Model-based Reinforcement Learning Transfer Learning +1

Conservative Offline Distributional Reinforcement Learning

1 code implementation NeurIPS 2021 Yecheng Jason Ma, Dinesh Jayaraman, Osbert Bastani

We prove that CODAC learns a conservative return distribution -- in particular, for finite MDPs, CODAC converges to an uniform lower bound on the quantiles of the return distribution; our proof relies on a novel analysis of the distributional Bellman operator.

D4RL Distributional Reinforcement Learning +5

How Are Learned Perception-Based Controllers Impacted by the Limits of Robust Control?

1 code implementation2 Apr 2021 Jingxi Xu, Bruce Lee, Nikolai Matni, Dinesh Jayaraman

The difficulty of optimal control problems has classically been characterized in terms of system properties such as minimum eigenvalues of controllability/observability gramians.

Reinforcement Learning (RL)

Likelihood-Based Diverse Sampling for Trajectory Forecasting

1 code implementation ICCV 2021 Yecheng Jason Ma, Jeevana Priya Inala, Dinesh Jayaraman, Osbert Bastani

We propose Likelihood-Based Diverse Sampling (LDS), a method for improving the quality and the diversity of trajectory samples from a pre-trained flow model.

Trajectory Forecasting

Model-Based Inverse Reinforcement Learning from Visual Demonstrations

no code implementations18 Oct 2020 Neha Das, Sarah Bechtle, Todor Davchev, Dinesh Jayaraman, Akshara Rai, Franziska Meier

Scaling model-based inverse reinforcement learning (IRL) to real robotic manipulation tasks with unknown dynamics remains an open problem.

Model Predictive Control reinforcement-learning +2

Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings

1 code implementation ICML 2020 Jesse Zhang, Brian Cheung, Chelsea Finn, Sergey Levine, Dinesh Jayaraman

Reinforcement learning (RL) in real-world safety-critical target settings like urban driving is hazardous, imperiling the RL agent, other agents, and the environment.

reinforcement-learning Reinforcement Learning +1

Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors

1 code implementation NeurIPS 2020 Karl Pertsch, Oleh Rybkin, Frederik Ebert, Chelsea Finn, Dinesh Jayaraman, Sergey Levine

In this work we propose a framework for visual prediction and planning that is able to overcome both of these limitations.

An Exploration of Embodied Visual Exploration

1 code implementation7 Jan 2020 Santhosh K. Ramakrishnan, Dinesh Jayaraman, Kristen Grauman

Embodied computer vision considers perception for robots in novel, unstructured environments.

Benchmarking

Morphology-Agnostic Visual Robotic Control

no code implementations31 Dec 2019 Brian Yang, Dinesh Jayaraman, Glen Berseth, Alexei Efros, Sergey Levine

Existing approaches for visuomotor robotic control typically require characterizing the robot in advance by calibrating the camera or performing system identification.

Goal-Conditioned Video Prediction

no code implementations25 Sep 2019 Oleh Rybkin, Karl Pertsch, Frederik Ebert, Dinesh Jayaraman, Chelsea Finn, Sergey Levine

Prior work on video generation largely focuses on prediction models that only observe frames from the beginning of the video.

Imitation Learning Video Generation +1

Hope For The Best But Prepare For The Worst: Cautious Adaptation In RL Agents

no code implementations25 Sep 2019 Jesse Zhang, Brian Cheung, Chelsea Finn, Dinesh Jayaraman, Sergey Levine

We study the problem of safe adaptation: given a model trained on a variety of past experiences for some task, can this model learn to perform that task in a new situation while avoiding catastrophic failure?

Domain Adaptation Meta Reinforcement Learning +2

Emergence of Exploratory Look-Around Behaviors through Active Observation Completion

1 code implementation Science Robotics 2019 Santhosh K. Ramakrishnan, Dinesh Jayaraman, Kristen Grauman

Standard computer vision systems assume access to intelligently captured inputs (e. g., photos from a human photographer), yet autonomously capturing good observations is a major challenge in itself.

Active Observation Completion Reinforcement Learning

Causal Confusion in Imitation Learning

2 code implementations NeurIPS 2019 Pim de Haan, Dinesh Jayaraman, Sergey Levine

Such discriminative models are non-causal: the training procedure is unaware of the causal structure of the interaction between the expert and the environment.

Imitation Learning

REPLAB: A Reproducible Low-Cost Arm Benchmark Platform for Robotic Learning

no code implementations17 May 2019 Brian Yang, Jesse Zhang, Vitchyr Pong, Sergey Levine, Dinesh Jayaraman

We envision REPLAB as a framework for reproducible research across manipulation tasks, and as a step in this direction, we define a template for a grasping benchmark consisting of a task definition, evaluation protocol, performance measures, and a dataset of 92k grasp attempts.

Benchmarking Machine Translation +2

Manipulation by Feel: Touch-Based Control with Deep Predictive Models

no code implementations11 Mar 2019 Stephen Tian, Frederik Ebert, Dinesh Jayaraman, Mayur Mudigonda, Chelsea Finn, Roberto Calandra, Sergey Levine

Touch sensing is widely acknowledged to be important for dexterous robotic manipulation, but exploiting tactile sensing for continuous, non-prehensile manipulation is challenging.

More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch

no code implementations28 May 2018 Roberto Calandra, Andrew Owens, Dinesh Jayaraman, Justin Lin, Wenzhen Yuan, Jitendra Malik, Edward H. Adelson, Sergey Levine

This model -- a deep, multimodal convolutional network -- predicts the outcome of a candidate grasp adjustment, and then executes a grasp by iteratively selecting the most promising actions.

Robotic Grasping

Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks

2 code implementations CVPR 2018 Dinesh Jayaraman, Kristen Grauman

It is common to implicitly assume access to intelligently captured inputs (e. g., photos from a human photographer), yet autonomously capturing good observations is itself a major challenge.

Reinforcement Learning

ShapeCodes: Self-Supervised Feature Learning by Lifting Views to Viewgrids

no code implementations ECCV 2018 Dinesh Jayaraman, Ruohan Gao, Kristen Grauman

We introduce an unsupervised feature learning approach that embeds 3D shape information into a single-view image representation.

Decoder Object +1

Pano2Vid: Automatic Cinematography for Watching 360$^{\circ}$ Videos

no code implementations7 Dec 2016 Yu-Chuan Su, Dinesh Jayaraman, Kristen Grauman

AutoCam leverages NFOV web video to discriminatively identify space-time "glimpses" of interest at each time instant, and then uses dynamic programming to select optimal human-like camera trajectories.

Object-Centric Representation Learning from Unlabeled Videos

no code implementations1 Dec 2016 Ruohan Gao, Dinesh Jayaraman, Kristen Grauman

Compared to existing temporal coherence methods, our idea has the advantage of lightweight preprocessing of the unlabeled video (no tracking required) while still being able to extract object-level regions from which to learn invariances.

Image Classification Object +3

Look-ahead before you leap: end-to-end active recognition by forecasting the effect of motion

no code implementations30 Apr 2016 Dinesh Jayaraman, Kristen Grauman

To verify this hypothesis, we attempt to induce this capacity in our active recognition pipeline, by simultaneously learning to forecast the effects of the agent's motions on its internal representation of the environment conditional on all past views.

Slow and steady feature analysis: higher order temporal coherence in video

no code implementations CVPR 2016 Dinesh Jayaraman, Kristen Grauman

While this standard approach captures the fact that high-level visual signals change slowly over time, it fails to capture *how* the visual content changes.

Action Recognition Temporal Action Localization

Learning image representations tied to ego-motion

1 code implementation ICCV 2015 Dinesh Jayaraman, Kristen Grauman

Understanding how images of objects and scenes behave in response to specific ego-motions is a crucial aspect of proper visual development, yet existing visual learning methods are conspicuously disconnected from the physical source of their images.

Autonomous Driving Scene Recognition

Zero-shot recognition with unreliable attributes

no code implementations NeurIPS 2014 Dinesh Jayaraman, Kristen Grauman

In principle, zero-shot learning makes it possible to train an object recognition model simply by specifying the category's attributes.

Attribute Object Recognition +1

Zero Shot Recognition with Unreliable Attributes

no code implementations15 Sep 2014 Dinesh Jayaraman, Kristen Grauman

In principle, zero-shot learning makes it possible to train a recognition model simply by specifying the category's attributes.

Attribute Zero-Shot Learning

Decorrelating Semantic Visual Attributes by Resisting the Urge to Share

no code implementations CVPR 2014 Dinesh Jayaraman, Fei Sha, Kristen Grauman

Existing methods to learn visual attributes are prone to learning the wrong thing---namely, properties that are correlated with the attribute of interest among training samples.

Attribute Multi-Task Learning

Cannot find the paper you are looking for? You can Submit a new open access paper.