Search Results for author: Amy Zhang

Found 68 papers, 30 papers with code

Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedback

no code implementations30 Oct 2024 Qinqing Zheng, Mikael Henaff, Amy Zhang, Aditya Grover, Brandon Amos

Our approach annotates the agent's collected experience via an asynchronous LLM server, which is then distilled into an intrinsic reward model.

Decision Making Language Modelling +3

SkiLD: Unsupervised Skill Discovery Guided by Factor Interactions

no code implementations24 Oct 2024 Zizhao Wang, Jiaheng Hu, Caleb Chuck, Stephen Chen, Roberto Martín-Martín, Amy Zhang, Scott Niekum, Peter Stone

However, in complex environments with many state factors (e. g., household environments with many objects), learning skills that cover all possible states is impossible, and naively encouraging state diversity often leads to simple skills that are not ideal for solving downstream tasks.

Diversity Inductive Bias +1

Learning a Fast Mixing Exogenous Block MDP using a Single Trajectory

1 code implementation3 Oct 2024 Alexander Levine, Peter Stone, Amy Zhang

Efroni et al. (2022b) has shown that this is possible with a sample complexity that depends only on the size of the controllable latent space, and not on the size of the noise factor.

Representation Learning

CAT: Caution Aware Transfer in Reinforcement Learning via Distributional Risk

no code implementations16 Aug 2024 Mohamad Fares El Hajj Chehade, Amrit Singh Bedi, Amy Zhang, Hao Zhu

To the best of our knowledge, this is the first work to explore the optimization of such a generalized risk notion within the context of transfer RL.

reinforcement-learning Reinforcement Learning +2

Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning

no code implementations29 Jul 2024 Liyuan Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang, Amy Zhang

In this work, we show that DICE-based methods can be viewed as a transformation from the behavior distribution to the optimal policy distribution.

Offline RL reinforcement-learning +1

Unified Auto-Encoding with Masked Diffusion

2 code implementations25 Jun 2024 Philippe Hansen-Estruch, Sriram Vishwanath, Amy Zhang, Manan Tomar

At the core of both successful generative and self-supervised representation learning models there is a reconstruction objective that incorporates some form of image corruption.

Computational Efficiency Representation Learning

A Dual Approach to Imitation Learning from Observations with Offline Datasets

no code implementations13 Jun 2024 Harshit Sikchi, Caleb Chuck, Amy Zhang, Scott Niekum

DILO reduces the learning from observations problem to that of simply learning an actor and a critic, bearing similar complexity to vanilla offline RL.

Imitation Learning Offline RL

Robot Air Hockey: A Manipulation Testbed for Robot Learning with Reinforcement Learning

no code implementations6 May 2024 Caleb Chuck, Carl Qi, Michael J. Munje, Shuozhe Li, Max Rudolph, Chang Shi, Siddhant Agarwal, Harshit Sikchi, Abhinav Peri, Sarthak Dayal, Evan Kuo, Kavan Mehta, Anthony Wang, Peter Stone, Amy Zhang, Scott Niekum

Reinforcement Learning is a promising tool for learning complex policies even in fast-moving and object-interactive domains where human teleoperation or hard-coded policies might fail.

Offline RL

Automated Discovery of Functional Actual Causes in Complex Environments

no code implementations16 Apr 2024 Caleb Chuck, Sankaran Vaidyanathan, Stephen Giguere, Amy Zhang, David Jensen, Scott Niekum

This paper introduces functional actual cause (FAC), a framework that uses context-specific independencies in the environment to restrict the set of actual causes.

Attribute Reinforcement Learning (RL)

Learning Action-based Representations Using Invariance

no code implementations25 Mar 2024 Max Rudolph, Caleb Chuck, Kevin Black, Misha Lvovsky, Scott Niekum, Amy Zhang

Robust reinforcement learning agents using high-dimensional observations must be able to identify relevant state features amidst many exogeneous distractors.

Multistep Inverse Is Not All You Need

1 code implementation18 Mar 2024 Alexander Levine, Peter Stone, Amy Zhang

In this work, we consider the Ex-BMDP model, first proposed by Efroni et al. (2022), which formalizes control problems where observations can be factorized into an action-dependent latent state which evolves deterministically, and action-independent time-correlated noise.

Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learning

no code implementations5 Feb 2024 Zihan Ding, Amy Zhang, Yuandong Tian, Qinqing Zheng

We introduce Diffusion World Model (DWM), a conditional diffusion model capable of predicting multistep future states and rewards concurrently.

D4RL Q-Learning

Zero-Shot Reinforcement Learning via Function Encoders

2 code implementations30 Jan 2024 Tyler Ingebrand, Amy Zhang, Ufuk Topcu

Although reinforcement learning (RL) can solve many challenging sequential decision making problems, achieving zero-shot transfer across related tasks remains a challenge.

Decision Making reinforcement-learning +3

An Investigation of Time Reversal Symmetry in Reinforcement Learning

1 code implementation28 Nov 2023 Brett Barkley, Amy Zhang, David Fridovich-Keil

We observe that utilizing the structure of time reversal in an MDP allows every environment transition experienced by an agent to be transformed into a feasible reverse-time transition, effectively doubling the number of experiences in the environment.

Data Augmentation Friction +3

Accelerating Exploration with Unlabeled Prior Data

1 code implementation NeurIPS 2023 Qiyang Li, Jason Zhang, Dibya Ghosh, Amy Zhang, Sergey Levine

Learning to solve tasks from a sparse reward signal is a major challenge for standard reinforcement learning (RL) algorithms.

Reinforcement Learning (RL)

SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learning

no code implementations3 Nov 2023 Harshit Sikchi, Rohan Chitnis, Ahmed Touati, Alborz Geramifard, Amy Zhang, Scott Niekum

Offline Goal-Conditioned Reinforcement Learning (GCRL) is tasked with learning to achieve multiple goals in an environment purely from offline datasets using sparse reward functions.

Contrastive Learning reinforcement-learning +2

Towards Robust Offline Reinforcement Learning under Diverse Data Corruption

2 code implementations19 Oct 2023 Rui Yang, Han Zhong, Jiawei Xu, Amy Zhang, Chongjie Zhang, Lei Han, Tong Zhang

Offline reinforcement learning (RL) presents a promising approach for learning reinforced policies from offline datasets without the need for costly or unsafe interactions with the environment.

Offline RL Q-Learning +3

$f$-Policy Gradients: A General Framework for Goal Conditioned RL using $f$-Divergences

no code implementations10 Oct 2023 Siddhant Agarwal, Ishan Durugkar, Peter Stone, Amy Zhang

We further introduce an entropy-regularized policy optimization objective, that we call $state$-MaxEnt RL (or $s$-MaxEnt RL) as a special case of our objective.

Efficient Exploration Policy Gradient Methods +1

Confidence Contours: Uncertainty-Aware Annotation for Medical Semantic Segmentation

1 code implementation15 Aug 2023 Andre Ye, Quan Ze Chen, Amy Zhang

However, as these annotations cannot represent an individual annotator's uncertainty, models trained on them produce uncertainty maps that are difficult to interpret.

Image Segmentation Medical Image Segmentation +2

Structure in Deep Reinforcement Learning: A Survey and Open Problems

no code implementations28 Jun 2023 Aditya Mohan, Amy Zhang, Marius Lindauer

We amalgamate these diverse methodologies under a unified framework, shedding light on the role of structure in the learning problem, and classify these methods into distinct patterns of incorporating structure.

reinforcement-learning Reinforcement Learning (RL)

Generalization Across Observation Shifts in Reinforcement Learning

no code implementations7 Jun 2023 Anuj Mahajan, Amy Zhang

We focus on bisimulation metrics, which provide a powerful means for abstracting task relevant components of the observation and learning a succinct representation space for training the agent using reinforcement learning.

reinforcement-learning Reinforcement Learning

LIV: Language-Image Representations and Rewards for Robotic Control

1 code implementation1 Jun 2023 Yecheng Jason Ma, William Liang, Vaidehi Som, Vikash Kumar, Amy Zhang, Osbert Bastani, Dinesh Jayaraman

We present Language-Image Value learning (LIV), a unified objective for vision-language representation and reward learning from action-free videos with text annotations.

Contrastive Learning Imitation Learning

A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem

no code implementations26 May 2023 Paul Barde, Jakob Foerster, Derek Nowrouzezahrai, Amy Zhang

Training multiple agents to coordinate is an essential problem with applications in robotics, game theory, economics, and social sciences.

Multi-agent Reinforcement Learning

When should we prefer Decision Transformers for Offline Reinforcement Learning?

1 code implementation23 May 2023 Prajjwal Bhargava, Rohan Chitnis, Alborz Geramifard, Shagun Sodhani, Amy Zhang

Three popular algorithms for offline RL are Conservative Q-Learning (CQL), Behavior Cloning (BC), and Decision Transformer (DT), from the class of Q-Learning, Imitation Learning, and Sequence Modeling respectively.

D4RL Imitation Learning +6

Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning

1 code implementation3 Apr 2023 Tongzhou Wang, Antonio Torralba, Phillip Isola, Amy Zhang

In goal-reaching reinforcement learning (RL), the optimal value function has a particular geometry, called quasimetric structure.

reinforcement-learning Reinforcement Learning +1

Neural Constraint Satisfaction: Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangement

no code implementations20 Mar 2023 Michael Chang, Alyssa L. Dayan, Franziska Meier, Thomas L. Griffiths, Sergey Levine, Amy Zhang

Object rearrangement is a challenge for embodied agents because solving these tasks requires generalizing across a combinatorially large set of configurations of entities and their locations.

Confidence-aware 3D Gaze Estimation and Evaluation Metric

no code implementations17 Mar 2023 Qiaojie Zheng, Jiucai Zhang, Amy Zhang, Xiaoli Zhang

To address the unreliable and overconfident issues, we introduce a confidence-aware model that predicts uncertainties together with gaze angle estimations.

Gaze Estimation

Dual RL: Unification and New Methods for Reinforcement and Imitation Learning

1 code implementation16 Feb 2023 Harshit Sikchi, Qinqing Zheng, Amy Zhang, Scott Niekum

For offline RL, our analysis frames a recent offline RL method XQL in the dual framework, and we further propose a new method f-DVL that provides alternative choices to the Gumbel regression loss that fixes the known training instability issue of XQL.

Imitation Learning Offline RL +2

Contrastive Distillation Is a Sample-Efficient Self-Supervised Loss Policy for Transfer Learning

no code implementations21 Dec 2022 Chris Lengerich, Gabriel Synnaeve, Amy Zhang, Hugh Leather, Kurt Shuster, François Charton, Charysse Redwood

Traditional approaches to RL have focused on learning decision policies directly from episodic decisions, while slowly and implicitly learning the semantics of compositional representations needed for generalization.

Few-Shot Learning Language Modelling +2

Language Control Diffusion: Efficiently Scaling through Space, Time, and Tasks

1 code implementation27 Oct 2022 Edwin Zhang, Yujie Lu, William Wang, Amy Zhang

Training generalist agents is difficult across several axes, requiring us to deal with high-dimensional inputs (space), long horizons (time), and generalization to novel tasks.

reinforcement-learning Reinforcement Learning (RL)

Latent State Marginalization as a Low-cost Approach for Improving Exploration

1 code implementation3 Oct 2022 Dinghuai Zhang, Aaron Courville, Yoshua Bengio, Qinqing Zheng, Amy Zhang, Ricky T. Q. Chen

While the maximum entropy (MaxEnt) reinforcement learning (RL) framework -- often touted for its exploration and robustness capabilities -- is usually motivated from a probabilistic perspective, the use of deep probabilistic models has not gained much traction in practice due to their inherent complexity.

continuous-control Continuous Control +2

VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training

1 code implementation30 Sep 2022 Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, Amy Zhang

Given the inherent cost and scarcity of in-domain, task-specific robot data, learning from large, diverse, offline human videos has emerged as a promising path towards acquiring a generally useful visual representation for control; however, how these human videos can be used for general-purpose reward learning remains an open question.

Offline RL Open-Ended Question Answering +2

Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning

no code implementations27 Apr 2022 Philippe Hansen-Estruch, Amy Zhang, Ashvin Nair, Patrick Yin, Sergey Levine

We learn this representation using a metric form of this abstraction, and show its ability to generalize to new goals in simulation manipulation tasks.

reinforcement-learning Reinforcement Learning +1

Robust Policy Learning over Multiple Uncertainty Sets

no code implementations14 Feb 2022 Annie Xie, Shagun Sodhani, Chelsea Finn, Joelle Pineau, Amy Zhang

Reinforcement learning (RL) agents need to be robust to variations in safety-critical environments.

Reinforcement Learning (RL)

Online Decision Transformer

2 code implementations11 Feb 2022 Qinqing Zheng, Amy Zhang, Aditya Grover

Recent work has shown that offline reinforcement learning (RL) can be formulated as a sequence modeling problem (Chen et al., 2021; Janner et al., 2021) and solved via approaches similar to large-scale language modeling.

D4RL Efficient Exploration +2

Learning Representations for Pixel-based Control: What Matters and Why?

no code implementations15 Nov 2021 Manan Tomar, Utkarsh A. Mishra, Amy Zhang, Matthew E. Taylor

A wide range of methods have been proposed to enable efficient learning, leading to sample complexities similar to those in the full state setting.

Contrastive Learning Representation Learning

Block Contextual MDPs for Continual Learning

no code implementations13 Oct 2021 Shagun Sodhani, Franziska Meier, Joelle Pineau, Amy Zhang

In this work, we propose to examine this continual reinforcement learning setting through the block contextual MDP (BC-MDP) framework, which enables us to relax the assumption of stationarity.

Continual Learning Generalization Bounds +3

Learning Minimal Representations with Model Invariance

no code implementations29 Sep 2021 Manan Tomar, Amy Zhang, Matthew E. Taylor

The common representation acts as a implicit invariance objective to avoid the different spurious correlations captured by individual predictors.

Self-Supervised Learning

Provably Efficient Representation Selection in Low-rank Markov Decision Processes: From Online to Offline RL

no code implementations22 Jun 2021 Weitong Zhang, Jiafan He, Dongruo Zhou, Amy Zhang, Quanquan Gu

For the offline counterpart, ReLEX-LCB, we show that the algorithm can find the optimal policy if the representation class can cover the state-action space and achieves gap-dependent sample complexity.

Offline RL reinforcement-learning +3

MBRL-Lib: A Modular Library for Model-based Reinforcement Learning

3 code implementations20 Apr 2021 Luis Pineda, Brandon Amos, Amy Zhang, Nathan O. Lambert, Roberto Calandra

MBRL-Lib is designed as a platform for both researchers, to easily develop, debug and compare new algorithms, and non-expert user, to lower the entry-bar of deploying state-of-the-art algorithms.

Model-based Reinforcement Learning reinforcement-learning +2

Multi-Task Reinforcement Learning with Context-based Representations

2 code implementations11 Feb 2021 Shagun Sodhani, Amy Zhang, Joelle Pineau

We posit that an efficient approach to knowledge transfer is through the use of multiple context-dependent, composable representations shared across a family of tasks.

Multi-Task Learning reinforcement-learning +2

Invariant Representations for Reinforcement Learning without Reconstruction

no code implementations ICLR 2021 Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, Sergey Levine

We study how representation learning can accelerate reinforcement learning from rich observations, such as images, without relying either on domain knowledge or pixel-reconstruction.

Causal Inference reinforcement-learning +3

Automating Document Classification with Distant Supervision to Increase the Efficiency of Systematic Reviews

no code implementations9 Dec 2020 Xiaoxiao Li, Rabah Al-Zaidy, Amy Zhang, Stefan Baral, Le Bao, C. Lee Giles

Conclusions: In sum, the automated procedure of document classification presented here could improve both the precision and efficiency of systematic reviews, as well as facilitating live reviews, where reviews are updated regularly.

Document Classification General Classification

Intervention Design for Effective Sim2Real Transfer

1 code implementation3 Dec 2020 Melissa Mozifian, Amy Zhang, Joelle Pineau, David Meger

The goal of this work is to address the recent success of domain randomization and data augmentation for the sim2real setting.

Causal Inference Data Augmentation

Learning Robust State Abstractions for Hidden-Parameter Block MDPs

2 code implementations ICLR 2021 Amy Zhang, Shagun Sodhani, Khimya Khetarpal, Joelle Pineau

Further, we provide transfer and generalization bounds based on task and state similarity, along with sample complexity bounds that depend on the aggregate number of samples across tasks, rather than the number of tasks, a significant improvement over prior work that use the same environment assumptions.

Generalization Bounds Meta Reinforcement Learning +3

Learning Invariant Representations for Reinforcement Learning without Reconstruction

2 code implementations18 Jun 2020 Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, Sergey Levine

We study how representation learning can accelerate reinforcement learning from rich observations, such as images, without relying either on domain knowledge or pixel-reconstruction.

Causal Inference reinforcement-learning +3

Plan2Vec: Unsupervised Representation Learning by Latent Plans

1 code implementation7 May 2020 Ge Yang, Amy Zhang, Ari S. Morcos, Joelle Pineau, Pieter Abbeel, Roberto Calandra

In this paper we introduce plan2vec, an unsupervised representation learning approach that is inspired by reinforcement learning.

Motion Planning reinforcement-learning +2

Invariant Causal Prediction for Block MDPs

1 code implementation ICML 2020 Amy Zhang, Clare Lyle, Shagun Sodhani, Angelos Filos, Marta Kwiatkowska, Joelle Pineau, Yarin Gal, Doina Precup

Generalization across environments is critical to the successful application of reinforcement learning algorithms to real-world challenges.

Causal Inference Reinforcement Learning +1

Stable Policy Optimization via Off-Policy Divergence Regularization

1 code implementation9 Mar 2020 Ahmed Touati, Amy Zhang, Joelle Pineau, Pascal Vincent

Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are among the most successful policy gradient approaches in deep reinforcement learning (RL).

Reinforcement Learning Reinforcement Learning (RL)

Learning Causal State Representations of Partially Observable Environments

no code implementations25 Jun 2019 Amy Zhang, Zachary C. Lipton, Luis Pineda, Kamyar Azizzadenesheli, Anima Anandkumar, Laurent Itti, Joelle Pineau, Tommaso Furlanello

In this paper, we propose an algorithm to approximate causal states, which are the coarsest partition of the joint history of actions and observations in partially-observable Markov decision processes (POMDP).

Causal Inference Reinforcement Learning

Natural Environment Benchmarks for Reinforcement Learning

2 code implementations14 Nov 2018 Amy Zhang, Yuxin Wu, Joelle Pineau

While current benchmark reinforcement learning (RL) tasks have been useful to drive progress in the field, they are in many ways poor substitutes for learning with real-world data.

reinforcement-learning Reinforcement Learning +1

Decoupling Dynamics and Reward for Transfer Learning

1 code implementation27 Apr 2018 Amy Zhang, Harsh Satija, Joelle Pineau

Current reinforcement learning (RL) methods can successfully learn single tasks but often generalize poorly to modest perturbations in task domain or training procedure.

reinforcement-learning Reinforcement Learning +2

Mapping the world population one building at a time

no code implementations15 Dec 2017 Tobias G. Tiecke, Xian-Ming Liu, Amy Zhang, Andreas Gros, Nan Li, Gregory Yetman, Talip Kilic, Siobhan Murray, Brian Blankespoor, Espen B. Prydz, Hai-Anh H. Dang

Obtaining high accuracy in estimation of population distribution in rural areas remains a very challenging task due to the simultaneous requirements of sufficient sensitivity and resolution to detect very sparse populations through remote sensing as well as reliable performance at a global scale.

Disaster Response

Building Detection from Satellite Images on a Global Scale

no code implementations27 Jul 2017 Amy Zhang, Xian-Ming Liu, Andreas Gros, Tobias Tiecke

Our work is some of the first to create population density maps from building detection on a large scale.

Density Estimation

Feedback Neural Network for Weakly Supervised Geo-Semantic Segmentation

no code implementations8 Dec 2016 Xianming Liu, Amy Zhang, Tobias Tiecke, Andreas Gros, Thomas S. Huang

Learning from weakly-supervised data is one of the main challenges in machine learning and computer vision, especially for tasks such as image semantic segmentation where labeling is extremely expensive and subjective.

Semantic Segmentation Weakly-supervised Learning

Cannot find the paper you are looking for? You can Submit a new open access paper.