Search Results for author:
Found 491 papers, 228 papers with code
Time-Contrastive Networks: Self-Supervised Learning from Video
While representations are learned from an unlabeled collection of task-related videos, robot behaviors such as pouring are learned by watching a single 3rd-person demonstration by a human.
Ranked #3 on
Video Alignment
on UPenn Action
Data-Efficient Hierarchical Reinforcement Learning
In this paper, we study how we can develop HRL algorithms that are general, in that they do not make onerous additional assumptions beyond standard RL algorithms, and efficient, in the sense that they can be used with modest numbers of interaction samples, making them suitable for real-world problems such as robotic control.
Hierarchical Reinforcement Learning
reinforcement-learning
+1
Cognitive Mapping and Planning for Visual Navigation
The accumulated belief of the world enables the agent to track visited regions of the environment.
MuProp: Unbiased Backpropagation for Stochastic Neural Networks
Deep neural networks are powerful parametric models that can be trained efficiently using the backpropagation algorithm.
Near-Optimal Representation Learning for Hierarchical Reinforcement Learning
We study the problem of representation learning in goal-conditioned hierarchical reinforcement learning.
Unsupervised Learning for Physical Interaction through Video Prediction
A core challenge for an agent learning to interact with the world is to predict how its actions affect objects in its environment.
Ranked #26 on
Video Generation
on BAIR Robot Pushing
High-Dimensional Continuous Control Using Generalized Advantage Estimation
Policy gradient methods are an appealing approach in reinforcement learning because they directly optimize the cumulative reward and can straightforwardly be used with nonlinear function approximators such as neural networks.
Search on the Replay Buffer: Bridging Planning and Reinforcement Learning
We introduce a general control algorithm that combines the strengths of planning and reinforcement learning to effectively solve these tasks.
TRAIL: Near-Optimal Imitation Learning with Suboptimal Data
In this work, we answer this question affirmatively and present training objectives that use offline datasets to learn a factored transition model whose structure enables the extraction of a latent action space.
Meta-Learning without Memorization
If this is not done, the meta-learner can ignore the task training data and learn a single model that performs all of the meta-training tasks zero-shot, but does not adapt effectively to new image classes.
Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design
We call our technique Protagonist Antagonist Induced Regret Environment Design (PAIRED).
Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification
Can we devise RL algorithms that instead enable users to specify tasks simply by providing examples of successful outcomes?
Data-Driven Offline Optimization For Architecting Hardware Accelerators
An alternative paradigm is to use a "data-driven", offline approach that utilizes logged simulation data, to architect hardware accelerators, without needing any form of simulations.
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
A platform for Applied Reinforcement Learning (Applied RL)
Ranked #1 on
Continuous Control
on Lunar Lander (OpenAI Gym)
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
We propose an algorithm for meta-learning that is model-agnostic, in the sense that it is compatible with any model trained with gradient descent and applicable to a variety of different learning problems, including classification, regression, and reinforcement learning.
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
In this tutorial article, we aim to provide the reader with the conceptual tools needed to get started on research on offline reinforcement learning algorithms: reinforcement learning algorithms that utilize previously collected data, without additional online data collection.
Conservative Q-Learning for Offline Reinforcement Learning
We theoretically show that CQL produces a lower bound on the value of the current policy and that it can be incorporated into a policy learning procedure with theoretical improvement guarantees.
Model-Based Reinforcement Learning for Atari
We describe Simulated Policy Learning (SimPLe), a complete model-based deep RL algorithm based on video prediction models and present a comparison of several model architectures, including a novel architecture that yields the best results in our setting.
Ranked #12 on
Atari Games 100k
on Atari 100k
VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation
Generative models that can model and predict sequences of future events can, in principle, learn to capture complex real-world phenomena, such as physical interactions.
Ranked #15 on
Video Generation
on BAIR Robot Pushing
Soft Actor-Critic Algorithms and Applications
A fork of OpenAI Baselines, implementations of reinforcement learning algorithms
Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
In this paper, we aim to develop a simple and scalable reinforcement learning algorithm that uses standard supervised learning methods as subroutines.
Ranked #1 on
OpenAI Gym
on Humanoid-v2
Trust Region Policy Optimization
We describe an iterative procedure for optimizing policies, with guaranteed monotonic improvement.
Reinforcement Learning with Deep Energy-Based Policies
We propose a method for learning expressive energy-based policies for continuous states and actions, which has been feasible only in tabular domains before.
Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
We explore how inverse optimal control (IOC) can be used to learn behaviors from demonstrations, with applications to torque control of high-dimensional robotic systems.
SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
Theoretically, we show that SQIL can be interpreted as a regularized variant of BC that uses a sparsity prior to encourage long-horizon imitation.
When to Trust Your Model: Model-Based Policy Optimization
Designing effective model-based reinforcement learning algorithms is difficult because the ease of data generation must be weighed against the bias of model-generated data.
Model-based Reinforcement Learning
reinforcement-learning
+1
Offline Reinforcement Learning with Implicit Q-Learning
The main insight in our work is that, instead of evaluating unseen actions from the latest policy, we can approximate the policy improvement step implicitly by treating the state value function as a random variable, with randomness determined by the action (while still integrating over the dynamics to avoid excessive optimism), and then taking a state conditional upper expectile of this random variable to estimate the value of the best actions in that state.
Planning with Diffusion for Flexible Behavior Synthesis
Model-based reinforcement learning methods often use learning only for the purpose of estimating an approximate dynamics model, offloading the rest of the decision-making work to classical trajectory optimizers.
Visual Reinforcement Learning with Imagined Goals
For an autonomous agent to fulfill a wide range of user-specified goals at test time, it must be able to learn broadly applicable and general-purpose skill repertoires.
Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
Autonomous agents that must exhibit flexible and broad capabilities will need to be equipped with large repertoires of skills.
AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
If we can instead allow RL algorithms to effectively use previously collected data to aid the online learning process, such applications could be made substantially more practical: the prior data would provide a starting point that mitigates challenges due to exploration and sample complexity, while the online training enables the agent to perfect the desired skill.
Robust Predictable Control
Many of the challenges facing today's reinforcement learning (RL) algorithms, such as robustness, generalization, transfer, and computational efficiency are closely related to compression.
DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
We further explore a number of methods for integrating multiple clips into the learning process to develop multi-skilled agents capable of performing a rich repertoire of diverse skills.
The Variational Bandwidth Bottleneck: Stochastic Evaluation on an Information Budget
This is typically the case when we have a standard conditioning input, such as a state observation, and a "privileged" input, which might correspond to the goal of a task, the output of a costly planning algorithm, or communication with another agent.
AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control
Our system produces high-quality motions that are comparable to those achieved by state-of-the-art tracking-based techniques, while also being able to easily accommodate large datasets of unstructured motion clips.
Adaptive Risk Minimization: Learning to Adapt to Domain Shift
A fundamental assumption of most machine learning algorithms is that the training and test data are drawn from the same underlying distribution.
WILDS: A Benchmark of in-the-Wild Distribution Shifts
Distribution shifts -- where the training distribution differs from the test distribution -- can substantially degrade the accuracy of machine learning (ML) systems deployed in the wild.
Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
Bootstrapping error is due to bootstrapping from actions that lie outside of the training data distribution, and it accumulates via the Bellman backup operator.
D4RL: Datasets for Deep Data-Driven Reinforcement Learning
In this work, we introduce benchmarks specifically designed for the offline setting, guided by key properties of datasets relevant to real-world applications of offline RL.
RT-1: Robotics Transformer for Real-World Control at Scale
By transferring knowledge from large, diverse, task-agnostic datasets, modern machine learning models can solve specific downstream tasks either zero-shot or with small task-specific datasets to a high level of performance.
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
Therefore, if the aim of these methods is to enable faster acquisition of entirely new behaviors, we must evaluate them on task distributions that are sufficiently broad to enable generalization to new behaviors.
Ranked #1 on
Meta-Learning
on ML10
The False Promise of Imitating Proprietary LLMs
This approach looks to cheaply imitate the proprietary model's capabilities using a weaker open-source model.
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
Model-based reinforcement learning (RL) algorithms can attain excellent sample efficiency, but often lag behind the best model-free algorithms in terms of asymptotic performance.
Model-based Reinforcement Learning
reinforcement-learning
+1
Deep Dynamics Models for Learning Dexterous Manipulation
Dexterous multi-fingered hands can provide robots with the ability to flexibly perform a wide range of manipulation skills.
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
The framework of reinforcement learning or optimal control provides a mathematical formalization of intelligent decision making that is powerful and broadly applicable.
Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow
By enforcing a constraint on the mutual information between the observations and the discriminator's internal representation, we can effectively modulate the discriminator's accuracy and maintain useful and informative gradients.
Guided Policy Search as Approximate Mirror Descent
Guided policy search algorithms can be used to optimize complex nonlinear policies, such as deep neural networks, without directly computing policy gradients in the high-dimensional parameter space.
Value Iteration Networks
We introduce the value iteration network (VIN): a fully differentiable neural network with a `planning module' embedded within.
Grasp2Vec: Learning Object Representations from Self-Supervised Grasping
We formulate an arithmetic relationship between feature vectors from this observation, and use it to learn a representation of scenes and objects that can then be used to identify object instances, localize them in the scene, and perform goal-directed grasping tasks where the robot must retrieve commanded objects from a bin.
Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning
We identify an implicit under-parameterization phenomenon in value-based deep RL methods that use bootstrapping: when value functions, approximated using deep neural networks, are trained with gradient descent using iterated regression onto target values generated by previous instances of the value network, more gradient updates decrease the expressivity of the current value network.
Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
In our approach, we perform online probabilistic filtering of latent task variables to infer how to solve a new task from small amounts of experience.
Offline Reinforcement Learning as One Big Sequence Modeling Problem
Reinforcement learning (RL) is typically concerned with estimating stationary policies or single-step models, leveraging the Markov property to factorize problems in time.
Reinforcement Learning as One Big Sequence Modeling Problem
However, we can also view RL as a sequence modeling problem, with the goal being to predict a sequence of actions that leads to a sequence of high rewards.
Composable Deep Reinforcement Learning for Robotic Manipulation
Second, we show that policies learned with soft Q-learning can be composed to create new policies, and that the optimality of the resulting policy can be bounded in terms of the divergence between the composed policies.
Learning Robust Rewards with Adversarial Inverse Reinforcement Learning
Reinforcement learning provides a powerful and general framework for decision making and control, but its application in practice is often hindered by the need for extensive feature and reward engineering.
Ranked #3 on
MuJoCo Games
on Ant
Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
We identify two issues with the family of algorithms based on the Adversarial Imitation Learning framework.
End-to-End Robotic Reinforcement Learning without Reward Engineering
In this paper, we propose an approach for removing the need for manual engineering of reward specifications by enabling a robot to learn from a modest number of examples of successful outcomes, followed by actively solicited queries, where the robot shows the user a state and asks for a label to determine whether that state represents successful completion of the task.
Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning
Model-free deep reinforcement learning algorithms have been shown to be capable of learning a wide range of robotic skills, but typically require a very large number of samples to achieve good performance.
Model-based Reinforcement Learning
Model Predictive Control
+2
Training Diffusion Models with Reinforcement Learning
However, most use cases of diffusion models are not concerned with likelihoods, but instead with downstream objectives such as human-perceived image quality or drug effectiveness.
OmniTact: A Multi-Directional High Resolution Touch Sensor
We compare with a state-of-the-art tactile sensor that is only sensitive on one side, as well as a state-of-the-art multi-directional tactile sensor, and find that OmniTact's combination of high-resolution and multi-directional sensing is crucial for reliably inserting the electrical connector and allows for higher accuracy in the state estimation task.
SFV: Reinforcement Learning of Physical Skills from Videos
In this paper, we propose a method that enables physically simulated characters to learn skills from videos (SFV).
Stochastic Adversarial Video Prediction
However, learning to predict raw future observations, such as frames in a video, is exceedingly challenging -- the ambiguous nature of the problem can cause a naively designed model to average together possible futures into a single, blurry prediction.
Ranked #1 on
Video Prediction
on KTH
(Cond metric)
Gradient Surgery for Multi-Task Learning
While deep learning and deep reinforcement learning (RL) systems have demonstrated impressive results in domains such as image classification, game playing, and robotic control, data efficiency remains a major challenge.
One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning
Humans and animals are capable of learning a new behavior by observing others perform the skill just once.
One-Shot Visual Imitation Learning via Meta-Learning
In this work, we present a meta-imitation learning method that enables a robot to learn how to learn more efficiently, allowing it to acquire new skills from just a single demonstration.
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Our goal is to enable a single end-to-end trained model to both learn to map robot observations to actions and enjoy the benefits of large-scale pretraining on language and vision-language data from the web.
Adversarial Policies: Attacking Deep Reinforcement Learning
Deep reinforcement learning (RL) policies are known to be vulnerable to adversarial perturbations to their observations, similar to adversarial examples for classifiers.
Inter-Level Cooperation in Hierarchical Reinforcement Learning
This allows us to draw on connections between communication and cooperation in multi-agent RL, and demonstrate the benefits of increased cooperation between sub-policies on the training performance of the overall policy.
Hierarchical Reinforcement Learning
reinforcement-learning
+1
A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
In particular, we demonstrate an equivalence between a sample-based algorithm for maximum entropy IRL and a GAN in which the generator's density can be evaluated and is provided as an additional input to the discriminator.
Probabilistic Model-Agnostic Meta-Learning
However, a critical challenge in few-shot learning is task ambiguity: even when a powerful prior can be meta-learned from a large number of prior tasks, a small dataset for a new task can simply be too ambiguous to acquire a single model (e. g., a classifier) for that task that is accurate.
A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning
Deep reinforcement learning is a promising approach to learning policies in uncontrolled environments that do not require domain knowledge.
GNM: A General Navigation Model to Drive Any Robot
Learning provides a powerful tool for vision-based navigation, but the capabilities of learning-based policies are constrained by limited training data.
MOPO: Model-based Offline Policy Optimization
We also characterize the trade-off between the gain and risk of leaving the support of the batch data.
COMBO: Conservative Offline Model-Based Policy Optimization
We overcome this limitation by developing a new model-based offline RL algorithm, COMBO, that regularizes the value function on out-of-support state-action tuples generated via rollouts under the learned model.
Learning from the Hindsight Plan -- Episodic MPC Improvement
To bring the next real-world execution closer to the hindsight plan, our approach learns to re-shape the original cost function with the goal of satisfying the following property: short horizon planning (as realistic during real executions) with respect to the shaped cost should result in mimicking the hindsight plan.
GenLoco: Generalized Locomotion Controllers for Quadrupedal Robots
In this work, we introduce a framework for training generalized locomotion (GenLoco) controllers for quadrupedal robots.
Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning
Although reinforcement learning methods can achieve impressive results in simulation, the real world presents two major challenges: generating samples is exceedingly expensive, and unexpected perturbations or unseen situations cause proficient but specialized policies to fail at test time.
PaLM-E: An Embodied Multimodal Language Model
Large language models excel at a wide range of complex tasks.
Ranked #2 on
Visual Question Answering (VQA)
on OK-VQA
Offline RL for Natural Language Generation with Implicit Language Q Learning
Large language models distill broad knowledge from text corpora.
Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?
Out-of-training-distribution (OOD) scenarios are a common challenge of learning agents at deployment, typically leading to arbitrary deductions and poorly-informed decisions.
Dynamics-Aware Unsupervised Discovery of Skills
Conventionally, model-based reinforcement learning (MBRL) aims to learn a global model for the dynamics of the environment.
Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning
Can we instead develop efficient reinforcement learning methods that acquire diverse skills without any reward function, and then repurpose these skills for downstream tasks?
Dynamics-Aware Unsupervised Skill Discovery
Conventionally, model-based reinforcement learning (MBRL) aims to learn a global model for the dynamics of the environment.
Self-Supervised Visual Planning with Temporal Skip Connections
One learning signal that is always available for autonomously collected data is prediction: if a robot can learn to predict the future, it can use this predictive model to take actions to produce desired outcomes, such as moving an object to a particular location.
Efficient Online Reinforcement Learning with Offline Data
Sample efficiency and exploration remain major challenges in online reinforcement learning (RL).
LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action
Goal-conditioned policies for robotic navigation can be trained on large, unannotated datasets, providing for good generalization to real-world settings.
Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
We analyze the connection between Q-Prop and existing model-free algorithms, and use control variate theory to derive two variants of Q-Prop with conservative and aggressive adaptation.
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
We show how low-level skills can be combined with large language models so that the language model provides high-level knowledge about the procedures for performing complex and temporally-extended instructions, while value functions associated with these skills provide the grounding necessary to connect this knowledge to a particular physical environment.
Few-Shot Segmentation Propagation with Guided Networks
Learning-based methods for visual segmentation have made progress on particular types of segmentation tasks, but are limited by the necessary supervision, the narrow definitions of fixed tasks, and the lack of control during inference for correcting errors.
Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
Deep reinforcement learning (RL) algorithms can use high-capacity deep networks to learn directly from image observations.
BADGR: An Autonomous Self-Supervised Learning-Based Navigation System
Mobile robot navigation is typically regarded as a geometric problem, in which the robot's objective is to perceive the geometry of the environment in order to plan collision-free paths towards a desired goal.
Learning Invariant Representations for Reinforcement Learning without Reconstruction
We study how representation learning can accelerate reinforcement learning from rich observations, such as images, without relying either on domain knowledge or pixel-reconstruction.
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
Deep reinforcement learning (RL) algorithms can learn complex robotic skills from raw sensory inputs, but have yet to achieve the kind of broad generalization and applicability demonstrated by deep learning methods in supervised domains.
Learning to Run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environments
In the NIPS 2017 Learning to Run challenge, participants were tasked with building a controller for a musculoskeletal model to make it run as fast as possible through an obstacle course.
Meta-Learning with Implicit Gradients
By drawing upon implicit differentiation, we develop the implicit MAML algorithm, which depends only on the solution to the inner level optimization and not the path taken by the inner loop optimizer.
Recurrent Independent Mechanisms
Learning modular structures which reflect the dynamics of the environment can lead to better generalization and robustness to changes which only affect a few of the underlying causes.
Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning
In this paper, we study a number of design decisions for the predictive model in visual MBRL algorithms, focusing specifically on methods that use a predictive model for planning.
Model-based Reinforcement Learning
Reinforcement Learning (RL)
Modular Multitask Reinforcement Learning with Policy Sketches
We describe a framework for multitask deep reinforcement learning guided by policy sketches.
Self-supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation
To address the need to learn complex policies with few samples, we propose a generalized computation graph that subsumes value-based model-free methods and model-based methods, with specific instantiations interpolating between model-free and model-based.
Multimodal Masked Autoencoders Learn Transferable Representations
We provide an empirical study of M3AE trained on a large-scale image-text dataset, and find that M3AE is able to learn generalizable representations that transfer well to downstream tasks.
PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings
For autonomous vehicles (AVs) to behave appropriately on roads populated by human-driven vehicles, they must be able to reason about the uncertain intentions and decisions of other drivers from rich perceptual information.
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction
We show that bootstrapping-based Q-learning algorithms do not necessarily benefit from this corrective feedback, and training on the experience collected by the algorithm is not sufficient to correct errors in the Q-function.
Ranked #3 on
Meta-Learning
on MT50
Continuous Deep Q-Learning with Model-based Acceleration
In this paper, we explore algorithms and representations to reduce the sample complexity of deep reinforcement learning for continuous control tasks.
Learning Human Objectives by Evaluating Hypothetical Behavior
To address this challenge, we propose an algorithm that safely and interactively learns a model of the user's reward function.
Benchmarks for Deep Off-Policy Evaluation
Off-policy evaluation (OPE) holds the promise of being able to leverage large, offline datasets for both evaluating and selecting complex policies for decision making.
BridgeData V2: A Dataset for Robot Learning at Scale
By publicly sharing BridgeData V2 and our pre-trained models, we aim to accelerate research in scalable robot learning methods.
Generalization through Simulation: Integrating Simulated and Real Data into Deep Reinforcement Learning for Vision-Based Autonomous Flight
Deep reinforcement learning provides a promising approach for vision-based control of real-world robots.
Reinforcement Learning from Passive Data via Latent Intentions
Passive observational data, such as human videos, is abundant and rich in information, yet remains largely untapped by current RL methods.
Learning Instance Segmentation by Interaction
The agent uses its current segmentation model to infer pixels that constitute objects and refines the segmentation model by interacting with these pixels.
Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimization
To address this, we present Design-Bench, a benchmark for offline MBO with a unified evaluation protocol and reference implementations of recent methods.
Adversarial Policies Beat Superhuman Go AIs
The core vulnerability uncovered by our attack persists even in KataGo agents adversarially trained to defend against our attack.
IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies
In this paper, we reinterpret IQL as an actor-critic method by generalizing the critic objective and connecting it to a behavior-regularized implicit actor.
Learning to Reach Goals via Iterated Supervised Learning
Current reinforcement learning (RL) algorithms can be brittle and difficult to use, especially when learning goal-reaching behaviors from sparse rewards.
Multi-Goal Reinforcement Learning
Reinforcement Learning (RL)
Shared Autonomy via Deep Reinforcement Learning
In shared autonomy, user input is combined with semi-autonomous control to achieve a common goal.
Deep Imitative Models for Flexible Inference, Planning, and Control
Yet, reward functions that evoke desirable behavior are often difficult to specify.
SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
Model-based reinforcement learning (RL) has proven to be a data efficient approach for learning control tasks but is difficult to utilize in domains with complex observations such as images.
Model-based Reinforcement Learning
reinforcement-learning
+1
RvS: What is Essential for Offline RL via Supervised Learning?
Recent work has shown that supervised learning alone, without temporal difference (TD) learning, can be remarkably effective for offline RL.
Learning to Poke by Poking: Experiential Learning of Intuitive Physics
We investigate an experiential learning paradigm for acquiring an internal model of intuitive physics.
Efficient Exploration via State Marginal Matching
The SMM objective can be viewed as a two-player, zero-sum game between a state density model and a parametric policy, an idea that we use to build an algorithm for optimizing the SMM objective.
FitVid: Overfitting in Pixel-Level Video Prediction
There is a growing body of evidence that underfitting on the training data is one of the primary causes for the low quality predictions.
Ranked #6 on
Video Generation
on BAIR Robot Pushing
Nonlinear Inverse Reinforcement Learning with Gaussian Processes
We present a probabilistic algorithm for nonlinear inverse reinforcement learning.
MELD: Meta-Reinforcement Learning from Images via Latent State Models
Meta-reinforcement learning algorithms can enable autonomous agents, such as robots, to quickly acquire new behaviors by leveraging prior experience in a set of related training tasks.
Divide-and-Conquer Reinforcement Learning
In this paper, we develop a novel algorithm that instead partitions the initial state space into "slices", and optimizes an ensemble of policies, each on a different slice.
Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
Our approach, calibrated Q-learning (Cal-QL), accomplishes this by learning a conservative value function initialization that underestimates the value of the learned policy from offline data, while also being calibrated, in the sense that the learned Q-values are at a reasonable scale.
HIQL: Offline Goal-Conditioned RL with Latent States as Actions
This structure can be very useful, as assessing the quality of actions for nearby goals is typically easier than for more distant goals.
Diversity is All You Need: Learning Skills without a Reward Function
On a variety of simulated robotic tasks, we show that this simple objective results in the unsupervised emergence of diverse skills, such as walking and jumping.
Extending the WILDS Benchmark for Unsupervised Adaptation
Unlabeled data can be a powerful point of leverage for mitigating these distribution shifts, as it is frequently much more available than labeled data and can often be obtained from distributions beyond the source distribution as well.
CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning
Conventionally, generation of natural language for dialogue agents may be viewed as a statistical learning problem: determine the patterns in human-provided data and generate appropriate responses with similar statistical properties.
ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL
In this paper, we develop a framework for building multi-turn RL algorithms for fine-tuning LLMs, that preserves the flexibility of existing single-turn RL methods for LLMs (e. g., proximal policy optimization), while accommodating multiple turns, long horizons, and delayed rewards effectively.
Entity Abstraction in Visual Model-Based Reinforcement Learning
This paper tests the hypothesis that modeling a scene in terms of entities and their local interactions, as opposed to modeling the scene globally, provides a significant benefit in generalizing to physical tasks in a combinatorial space the learner has not encountered before.
Learning to Walk in the Real World with Minimal Human Effort
In this paper, we develop a system for learning legged locomotion policies with deep RL in the real world with minimal human effort.
LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models
Developing such algorithms requires tasks that can gauge progress on algorithm design, provide accessible and reproducible evaluations for multi-turn interactions, and cover a range of task properties and challenges in improving reinforcement learning algorithms.
Foundation Policies with Hilbert Representations
While a number of methods have been proposed to enable generic self-supervised RL, based on principles such as goal-conditioned RL, behavioral cloning, and unsupervised skill learning, such methods remain limited in terms of either the diversity of the discovered behaviors, the need for high-quality demonstration data, or the lack of a clear prompting or adaptation mechanism for downstream tasks.
Learning with Latent Language
The named concepts and compositional operators present in natural language provide a rich source of information about the kinds of abstractions humans use to navigate the world.
Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation
Imitation learning is an effective approach for autonomous systems to acquire control policies when an explicit reward function is unavailable, using supervision provided as demonstrations from an expert, typically a human operator.
Conservative Objective Models for Effective Offline Model-Based Optimization
Computational design problems arise in a number of settings, from synthetic biology to computer architectures.
Planning with Goal-Conditioned Policies
Planning methods can solve temporally extended sequential decision making problems by composing simple behaviors.
MEMO: Test Time Robustness via Adaptation and Augmentation
We study the problem of test time robustification, i. e., using the test input to improve model robustness.
Evolving Reinforcement Learning Algorithms
Learning from scratch on simple classical control and gridworld tasks, our method rediscovers the temporal-difference (TD) algorithm.
Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstration
We propose a technique for multi-task learning from demonstration that trains the controller of a low-cost robotic arm to accomplish several complex picking and placing tasks, as well as non-prehensile manipulation.
Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
In this work we propose a framework for visual prediction and planning that is able to overcome both of these limitations.
Offline Meta-Reinforcement Learning with Advantage Weighting
That is, in offline meta-RL, we meta-train on fixed, pre-collected data from several tasks in order to adapt to a new task with a very small amount (less than 5 trajectories) of data from the new task.
Causal Confusion in Imitation Learning
Such discriminative models are non-causal: the training procedure is unaware of the causal structure of the interaction between the expert and the environment.
Meta-Reinforcement Learning of Structured Exploration Strategies
Exploration is a fundamental challenge in reinforcement learning (RL).
Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings
Can we pre-train a generalist agent from a large amount of unlabeled offline trajectories such that it can be immediately adapted to any new downstream tasks in a zero-shot manner?
LaND: Learning to Navigate from Disengagements
However, we believe that these disengagements not only show where the system fails, which is useful for troubleshooting, but also provide a direct learning signal by which the robot can learn to navigate.
METRA: Scalable Unsupervised RL with Metric-Aware Abstraction
Through our experiments in five locomotion and manipulation environments, we demonstrate that METRA can discover a variety of useful behaviors even in complex, pixel-based environments, being the first unsupervised RL method that discovers diverse locomotion behaviors in pixel-based Quadruped and Humanoid.
Generative Temporal Difference Learning for Infinite-Horizon Prediction
We introduce the $\gamma$-model, a predictive model of environment dynamics with an infinite probabilistic horizon.
Gamma-Models: Generative Temporal Difference Learning for Infinite-Horizon Prediction
We introduce the gamma-model, a predictive model of environment dynamics with an infinite, probabilistic horizon.
EMI: Exploration with Mutual Information
Reinforcement learning algorithms struggle when the reward signal is very sparse.
Autonomous Evaluation and Refinement of Digital Agents
We show that domain-general automatic evaluators can significantly improve the performance of agents for web navigation and device control.
Autonomous Reinforcement Learning: Formalism and Benchmarking
In this paper, we aim to address this discrepancy by laying out a framework for Autonomous Reinforcement Learning (ARL): reinforcement learning where the agent not only learns through its own experience, but also contends with lack of human supervision to reset between trials.
Universal Planning Networks
We find that the representations learned are not only effective for goal-directed visual imitation via gradient-based trajectory optimization, but can also provide a metric for specifying goals using images.
Learning Visual Servoing with Deep Features and Fitted Q-Iteration
Our approach is based on servoing the camera in the space of learned visual features, rather than image pixels or manually-designed keypoints.
Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigation
We show that a simulated robotic car and a real-world RC car can gather data and train fully autonomously without any human-provided labels beyond those needed to train the detectors, and then at test-time be able to accomplish a variety of different tasks.
Universal Planning Networks: Learning Generalizable Representations for Visuomotor Control
A key challenge in complex visuomotor control is learning abstract representations that are effective for specifying goals, planning, and generalization.
COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning
Reinforcement learning has been applied to a wide variety of robotics problems, but most of such applications involve collecting data from scratch for each new task.
Contingencies from Observations: Tractable Contingency Planning with Learned Behavior Models
Humans have a remarkable ability to make decisions by accurately reasoning about future events, including the future behaviors and states of mind of other agents.
Model-Based Reinforcement Learning via Latent-Space Collocation
The resulting latent collocation method (LatCo) optimizes trajectories of latent states, which improves over previously proposed shooting methods for visual model-based RL on tasks with sparse rewards and long-term goals.
Model-based Reinforcement Learning
reinforcement-learning
+1
Stochastic Variational Video Prediction
We find that our proposed method produces substantially improved video predictions when compared to the same model without stochasticity, and to other stochastic video prediction methods.
Ranked #5 on
Video Prediction
on KTH
Simple and Effective VAE Training with Calibrated Decoders
We perform the first comprehensive comparative analysis of calibrated decoder and provide recommendations for simple and effective VAE training.
Reinforcement Learning with Videos: Combining Offline Observations with Interaction
In this paper, we consider the question: can we perform reinforcement learning directly on experience collected by humans?
Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trials
To our knowledge, PTR is the first RL method that succeeds at learning new tasks in a new domain on a real WidowX robot with as few as 10 task demonstrations, by effectively leveraging an existing dataset of diverse multi-task robot data collected in a variety of toy kitchens.
Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inference
The key idea is to apply a variant of contrastive learning to time series data.
Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior
Inferring intent from observed behavior has been studied extensively within the frameworks of Bayesian inverse planning and inverse reinforcement learning.
Predictable MDP Abstraction for Unsupervised Model-Based RL
A key component of model-based reinforcement learning (RL) is a dynamics model that predicts the outcomes of actions.
Model-based Reinforcement Learning
Reinforcement Learning (RL)
Automatically Composing Representation Transformations as a Means for Generalization
A generally intelligent learner should generalize to more complex tasks than it has previously encountered, but the two common paradigms in machine learning -- either training a separate learner per task or training a single learner for all tasks -- both have difficulty with such generalization because they do not leverage the compositional structure of the task distribution.
Model-Based Meta-Reinforcement Learning for Flight with Suspended Payloads
Our experiments demonstrate that our online adaptation approach outperforms non-adaptive methods on a series of challenging suspended payload transportation tasks.
First Contact: Unsupervised Human-Machine Co-Adaptation via Mutual Information Maximization
How can we train an assistive human-machine interface (e. g., an electromyography-based limb prosthesis) to translate a user's raw command signals into the actions of a robot or computer when there is no prior mapping, we cannot ask the user for supervision in the form of action labels or reward feedback, and we do not have prior knowledge of the tasks the user is trying to accomplish?
Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets
Robot learning holds the promise of learning policies that generalize broadly.
Mismatched No More: Joint Model-Policy Optimization for Model-Based RL
Many model-based reinforcement learning (RL) methods follow a similar template: fit a model to previously observed data, and then use data from that model for RL or planning.
Model-based Reinforcement Learning
Reinforcement Learning (RL)
The Information Geometry of Unsupervised Reinforcement Learning
In this work, we show that unsupervised skill discovery algorithms based on mutual information maximization do not learn skills that are optimal for every possible reward function.
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning
We present relay policy learning, a method for imitation and reinforcement learning that can solve multi-stage, long-horizon robotic tasks.
Deep Object-Centric Representations for Generalizable Robot Learning
We devise an object-level attentional mechanism that can be used to determine relevant objects from a few trajectories or demonstrations, and then immediately incorporate those objects into a learned policy.
Continual Learning of Control Primitives: Skill Discovery via Reset-Games
Reinforcement learning has the potential to automate the acquisition of behavior in complex settings, but in order for it to be successfully deployed, a number of practical challenges must be addressed.
Model-Based Visual Planning with Self-Supervised Functional Distances
In our experiments, we find that our method can successfully learn models that perform a variety of tasks at test-time, moving objects amid distractors with a simulated robotic arm and even learning to open and close a drawer using a real-world robot.
Offline Reinforcement Learning for Visual Navigation
Reinforcement learning can enable robots to navigate to distant goals while optimizing user-specified reward functions, including preferences for following lanes, staying on paved paths, or avoiding freshly mowed grass.
Accelerating Exploration with Unlabeled Prior Data
Learning to solve tasks from a sparse reward signal is a major challenge for standard reinforcement learning (RL) algorithms.
Diagnosing Bottlenecks in Deep Q-learning Algorithms
Q-learning methods represent a commonly used class of algorithms in reinforcement learning: they are generally efficient and simple, and can be combined readily with function approximators for deep reinforcement learning (RL).
The Mirage of Action-Dependent Baselines in Reinforcement Learning
Policy gradient methods are a widely used class of model-free reinforcement learning algorithms where a state-dependent baseline is used to reduce gradient estimator variance.
Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience
IRL can provide a generalizable and compact representation for apprenticeship learning, and enable accurately inferring the preferences of a human in order to assist them.
Contextual Imagined Goals for Self-Supervised Robotic Learning
When the robot's environment and available objects vary, as they do in most open-world settings, the robot must propose to itself only those goals that it can accomplish in its present setting with the objects that are at hand.
Reward-Conditioned Policies
By then conditioning the policy on the numerical value of the reward, we can obtain a policy that generalizes to larger returns.
Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning
In this work, we propose an autonomous method for safe and efficient reinforcement learning that simultaneously learns a forward and reset policy, with the reset policy resetting the environment for a subsequent attempt.
Robustness via Retrying: Closed-Loop Robotic Manipulation with Self-Supervised Learning
We demonstrate that this idea can be combined with a video-prediction based controller to enable complex behaviors to be learned from scratch using only raw visual inputs, including grasping, repositioning objects, and non-prehensile manipulation.
Offline Reinforcement Learning with In-sample Q-Learning
The main insight in our work is that, instead of evaluating unseen actions from the latest policy, we can approximate the policy improvement step implicitly by treating the state value function as a random variable, with randomness determined by the action (while still integrating over the dynamics to avoid excessive optimism), and then taking a state conditional upper expectile of this random variable to estimate the value of the best actions in that state.
What Can I Do Here? Learning New Skills by Imagining Visual Affordances
In effect, prior data is used to learn what kinds of outcomes may be possible, such that when the robot encounters an unfamiliar setting, it can sample potential outcomes from its model, attempt to reach them, and thereby update both its skills and its outcome model.
Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias
Can we leverage offline RL to recover better policies from online interaction?
Deep Visual Foresight for Planning Robot Motion
A key challenge in scaling up robot learning to many skills and environments is removing the need for human supervision, so that robots can collect their own data and improve their own performance without being limited by the cost of requesting human feedback.
Model-based Reinforcement Learning
Model Predictive Control
+2
Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings
Reinforcement learning (RL) in real-world safety-critical target settings like urban driving is hazardous, imperiling the RL agent, other agents, and the environment.
Offline Meta-Reinforcement Learning with Online Self-Supervision
If we can meta-train on offline data, then we can reuse the same static dataset, labeled once with rewards for different tasks, to meta-train policies that adapt to a variety of new tasks at meta-test time.
Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Data
Robotic systems that rely primarily on self-supervised learning have the potential to decrease the amount of human annotation and engineering effort required to learn control strategies.
Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping
We extensively evaluate our approaches with a total of more than 25, 000 physical test grasps, studying a range of simulation conditions and domain adaptation methods, including a novel extension of pixel-level domain adaptation that we term the GraspGAN.
Understanding the World Through Action
The recent history of machine learning research has taught us that machine learning methods can be most effective when they are provided with very large, high-capacity models, and trained on very large and diverse datasets.
MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies
In this work, we propose multiplicative compositional policies (MCP), a method for learning reusable motor skills that can be composed to produce a range of complex behaviors.
Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
In this paper, we show that hindsight relabeling is inverse RL, an observation that suggests that we can use inverse RL in tandem for RL algorithms to efficiently solve many tasks.
Pragmatic Image Compression for Human-in-the-Loop Decision-Making
Standard lossy image compression algorithms aim to preserve an image's appearance, while minimizing the number of bits needed to transmit it.
A Workflow for Offline Model-Free Robotic Reinforcement Learning
To this end, we devise a set of metrics and conditions that can be tracked over the course of offline training, and can inform the practitioner about how the algorithm and model architecture should be adjusted to improve final performance.
EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
Deep reinforcement learning algorithms have been shown to learn complex tasks using highly general policy classes.
Learning Powerful Policies by Using Consistent Dynamics Model
There is enough evidence that humans build a model of the environment, not only by observing the environment but also by interacting with the environment.
