Search Results for author:
Found 237 papers, 109 papers with code
Learning a Prior over Intent via Meta-Inverse Reinforcement Learning
A significant challenge for the practical application of reinforcement learning in the real world is the need to specify an oracle reward function that correctly defines a task.
Unsupervised Meta-Learning for Reinforcement Learning
In the context of reinforcement learning, meta-learning algorithms acquire reinforcement learning procedures to solve new problems more efficiently by utilizing experience from prior tasks.
Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
Learning to learn is a powerful paradigm for enabling models to learn from data more effectively and efficiently.
Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
Meta-learning allows an intelligent agent to leverage prior learning episodes as a basis for quickly improving performance on a novel task.
Adapting Deep Visuomotor Representations with Weak Pairwise Constraints
We propose a novel, more powerful combination of both distribution and pairwise image alignment, and remove the requirement for expensive annotation by using weakly aligned pairs of images in the source and target domains.
Generalizing Skills with Semi-Supervised Reinforcement Learning
We evaluate our method on challenging tasks that require control directly from images, and show that our approach can improve the generalization of a learned deep neural network policy by using experience for which no reward function is available.
Reset-Free Guided Policy Search: Efficient Deep Reinforcement Learning with Stochastic Initial States
Autonomous learning of robotic skills can allow general-purpose robots to learn wide behavioral repertoires without requiring extensive manual engineering.
End-to-End Training of Deep Visuomotor Policies
Policy search methods can allow robots to learn control policies for a wide range of tasks, but practical applications of policy search often require hand-engineered components for perception, state estimation, and low-level control.
Learning Deep Neural Network Policies with Continuous Memory States
We evaluate our method on tasks involving continuous control in manipulation and navigation settings, and show that our method can learn complex policies that successfully complete a range of tasks that require memory.
Learning Compact Convolutional Neural Networks with Nested Dropout
Recently, nested dropout was proposed as a method for ordering representation units in autoencoders by their information content, without diminishing reconstruction cost.
Time Reversal as Self-Supervision
We test our method on the domain of assembly, specifically the mating of tetris-style block pairs.
Few-Shot Goal Inference for Visuomotor Learning and Planning
To that end, we formulate the few-shot objective learning problem, where the goal is to learn a task objective from only a few example images of successful end states for that task.
Unsupervised Learning via Meta-Learning
A central goal of unsupervised learning is to acquire representations from unlabeled data or experience that can be used for more effective learning of downstream tasks from modest amounts of labeled data.
One-Shot Hierarchical Imitation Learning of Compound Visuomotor Tasks
We consider the problem of learning multi-stage vision-based tasks on a real robot from a single video of a human performing the task, while leveraging demonstration data of subtasks with other objects.
Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL
The goal in this paper is to develop a method for continual online learning from an incoming stream of data, using deep neural network models.
Learning to Reinforcement Learn by Imitation
Meta-reinforcement learning aims to learn fast reinforcement learning (RL) procedures that can be applied to new tasks or environments.
Few-Shot Intent Inference via Meta-Inverse Reinforcement Learning
A significant challenge for the practical application of reinforcement learning toreal world problems is the need to specify an oracle reward function that correctly defines a task.
Reasoning About Physical Interactions with Object-Centric Models
Object-based factorizations provide a useful level of abstraction for interacting with the world.
Self-Supervised Learning of Object Motion Through Adversarial Video Prediction
In this paper, we study the problem of multi-step video prediction, where the goal is to predict a sequence of future frames conditioned on a short context.
Reasoning About Physical Interactions with Object-Oriented Prediction and Planning
Object-based factorizations provide a useful level of abstraction for interacting with the world.
Online Meta-Learning
Meta-learning views this problem as learning a prior over model parameters that is amenable for fast adaptation on a new task, but typically assumes the set of tasks are available together as a batch.
Manipulation by Feel: Touch-Based Control with Deep Predictive Models
Touch sensing is widely acknowledged to be important for dexterous robotic manipulation, but exploiting tactile sensing for continuous, non-prehensile manipulation is challenging.
Guided Meta-Policy Search
Reinforcement learning (RL) algorithms have demonstrated promising results on complex tasks, yet often require impractical numbers of samples since they learn from scratch.
Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresight
Machine learning techniques have enabled robots to learn narrow, yet complex tasks and also perform broad, yet simple skills with a wide variety of objects.
Watch, Try, Learn: Meta-Learning from Demonstrations and Reward
Imitation learning allows agents to learn complex behaviors from demonstrations.
Learning to Interactively Learn and Assist
Most importantly, we find that our approach produces an agent that is capable of learning interactively from a human user, without a set of explicit demonstrations or a reward function, and achieving significantly better performance cooperatively with a human than a human performing the task alone.
Training an Interactive Helper
In this paper, we investigate if, and how, a "helper" agent can be trained to interactively adapt their behavior to maximize the reward of another agent, whom we call the "prime" agent, without observing their reward or receiving explicit demonstrations.
RoboNet: Large-Scale Multi-Robot Learning
This leads to a frequent tension in robotic learning: how can we learn generalizable robotic controllers without having to collect impractically large amounts of data for each separate experiment?
Unsupervised Curricula for Visual Meta-Reinforcement Learning
In experiments on vision-based navigation and manipulation domains, we show that the algorithm allows for unsupervised meta-learning that transfers to downstream tasks specified by hand-crafted reward functions and serves as pre-training for more efficient supervised meta-learning of test task distributions.
Learning Predictive Models From Observation and Interaction
Learning predictive models from interaction with the world allows an agent, such as a robot, to learn about how the world works, and then use this learned model to plan coordinated sequences of actions to bring about desired outcomes.
Rapidly Adaptable Legged Robots via Evolutionary Meta-Learning
Learning adaptable policies is crucial for robots to operate autonomously in our complex and quickly changing world.
Scalable Multi-Task Imitation Learning with Autonomous Improvement
In this work, we target this challenge, aiming to build an imitation learning system that can continuously improve through autonomous data collection, while simultaneously avoiding the explicit use of reinforcement learning, to maintain the stability, simplicity, and scalability of supervised imitation.
Weakly-Supervised Reinforcement Learning for Controllable Behavior
Reinforcement learning (RL) is a powerful framework for learning to take actions to solve tasks.
Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning
One of the great promises of robot learning systems is that they will be able to learn from their mistakes and continuously adapt to ever-changing environments.
Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling
Our method is based on a simple insight: we recognize that dynamics models can be adapted efficiently and consistently with off-policy data, more easily than policies and value functions.
Deep Reinforcement Learning amidst Lifelong Non-Stationarity
As humans, our goals and our environment are persistently changing throughout our lifetime based on our experiences, actions, and internal and external drives.
Goal-Aware Prediction: Learning to Model What Matters
In this paper, we propose to direct prediction towards task relevant information, enabling the model to be aware of the current task and encouraging it to only model relevant quantities of the state space, resulting in a learning objective that more closely matches the downstream task.
Watch, Try, Learn: Meta-Learning from Demonstrations and Rewards
Imitation learning allows agents to learn complex behaviors from demonstrations.
Model Based Reinforcement Learning for Atari
We describe Simulated Policy Learning (SimPLe), a complete model-based deep RL algorithm based on video prediction models and present a comparison of several model architectures, including a novel architecture that yields the best results in our setting.
Variable-Shot Adaptation for Incremental Meta-Learning
Few-shot meta-learning methods consider the problem of learning new tasks from a small, fixed number of examples, by meta-learning across static data from a set of previous tasks.
On Trade-offs of Image Prediction in Visual Model-Based Reinforcement Learning
In this paper, we study a number of design decisions for the predictive model in visual MBRL algorithms, focusing specifically on methods that use a predictive model for planning.
Model-based Reinforcement Learning
reinforcement-learning
+1
Information Transfer in Multi-Task Learning
Multi-task learning can leverage information learned by one task to benefit the training of other tasks.
One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
While reinforcement learning algorithms can learn effective policies for complex tasks, these policies are often brittle to even minor task variations, especially when variations are not explicitly provided during training.
Learning to be Safe: Deep RL with a Safety Critic
Safety is an essential component for deploying reinforcement learning (RL) algorithms in real-world scenarios, and is critical during the learning process itself.
Measuring and Harnessing Transference in Multi-Task Learning
Multi-task learning can leverage information learned by one task to benefit the training of other tasks.
Continual Learning of Control Primitives : Skill Discovery via Reset-Games
First, in real world settings, when an agent attempts a tasks and fails, the environment must somehow "reset" so that the agent can attempt the task again.
Learning Latent Representations to Influence Multi-Agent Interaction
We propose a reinforcement learning-based framework for learning latent representations of an agent's policy, where the ego agent identifies the relationship between its behavior and the other agent's future strategy.
Variable-Shot Adaptation for Online Meta-Learning
Few-shot meta-learning methods consider the problem of learning new tasks from a small, fixed number of examples, by meta-learning across static data from a set of previous tasks.
How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
Learning to perceive and move in the real world presents numerous challenges, some of which are easier to address than others, and some of which are often not considered in RL research that focuses only on simulated domains.
Bayesian Meta-Learning for Few-Shot Policy Adaptation Across Robotic Platforms
Our results show that the proposed method can successfully adapt a trained policy to different robotic platforms with novel physical parameters and the superiority of our meta-learning algorithm compared to state-of-the-art methods for the introduced few-shot policy adaptation problem.
Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction
Our key insight is that greedy and modular optimization of hierarchical autoencoders can simultaneously address both the memory constraints and the optimization challenges of large-scale video prediction.
Ranked #1 on
Video Prediction
on Cityscapes 128x128
Discriminator Augmented Model-Based Reinforcement Learning
By planning through a learned dynamics model, model-based reinforcement learning (MBRL) offers the prospect of good performance with little environment interaction.
Model-based Reinforcement Learning
reinforcement-learning
+1
Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos
We find that by leveraging diverse human datasets, this reward function (a) can generalize zero shot to unseen environments, (b) generalize zero shot to unseen tasks, and (c) can be combined with visual model predictive control to solve robotic manipulation tasks on a real WidowX200 robot in an unseen environment from a single human demo.
MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
In this paper, we study how a large-scale collective robotic learning system can acquire a repertoire of behaviors simultaneously, sharing exploration, experience, and representations across tasks.
Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
We consider the problem of learning useful robotic skills from previously collected offline data without access to manually specified rewards or additional online exploration, a setting that is becoming increasingly important for scaling robot learning by reusing past robotic data.
Visual Adversarial Imitation Learning using Variational Models
We consider a setting where an agent is provided a fixed dataset of visual demonstrations illustrating how to perform a task, and must learn to solve the task using the provided demonstrations and unsupervised environment interactions.
Differentiable Annealed Importance Sampling and the Perils of Gradient Noise
To this end, we propose Differentiable AIS (DAIS), a variant of AIS which ensures differentiability by abandoning the Metropolis-Hastings corrections.
Autonomous Reinforcement Learning via Subgoal Curricula
Reinforcement learning (RL) promises to enable autonomous acquisition of complex behaviors for diverse agents.
Bayesian Embeddings for Few-Shot Open World Recognition
As autonomous decision-making agents move from narrow operating environments to unstructured worlds, learning systems must move from a closed-world formulation to an open-world and few-shot setting in which agents continuously learn new classes from small amounts of information.
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation
However, goal images also have a number of drawbacks: they are inconvenient for humans to provide, they can over-specify the desired behavior leading to a sparse reward signal, or under-specify task information in the case of non-goal reaching tasks.
Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
We argue that a natural use case of offline RL is in settings where we can pool large amounts of data collected in various scenarios for solving different tasks, and utilize all of this data to learn behaviors for all the tasks more effectively rather than training each one in isolation.
Lifelong Robotic Reinforcement Learning by Retaining Experiences
Multi-task learning ideally allows robots to acquire a diverse repertoire of useful skills.
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks
In this paper, we study the problem of learning a repertoire of low-level skills from raw images that can be sequenced to complete long-horizon visuomotor tasks.
Model-based Reinforcement Learning
reinforcement-learning
+1
Data Sharing without Rewards in Multi-Task Offline Reinforcement Learning
However, these benefits come at a cost -- for data to be shared between tasks, each transition must be annotated with reward labels corresponding to other tasks.
Test Time Robustification of Deep Models via Adaptation and Augmentation
We study the problem of test time robustification, i. e., using the test input to improve model robustness.
FitVid: High-Capacity Pixel-Level Video Prediction
Furthermore, such an agent can internally represent the complex dynamics of the real-world and therefore can acquire a representation useful for a variety of visual perception tasks.
Goal-Conditioned Video Prediction
Prior work on video generation largely focuses on prediction models that only observe frames from the beginning of the video.
SMiRL: Surprise Minimizing RL in Entropic Environments
All living organisms struggle against the forces of nature to carve out niches where they can maintain relative stasis.
Unsupervised Pre-training
Unsupervised Reinforcement Learning
Hope For The Best But Prepare For The Worst: Cautious Adaptation In RL Agents
We study the problem of safe adaptation: given a model trained on a variety of past experiences for some task, can this model learn to perform that task in a new situation while avoiding catastrophic failure?
Mint: Matrix-Interleaving for Multi-Task Learning
Deep learning enables training of large and flexible function approximators from scratch at the cost of large amounts of data.
Consistent Meta-Reinforcement Learning via Model Identification and Experience Relabeling
Reinforcement learning algorithms can acquire policies for complex tasks automatically, however the number of samples required to learn a diverse set of skills can be prohibitively large.
Adaptive Risk Minimization: A Meta-Learning Approach for Tackling Group Shift
A fundamental assumption of most machine learning algorithms is that the training and test data are drawn from the same underlying distribution.
FEW-SHOTLEARNING WITH WEAK SUPERVISION
Few-shot meta-learning methods aim to learn the common structure shared across a set of tasks to facilitate learning new tasks with small amounts of data.
Variational Model-Based Imitation Learning in High-Dimensional Observation Spaces
We consider the problem setting of imitation learning where the agent is provided a fixed dataset of demonstrations.
Explore then Execute: Adapting without Rewards via Factorized Meta-Reinforcement Learning
In principle, meta-reinforcement learning approaches can exploit this shared structure, but in practice, they fail to adapt to new environments when adaptation requires targeted exploration (e. g., exploring the cabinets to find ingredients in a new kitchen).
Efficient Adaptation for End-to-End Vision-Based Robotic Manipulation
One of the great promises of robot learning systems is that they will be able to learn from their mistakes and continuously adapt to ever-changing environments, but most robot learning systems today are deployed as fixed policies which do not adapt after deployment.
Intrinsic Control of Variational Beliefs in Dynamic Partially-Observed Visual Environments
We study this question in dynamic partially-observed environments, and argue that a compact and general learning objective is to minimize the entropy of the agent's state visitation estimated using a latent state-space model.
Information is Power: Intrinsic Control via Information Capture
We study this question in dynamic partially-observed environments, and argue that a compact and general learning objective is to minimize the entropy of the agent's state visitation estimated using a latent state-space model.
MESA: Offline Meta-RL for Safe Adaptation and Fault Tolerance
Safe exploration is critical for using reinforcement learning (RL) in risk-sensitive environments.
Noether Networks: Meta-Learning Useful Conserved Quantities
Progress in machine learning (ML) stems from a combination of data availability, computational resources, and an appropriate encoding of inductive biases.
CoMPS: Continual Meta Policy Search
Beyond simply transferring past experience to new tasks, our goal is to devise continual reinforcement learning algorithms that learn to learn, using their experience on previous tasks to learn new tasks more quickly.
Fully Online Meta-Learning Without Task Boundaries
In this paper, we study how meta-learning can be applied to tackle online problems of this nature, simultaneously adapting to changing tasks and input distributions and meta-training the model in order to adapt more quickly in the future.
How to Leverage Unlabeled Data in Offline Reinforcement Learning
One natural solution is to learn a reward function from the labeled data and use it to label the unlabeled data.
BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning
In this paper, we study the problem of enabling a vision-based robotic manipulation system to generalize to novel tasks, a long-standing challenge in robot learning.
Robust Policy Learning over Multiple Uncertainty Sets
Reinforcement learning (RL) agents need to be robust to variations in safety-critical environments.
Policy Architectures for Compositional Generalization in Control
Many tasks in control, robotics, and planning can be specified using desired goal configurations for various entities in the environment.
Vision-Based Manipulators Need to Also See from Their Hands
We study how the choice of visual perspective affects learning and generalization in the context of physical manipulation from raw sensor observations.
Training and Evaluation of Deep Policies using Reinforcement Learning and Generative Models
We present a data-efficient framework for solving sequential decision-making problems which exploits the combination of reinforcement learning (RL) and latent variable generative models.
Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learning
In a set of simulated tasks, we find that our system benefits from using audio, and that by using online interventions we are able to improve the success rate of offline imitation learning by ~20%.
Offline Reinforcement Learning at Multiple Frequencies
In this paper, we focus on one particular aspect of heterogeneity: learning from offline data collected at different control frequencies.
Learning to Reason With Relational Abstractions
Large language models have recently shown promising progress in mathematical reasoning when fine-tuned with human-generated sequences walking through a sequence of solution steps.
Knowledge-Driven New Drug Recommendation
However, newly approved drugs do not have much historical prescription data and cannot leverage existing drug recommendation methods.
You Only Live Once: Single-Life Reinforcement Learning
We formalize this problem setting, which we call single-life reinforcement learning (SLRL), where an agent must complete a task within a single episode without interventions, utilizing its prior experience while contending with some form of novelty.
Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inference
While large pre-trained language models are powerful, their predictions often lack logical consistency across test inputs.
A Survey of Meta-Reinforcement Learning
Meta-RL is most commonly studied in a problem setting where, given a distribution of tasks, the goal is to learn a policy that is capable of adapting to any new task from the task distribution with as little data as possible.
NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesis
Expert demonstrations are a rich source of supervision for training visual robotic manipulation policies, but imitation learning methods often require either a large number of demonstrations or expensive online expert supervision to learn reactive closed-loop behaviors.
Improving Domain Generalization with Domain Relations
Distribution shift presents a significant challenge in machine learning, where models often underperform during the test stage when faced with a different distribution than the one they were trained on.
Project and Probe: Sample-Efficient Domain Adaptation by Interpolating Orthogonal Features
Transfer learning with a small amount of target data is an effective and common approach to adapting a pre-trained model to distribution shifts.
Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning
In this work, we propose MEDAL++, a novel design for self-improving robotic systems: given a small set of expert demonstrations at the start, the robot autonomously practices the task by learning to both do and undo the task, simultaneously inferring the reward function from the demonstrations.
Open-World Object Manipulation using Pre-trained Vision-Language Models
This brings up a notably difficult challenge for robots: while robot learning approaches allow robots to learn many different behaviors from first-hand experience, it is impractical for robots to have first-hand experiences that span all of this semantic information.
Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasets
Concretely, we propose a simple approach that uses a small amount of downstream expert data to selectively query relevant behaviors from an offline, unlabeled dataset (including many sub-optimal behaviors).
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
Fine manipulation tasks, such as threading cable ties or slotting a battery, are notoriously difficult for robots because they require precision, careful coordination of contact forces, and closed-loop visual feedback.
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback
A trustworthy real-world prediction system should produce well-calibrated confidence scores; that is, its confidence in an answer should be indicative of the likelihood that the answer is correct, enabling deferral to an expert in cases of low-confidence predictions.
Simple Embodied Language Learning as a Byproduct of Meta-Reinforcement Learning
Specifically, we design an office navigation environment, where the agent's goal is to find a particular office, and office locations differ in different buildings (i. e., tasks).
Confidence-Based Model Selection: When to Take Shortcuts for Subpopulation Shifts
Effective machine learning models learn both robust features that directly determine the outcome of interest (e. g., an object with wheels is more likely to be a car), and shortcut features (e. g., an object on a road is more likely to be a car).
Polybot: Training One Policy Across Robots While Embracing Variability
Reusing large datasets is crucial to scale vision-based robotic manipulators to everyday scenarios due to the high cost of collecting robotic datasets.
Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation
Towards an answer to this question, we study imitation learning policies in simulation and on a real robot language-conditioned manipulation task to quantify the difficulty of generalization to different (sets of) factors.
Giving Robots a Hand: Learning Generalizable Manipulation with Eye-in-Hand Human Video Demonstrations
Eye-in-hand cameras have shown promise in enabling greater sample efficiency and generalization in vision-based robotic manipulation.
Waypoint-Based Imitation Learning for Robotic Manipulation
AWE can be combined with any BC algorithm, and we find that AWE can increase the success rate of state-of-the-art algorithms by up to 25% in simulation and by 4-28% on real-world bimanual manipulation tasks, reducing the decision making horizon by up to a factor of 10.
Robot Parkour Learning
Parkour is a grand challenge for legged locomotion that requires robots to overcome various obstacles rapidly in complex environments.
Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions
In this work, we present a scalable reinforcement learning method for training multi-task policies from large offline datasets that can leverage both human demonstrations and autonomously collected data.
Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning
We aim to enable this paradigm in robotic reinforcement learning, allowing a robot to learn a new task with little human effort by leveraging data and models from the Internet.
Adapt On-the-Go: Behavior Modulation for Single-Life Robot Deployment
We provide theoretical analysis of our selection mechanism and demonstrate that ROAM enables a robot to adapt rapidly to changes in dynamics both in simulation and on a real Go1 quadruped, even successfully moving forward with roller skates on its feet.
RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches
Generalization remains one of the most important desiderata for robust robot learning systems.
Fine-tuning Language Models for Factuality
The fluency and creativity of large pre-trained language models (LLMs) have led to their widespread use, sometimes even as a replacement for traditional search engines.
What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?
Inspired by the success of transfer learning in computer vision, roboticists have investigated visual pre-training as a means to improve the learning efficiency and generalization ability of policies learned from pixels.
Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation
We first present Mobile ALOHA, a low-cost and whole-body teleoperation system for data collection.
MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning
We study the problem of offline pre-training and online fine-tuning for reinforcement learning from high-dimensional observations in the context of realistic robot tasks.
AutoFT: Learning an Objective for Robust Fine-Tuning
Given a task, AutoFT searches for a fine-tuning procedure that enhances out-of-distribution (OOD) generalization.
AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents
We experimentally show that such "in-the-wild" data collected by AutoRT is significantly more diverse, and that AutoRT's use of LLMs allows for instruction following data collection robots that can align to human preferences.
SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning
We posit that a significant challenge to widespread adoption of robotic RL, as well as further development of robotic RL methods, is the comparative inaccessibility of such methods.
Clarify: Improving Model Robustness With Natural Language Corrections
Additionally, we use Clarify to find and rectify 31 novel hard subpopulations in the ImageNet dataset, improving minority-split accuracy from 21. 1% to 28. 7%.
PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
In each iteration, the image is annotated with a visual representation of proposals that the VLM can refer to (e. g., candidate robot actions, localizations, or trajectories).
Efficient Data Collection for Robotic Manipulation via Compositional Generalization
Recent works on large-scale robotic data collection typically vary a wide range of environmental factors during data collection, such as object types and table textures.
Yell At Your Robot: Improving On-the-Fly from Language Corrections
In this paper, we make the following observation: high-level policies that index into sufficiently rich and expressive low-level language-conditioned skills can be readily supervised with human feedback in the form of language corrections.
Disentangling Length from Quality in Direct Preference Optimization
A number of approaches have been developed to control those biases in the classical RLHF literature, but the problem remains relatively under-explored for Direct Alignment Algorithms such as Direct Preference Optimization (DPO).
Tripod: Three Complementary Inductive Biases for Disentangled Representation Learning
Inductive biases are crucial in disentangled representation learning for narrowing down an underspecified solution set.
From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function
Standard RLHF deploys reinforcement learning in a specific token-level MDP, while DPO is derived as a bandit problem in which the whole response of the model is treated as a single arm.
Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methods
In this paper, we explore deep reinforcement learning algorithms for vision-based robotic grasping.
Conservative Prediction via Data-Driven Confidence Minimization
Taking inspiration from this result, we present data-driven confidence minimization (DCM), which minimizes confidence on an uncertainty dataset containing examples that the model is likely to misclassify at test time.
Self-Guided Masked Autoencoders for Domain-Agnostic Self-Supervised Learning
Self-supervised learning excels in learning representations from large amounts of unlabeled data, demonstrating success across multiple data modalities.
A State-Distribution Matching Approach to Non-Episodic Reinforcement Learning
Prior works have considered an alternating approach where a forward policy learns to solve the task and the backward policy learns to reset the environment, but what initial state distribution should the backward policy reset the agent to?
Multi-Domain Long-Tailed Learning by Augmenting Disentangled Representations
We study this multi-domain long-tailed learning problem and aim to produce a model that generalizes well across all classes and domains.
SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments
Every living organism struggles against disruptive environmental forces to carve out and maintain an orderly niche.
Bitrate-Constrained DRO: Beyond Worst Case Robustness To Unknown Group Shifts
Some robust training algorithms (e. g., Group DRO) specialize to group shifts and require group information on all training points.
Contrastive Example-Based Control
In this paper, we propose a method for offline, example-based control that learns an implicit model of multi-step transitions, rather than a reward function.
Deep Spatial Autoencoders for Visuomotor Learning
Our method uses a deep spatial autoencoder to acquire a set of feature points that describe the environment for the current task, such as the positions of objects, and then learns a motion skill with these feature points using an efficient reinforcement learning method based on local linear models.
When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learning
A long-term goal of reinforcement learning is to design agents that can autonomously interact and learn in the world.
Giving Feedback on Interactive Student Programs with Meta-Exploration
However, teaching and giving feedback on such software is time-consuming -- standard approaches require instructors to manually grade student-implemented interactive programs.
Unsupervised Visuomotor Control through Distributional Planning Networks
While reinforcement learning (RL) has the potential to enable robots to autonomously acquire a wide range of skills, in practice, RL usually requires manual, per-task engineering of reward functions, especially in real world settings where aspects of the environment needed to compute progress are not directly accessible.
Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation
Video prediction models combined with planning algorithms have shown promise in enabling robots to learn to perform many vision-based tasks through only self-supervision, reaching novel goals in cluttered scenes with unseen objects.
A Workflow for Offline Model-Free Robotic Reinforcement Learning
To this end, we devise a set of metrics and conditions that can be tracked over the course of offline training, and can inform the practitioner about how the algorithm and model architecture should be adjusted to improve final performance.
Latent-Variable Advantage-Weighted Policy Optimization for Offline RL
Offline reinforcement learning methods hold the promise of learning policies from pre-collected datasets without the need to query the environment for new transitions.
Batch Exploration with Examples for Scalable Robotic Reinforcement Learning
Concretely, we propose an exploration technique, Batch Exploration with Examples (BEE), that explores relevant regions of the state-space, guided by a modest number of human provided images of important states.
Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Models
A growing ecosystem of large, open-source foundation models has reduced the labeled data and technical expertise necessary to apply machine learning to many new problems.
Deep Visual Foresight for Planning Robot Motion
A key challenge in scaling up robot learning to many skills and environments is removing the need for human supervision, so that robots can collect their own data and improve their own performance without being limited by the cost of requesting human feedback.
Model-based Reinforcement Learning
Model Predictive Control
+2
Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings
Reinforcement learning (RL) in real-world safety-critical target settings like urban driving is hazardous, imperiling the RL agent, other agents, and the environment.
A Control-Centric Benchmark for Video Prediction
Video is a promising source of knowledge for embodied agents to learn models of the world's dynamics.
ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback
High-quality computer science education is limited by the difficulty of providing instructor feedback to students at scale.
Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias
Can we leverage offline RL to recover better policies from online interaction?
Robustness via Retrying: Closed-Loop Robotic Manipulation with Self-Supervised Learning
We demonstrate that this idea can be combined with a video-prediction based controller to enable complex behaviors to be learned from scratch using only raw visual inputs, including grasping, repositioning objects, and non-prehensile manipulation.
Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data
Our main finding is that, in general, approaches that use on-policy sampling or attempt to push down the likelihood on certain responses (i. e., employ a "negative gradient") outperform offline and maximum likelihood objectives.
Meta-learning with an Adaptive Task Scheduler
In ATS, for the first time, we design a neural scheduler to decide which meta-training tasks to use next by predicting the probability being sampled for each candidate task, and train the scheduler to optimize the generalization capacity of the meta-model to unseen tasks.
Continual Learning of Control Primitives: Skill Discovery via Reset-Games
Reinforcement learning has the potential to automate the acquisition of behavior in complex settings, but in order for it to be successfully deployed, a number of practical challenges must be addressed.
Model-Based Visual Planning with Self-Supervised Functional Distances
In our experiments, we find that our method can successfully learn models that perform a variety of tasks at test-time, moving objects amid distractors with a simulated robotic arm and even learning to open and close a drawer using a real-world robot.
An Emulator for Fine-Tuning Large Language Models using Small Language Models
To aid in doing so, we introduce a novel technique for decoupling the knowledge and skills gained in these two stages, enabling a direct answer to the question, "What would happen if we combined the knowledge learned by a large model during pre-training with the knowledge learned by a small model during fine-tuning (or vice versa)?"
Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations
As ERM models can be good spurious attribute predictors, CNC works by (1) using a trained ERM model's outputs to identify samples with the same class but dissimilar spurious features, and (2) training a robust model with contrastive learning to learn similar representations for same-class samples.
Meta-Learning Online Adaptation of Language Models
We meta-train a small, autoregressive model to reweight the language modeling loss for each token during online fine-tuning, with the objective of maximizing the out-of-date base question-answering model's ability to answer questions about a document after a single weighted gradient step.
Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets
Robot learning holds the promise of learning policies that generalize broadly.
Learning Options via Compression
Identifying statistical regularities in solutions to some tasks in multi-task reinforcement learning can accelerate the learning of new tasks.
PIGEON: Predicting Image Geolocations
We train two models for evaluations on street-level data and general-purpose image geolocalization; the first model, PIGEON, is trained on data from the game of Geoguessr and is capable of placing over 40% of its guesses within 25 kilometers of the target location globally.
Ranked #1 on
Photo geolocation estimation
on YFCC4k
Disentanglement via Latent Quantization
In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space.
RLVF: Learning from Verbal Feedback without Overgeneralization
The diversity of contexts in which large language models (LLMs) are deployed requires the ability to modify or customize default model behaviors to incorporate nuanced requirements and preferences.
Do Deep Networks Transfer Invariances Across Classes?
Based on this analysis, we show how a generative approach for learning the nuisance transformations can help transfer invariances across classes and improve performance on a set of imbalanced image classification benchmarks.
Ranked #21 on
Long-tail Learning
on CIFAR-10-LT (ρ=100)
Surgical Fine-Tuning Improves Adaptation to Distribution Shifts
A common approach to transfer learning under distribution shift is to fine-tune the last few layers of a pre-trained model, preserving learned features while also adapting to the new task.
Stochastic Variational Video Prediction
We find that our proposed method produces substantially improved video predictions when compared to the same model without stochasticity, and to other stochastic video prediction methods.
Ranked #5 on
Video Prediction
on KTH
Reinforcement Learning with Videos: Combining Offline Observations with Interaction
In this paper, we consider the question: can we perform reinforcement learning directly on experience collected by humans?
Offline Reinforcement Learning from Images with Latent Space Models
In this work, we build on recent advances in model-based algorithms for offline RL, and extend them to high-dimensional visual observation spaces.
Meta-Learning with Fewer Tasks through Task Interpolation
Meta-learning enables algorithms to quickly learn a newly encountered task with just a few labeled examples by transferring previously learned knowledge.
Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trials
To our knowledge, PTR is the first RL method that succeeds at learning new tasks in a new domain on a real WidowX robot with as few as 10 task demonstrations, by effectively leveraging an existing dataset of diverse multi-task robot data collected in a variety of toy kitchens.
Continuous Meta-Learning without Tasks
In this work, we enable the application of generic meta-learning algorithms to settings where this task segmentation is unavailable, such as continual online learning with a time-varying task.
Universal Planning Networks
We find that the representations learned are not only effective for goal-directed visual imitation via gradient-based trajectory optimization, but can also provide a metric for specifying goals using images.
Active One-shot Learning
Recent advances in one-shot learning have produced models that can learn from a handful of labeled examples, for passive classification and regression tasks.
Universal Planning Networks: Learning Generalizable Representations for Visuomotor Control
A key challenge in complex visuomotor control is learning abstract representations that are effective for specifying goals, planning, and generalization.
Train Offline, Test Online: A Real Robot Learning Benchmark
Three challenges limit the progress of robot learning research: robots are expensive (few labs can participate), everyone uses different robots (findings do not generalize across labs), and we lack internet-scale robotics data.
Autonomous Reinforcement Learning: Formalism and Benchmarking
In this paper, we aim to address this discrepancy by laying out a framework for Autonomous Reinforcement Learning (ARL): reinforcement learning where the agent not only learns through its own experience, but also contends with lack of human supervision to reset between trials.
Aligning Modalities in Vision Large Language Models via Preference Fine-tuning
This procedure is not perfect and can cause the model to hallucinate - provide answers that do not accurately reflect the image, even when the core LLM is highly factual and the vision backbone has sufficiently complete representations.
Diversify and Disambiguate: Learning From Underspecified Data
Many datasets are underspecified: there exist multiple equally viable solutions to a given task.
Universal Neural Functionals
A challenging problem in many modern machine learning tasks is to process weight-space features, i. e., to transform or extract information from the weights and gradients of a neural network.
Offline Meta-Reinforcement Learning with Advantage Weighting
That is, in offline meta-RL, we meta-train on fixed, pre-collected data from several tasks in order to adapt to a new task with a very small amount (less than 5 trajectories) of data from the new task.
Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
In this work we propose a framework for visual prediction and planning that is able to overcome both of these limitations.
MEMO: Test Time Robustness via Adaptation and Augmentation
We study the problem of test time robustification, i. e., using the test input to improve model robustness.
Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones
Safety remains a central obstacle preventing widespread use of RL in the real world: learning new tasks in uncertain environments requires extensive exploration, but safety requires limiting exploration.
Memory-Based Model Editing at Scale
We find that only SERAC achieves high performance on all three problems, consistently outperforming existing approaches to model editing by a significant margin.
Entity Abstraction in Visual Model-Based Reinforcement Learning
This paper tests the hypothesis that modeling a scene in terms of entities and their local interactions, as opposed to modeling the scene globally, provides a significant benefit in generalizing to physical tasks in a combinatorial space the learner has not encountered before.
Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time
Temporal shifts -- distribution shifts arising from the passage of time -- often occur gradually and have the additional structure of timestamp metadata.
Extending the WILDS Benchmark for Unsupervised Adaptation
Unlabeled data can be a powerful point of leverage for mitigating these distribution shifts, as it is frequently much more available than labeled data and can often be obtained from distributions beyond the source distribution as well.
C-Mixup: Improving Generalization in Regression
In this paper, we propose a simple yet powerful algorithm, C-Mixup, to improve generalization on regression tasks.
MELD: Meta-Reinforcement Learning from Images via Latent State Models
Meta-reinforcement learning algorithms can enable autonomous agents, such as robots, to quickly acquire new behaviors by leveraging prior experience in a set of related training tasks.
Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
Our approach, calibrated Q-learning (Cal-QL), accomplishes this by learning a conservative value function initialization that underestimates the value of the learned policy from offline data, while also being calibrated, in the sense that the learned Q-values are at a reasonable scale.
FitVid: Overfitting in Pixel-Level Video Prediction
There is a growing body of evidence that underfitting on the training data is one of the primary causes for the low quality predictions.
Ranked #6 on
Video Generation
on BAIR Robot Pushing
Just Train Twice: Improving Group Robustness without Training Group Information
Standard training via empirical risk minimization (ERM) can produce models that achieve high accuracy on average but low accuracy on certain groups, especially in the presence of spurious correlations between the input and label.
Ranked #1 on
Out-of-Distribution Generalization
on ImageNet-W
Meta-Inverse Reinforcement Learning with Probabilistic Context Variables
Critically, our model can infer rewards for new, structurally-similar tasks from a single demonstration.
Ranked #1 on
MuJoCo Games
on Sawyer Pusher
A Critical Evaluation of AI Feedback for Aligning Large Language Models
RLAIF first performs supervised fine-tuning (SFT) using demonstrations from a teacher model and then further fine-tunes the model with reinforcement learning (RL), using feedback from a critic model.
BridgeData V2: A Dataset for Robot Learning at Scale
By publicly sharing BridgeData V2 and our pre-trained models, we aim to accelerate research in scalable robot learning methods.
Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
Large vision-language models (LVLMs) have shown remarkable abilities in understanding visual information with human languages.
Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning
In this paper, we study a number of design decisions for the predictive model in visual MBRL algorithms, focusing specifically on methods that use a predictive model for planning.
Model-based Reinforcement Learning
Reinforcement Learning (RL)
Meta-Learning with Implicit Gradients
By drawing upon implicit differentiation, we develop the implicit MAML algorithm, which depends only on the solution to the inner level optimization and not the path taken by the inner loop optimizer.
Language as an Abstraction for Hierarchical Deep Reinforcement Learning
We find that, using our approach, agents can learn to solve to diverse, temporally-extended tasks such as object sorting and multi-object rearrangement, including from raw pixel observations.
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
Deep reinforcement learning (RL) algorithms can learn complex robotic skills from raw sensory inputs, but have yet to achieve the kind of broad generalization and applicability demonstrated by deep learning methods in supervised domains.
