Search Results for author:
Found 50 papers, 16 papers with code
$π_0$: A Vision-Language-Action Flow Model for General Robot Control
Robot learning holds tremendous promise to unlock the full potential of flexible, general, and dexterous robot systems, as well as to address some of the deepest questions in artificial intelligence.
GenCHiP: Generating Robot Policy Code for High-Precision and Contact-Rich Manipulation Tasks
Large Language Models (LLMs) have been successful at generating robot policy code, but so far these results have been limited to high-level tasks that do not require precise movement.
PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
In each iteration, the image is annotated with a visual representation of proposals that the VLM can refer to (e. g., candidate robot actions, localizations, or trajectories).
AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents
We experimentally show that such "in-the-wild" data collected by AutoRT is significantly more diverse, and that AutoRT's use of LLMs allows for instruction following data collection robots that can align to human preferences.
Chain of Code: Reasoning with a Language Model-Augmented Code Emulator
For example, consider prompting an LM to write code that counts the number of times it detects sarcasm in an essay: the LM may struggle to write an implementation for "detect_sarcasm(string)" that can be executed by the interpreter (handling the edge cases would be insurmountable).
Foundations for Transfer in Reinforcement Learning: A Taxonomy of Knowledge Modalities
Contemporary artificial intelligence systems exhibit rapidly growing abilities accompanied by the growth of required resources, expansive datasets and corresponding investments into computing infrastructure.
SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attention
We present Self-Adaptive Robust Attention for Robotics Transformers (SARA-RT): a new paradigm for addressing the emerging challenge of scaling up Robotics Transformers (RT) for on-robot deployment.
What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?
Inspired by the success of transfer learning in computer vision, roboticists have investigated visual pre-training as a means to improve the learning efficiency and generalization ability of policies learned from pixels.
RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches
Generalization remains one of the most important desiderata for robust robot learning systems.
Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions
In this work, we present a scalable reinforcement learning method for training multi-task policies from large offline datasets that can leverage both human demonstrations and autonomously collected data.
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Our goal is to enable a single end-to-end trained model to both learn to map robot observations to actions and enjoy the benefits of large-scale pretraining on language and vision-language data from the web.
PaLM-E: An Embodied Multimodal Language Model
Large language models excel at a wide range of complex tasks.
Ranked #2 on
Visual Question Answering (VQA)
on OK-VQA
Open-World Object Manipulation using Pre-trained Vision-Language Models
This brings up a notably difficult challenge for robots: while robot learning approaches allow robots to learn many different behaviors from first-hand experience, it is impractical for robots to have first-hand experiences that span all of this semantic information.
Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agents
Recent progress in large language models (LLMs) has demonstrated the ability to learn and leverage Internet-scale knowledge through pre-training with autoregressive models.
Scaling Robot Learning with Semantically Imagined Experience
Specifically, we make use of the state of the art text-to-image diffusion models and perform aggressive data augmentation on top of our existing robotic manipulation datasets via inpainting various unseen objects for manipulation, backgrounds, and distractors with text guidance.
RT-1: Robotics Transformer for Real-World Control at Scale
By transferring knowledge from large, diverse, task-agnostic datasets, modern machine learning models can solve specific downstream tasks either zero-shot or with small task-specific datasets to a high level of performance.
Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models
To accomplish this, we introduce Data-driven Instruction Augmentation for Language-conditioned control (DIAL): we utilize semi-supervised language labels leveraging the semantic understanding of CLIP to propagate knowledge onto large datasets of unlabelled demonstration data and then train language-conditioned policies on the augmented datasets.
Offline Reinforcement Learning at Multiple Frequencies
In this paper, we focus on one particular aspect of heterogeneity: learning from offline data collected at different control frequencies.
Inner Monologue: Embodied Reasoning through Planning with Language Models
We investigate a variety of sources of feedback, such as success detection, scene description, and human interaction.
Jump-Start Reinforcement Learning
In addition, we provide an upper bound on the sample complexity of JSRL and show that with the help of a guide-policy, one can improve the sample complexity for non-optimism exploration methods from exponential in horizon to polynomial.
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
We show how low-level skills can be combined with large language models so that the language model provides high-level knowledge about the procedures for performing complex and temporally-extended instructions, while value functions associated with these skills provide the grounding necessary to connect this knowledge to a particular physical environment.
How to Leverage Unlabeled Data in Offline Reinforcement Learning
One natural solution is to learn a reward function from the labeled data and use it to label the unlabeled data.
Autonomous Reinforcement Learning: Formalism and Benchmarking
In this paper, we aim to address this discrepancy by laying out a framework for Autonomous Reinforcement Learning (ARL): reinforcement learning where the agent not only learns through its own experience, but also contends with lack of human supervision to reset between trials.
Data Sharing without Rewards in Multi-Task Offline Reinforcement Learning
However, these benefits come at a cost -- for data to be shared between tasks, each transition must be annotated with reward labels corresponding to other tasks.
Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
We argue that a natural use case of offline RL is in settings where we can pool large amounts of data collected in various scenarios for solving different tasks, and utilize all of this data to learn behaviors for all the tasks more effectively rather than training each one in isolation.
Autonomous Reinforcement Learning via Subgoal Curricula
Reinforcement learning (RL) promises to enable autonomous acquisition of complex behaviors for diverse agents.
MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
In this paper, we study how a large-scale collective robotic learning system can acquire a repertoire of behaviors simultaneously, sharing exploration, experience, and representations across tasks.
Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
We consider the problem of learning useful robotic skills from previously collected offline data without access to manually specified rewards or additional online exploration, a setting that is becoming increasingly important for scaling robot learning by reusing past robotic data.
A Geometric Perspective on Self-Supervised Policy Adaptation
One of the most challenging aspects of real-world reinforcement learning (RL) is the multitude of unpredictable and ever-changing distractions that could divert an agent from what was tasked to do in its training environment.
Confidence-rich grid mapping
Representing the environment is a fundamental task in enabling robots to act autonomously in unknown environments.
Efficient Adaptation for End-to-End Vision-Based Robotic Manipulation
One of the great promises of robot learning systems is that they will be able to learn from their mistakes and continuously adapt to ever-changing environments, but most robot learning systems today are deployed as fixed policies which do not adapt after deployment.
Modeling Long-horizon Tasks as Sequential Interaction Landscapes
We show that complex plans can be carried out when executing the robotic task and the robot can interactively adapt to changes in the environment and recover from failure cases.
Dynamics-Aware Unsupervised Skill Discovery
Conventionally, model-based reinforcement learning (MBRL) aims to learn a global model for the dynamics of the environment.
Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning
Can we instead develop efficient reinforcement learning methods that acquire diverse skills without any reward function, and then repurpose these skills for downstream tasks?
Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning
One of the great promises of robot learning systems is that they will be able to learn from their mistakes and continuously adapt to ever-changing environments.
Thinking While Moving: Deep Reinforcement Learning with Concurrent Control
We study reinforcement learning in settings where sampling an action from the policy must be done concurrently with the time evolution of the controlled system, such as when a robot must decide on the next action while still performing the previous action.
Gradient Surgery for Multi-Task Learning
While deep learning and deep reinforcement learning (RL) systems have demonstrated impressive results in domains such as image classification, game playing, and robotic control, data efficiency remains a major challenge.
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning
We present relay policy learning, a method for imitation and reinforcement learning that can solve multi-stage, long-horizon robotic tasks.
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
Therefore, if the aim of these methods is to enable faster acquisition of entirely new behaviors, we must evaluate them on task distributions that are sufficiently broad to enable generalization to new behaviors.
Ranked #1 on
Meta-Learning
on ML10
Quantile QT-Opt for Risk-Aware Vision-Based Robotic Grasping
The absence of an actor in Q2-Opt allows us to directly draw a parallel to the previous discrete experiments in the literature without the additional complexities induced by an actor-critic architecture.
Mint: Matrix-Interleaving for Multi-Task Learning
Deep learning enables training of large and flexible function approximators from scratch at the cost of large amounts of data.
Dynamics-Aware Unsupervised Discovery of Skills
Conventionally, model-based reinforcement learning (MBRL) aims to learn a global model for the dynamics of the environment.
Training an Interactive Helper
In this paper, we investigate if, and how, a "helper" agent can be trained to interactively adapt their behavior to maximize the reward of another agent, whom we call the "prime" agent, without observing their reward or receiving explicit demonstrations.
Learning to Interactively Learn and Assist
Most importantly, we find that our approach produces an agent that is capable of learning interactively from a human user, without a set of explicit demonstrations or a reward function, and achieving significantly better performance cooperatively with a human than a human performing the task alone.
Simulator Predictive Control: Using Learned Task Representations and MPC for Zero-Shot Generalization and Sequencing
We complete unseen tasks by choosing new sequences of skill latents to control the robot using MPC, where our MPC model is composed of the pre-trained skill policy executed in the simulation environment, run in parallel with the real robot.
Scaling simulation-to-real transfer by learning composable robot skills
In particular, we first use simulation to jointly learn a policy for a set of low-level skills, and a "skill embedding" parameterization which can be used to compose them.
Region Growing Curriculum Generation for Reinforcement Learning
In this work, we introduce a method based on region-growing that allows learning in an environment with any pair of initial and goal states.
Learning an Embedding Space for Transferable Robot Skills
We present a method for reinforcement learning of closely related skills that are parameterized via a skill embedding space.
Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets
Imitation learning has traditionally been applied to learn a single task from demonstrations thereof.
Interactive Perception: Leveraging Action in Perception and Perception in Action
Recent approaches in robotics follow the insight that perception is facilitated by interaction with the environment.
Robotics
