Search Results for author:
Found 34 papers, 12 papers with code
Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models
Generalist robots that can perform a range of different tasks in open-world settings must be able to not only reason about the steps needed to accomplish their goals, but also process complex instructions, prompts, and even feedback during task execution.
FAST: Efficient Action Tokenization for Vision-Language-Action Models
However, such models require us to choose a tokenization of our continuous action signals, which determines how the discrete symbols predicted by the model map to continuous robot actions.
What's the Move? Hybrid Imitation Learning via Salient Points
Given 3D point cloud observations, SPHINX learns to infer task-relevant points within a point cloud, or salient points, which support spatial generalization by focusing on semantically meaningful features.
$π_0$: A Vision-Language-Action Flow Model for General Robot Control
Robot learning holds tremendous promise to unlock the full potential of flexible, general, and dexterous robot systems, as well as to address some of the deepest questions in artificial intelligence.
Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidance
In this paper, we present a new approach for language-driven 6-DoF grasp detection in cluttered point clouds.
OpenVLA: An Open-Source Vision-Language-Action Model
Large policies pretrained on a combination of Internet-scale vision-language data and diverse robot demonstrations have the potential to change how we teach robots new skills: rather than training new behaviors from scratch, we can fine-tune such vision-language-action (VLA) models to obtain robust, generalizable policies for visuomotor control.
Ranked #7 on
Robot Manipulation
on SimplerEnv-Widow X
(using extra training data)
Octo: An Open-Source Generalist Robot Policy
In experiments across 9 robotic platforms, we demonstrate that Octo serves as a versatile policy initialization that can be effectively finetuned to new observation and action spaces.
Ranked #3 on
Robot Manipulation
on SimplerEnv-Widow X
(using extra training data)
Evaluating Real-World Robot Manipulation Policies in Simulation
We then employ these approaches to create SIMPLER, a collection of simulated environments for manipulation policy evaluation on common real robot setups.
Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers
Vid2Robot uses cross-attention transformer layers between video features and the current robot state to produce the actions and perform the same task as shown in the video.
Stop Regressing: Training Value Functions via Classification for Scalable Deep RL
Observing this discrepancy, in this paper, we investigate whether the scalability of deep RL can also be improved simply by using classification in place of regression for training value functions.
PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
In each iteration, the image is annotated with a visual representation of proposals that the VLM can refer to (e. g., candidate robot actions, localizations, or trajectories).
AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents
We experimentally show that such "in-the-wild" data collected by AutoRT is significantly more diverse, and that AutoRT's use of LLMs allows for instruction following data collection robots that can align to human preferences.
SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attention
We present Self-Adaptive Robust Attention for Robotics Transformers (SARA-RT): a new paradigm for addressing the emerging challenge of scaling up Robotics Transformers (RT) for on-robot deployment.
RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches
Generalization remains one of the most important desiderata for robust robot learning systems.
Robotic Offline RL from Internet Videos via Value-Function Pre-Training
Our system, called V-PTR, combines the benefits of pre-training on video data with robotic offline RL approaches that train on diverse robot data, resulting in value functions and policies for manipulation tasks that perform better, act robustly, and generalize broadly.
Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions
In this work, we present a scalable reinforcement learning method for training multi-task policies from large offline datasets that can leverage both human demonstrations and autonomously collected data.
BridgeData V2: A Dataset for Robot Learning at Scale
By publicly sharing BridgeData V2 and our pre-trained models, we aim to accelerate research in scalable robot learning methods.
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Our goal is to enable a single end-to-end trained model to both learn to map robot observations to actions and enjoy the benefits of large-scale pretraining on language and vision-language data from the web.
Ranked #4 on
Robot Manipulation
on SimplerEnv-Google Robot
(using extra training data)
PaLM-E: An Embodied Multimodal Language Model
Large language models excel at a wide range of complex tasks.
Ranked #2 on
Visual Question Answering (VQA)
on OK-VQA
Open-World Object Manipulation using Pre-trained Vision-Language Models
This brings up a notably difficult challenge for robots: while robot learning approaches allow robots to learn many different behaviors from first-hand experience, it is impractical for robots to have first-hand experiences that span all of this semantic information.
RT-1: Robotics Transformer for Real-World Control at Scale
By transferring knowledge from large, diverse, task-agnostic datasets, modern machine learning models can solve specific downstream tasks either zero-shot or with small task-specific datasets to a high level of performance.
Ranked #6 on
Robot Manipulation
on SimplerEnv-Google Robot
(using extra training data)
Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints
Both theoretically and empirically, we show that typical offline RL methods, which are based on distribution constraints fail to learn from data with such non-uniform variability, due to the requirement to stay close to the behavior policy to the same extent across the state space.
Dual Generator Offline Reinforcement Learning
In this paper, we show that the issue of conflicting objectives can be resolved by training two generators: one that maximizes return, with the other capturing the ``remainder'' of the data distribution in the offline dataset, such that the mixture of the two is close to the behavior policy.
Single RGB-D Camera Teleoperation for General Robotic Manipulation
We propose a teleoperation system that uses a single RGB-D camera as the human motion capture device.
First Order Constrained Optimization in Policy Space
We propose a novel approach called First Order Constrained Optimization in Policy Space (FOCOPS) which maximizes an agent's overall reward while ensuring the agent satisfies a set of cost constraints.
Better Exploration with Optimistic Actor Critic
To address both of these phenomena, we introduce a new algorithm, Optimistic Actor Critic, which approximates a lower and upper confidence bound on the state-action value function.
Better Exploration with Optimistic Actor-Critic
To address both of these phenomena, we introduce a new algorithm, Optimistic Actor Critic, which approximates a lower and upper confidence bound on the state-action value function.
Striving for Simplicity and Performance in Off-Policy DRL: Output Normalization and Non-Uniform Sampling
We aim to develop off-policy DRL algorithms that not only exceed state-of-the-art performance but are also simple and minimalistic.
Pre-training as Batch Meta Reinforcement Learning with tiMe
Combining ideas from Batch RL and Meta RL, we propose tiMe, which learns distillation of multiple value functions and MDP embeddings from only existing data.
Towards Simplicity in Deep Reinforcement Learning: Streamlined Off-Policy Learning
The field of Deep Reinforcement Learning (DRL) has recently seen a surge in the popularity of maximum entropy reinforcement learning algorithms.
Multi-task Batch Reinforcement Learning with Metric Learning
To perform well, the policy must infer the task identity from collected transitions by modelling its dependency on states, actions and rewards.
SUPERVISED POLICY UPDATE
We show how the Natural Policy Gradient and Trust Region Policy Optimization (NPG/TRPO) problems, and the Proximal Policy Optimization (PPO) problem can be addressed by this methodology.
How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies?
A human-specified design choice in domain randomization is the form and parameters of the distribution of simulated environments.
Supervised Policy Update for Deep Reinforcement Learning
We show how the Natural Policy Gradient and Trust Region Policy Optimization (NPG/TRPO) problems, and the Proximal Policy Optimization (PPO) problem can be addressed by this methodology.
