Search Results for author:
Found 28 papers, 14 papers with code
Multimodal Spatial Language Maps for Robot Navigation and Manipulation
Experiments in simulation and real-world settings demonstrate that our multimodal spatial language maps enable zero-shot spatial and multimodal goal navigation and improve recall by 50% in ambiguous scenarios.
FAST: Efficient Action Tokenization for Vision-Language-Action Models
However, such models require us to choose a tokenization of our continuous action signals, which determines how the discrete symbols predicted by the model map to continuous robot actions.
Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding
However, state-of-the-art generalist robot policies are typically trained on large datasets to predict robot actions solely from visual and proprioceptive observations.
GHIL-Glue: Hierarchical Control with Filtered Subgoal Images
Image and video generative models that are pre-trained on Internet-scale data can greatly increase the generalization capacity of robot learning systems.
Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Models
A central challenge towards developing robots that can relate human language to their perception and actions is the scarcity of natural language annotations in diverse robot datasets.
Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance
Large, general-purpose robotic policies trained on diverse demonstration datasets have been shown to be remarkably effective both for controlling a variety of robots in a range of different scenes, and for acquiring broad repertoires of manipulation skills.
The Ingredients for Robotic Diffusion Transformers
By combining the results of our investigation with our improved model components, we are able to present a novel architecture, named \method, that significantly outperforms the state of the art in solving long-horizon ($1500+$ time-steps) dexterous tasks on a bi-manual ALOHA robot.
Policy Adaptation via Language Optimization: Decomposing Tasks for Few-Shot Imitation
Learned language-conditioned robot policies often struggle to effectively adapt to new real-world tasks even when pre-trained across a diverse set of instructions.
Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviation
By training a single policy across many different kinds of robots, a robot learning method can leverage much broader and more diverse datasets, which in turn can lead to better generalization and robustness.
Autonomous Improvement of Instruction Following Skills via Foundation Models
Intelligent instruction-following robots capable of improving from autonomously collected experience have the potential to transform robot learning: instead of collecting costly teleoperated demonstration data, large-scale deployment of fleets of robots can quickly collect larger quantities of autonomous data that can collectively improve their performance.
Robotic Control via Embodied Chain-of-Thought Reasoning
Recent works on vision-language-action models (VLAs) have shown that the use of large, internet pre-trained vision-language models as the backbone of learned robot policies can substantially improve their robustness and generalization ability.
Octo: An Open-Source Generalist Robot Policy
In experiments across 9 robotic platforms, we demonstrate that Octo serves as a versatile policy initialization that can be effectively finetuned to new observation and action spaces.
Ranked #3 on
Robot Manipulation
on SimplerEnv-Widow X
(using extra training data)
Evaluating Real-World Robot Manipulation Policies in Simulation
We then employ these approaches to create SIMPLER, a collection of simulated environments for manipulation policy evaluation on common real robot setups.
Vision-Language Models Provide Promptable Representations for Reinforcement Learning
We find that our policies trained on embeddings from off-the-shelf, general-purpose VLMs outperform equivalent policies trained on generic, non-promptable image embeddings.
Audio Visual Language Maps for Robot Navigation
While interacting in the world is a multi-sensory experience, many robots continue to predominantly rely on visual perception to map and navigate in their environments.
Visual Language Maps for Robot Navigation
Grounding language to the visual observations of a navigating agent can be performed using off-the-shelf visual-language models pretrained on Internet-scale data (e. g., image captions).
Grounding Language with Visual Affordances over Unstructured Data
Recent works have shown that Large Language Models (LLMs) can be applied to ground natural language to a wide variety of robot skills.
Ranked #1 on
Avg. sequence length
on CALVIN
Latent Plans for Task-Agnostic Offline Reinforcement Learning
Concretely, we combine a low-level policy that learns latent skills via imitation learning and a high-level policy learned from offline reinforcement learning for skill-chaining the latent behavior priors.
What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data
We have open-sourced our implementation to facilitate future research in learning to perform many complex manipulation skills in a row specified with natural language.
Affordance Learning from Play for Sample-Efficient Policy Learning
Robots operating in human-centered environments should have the ability to understand how objects function: what can be done with each object, where this interaction may occur, and how the object is used to achieve a goal.
CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks
We show that a baseline model based on multi-context imitation learning performs poorly on CALVIN, suggesting that there is significant room for developing innovative agents that learn to relate human language to their world models with this benchmark.
Composing Pick-and-Place Tasks By Grounding Language
Controlling robots to perform tasks via natural language is one of the most challenging topics in human-robot interaction.
Hindsight for Foresight: Unsupervised Structured Dynamics Models from Physical Interaction
A key challenge for an agent learning to interact with the world is to reason about physical properties of objects and to foresee their dynamics under the effect of applied forces.
Learning Object Placements For Relational Instructions by Hallucinating Scene Representations
One particular requirement for such robots is that they are able to understand spatial relations and can place objects in accordance with the spatial relations expressed by their user.
Adversarial Skill Networks: Unsupervised Robot Skill Learning from Video
Our method learns a general skill embedding independently from the task context by using an adversarial loss.
Self-supervised 3D Shape and Viewpoint Estimation from Single Images for Robotics
We present a convolutional neural network for joint 3D shape prediction and viewpoint estimation from a single input image.
3D Object Reconstruction From A Single Image
3D Reconstruction
+3
Choosing Smartly: Adaptive Multimodal Fusion for Object Detection in Changing Environments
Object detection is an essential task for autonomous robots operating in dynamic and changing environments.
Metric Learning for Generalizing Spatial Relations to New Objects
Human-centered environments are rich with a wide variety of spatial relations between everyday objects.
