Search Results for author:
Found 24 papers, 8 papers with code
Physically Grounded Vision-Language Models for Robotic Manipulation
However, current VLMs are limited in their understanding of the physical concepts (e. g., material, fragility) of common objects, which restricts their usefulness for robotic manipulation tasks that involve interaction and physical reasoning about such objects.
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Our goal is to enable a single end-to-end trained model to both learn to map robot observations to actions and enjoy the benefits of large-scale pretraining on language and vision-language data from the web.
Large Language Models as General Pattern Machines
We observe that pre-trained large language models (LLMs) are capable of autoregressively completing complex token sequences -- from arbitrary ones procedurally generated by probabilistic context-free grammars (PCFG), to more rich spatial patterns found in the Abstract Reasoning Corpus (ARC), a general AI benchmark, prompted in the style of ASCII art.
Language to Rewards for Robotic Skill Synthesis
However, since low-level robot actions are hardware-dependent and underrepresented in LLM training corpora, existing efforts in applying LLMs to robotics have largely treated LLMs as semantic planners or relied on human-engineered control primitives to interface with the robot.
PaLM-E: An Embodied Multimodal Language Model
Large language models excel at a wide range of complex tasks.
Ranked #1 on
Visual Question Answering (VQA)
on OK-VQA
Grounded Decoding: Guiding Text Generation with Grounded Models for Robot Control
Recent progress in large language models (LLMs) has demonstrated the ability to learn and leverage Internet-scale knowledge through pre-training with autoregressive models.
Scaling Robot Learning with Semantically Imagined Experience
Specifically, we make use of the state of the art text-to-image diffusion models and perform aggressive data augmentation on top of our existing robotic manipulation datasets via inpainting various unseen objects for manipulation, backgrounds, and distractors with text guidance.
RT-1: Robotics Transformer for Real-World Control at Scale
By transferring knowledge from large, diverse, task-agnostic datasets, modern machine learning models can solve specific downstream tasks either zero-shot or with small task-specific datasets to a high level of performance.
Open-vocabulary Queryable Scene Representations for Real World Planning
Large language models (LLMs) have unlocked new capabilities of task planning from human instructions.
Inner Monologue: Embodied Reasoning through Planning with Language Models
We investigate a variety of sources of feedback, such as success detection, scene description, and human interaction.
LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action
Goal-conditioned policies for robotic navigation can be trained on large, unannotated datasets, providing for good generalization to real-world settings.
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
We show how low-level skills can be combined with large language models so that the language model provides high-level knowledge about the procedures for performing complex and temporally-extended instructions, while value functions associated with these skills provide the grounding necessary to connect this knowledge to a particular physical environment.
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
Large pretrained (e. g., "foundation") models exhibit distinct capabilities depending on the domain of data they are trained on.
Ranked #1 on
Zero-Shot Video Retrieval
on MSR-VTT
(using extra training data)
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
We explore how generating a chain of thought -- a series of intermediate reasoning steps -- significantly improves the ability of large language models to perform complex reasoning.
Mechanical Search on Shelves using a Novel "Bluction" Tool
Shelves are common in homes, warehouses, and commercial settings due to their storage efficiency.
Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning
Hierarchical reinforcement learning aims to enable this by providing a bank of low-level skills as action abstractions.
Hierarchical Reinforcement Learning
reinforcement-learning
+1
Multi-Task Learning with Sequence-Conditioned Transporter Networks
Enabling robots to solve multiple manipulation tasks has a wide range of industrial applications.
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation
However, goal images also have a number of drawbacks: they are inconvenient for humans to provide, they can over-specify the desired behavior leading to a sparse reward signal, or under-specify task information in the case of non-goal reaching tasks.
Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning
This task space can be quite general and abstract; its only requirements are to be sampleable and to well-cover the space of useful tasks.
Model-based Reinforcement Learning for Decentralized Multiagent Rendezvous
We propose hierarchical predictive planning (HPP), a model-based reinforcement learning method for decentralized multiagent rendezvous.
Model-based Reinforcement Learning
reinforcement-learning
+1
Neural Collision Clearance Estimator for Batched Motion Planning
We present a neural network collision checking heuristic, ClearanceNet, and a planning algorithm, CN-RRT.
Learned Critical Probabilistic Roadmaps for Robotic Motion Planning
Critical PRMs are demonstrated to achieve up to three orders of magnitude improvement over uniform sampling, while preserving the guarantees and complexity of sampling-based motion planning.
Zero-shot Imitation Learning from Demonstrations for Legged Robot Visual Navigation
Here, we propose a zero-shot imitation learning approach for training a visual navigation policy on legged robots from human (third-person perspective) demonstrations, enabling high-quality navigation and cost-effective data collection.
Learning Sampling Distributions for Robot Motion Planning
This paper proposes a methodology for non-uniform sampling, whereby a sampling distribution is learned from demonstrations, and then used to bias sampling.
