Search Results for author:
Found 13 papers, 7 papers with code
Generative Image as Action Models
We present GENIMA, a behavior-cloning agent that fine-tunes Stable Diffusion to 'draw joint-actions' as targets on RGB images.
Ranked #1 on
Robot Manipulation Generalization
on The COLOSSEUM
PerAct2: Benchmarking and Learning for Robotic Bimanual Manipulation Tasks
To kickstart the benchmark, we extended several state-of-the art methods to bimanual manipulation and also present a language-conditioned behavioral cloning agent -- PerAct2, which enables the learning and execution of bimanual 6-DoF manipulation tasks.
GenSim: Generating Robotic Simulation Tasks via Large Language Models
Collecting large amounts of real-world interaction data to train general robotic policies is often prohibitively expensive, thus motivating the use of simulation data.
AR2-D2:Training a Robot Without a Robot
By contrast, we introduce AR2-D2: a system for collecting demonstrations which (1) does not require people with specialized training, (2) does not require any real robots during data collection, and therefore, (3) enables manipulation of diverse objects with a real robot.
Retrospectives on the Embodied AI Workshop
We present a retrospective on the state of Embodied AI research.
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation
With this formulation, we train a single multi-task Transformer for 18 RLBench tasks (with 249 variations) and 7 real-world tasks (with 18 variations) from just a few demonstrations per task.
Ranked #5 on
Robot Manipulation
on RLBench
CLIPort: What and Where Pathways for Robotic Manipulation
We even learn one multi-task policy for 10 simulated and 9 real-world tasks that is better or comparable to single-task policies.
Language Grounding with 3D Objects
We introduce several CLIP-based models for distinguishing objects and demonstrate that while recent advances in jointly modeling vision and language are useful for robotic language understanding, it is still the case that these image-based models are weaker at understanding the 3D nature of objects -- properties which play a key role in manipulation.
BUTLER: Building Understanding in TextWorld via Language for Embodied Reasoning
ALFWorld enables the creation of a new BUTLER agent whose abstract knowledge, learned in TextWorld, corresponds directly to concrete, visually grounded actions.
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
ALFWorld enables the creation of a new BUTLER agent whose abstract knowledge, learned in TextWorld, corresponds directly to concrete, visually grounded actions.
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
We present ALFRED (Action Learning From Realistic Environments and Directives), a benchmark for learning a mapping from natural language instructions and egocentric vision to sequences of actions for household tasks.
Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction
The first stage uses a neural network to generate visual descriptions of objects, compares them with the input language expression, and identifies a set of candidate objects.
Grounding Spatio-Semantic Referring Expressions for Human-Robot Interaction
A core issue for the system is semantic and spatial grounding, which is to infer objects and their spatial relationships from images and natural language expressions.
