Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policy

2 Oct 2024  ·  Ricardo Garcia, ShiZhe Chen, Cordelia Schmid ·

Generalizing language-conditioned robotic policies to new tasks remains a significant challenge, hampered by the lack of suitable simulation benchmarks. In this paper, we address this gap by introducing GemBench, a novel benchmark to assess generalization capabilities of vision-language robotic manipulation policies. GemBench incorporates seven general action primitives and four levels of generalization, spanning novel placements, rigid and articulated objects, and complex long-horizon tasks. We evaluate state-of-the-art approaches on GemBench and also introduce a new method. Our approach 3D-LOTUS leverages rich 3D information for action prediction conditioned on language. While 3D-LOTUS excels in both efficiency and performance on seen tasks, it struggles with novel tasks. To address this, we present 3D-LOTUS++, a framework that integrates 3D-LOTUS's motion planning capabilities with the task planning capabilities of LLMs and the object grounding accuracy of VLMs. 3D-LOTUS++ achieves state-of-the-art performance on novel tasks of GemBench, setting a new standard for generalization in robotic manipulation. The benchmark, codes and trained models are available at https://www.di.ens.fr/willow/research/gembench/.

PDF Abstract

Datasets


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Robot Manipulation Generalization GEMBench 3D-LOTUS++ Average Success Rate (L1) 68.7±0.6 # 5
Average Success Rate (L2) 64.5±0.9 # 1
Average Success Rate (L3) 41.5±1.8 # 1
Average Success Rate (L4) 17.4±0.4 # 1
Average Success Rate 48.0 # 1
Robot Manipulation Generalization GEMBench 3D-LOTUS Average Success Rate (L1) 94.3±1.4 # 1
Average Success Rate (L2) 49.9±2.2 # 3
Average Success Rate (L3) 38.1±1.1 # 3
Average Success Rate (L4) 0.3±0.3 # 2
Average Success Rate 45.7 # 2
Robot Manipulation RLBench 3D-LOTUS Succ. Rate (18 tasks, 100 demo/task) 83.1 # 4
Training Time (V100 x 8 x day) 0.28 # 2
Inference Speed (fps) 9.5 # 3
Input Image Size 256 # 10
Training Time (A100 x hour) 40 # 2

Methods


No methods listed for this paper. Add relevant methods here