PLA: Language-Driven Open-Vocabulary 3D Scene Understanding

Open-vocabulary scene understanding aims to localize and recognize unseen categories beyond the annotated label space. The recent breakthrough of 2D open-vocabulary perception is largely driven by Internet-scale paired image-text data with rich vocabulary concepts. However, this success cannot be directly transferred to 3D scenarios due to the inaccessibility of large-scale 3D-text pairs. To this end, we propose to distill knowledge encoded in pre-trained vision-language (VL) foundation models through captioning multi-view images from 3D, which allows explicitly associating 3D and semantic-rich captions. Further, to foster coarse-to-fine visual-semantic representation learning from captions, we design hierarchical 3D-caption pairs, leveraging geometric constraints between 3D scenes and multi-view images. Finally, by employing contrastive learning, the model learns language-aware embeddings that connect 3D and text for open-vocabulary tasks. Our method not only remarkably outperforms baseline methods by 25.8% $\sim$ 44.7% hIoU and 14.5% $\sim$ 50.4% hAP$_{50}$ in open-vocabulary semantic and instance segmentation, but also shows robust transferability on challenging zero-shot domain transfer tasks. See the project website at

PDF Abstract CVPR 2023 PDF CVPR 2023 Abstract


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
3D Open-Vocabulary Instance Segmentation S3DIS PLA AP50 Base B8/N4 59.0 # 2
AP50 Novel B8/N4 8.6 # 4
AP50 Base B6/N6 46.9 # 3
AP50 Novel B6/N6 9.8 # 4


No methods listed for this paper. Add relevant methods here