Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following

We introduce Point-Bind, a 3D multi-modality model aligning point clouds with 2D image, language, audio, and video. Guided by ImageBind, we construct a joint embedding space between 3D and multi-modalities, enabling many promising applications, e.g., any-to-3D generation, 3D embedding arithmetic, and 3D open-world understanding. On top of this, we further present Point-LLM, the first 3D large language model (LLM) following 3D multi-modal instructions. By parameter-efficient fine-tuning techniques, Point-LLM injects the semantics of Point-Bind into pre-trained LLMs, e.g., LLaMA, which requires no 3D instruction data, but exhibits superior 3D and multi-modal question-answering capacity. We hope our work may cast a light on the community for extending 3D point clouds to multi-modality applications. Code is available at https://github.com/ZiyuGuo99/Point-Bind_Point-LLM.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
3D Question Answering (3D-QA) 3D MM-Vet Point-Bind & Point-LLM Overall Accuracy 23.5 # 5
Generative 3D Object Classification ModelNet40 Point-Bind LLM ModelNet40 (Average) 45.81 # 6
Generative 3D Object Classification Objaverse Point-Bind LLM Objaverse (I) 6.00 # 5
Objaverse (Average) 5.25 # 7
Objaverse (C) 4.50 # 5

Methods


No methods listed for this paper. Add relevant methods here