VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding

1 code implementation • 17 Oct 2024 • Runsen Xu, Zhiwei Huang, Tai Wang, Yilun Chen, Jiangmiao Pang, Dahua Lin

3D visual grounding is crucial for robots, requiring integration of natural language and 3D scene understanding.

3D geometry 3D visual grounding +2

MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations

1 code implementation • 13 Jun 2024 • Ruiyuan Lyu, Tai Wang, Jingli Lin, Shuai Yang, Xiaohan Mao, Yilun Chen, Runsen Xu, Haifeng Huang, Chenming Zhu, Dahua Lin, Jiangmiao Pang

With the emergence of LLMs and their integration with other data modalities, multi-modal 3D perception attracts more attention due to its connectivity to the physical world and makes rapid progress.

3D visual grounding Attribute +1

Grounded 3D-LLM with Referent Tokens

1 code implementation • 16 May 2024 • Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Runsen Xu, Ruiyuan Lyu, Dahua Lin, Jiangmiao Pang

To facilitate the use of referent tokens in subsequent language modeling, we provide a large-scale, automatically curated grounded scene-text dataset with over 1 million phrase-to-region correspondences and introduce Contrastive Language-Scene Pre-training (CLASP) to perform phrase-level scene-text alignment using this data.

Dense Captioning Diversity +7

EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI

1 code implementation • CVPR 2024 • Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, Xihui Liu, Cewu Lu, Dahua Lin, Jiangmiao Pang

In the realm of computer vision and robotics, embodied agents are expected to explore their environment and carry out human instructions.

Scene Understanding

Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers

2 code implementations • 13 Dec 2023 • Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, Zhou Zhao

Recent advancements in 3D Large Language Models (LLMs) have demonstrated promising capabilities for 3D scene understanding.

Attribute Object +2

PointLLM: Empowering Large Language Models to Understand Point Clouds

3 code implementations • 31 Aug 2023 • Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, Dahua Lin

The unprecedented advancements in Large Language Models (LLMs) have shown a profound impact on natural language processing but are yet to fully embrace the realm of 3D understanding.

3D Object Captioning 3D Question Answering (3D-QA) +3

Fine-Grained Cross-View Geo-Localization Using a Correlation-Aware Homography Estimator

1 code implementation • NeurIPS 2023 • Xiaolong Wang, Runsen Xu, Zuofan Cui, Zeyu Wan, Yu Zhang

In this paper, we introduce a novel approach to fine-grained cross-view geo-localization.

geo-localization Homography Estimation

MV-JAR: Masked Voxel Jigsaw and Reconstruction for LiDAR-Based Self-Supervised Pre-Training

1 code implementation • CVPR 2023 • Runsen Xu, Tai Wang, Wenwei Zhang, Runjian Chen, Jinkun Cao, Jiangmiao Pang, Dahua Lin

This paper introduces the Masked Voxel Jigsaw and Reconstruction (MV-JAR) method for LiDAR-based self-supervised pre-training and a carefully designed data-efficient 3D object detection benchmark on the Waymo dataset.

3D Object Detection object-detection

CO^3: Cooperative Unsupervised 3D Representation Learning for Autonomous Driving

1 code implementation • 8 Jun 2022 • Runjian Chen, Yao Mu, Runsen Xu, Wenqi Shao, Chenhan Jiang, Hang Xu, Zhenguo Li, Ping Luo

In this paper, we propose CO^3, namely Cooperative Contrastive Learning and Contextual Shape Prediction, to learn 3D representation for outdoor-scene point clouds in an unsupervised manner.

Autonomous Driving Contrastive Learning +1

LIFE: Lighting Invariant Flow Estimation

no code implementations • 7 Apr 2021 • Zhaoyang Huang, Xiaokun Pan, Runsen Xu, Yan Xu, Ka Chun Cheung, Guofeng Zhang, Hongsheng Li

However, local image contents are inevitably ambiguous and error-prone during the cross-image feature matching process, which hinders downstream tasks.