V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer

20 Mar 2022  ·  Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuan Yang, Jiaqi Ma ·

In this paper, we investigate the application of Vehicle-to-Everything (V2X) communication to improve the perception performance of autonomous vehicles. We present a robust cooperative perception framework with V2X communication using a novel vision Transformer. Specifically, we build a holistic attention model, namely V2X-ViT, to effectively fuse information across on-road agents (i.e., vehicles and infrastructure). V2X-ViT consists of alternating layers of heterogeneous multi-agent self-attention and multi-scale window self-attention, which captures inter-agent interaction and per-agent spatial relationships. These key modules are designed in a unified Transformer architecture to handle common V2X challenges, including asynchronous information sharing, pose errors, and heterogeneity of V2X components. To validate our approach, we create a large-scale V2X perception dataset using CARLA and OpenCDA. Extensive experimental results demonstrate that V2X-ViT sets new state-of-the-art performance for 3D object detection and achieves robust performance even under harsh, noisy environments. The code is available at https://github.com/DerrickXuNu/v2x-vit.

PDF Abstract

Datasets


Introduced in the Paper:

V2XSet

Used in the Paper:

CARLA OPV2V

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
3D Object Detection V2XSet V2X-ViT AP0.5 (Perfect) 0.882 # 1
AP0.7 (Perfect) 0.712 # 2
AP0.5 (Noisy) 0.836 # 1
AP0.7 (Noisy) 0.614 # 1

Methods