DSVT: Dynamic Sparse Voxel Transformer with Rotated Sets

Designing an efficient yet deployment-friendly 3D backbone to handle sparse point clouds is a fundamental problem in 3D perception. Compared with the customized sparse convolution, the attention mechanism in Transformers is more appropriate for flexibly modeling long-range relationships and is easier to be deployed in real-world applications. However, due to the sparse characteristics of point clouds, it is non-trivial to apply a standard transformer on sparse points. In this paper, we present Dynamic Sparse Voxel Transformer (DSVT), a single-stride window-based voxel Transformer backbone for outdoor 3D perception. In order to efficiently process sparse points in parallel, we propose Dynamic Sparse Window Attention, which partitions a series of local regions in each window according to its sparsity and then computes the features of all regions in a fully parallel manner. To allow the cross-set connection, we design a rotated set partitioning strategy that alternates between two partitioning configurations in consecutive self-attention layers. To support effective downsampling and better encode geometric information, we also propose an attention-style 3D pooling module on sparse points, which is powerful and deployment-friendly without utilizing any customized CUDA operations. Our model achieves state-of-the-art performance with a broad range of 3D perception tasks. More importantly, DSVT can be easily deployed by TensorRT with real-time inference speed (27Hz). Code will be available at \url{https://github.com/Haiyang-W/DSVT}.

PDF Abstract CVPR 2023 PDF CVPR 2023 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
3D Object Detection nuScenes DSVT NDS 0.73 # 26
mATE 0.25 # 305
mASE 0.23 # 322
mAOE 0.30 # 348
mAVE 0.25 # 285
mAAE 0.14 # 70
3D Object Detection nuScenes LiDAR only DSVT NDS 72.7 # 2
mAP 68.4 # 2
NDS (val) 71.1 # 2
mAP (val) 66.4 # 2
3D Object Detection waymo cyclist DSVT(val) APH/L2 78.0 # 1
3D Object Detection Waymo Open Dataset DSVT mAPH/L2 72.1 # 4
3D Object Detection waymo pedestrian DSVT(val) APH/L2 76.4 # 1
3D Object Detection waymo vehicle DSVT(val) APH/L2 74.1 # 2
L1 mAP 82.1 # 1

Methods