Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection

21 Mar 2023  ·  Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, Xiangyu Zhang ·

In this paper, we propose a long-sequence modeling framework, named StreamPETR, for multi-view 3D object detection. Built upon the sparse query design in the PETR series, we systematically develop an object-centric temporal mechanism. The model is performed in an online manner and the long-term historical information is propagated through object queries frame by frame. Besides, we introduce a motion-aware layer normalization to model the movement of the objects. StreamPETR achieves significant performance improvements only with negligible computation cost, compared to the single-frame baseline. On the standard nuScenes benchmark, it reaches a new state-of-the-art performance (63.6% NDS). The lightweight version realizes 45.0% mAP and 31.7 FPS, outperforming the state-of-the-art method (SOLOFusion) by 2.3% mAP and 1.8x faster FPS. Code will be available at https://github.com/exiawsh/StreamPETR.git.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Benchmark
3D Multi-Object Tracking nuScenes Camera Only StreamPETR-Large AMOTA 65.3 # 1
3D Object Detection nuScenes Camera Only StreamPETR-Large NDS 67.6 # 2