Self-Supervised Multi-Frame Monocular Scene Flow
Estimating 3D scene flow from a sequence of monocular images has been gaining increased attention due to the simple, economical capture setup. Owing to the severe ill-posedness of the problem, the accuracy of current methods has been limited, especially that of efficient, real-time approaches. In this paper, we introduce a multi-frame monocular scene flow network based on self-supervised learning, improving the accuracy over previous networks while retaining real-time efficiency. Based on an advanced two-frame baseline with a split-decoder design, we propose (i) a multi-frame model using a triple frame input and convolutional LSTM connections, (ii) an occlusion-aware census loss for better accuracy, and (iii) a gradient detaching strategy to improve training stability. On the KITTI dataset, we observe state-of-the-art accuracy among monocular scene flow methods based on self-supervised learning.
PDF Abstract CVPR 2021 PDF CVPR 2021 AbstractResults from the Paper
Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Benchmark |
---|---|---|---|---|---|---|
Scene Flow Estimation | KITTI 2015 Scene Flow Test | Multi-Mono-SF | D1-all | 30.78 | # 2 | |
D2-all | 34.41 | # 2 | ||||
Fl-all | 19.54 | # 2 | ||||
SF-all | 44.04 | # 3 | ||||
Runtime (s) | 0.063 | # 1 | ||||
Scene Flow Estimation | KITTI 2015 Scene Flow Training | Multi-Mono-SF | D1-all | 27.33 | # 3 | |
D2-all | 30.44 | # 1 | ||||
Fl-all | 18.92 | # 1 | ||||
SF-all | 39.82 | # 1 | ||||
Runtime (s) | 0.063 | # 3 |