Non-Local Spatial Propagation Network for Depth Completion

ECCV 2020  ยท  Jinsun Park, Kyungdon Joo, Zhe Hu, Chi-Kuei Liu, In So Kweon ยท

In this paper, we propose a robust and efficient end-to-end non-local spatial propagation network for depth completion. The proposed network takes RGB and sparse depth images as inputs and estimates non-local neighbors and their affinities of each pixel, as well as an initial depth map with pixel-wise confidences. The initial depth prediction is then iteratively refined by its confidence and non-local spatial propagation procedure based on the predicted non-local neighbors and corresponding affinities. Unlike previous algorithms that utilize fixed-local neighbors, the proposed algorithm effectively avoids irrelevant local neighbors and concentrates on relevant non-local neighbors during propagation. In addition, we introduce a learnable affinity normalization to better learn the affinity combinations compared to conventional methods. The proposed algorithm is inherently robust to the mixed-depth problem on depth boundaries, which is one of the major issues for existing depth estimation/completion algorithms. Experimental results on indoor and outdoor datasets demonstrate that the proposed algorithm is superior to conventional algorithms in terms of depth completion accuracy and robustness to the mixed-depth problem. Our implementation is publicly available on the project page.

PDF Abstract ECCV 2020 PDF ECCV 2020 Abstract

Datasets


Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Depth Completion KITTI Depth Completion NLSPN iRMSE 1.99 # 1
iMAE 0.84 # 1
RMSE 741.68 # 4
MAE 199.59 # 1
Runtime [ms] 220 # 11
Stereo-LiDAR Fusion KITTI Depth Completion Validation NLSPN RMSE 771.8 # 5
Depth Completion NYU-Depth V2 NLSPN RMSE 0.092 # 1
REL 0.012 # 1
Depth Completion VOID NLSPN MAE 26.736 # 1
RMSE 79.121 # 1
iMAE 12.703 # 1
iRMSE 33.876 # 1

Methods


No methods listed for this paper. Add relevant methods here