Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks

The use of RGB-D information for salient object detection has been extensively explored in recent years. However, relatively few efforts have been put towards modeling salient object detection in real-world human activity scenes with RGBD. In this work, we fill the gap by making the following contributions to RGB-D salient object detection. (1) We carefully collect a new SIP (salient person) dataset, which consists of ~1K high-resolution images that cover diverse real-world scenes from various viewpoints, poses, occlusions, illuminations, and backgrounds. (2) We conduct a large-scale (and, so far, the most comprehensive) benchmark comparing contemporary methods, which has long been missing in the field and can serve as a baseline for future research. We systematically summarize 32 popular models and evaluate 18 parts of 32 models on seven datasets containing a total of about 97K images. (3) We propose a simple general architecture, called Deep Depth-Depurator Network (D3Net). It consists of a depth depurator unit (DDU) and a three-stream feature learning module (FLM), which performs low-quality depth map filtering and cross-modal feature learning respectively. These components form a nested structure and are elaborately designed to be learned jointly. D3Net exceeds the performance of any prior contenders across all five metrics under consideration, thus serving as a strong model to advance research in this field. We also demonstrate that D3Net can be used to efficiently extract salient object masks from real scenes, enabling effective background changing application with a speed of 65fps on a single GPU. All the saliency maps, our new SIP dataset, the D3Net model, and the evaluation tools are publicly available at https://github.com/DengPingFan/D3NetBenchmark.

PDF Abstract

Datasets


Introduced in the Paper:

SIP

Used in the Paper:

NLPR LFSD NJU2K
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
RGB-D Salient Object Detection LFSD D3Net S-Measure 82.5 # 7
Average MAE 0.095 # 7
max E-Measure 86.2 # 4
max F-Measure 81.0 # 4
RGB-D Salient Object Detection NJU2K D3Net S-Measure 90.0 # 13
Average MAE 0.046 # 15
max E-Measure 93.9 # 7
max F-Measure 90.0 # 9
RGB-D Salient Object Detection NLPR D3Net S-Measure 91.2 # 11
Average MAE 0.030 # 11
max F-Measure 89.7 # 8
max E-Measure 95.3 # 7
RGB-D Salient Object Detection RGBD135 D3Net S-Measure 85.7 # 4
Average MAE 0.058 # 4
max F-Measure 83.4 # 4
max E-Measure 91.0 # 3
RGB-D Salient Object Detection SIP D3Net S-Measure 86.0 # 12
max E-Measure 90.9 # 8
max F-Measure 86.1 # 8
Average MAE 0.063 # 12
RGB-D Salient Object Detection STERE D3Net S-Measure 89.9 # 10
Average MAE 0.046 # 11
max F-Measure 89.1 # 8
max E-Measure 93.8 # 7

Methods