Recurrent Scene Parsing with Perspective Understanding in the Loop

CVPR 2018  ·  Shu Kong, Charless Fowlkes ·

Objects may appear at arbitrary scales in perspective images of a scene, posing a challenge for recognition systems that process images at a fixed resolution. We propose a depth-aware gating module that adaptively selects the pooling field size in a convolutional network architecture according to the object scale (inversely proportional to the depth) so that small details are preserved for distant objects while larger receptive fields are used for those nearby. The depth gating signal is provided by stereo disparity or estimated directly from monocular input. We integrate this depth-aware gating into a recurrent convolutional neural network to perform semantic segmentation. Our recurrent module iteratively refines the segmentation results, leveraging the depth and semantic predictions from the previous iterations. Through extensive experiments on four popular large-scale RGB-D datasets, we demonstrate this approach achieves competitive semantic segmentation performance with a model which is substantially more compact. We carry out extensive analysis of this architecture including variants that operate on monocular RGB but use depth as side-information during training, unsupervised gating as a generic attentional mechanism, and multi-resolution gating. We find that gated pooling for joint semantic segmentation and depth yields state-of-the-art results for quantitative monocular depth estimation.

PDF Abstract CVPR 2018 PDF CVPR 2018 Abstract

Results from the Paper


Ranked #35 on Semantic Segmentation on SUN-RGBD (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Semantic Segmentation Cityscapes test DepthSeg (ResNet-101) Mean IoU (class) 78.2% # 60
Semantic Segmentation NYU Depth v2 RecurrentSceneParsing Mean IoU 44.5% # 90
Semantic Segmentation SUN-RGBD DPLNet Mean IoU 45.1% # 35

Methods