Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

22 Dec 2014  ·  Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L. Yuille ·

Deep Convolutional Neural Networks (DCNNs) have recently shown state of the art performance in high level vision tasks, such as image classification and object detection. This work brings together methods from DCNNs and probabilistic graphical models for addressing the task of pixel-level classification (also called "semantic image segmentation"). We show that responses at the final layer of DCNNs are not sufficiently localized for accurate object segmentation. This is due to the very invariance properties that make DCNNs good for high level tasks. We overcome this poor localization property of deep networks by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF). Qualitatively, our "DeepLab" system is able to localize segment boundaries at a level of accuracy which is beyond previous methods. Quantitatively, our method sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 71.6% IOU accuracy in the test set. We show how these results can be obtained efficiently: Careful network re-purposing and a novel application of the 'hole' algorithm from the wavelet community allow dense computation of neural net responses at 8 frames per second on a modern GPU.

PDF Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Real-Time Semantic Segmentation CamVid DeepLab mIoU 61.6% # 24
Time (ms) 203 # 17
Frame (fps) 4.9 # 15
Semantic Segmentation CamVid DeepLab-MSc-CRF-LargeFOV Mean IoU 61.6% # 14
Semantic Segmentation Cityscapes test DeepLab Mean IoU (class) 63.1% # 89
Real-Time Semantic Segmentation Cityscapes test DeepLab mIoU 63.1% # 35
Time (ms) 4000 # 25
Frame (fps) 0.25 # 30
Semantic Segmentation PASCAL VOC 2012 test DeepLab-MSc-CRF-LargeFOV (VGG-16) Mean IoU 71.6% # 46
Scene Segmentation SUN-RGBD DeepLab-LargeFOV Mean IoU 32.08 # 3