Learning to Fuse Things and Stuff

4 Dec 2018  ·  Jie Li, Allan Raventos, Arjun Bhargava, Takaaki Tagawa, Adrien Gaidon ·

We propose an end-to-end learning approach for panoptic segmentation, a novel task unifying instance (things) and semantic (stuff) segmentation. Our model, TASCNet, uses feature maps from a shared backbone network to predict in a single feed-forward pass both things and stuff segmentations. We explicitly constrain these two output distributions through a global things and stuff binary mask to enforce cross-task consistency. Our proposed unified network is competitive with the state of the art on several benchmarks for panoptic segmentation as well as on the individual semantic and instance segmentation tasks.

PDF Abstract

Results from the Paper


Ranked #20 on Panoptic Segmentation on Cityscapes val (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Panoptic Segmentation Cityscapes val TASCNet (ResNet-50) PQ 59.2 # 22
PQst 61.5 # 16
PQth 56 # 9
mIoU 77.8 # 18
AP 37.6 # 16
Panoptic Segmentation Cityscapes val TASCNet (ResNet-50, multi-scale) PQ 60.4 # 20
PQst 63.3 # 9
PQth 56.1 # 8
mIoU 78 # 17
AP 39 # 12
Panoptic Segmentation COCO test-dev TASCNet PQ 40.7 # 32
PQst 31.0 # 29
PQth 47.0 # 28

Methods