Dense Gaussian Processes for Few-Shot Segmentation

7 Oct 2021  ยท  Joakim Johnander, Johan Edstedt, Michael Felsberg, Fahad Shahbaz Khan, Martin Danelljan ยท

Few-shot segmentation is a challenging dense prediction task, which entails segmenting a novel query image given only a small annotated support set. The key problem is thus to design a method that aggregates detailed information from the support set, while being robust to large variations in appearance and context. To this end, we propose a few-shot segmentation method based on dense Gaussian process (GP) regression. Given the support set, our dense GP learns the mapping from local deep image features to mask values, capable of capturing complex appearance distributions. Furthermore, it provides a principled means of capturing uncertainty, which serves as another powerful cue for the final segmentation, obtained by a CNN decoder. Instead of a one-dimensional mask output, we further exploit the end-to-end learning capabilities of our approach to learn a high-dimensional output space for the GP. Our approach sets a new state-of-the-art on the PASCAL-5$^i$ and COCO-20$^i$ benchmarks, achieving an absolute gain of $+8.4$ mIoU in the COCO-20$^i$ 5-shot setting. Furthermore, the segmentation quality of our approach scales gracefully when increasing the support set size, while achieving robust cross-dataset transfer. Code and trained models are available at \url{https://github.com/joakimjohnander/dgpnet}.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Few-Shot Semantic Segmentation COCO-20i (10-shot) DGPNet (ResNet-101) Mean IoU 60.2 # 1
Few-Shot Semantic Segmentation COCO-20i (1-shot) DGPNet (ResNet-101) Mean IoU 46.7 # 18
Few-Shot Semantic Segmentation COCO-20i (1-shot) DGPNet (ResNet-50) Mean IoU 45 # 30
Few-Shot Semantic Segmentation COCO-20i (5-shot) DGPNet (ResNet-50) Mean IoU 56.2 # 11
Few-Shot Semantic Segmentation COCO-20i (5-shot) DGPNet (ResNet-101) Mean IoU 57.9 # 7
Few-Shot Semantic Segmentation COCO-20i -> Pascal VOC (1-shot) DGPNet (ResNet-101) Mean IoU 70.1 # 3
Few-Shot Semantic Segmentation COCO-20i -> Pascal VOC (1-shot) DGPNet (ResNet-50) Mean IoU 68.9 # 6
Few-Shot Semantic Segmentation COCO-20i -> Pascal VOC (5-shot) DGPNet (ResNet-101) Mean IoU 78.5 # 2
Few-Shot Semantic Segmentation COCO-20i -> Pascal VOC (5-shot) DGPNet (ResNet-50) Mean IoU 77.5 # 3
Few-Shot Semantic Segmentation PASCAL-5i (10-Shot) DGPNet (ResNet-101) Mean IoU 77.7 # 1
Few-Shot Semantic Segmentation PASCAL-5i (1-Shot) DGPNet (ResNet-101) Mean IoU 64.8 # 49
Few-Shot Semantic Segmentation PASCAL-5i (1-Shot) DGPNet (ResNet-50) Mean IoU 63.5 # 63
Few-Shot Semantic Segmentation PASCAL-5i (5-Shot) DGPNet (ResNet-101) Mean IoU 75.4 # 5
Few-Shot Semantic Segmentation PASCAL-5i (5-Shot) DGPNet (ResNet-50) Mean IoU 73.5 # 11

Methods