Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation

22 Jul 2022  ·  Sunghwan Hong, Seokju Cho, Jisu Nam, Stephen Lin, Seungryong Kim ·

This paper presents a novel cost aggregation network, called Volumetric Aggregation with Transformers (VAT), for few-shot segmentation. The use of transformers can benefit correlation map aggregation through self-attention over a global receptive field. However, the tokenization of a correlation map for transformer processing can be detrimental, because the discontinuity at token boundaries reduces the local context available near the token edges and decreases inductive bias. To address this problem, we propose a 4D Convolutional Swin Transformer, where a high-dimensional Swin Transformer is preceded by a series of small-kernel convolutions that impart local context to all pixels and introduce convolutional inductive bias. We additionally boost aggregation performance by applying transformers within a pyramidal structure, where aggregation at a coarser level guides aggregation at a finer level. Noise in the transformer output is then filtered in the subsequent decoder with the help of the query's appearance embedding. With this model, a new state-of-the-art is set for all the standard benchmarks in few-shot segmentation. It is shown that VAT attains state-of-the-art performance for semantic correspondence as well, where cost aggregation also plays a central role.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Few-Shot Semantic Segmentation COCO-20i (1-shot) VAT (ResNet-101) Mean IoU 41.3 # 48
FB-IoU 68.8 # 22
Few-Shot Semantic Segmentation COCO-20i (5-shot) VAT (ResNet-101) Mean IoU 47.9 # 42
FB-IoU 72.4 # 15
Few-Shot Semantic Segmentation FSS-1000 (1-shot) VAT (ResNet-101) Mean IoU 90.3 # 4
FB-IoU 94 # 1
Few-Shot Semantic Segmentation FSS-1000 (1-shot) VAT (ResNet-50) Mean IoU 90.1 # 6
FB-IoU 93.8 # 2
Few-Shot Semantic Segmentation FSS-1000 (5-shot) VAT (ResNet-101) Mean IoU 90.8 # 3
FB-IoU 94.4 # 1
Few-Shot Semantic Segmentation FSS-1000 (5-shot) VAT (ResNet-50) Mean IoU 90.7 # 4
FB-IoU 94.2 # 2
Few-Shot Semantic Segmentation PASCAL-5i (1-Shot) VAT (ResNet-50) Mean IoU 65.5 # 43
FB-IoU 77.8 # 23
Few-Shot Semantic Segmentation PASCAL-5i (1-Shot) VAT (ResNet-101) Mean IoU 67.9 # 18
FB-IoU 79.6 # 8
Few-Shot Semantic Segmentation PASCAL-5i (5-Shot) VAT (ResNet-50) Mean IoU 70.1 # 35
FB-IoU 80.9 # 21
Few-Shot Semantic Segmentation PASCAL-5i (5-Shot) VAT (ResNet-101) Mean IoU 72 # 16
FB-IoU 83.2 # 5
Semantic correspondence PF-PASCAL VAT (ECCV) PCK 92.3 # 3
Semantic correspondence PF-WILLOW VAT (ECCV) PCK 81.6 # 1
Semantic correspondence SPair-71k VAT (ECCV) PCK 55.5 # 5

Methods