Semantic Correspondence as an Optimal Transport Problem

CVPR 2020  ·  Yanbin Liu, Linchao Zhu, Makoto Yamada, Yi Yang ·

Establishing dense correspondences across semantically similar images is a challenging task. Due to the large intra-class variation and background clutter, two common issues occur in current approaches... First, many pixels in a source image are assigned to one target pixel, i.e., many to one matching. Second, some object pixels are assigned to the background pixels, i.e., background matching. We solve the first issue by global feature matching, which maximizes the total matching correlations between images to obtain a global optimal matching matrix. The row sum and column sum constraints are enforced on the matching matrix to induce a balanced solution, thus suppressing the many to one matching. We solve the second issue by applying a staircase function on the class activation maps to re-weight the importance of pixels into four levels from foreground to background. The whole procedure is combined into a unified optimal transport algorithm by converting the maximization problem to the optimal transport formulation and incorporating the staircase weights into optimal transport algorithm to act as empirical distributions. The proposed algorithm achieves state-of-the-art performance on four benchmark datasets. Notably, a 26% relative improvement is achieved on the large-scale SPair-71k dataset. read more

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Semantic correspondence PF-PASCAL SCOT PCK 88.8 # 4
Semantic correspondence PF-WILLOW SCOT PCK 78.1 # 3
Semantic correspondence SPair-71k SCOT PCK 35.6 # 4

Methods


No methods listed for this paper. Add relevant methods here