COTR: Correspondence Transformer for Matching Across Images

We propose a novel framework for finding correspondences in images based on a deep neural network that, given two images and a query point in one of them, finds its correspondence in the other. By doing so, one has the option to query only the points of interest and retrieve sparse correspondences, or to query all points in an image and obtain dense mappings. Importantly, in order to capture both local and global priors, and to let our model relate between image regions using the most relevant among said priors, we realize our network using a transformer. At inference time, we apply our correspondence network by recursively zooming in around the estimates, yielding a multiscale pipeline able to provide highly-accurate correspondences. Our method significantly outperforms the state of the art on both sparse and dense correspondence problems on multiple datasets and tasks, ranging from wide-baseline stereo to optical flow, without any retraining for a specific dataset. We commit to releasing data, code, and all the tools necessary to train from scratch and ensure reproducibility.

PDF Abstract ICCV 2021 PDF ICCV 2021 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Dense Pixel Correspondence Estimation ETH3D COTR +Interp. AEPE (rate=5) 1.71 # 1
Dense Pixel Correspondence Estimation ETH3D COTR AEPE (rate=3) 1.66 # 1
Dense Pixel Correspondence Estimation HPatches COTR Viewpoint I AEPE 7.75 # 6
PCK-1px 40.91 # 1
PCK-3px 82.37 # 1
PCK-5px 91.1 # 2
Dense Pixel Correspondence Estimation HPatches COTR +Interp. Viewpoint I AEPE 7.98 # 7
PCK-1px 33.08 # 2
PCK-3px 77.09 # 2
PCK-5px 86.33 # 3
Dense Pixel Correspondence Estimation KITTI 2012 COTR +Interp. Average End-Point Error 2.62 # 2
Dense Pixel Correspondence Estimation KITTI 2012 COTR Average End-Point Error 1.28 # 1
Dense Pixel Correspondence Estimation KITTI 2015 COTR +Interp. Average End-Point Error 6.12 # 2
Dense Pixel Correspondence Estimation KITTI 2015 COTR Average End-Point Error 2.26 # 1

Methods


No methods listed for this paper. Add relevant methods here