Neural Nearest Neighbors Networks

NeurIPS 2018  ·  Tobias Plötz, Stefan Roth ·

Non-local methods exploiting the self-similarity of natural signals have been well studied, for example in image analysis and restoration. Existing approaches, however, rely on k-nearest neighbors (KNN) matching in a fixed feature space. The main hurdle in optimizing this feature space w.r.t. application performance is the non-differentiability of the KNN selection rule. To overcome this, we propose a continuous deterministic relaxation of KNN selection that maintains differentiability w.r.t. pairwise distances, but retains the original KNN as the limit of a temperature parameter approaching zero. To exploit our relaxation, we propose the neural nearest neighbors block (N3 block), a novel non-local processing layer that leverages the principle of self-similarity and can be used as building block in modern neural network architectures. We show its effectiveness for the set reasoning task of correspondence classification as well as for image restoration, including image denoising and single image super-resolution, where we outperform strong convolutional neural network (CNN) baselines and recent non-local models that rely on KNN selection in hand-chosen features spaces.

PDF Abstract NeurIPS 2018 PDF NeurIPS 2018 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Grayscale Image Denoising BSD68 sigma25 N3Net PSNR 29.3 # 8
Grayscale Image Denoising BSD68 sigma50 N3Net PSNR 26.39 # 8
Grayscale Image Denoising BSD68 sigma70 N3Net PSNR 25.14 # 1
Grayscale Image Denoising Set12 sigma25 N3Net PSNR 30.55 # 4
Grayscale Image Denoising Set12 sigma50 N3Net PSNR 27.43 # 5
Grayscale Image Denoising Set12 sigma70 N3Net PSNR 25.9 # 1
Image Super-Resolution Set5 - 2x upscaling N3Net PSNR 37.57 # 22
Image Super-Resolution Set5 - 3x upscaling N3Net PSNR 33.84 # 16
Image Super-Resolution Set5 - 4x upscaling N3Net PSNR 31.5 # 43
Grayscale Image Denoising Urban100 sigma25 N3Net PSNR 30.19 # 8
SSIM 0.892 # 1
Grayscale Image Denoising Urban100 sigma50 N3Net PSNR 26.82 # 9
Grayscale Image Denoising Urban100 sigma70 N3Net PSNR 25.15 # 2


No methods listed for this paper. Add relevant methods here