Palette: Image-to-Image Diffusion Models

This paper develops a unified framework for image-to-image translation based on conditional diffusion models and evaluates this framework on four challenging image-to-image translation tasks, namely colorization, inpainting, uncropping, and JPEG restoration. Our simple implementation of image-to-image diffusion models outperforms strong GAN and regression baselines on all tasks, without task-specific hyper-parameter tuning, architecture customization, or any auxiliary loss or sophisticated new techniques needed. We uncover the impact of an L2 vs. L1 loss in the denoising diffusion objective on sample diversity, and demonstrate the importance of self-attention in the neural architecture through empirical studies. Importantly, we advocate a unified evaluation protocol based on ImageNet, with human evaluation and sample quality scores (FID, Inception Score, Classification Accuracy of a pre-trained ResNet-50, and Perceptual Distance against original images). We expect this standardized evaluation protocol to play a role in advancing image-to-image translation research. Finally, we show that a generalist, multi-task diffusion model performs as well or better than task-specific specialist counterparts. Check out https://diffusion-palette.github.io for an overview of the results.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
JPEG Decompression ImageNet Palette (QF: 5) FID-5K 8.3 # 3
IS 133.6 # 4
CA 64.2 # 4
PD 95.5 # 4
JPEG Decompression ImageNet Palette (QF: 20) FID-5K 4.3 # 1
IS 208.7 # 1
CA 73.5 # 1
PD 37.1 # 1
JPEG Decompression ImageNet Regression (QF: 20) FID-5K 11.5 # 4
IS 158.7 # 3
CA 69.7 # 3
PD 65.4 # 3
JPEG Decompression ImageNet Palette (QF: 10) FID-5K 5.4 # 2
IS 180.5 # 2
CA 70.7 # 2
PD 58.3 # 2
JPEG Decompression ImageNet Regression (QF: 10) FID-5K 18.0 # 5
IS 117.2 # 5
CA 63.5 # 5
PD 102.2 # 5
JPEG Decompression ImageNet Regression (QF: 5) FID-5K 29.0 # 6
IS 73.9 # 6
CA 52.8 # 6
PD 155.4 # 6
Colorization ImageNet ctest10k Palette FID 3.4 # 1
Colorization ImageNet val Palette FID-5K 15.78 # 1
Uncropping Places2 val Palette FID 3.53 # 1
PD 103.3 # 1
Fool rate 39.9 # 1
Image Inpainting Places2 val Palatte (20-30% free form) FID 11.7 # 1
PD 35.0 # 1
Image Inpainting Places2 val Palette (128ร—128 center mask) FID 11.9 # 2
PD 57.3 # 2

Methods