Diffusion-GAN: Training GANs with Diffusion

5 Jun 2022  ·  Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, Mingyuan Zhou ·

Generative adversarial networks (GANs) are challenging to train stably, and a promising remedy of injecting instance noise into the discriminator input has not been very effective in practice. In this paper, we propose Diffusion-GAN, a novel GAN framework that leverages a forward diffusion chain to generate Gaussian-mixture distributed instance noise. Diffusion-GAN consists of three components, including an adaptive diffusion process, a diffusion timestep-dependent discriminator, and a generator. Both the observed and generated data are diffused by the same adaptive diffusion process. At each diffusion timestep, there is a different noise-to-data ratio and the timestep-dependent discriminator learns to distinguish the diffused real data from the diffused generated data. The generator learns from the discriminator's feedback by backpropagating through the forward diffusion chain, whose length is adaptively adjusted to balance the noise and data levels. We theoretically show that the discriminator's timestep-dependent strategy gives consistent and helpful guidance to the generator, enabling it to match the true data distribution. We demonstrate the advantages of Diffusion-GAN over strong GAN baselines on various datasets, showing that it can produce more realistic images with higher stability and data efficiency than state-of-the-art GANs.

PDF Abstract

Results from the Paper


 Ranked #1 on Image Generation on AFHQ Wild (FID metric)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Image Generation AFHQ Cat Diffusion InsGen FID 2.40 # 2
Image Generation AFHQ Dog Diffusion InsGen FID 4.83 # 3
Image Generation AFHQ Wild Diffusion InsGen FID 1.51 # 1
Image Generation CelebA 64x64 Diffusion StyleGAN2 FID 1.69 # 3
Image Generation CIFAR-10 Diffusion StyleGAN2 FID 3.19 # 31
Image Generation CIFAR-10 Diffusion StyleGAN2 + ADA FID 2.67 # 27
Image Generation FFHQ 1024 x 1024 Diffusion StyleGAN2 FID 2.83 # 5
Image Generation LSUN Bedroom 256 x 256 Diffusion ProjectedGAN FID 1.43 # 1
Image Generation LSUN Bedroom 256 x 256 Diffusion StyleGAN2 FID 3.65 # 5
Image Generation LSUN Churches 256 x 256 Diffusion StyleGAN2 FID 3.17 # 6
Image Generation LSUN Churches 256 x 256 Diffusion ProjectedGAN FID 1.85 # 3
Image Generation STL-10 Diffusion StyleGAN2 FID 11.53 # 3
Image Generation STL-10 Diffusion ProjectedGAN FID 6.91 # 1

Methods