Introspective Adversarial Network

The Introspective Adversarial Network (IAN) is a hybridization of GANs and VAEs that leverages the power of the adversarial objective while maintaining the VAE’s efficient inference mechanism. It uses the discriminator of the GAN, $D$, as a feature extractor for an inference subnetwork, $E$, which is implemented as a fully-connected layer on top of the final convolutional layer of the discriminator. We infer latent values $Z \sim E\left(X\right) = q\left(Z\mid{X}\right)$ for reconstruction and sample random values $Z \sim p\left(Z\right)$ from a standard normal for random image generation using the generator network, $G$.

Three distinct loss functions are used:

• $\mathcal{L}_{img}$, the L1 pixel-wise reconstruction loss, which is preferred to the L2 reconstruction loss for its higher average gradient.
• $\mathcal{L_{feature}}$, the feature-wise reconstruction loss, evaluated as the L2 difference between the original and reconstruction in the space of the hidden layers of the discriminator.
• $\mathcal{L}_{adv}$, the ternary adversarial loss, a modification of the adversarial loss that forces the discriminator to label a sample as real, generated, or reconstructed (as opposed to a binary real vs. generated label).

Including the VAE’s KL divergence between the inferred latents $E\left(X\right)$ and the prior $p\left(Z\right)$, the loss function for the generator and encoder network is thus:

$$\mathcal{L}_{E, G} = \lambda_{adv}\mathcal{L}_{G_{adv}} + \lambda_{img}\mathcal{L}_{img} + \lambda_{feature}\mathcal{L}_{feature} + D_{KL}\left(E\left(X\right) || p\left(Z\right)\right) $$

Where the $\lambda$ terms weight the relative importance of each loss. We set $\lambda_{img}$ to 3 and leave the other terms at 1. The discriminator is updated solely using the ternary adversarial loss. During each training step, the generator produces reconstructions $G\left(E\left(X\right)\right)$ (using the standard VAE reparameterization trick) from data $X$ and random samples $G\left(Z\right)$, while the discriminator observes $X$ as well as the reconstructions and random samples, and both networks are simultaneously updated.