Taming Transformers for High-Resolution Image Synthesis

CVPR 2021  ·  Patrick Esser, Robin Rombach, Björn Ommer ·

Designed to learn long-range interactions on sequential data, transformers continue to show state-of-the-art results on a wide variety of tasks. In contrast to CNNs, they contain no inductive bias that prioritizes local interactions. This makes them expressive, but also computationally infeasible for long sequences, such as high-resolution images. We demonstrate how combining the effectiveness of the inductive bias of CNNs with the expressivity of transformers enables them to model and thereby synthesize high-resolution images. We show how to (i) use CNNs to learn a context-rich vocabulary of image constituents, and in turn (ii) utilize transformers to efficiently model their composition within high-resolution images. Our approach is readily applied to conditional synthesis tasks, where both non-spatial information, such as object classes, and spatial information, such as segmentations, can control the generated image. In particular, we present the first results on semantically-guided synthesis of megapixel images with transformers and obtain the state of the art among autoregressive models on class-conditional ImageNet. Code and pretrained models can be found at https://github.com/CompVis/taming-transformers .

PDF Abstract CVPR 2021 PDF CVPR 2021 Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Image-to-Image Translation ADE20K Labels-to-Photos VQGAN+Transformer FID 35.5 # 11
Image Generation CelebA 256x256 VQGAN FID 10.2 # 4
Image Generation CelebA-HQ 256x256 VQGAN+Transformer FID 10.2 # 11
Image-to-Image Translation COCO-Stuff Labels-to-Photos VQGAN+Transformer FID 22.4 # 8
Text-to-Image Generation Conceptual Captions VQ-GAN FID 28.86 # 5
DeepFake Detection FakeAVCeleb VQGAN ROC AUC 51.8 # 9
AP 55.0 # 9
Image Generation FFHQ 256 x 256 VQGAN+Transformer FID 9.6 # 26
Image Generation ImageNet 256x256 VQGAN+Transformer (k=600, p=1.0, a=0.05) FID 5.2 # 39
Image Generation ImageNet 256x256 VQGAN+Transformer (k=mixed, p=1.0, a=0.005) FID 6.59 # 41
Image Outpainting LHQC Taming Block-FID (Right Extend) 22.53 # 4
Block-FID (Left Extend) - # 4
Block-FID (Down Extend) 26.38 # 4
Block-FID (Up Extend) - # 4
Text-to-Image Generation LHQC Taming Block-FID 38.89 # 3