NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion

24 Nov 2021  ·  Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, Nan Duan ·

This paper presents a unified multimodal pre-trained model called N\"UWA that can generate new or manipulate existing visual data (i.e., images and videos) for various visual synthesis tasks. To cover language, image, and video at the same time for different scenarios, a 3D transformer encoder-decoder framework is designed, which can not only deal with videos as 3D data but also adapt to texts and images as 1D and 2D data, respectively. A 3D Nearby Attention (3DNA) mechanism is also proposed to consider the nature of the visual data and reduce the computational complexity. We evaluate N\"UWA on 8 downstream tasks. Compared to several strong baselines, N\"UWA achieves state-of-the-art results on text-to-image generation, text-to-video generation, video prediction, etc. Furthermore, it also shows surprisingly good zero-shot capabilities on text-guided image and video manipulation tasks. Project repo is https://github.com/microsoft/NUWA.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Video Generation BAIR Robot Pushing NUWA FVD score 86.9 # 2
Cond 1 # 1
Pred 15 # 8
Train 15 # 2
Text-to-Image Generation COCO 256 x 256 CogView FID-0 27.1 # 5
IS 18.2 # 6
Text-to-Image Generation COCO 256 x 256 AttnGAN FID-0 35.2 # 7
IS 23.3 # 4
Text-to-Image Generation COCO 256 x 256 DM-GAN FID-0 26.0 # 3
IS 32.2 # 1
Text-to-Image Generation COCO 256 x 256 DF-GAN FID-0 26.0 # 3
IS 18.7 # 5
Text-to-Image Generation COCO 256 x 256 DALL-E FID-0 27.5 # 6
IS 17.9 # 7
Text-to-Image Generation COCO 256 x 256 XMC-GAN FID-0 9.3 # 1
IS 30.5 # 2
Text-to-Image Generation COCO 256 x 256 NÜWA FID-0 12.9 # 2
IS 27.2 # 3
Text-to-Video Generation Kinetics NUWA (128×128) Accuracy 77.9 # 1

Methods