NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion

24 Nov 2021  ·  Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, Nan Duan ·

This paper presents a unified multimodal pre-trained model called N\"UWA that can generate new or manipulate existing visual data (i.e., images and videos) for various visual synthesis tasks. To cover language, image, and video at the same time for different scenarios, a 3D transformer encoder-decoder framework is designed, which can not only deal with videos as 3D data but also adapt to texts and images as 1D and 2D data, respectively. A 3D Nearby Attention (3DNA) mechanism is also proposed to consider the nature of the visual data and reduce the computational complexity. We evaluate N\"UWA on 8 downstream tasks. Compared to several strong baselines, N\"UWA achieves state-of-the-art results on text-to-image generation, text-to-video generation, video prediction, etc. Furthermore, it also shows surprisingly good zero-shot capabilities on text-guided image and video manipulation tasks. Project repo is https://github.com/microsoft/NUWA.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Video Generation BAIR Robot Pushing NUWA FVD score 86.9 # 3
Cond 1 # 1
Pred 15 # 8
Train 15 # 2
Text-to-Video Generation Kinetics NUWA (128×128) Accuracy 77.9 # 1
Text-to-Image Generation MS COCO NÜWA (256 x 256) FID 12.9 # 38
Inception score 27.2 # 13
Text-to-Image Generation MS COCO XMC-GAN (256 x 256) FID 9.3 # 26
Inception score 30.5 # 9
Text-to-Image Generation MS COCO CogView (256 x 256) FID 27.1 # 56
Inception score 18.2 # 20
Text-to-Image Generation MS COCO DALL-E (256 x 256) FID 27.5 # 58
Inception score 17.9 # 22
Text-to-Image Generation MS COCO DF-GAN (256 x 256) Inception score 18.7 # 19
Text-to-Image Generation MS COCO AttnGAN (256 x 256) FID 35.2 # 65
Inception score 23.3 # 18
Text-to-Image Generation MS COCO DM-GAN (256 x 256) FID 26.0 # 54
Inception score 32.2 # 7
Text-to-Video Generation MSR-VTT NUWA FID 47.68 # 9
CLIPSIM 0.2439 # 15
CLIP-FID 47.68 # 6

Methods