Object-Centric Image Generation from Layouts

16 Mar 2020  ·  Tristan Sylvain, Pengchuan Zhang, Yoshua Bengio, R. Devon Hjelm, Shikhar Sharma ·

Despite recent impressive results on single-object and single-domain image generation, the generation of complex scenes with multiple objects remains challenging. In this paper, we start with the idea that a model must be able to understand individual objects and relationships between objects in order to generate complex scenes well. Our layout-to-image-generation method, which we call Object-Centric Generative Adversarial Network (or OC-GAN), relies on a novel Scene-Graph Similarity Module (SGSM). The SGSM learns representations of the spatial relationships between objects in the scene, which lead to our model's improved layout-fidelity. We also propose changes to the conditioning mechanism of the generator that enhance its object instance-awareness. Apart from improving image quality, our contributions mitigate two failure modes in previous approaches: (1) spurious objects being generated without corresponding bounding boxes in the layout, and (2) overlapping bounding boxes in the layout leading to merged objects in images. Extensive quantitative evaluation and ablation studies demonstrate the impact of our contributions, with our model outperforming previous state-of-the-art approaches on both the COCO-Stuff and Visual Genome datasets. Finally, we address an important limitation of evaluation metrics used in previous works by introducing SceneFID -- an object-centric adaptation of the popular Fr{\'e}chet Inception Distance metric, that is better suited for multi-object images.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Layout-to-Image Generation COCO-Stuff 128x128 OC-GAN FID 36.31 # 4
Inception Score 14.6 # 2
SceneFID 16.76 # 1
Layout-to-Image Generation COCO-Stuff 256x256 OC-GAN Inception Score 17.8 # 1
FID 41.65 # 1
Layout-to-Image Generation COCO-Stuff 64x64 OC-GAN FID 29.57 # 1
Inception Score 10.8 # 1
Layout-to-Image Generation Visual Genome 128x128 OC-GAN FID 28.26 # 2
Inception Score 12.3 # 2
SceneFID 9.63 # 1
Layout-to-Image Generation Visual Genome 256x256 OC-GAN Inception Score 14.7 # 1
FID 40.85 # 2
Layout-to-Image Generation Visual Genome 64x64 OC-GAN FID 20.27 # 1
Inception Score 9.3 # 1

Methods


No methods listed for this paper. Add relevant methods here