GAUDI: A Neural Architect for Immersive 3D Scene Generation

We introduce GAUDI, a generative model capable of capturing the distribution of complex and realistic 3D scenes that can be rendered immersively from a moving camera. We tackle this challenging problem with a scalable yet powerful approach, where we first optimize a latent representation that disentangles radiance fields and camera poses. This latent representation is then used to learn a generative model that enables both unconditional and conditional generation of 3D scenes. Our model generalizes previous works that focus on single objects by removing the assumption that the camera pose distribution can be shared across samples. We show that GAUDI obtains state-of-the-art performance in the unconditional generative setting across multiple datasets and allows for conditional generation of 3D scenes given conditioning variables like sparse image observations or text that describes the scene.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Image Generation ARKitScenes GAUDI FID 37.35 # 1
FID (SwAV) 4.14 # 4
Image Generation ARKitScenes GSN FID 79.54 # 2
FID (SwAV) 10.21 # 3
Image Generation ARKitScenes π-GAN FID 134.8 # 4
FID (SwAV) 15.58 # 1
Image Generation ARKitScenes GRAF FID 87.06 # 3
FID (SwAV) 13.44 # 2
Image Generation Replica GRAF FID 65.37 # 3
FID (SwAV) 5.76 # 2
Image Generation Replica π-GAN FID 166.55 # 4
FID (SwAV) 13.17 # 1
Image Generation Replica GSN FID 41.75 # 2
FID (SwAV) 4.14 # 3
Image Generation Replica GAUDI FID 18.75 # 1
FID (SwAV) 1.76 # 4
Image Generation VizDoom GAUDI FID 33.7 # 1
FID (SwAV) 3.24 # 4
Image Generation VizDoom GSN FID 37.21 # 2
FID (SwAV) 4.56 # 3
Image Generation VizDoom π-GAN FID 143.55 # 4
FID (SwAV) 15.26 # 1
Image Generation VizDoom GRAF FID 47.5 # 3
FID (SwAV) 5.44 # 2
Image Generation VLN-CE GRAF FID 90.43 # 3
FID (SwAV) 8.65 # 2
Image Generation VLN-CE GAUDI FID 18.52 # 1
FID (SwAV) 3.63 # 4
Image Generation VLN-CE π-GAN FID 151.26 # 4
FID (SwAV) 14.07 # 1
Image Generation VLN-CE GSN FID 43.32 # 2
FID (SwAV) 6.19 # 3

Methods


No methods listed for this paper. Add relevant methods here