Holistic 3D Scene Understanding from a Single Image with Implicit Representation

We present a new pipeline for holistic 3D scene understanding from a single image, which could predict object shapes, object poses, and scene layout. As it is a highly ill-posed problem, existing methods usually suffer from inaccurate estimation of both shapes and layout especially for the cluttered scene due to the heavy occlusion between objects. We propose to utilize the latest deep implicit representation to solve this challenge. We not only propose an image-based local structured implicit network to improve the object shape estimation, but also refine the 3D object pose and scene layout via a novel implicit scene graph neural network that exploits the implicit local object features. A novel physical violation loss is also proposed to avoid incorrect context between objects. Extensive experiments demonstrate that our method outperforms the state-of-the-art methods in terms of object shape, scene layout estimation, and 3D object detection.

PDF Abstract CVPR 2021 PDF CVPR 2021 Abstract


Results from the Paper

 Ranked #1 on Monocular 3D Object Detection on SUN RGB-D (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
3D Shape Reconstruction Pix3D IM3D CD 0.0672 # 1
EMD N/A # 3
IoU N/A # 2
Monocular 3D Object Detection SUN RGB-D IM3D AP@0.15 (10 / NYU-37) 45.21 # 1
AP@0.15 (NYU-37) 24.10 # 1
Room Layout Estimation SUN RGB-D IM3D IoU 64.4 # 1
Camera Pitch 2.98 # 2
Camera Roll 2.11 # 3