Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning

ICCV 2019  ·  Shichen Liu, Tianye Li, Weikai Chen, Hao Li ·

Rendering bridges the gap between 2D vision and 3D scenes by simulating the physical process of image formation. By inverting such renderer, one can think of a learning approach to infer 3D information from 2D images. However, standard graphics renderers involve a fundamental discretization step called rasterization, which prevents the rendering process to be differentiable, hence able to be learned. Unlike the state-of-the-art differentiable renderers, which only approximate the rendering gradient in the back propagation, we propose a truly differentiable rendering framework that is able to (1) directly render colorized mesh using differentiable functions and (2) back-propagate efficient supervision signals to mesh vertices and their attributes from various forms of image representations, including silhouette, shading and color images. The key to our framework is a novel formulation that views rendering as an aggregation function that fuses the probabilistic contributions of all mesh triangles with respect to the rendered pixels. Such formulation enables our framework to flow gradients to the occluded and far-range vertices, which cannot be achieved by the previous state-of-the-arts. We show that by using the proposed renderer, one can achieve significant improvement in 3D unsupervised single-view reconstruction both qualitatively and quantitatively. Experiments also demonstrate that our approach is able to handle the challenging tasks in image-based shape fitting, which remain nontrivial to existing differentiable renderers.

PDF Abstract ICCV 2019 PDF ICCV 2019 Abstract


Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
3D Object Reconstruction ShapeNet SoftRas (full) 3DIoU 0.6464 # 1
Single-View 3D Reconstruction ShapeNet SoftRas (full) 3DIoU 0.6464 # 3
Single-View 3D Reconstruction ShapeNet NMR [19] 3DIoU 0.6015 # 5
Single-View 3D Reconstruction ShapeNet voxel [47] 3DIoU 0.5736 # 6
Single-View 3D Reconstruction ShapeNet retrieval [47] 3DIoU 0.4766 # 7


No methods listed for this paper. Add relevant methods here