NeRFLiX: High-Quality Neural View Synthesis by Learning a Degradation-Driven Inter-viewpoint MiXer

Neural radiance fields (NeRF) show great success in novel view synthesis. However, in real-world scenes, recovering high-quality details from the source images is still challenging for the existing NeRF-based approaches, due to the potential imperfect calibration information and scene representation inaccuracy. Even with high-quality training frames, the synthetic novel views produced by NeRF models still suffer from notable rendering artifacts, such as noise, blur, etc. Towards to improve the synthesis quality of NeRF-based approaches, we propose NeRFLiX, a general NeRF-agnostic restorer paradigm by learning a degradation-driven inter-viewpoint mixer. Specially, we design a NeRF-style degradation modeling approach and construct large-scale training data, enabling the possibility of effectively removing NeRF-native rendering artifacts for existing deep neural networks. Moreover, beyond the degradation removal, we propose an inter-viewpoint aggregation framework that is able to fuse highly related high-quality training images, pushing the performance of cutting-edge NeRF models to entirely new levels and producing highly photo-realistic synthetic views.

PDF Abstract CVPR 2023 PDF CVPR 2023 Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Novel View Synthesis LLFF TensoRF + NeRFLiX PSNR 27.39 # 1
SSIM 0.867 # 1
LPIPS 0.149 # 6
Novel View Synthesis LLFF Plenoxels + NeRFLiX PSNR 26.9 # 4
SSIM 0.864 # 2
LPIPS 0.156 # 5
Novel View Synthesis Tanks and Temples DIVeR + NeRFLiX SSIM 0.924 # 2
Novel View Synthesis Tanks and Temples Plenoxels + NeRFLiX PSNR 28.61 # 3
Novel View Synthesis Tanks and Temples TensoRF + NeRFLiX PSNR 28.94 # 2
SSIM 0.93 # 1