HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling
Volumetric scene representations enable photorealistic view synthesis for static scenes and form the basis of several existing 6-DoF video techniques. However, the volume rendering procedures that drive these representations necessitate careful trade-offs in terms of quality, rendering speed, and memory efficiency. In particular, existing methods fail to simultaneously achieve real-time performance, small memory footprint, and high-quality rendering for challenging real-world scenes. To address these issues, we present HyperReel -- a novel 6-DoF video representation. The two core components of HyperReel are: (1) a ray-conditioned sample prediction network that enables high-fidelity, high frame rate rendering at high resolutions and (2) a compact and memory-efficient dynamic volume representation. Our 6-DoF video pipeline achieves the best performance compared to prior and contemporary approaches in terms of visual quality with small memory requirements, while also rendering at up to 18 frames-per-second at megapixel resolution without any custom CUDA code.
PDF Abstract CVPR 2023 PDF CVPR 2023 AbstractCode
Datasets
DONeRF: Evaluation Dataset
Results from the Paper
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Benchmark |
|---|---|---|---|---|---|---|
| Novel View Synthesis | DONeRF: Evaluation Dataset | HyperReel | PSNR | 35.1 | # 1 | |
| Novel View Synthesis | DONeRF: Evaluation Dataset | TermiNeRF | PSNR | 29.8 | # 6 | |
| Novel View Synthesis | DONeRF: Evaluation Dataset | AdaNeRF | PSNR | 30.9 | # 3 | |
| Novel View Synthesis | DONeRF: Evaluation Dataset | DoNeRF | PSNR | 30.8 | # 5 | |
| Novel View Synthesis | DONeRF: Evaluation Dataset | Instant NGP | PSNR | 33.1 | # 2 | |
| Novel View Synthesis | DONeRF: Evaluation Dataset | NeRF | PSNR | 30.9 | # 3 | |
| Novel View Synthesis | LLFF | HyperReel | PSNR | 26.2 | # 9 | |
| Novel View Synthesis | LLFF | TermiNeRF | PSNR | 23.6 | # 14 | |
| Novel View Synthesis | LLFF | AdaNeRF | PSNR | 25.7 | # 11 | |
| Novel View Synthesis | LLFF | DoNeRF | PSNR | 22.9 | # 15 | |
| Novel View Synthesis | LLFF | Instant NGP | PSNR | 25.6 | # 13 |
