Towards Robust Blind Face Restoration with Codebook Lookup Transformer

22 Jun 2022  Β·  Shangchen Zhou, Kelvin C. K. Chan, Chongyi Li, Chen Change Loy Β·

Blind face restoration is a highly ill-posed problem that often requires auxiliary guidance to 1) improve the mapping from degraded inputs to desired outputs, or 2) complement high-quality details lost in the inputs. In this paper, we demonstrate that a learned discrete codebook prior in a small proxy space largely reduces the uncertainty and ambiguity of restoration mapping by casting blind face restoration as a code prediction task, while providing rich visual atoms for generating high-quality faces. Under this paradigm, we propose a Transformer-based prediction network, named CodeFormer, to model the global composition and context of the low-quality faces for code prediction, enabling the discovery of natural faces that closely approximate the target faces even when the inputs are severely degraded. To enhance the adaptiveness for different degradation, we also propose a controllable feature transformation module that allows a flexible trade-off between fidelity and quality. Thanks to the expressive codebook prior and global modeling, CodeFormer outperforms the state of the arts in both quality and fidelity, showing superior robustness to degradation. Extensive experimental results on synthetic and real-world datasets verify the effectiveness of our method.

PDF Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Blind Face Restoration CelebA-Test CodeFormer LPIPS 29.9 # 1
FID 60.62 # 11
PSNR 22.18 # 10
SSIM 0.61 # 12
IDS 60 # 8
Blind Face Restoration LFW CodeFormer FID 52.02 # 5
Blind Face Restoration LFW PULSE FID 64.86 # 9
Blind Face Restoration LFW GPEN FID 57.58 # 7
Blind Face Restoration LFW GFP-GAN FID 49.96 # 3
Blind Face Restoration LFW GLEAN FID 53.49 # 6
Blind Face Restoration LFW PSFRGAN FID 51.89 # 4
Blind Face Restoration LFW DFDNet FID 62.57 # 8
Blind Face Restoration WIDER CodeFormer FID 39.06 # 3
Blind Face Restoration WIDER PSFRGAN FID 51.16 # 7
Blind Face Restoration WIDER GPEN FID 46.99 # 5
Blind Face Restoration WIDER PULSE FID 73.59 # 9
Blind Face Restoration WIDER GFP-GAN FID 40.59 # 4
Blind Face Restoration WIDER DFDNet FID 57.84 # 8
Blind Face Restoration WIDER GLEAN FID 47.11 # 6


No methods listed for this paper. Add relevant methods here