Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial Geometry

19 Oct 2021  ·  Cho-Ying Wu, Qiangeng Xu, Ulrich Neumann ·

This work studies learning from a synergy process of 3D Morphable Models (3DMM) and 3D facial landmarks to predict complete 3D facial geometry, including 3D alignment, face orientation, and 3D face modeling. Our synergy process leverages a representation cycle for 3DMM parameters and 3D landmarks. 3D landmarks can be extracted and refined from face meshes built by 3DMM parameters. We next reverse the representation direction and show that predicting 3DMM parameters from sparse 3D landmarks improves the information flow. Together we create a synergy process that utilizes the relation between 3D landmarks and 3DMM parameters, and they collaboratively contribute to better performance. We extensively validate our contribution on full tasks of facial geometry prediction and show our superior and robust performance on these tasks for various scenarios. Particularly, we adopt only simple and widely-used network operations to attain fast and accurate facial geometry prediction. Codes and data:

PDF Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Face Alignment AFLW SynergyNet Mean NME 4.06 # 1
Head Pose Estimation AFLW2000 SynergyNet MAE 3.35 # 2
Face Alignment AFLW2000-3D SynergyNet-Reannotated Mean NME 2.65% # 2
Face Alignment AFLW2000-3D SynergyNet Mean NME 3.41% # 4
3D Face Reconstruction NoW Benchmark SynergyNet Mean Reconstruction Error (mm) 1.59 # 13
Stdev Reconstruction Error (mm) 1.31 # 9
Median Reconstruction Error 1.27 # 13
3D Face Reconstruction REALY SynergyNet @nose 2.026 (±0.532) # 16
@mouth 1.731 (±0.502) # 10
@forehead 2.679 (±0.741) # 18
@cheek 1.647 (±0.622) # 14
all 2.021 # 14
3D Face Reconstruction REALY (side-view) SynergyNet @nose 2.008 (±0.526) # 14
all 2.008 # 9
@mouth 1.725 (±0.533) # 8
@forehead 2.638 (±0.719) # 15
@cheek 1.662 (±0.627) # 11


No methods listed for this paper. Add relevant methods here