Using Self-Supervised Auxiliary Tasks to Improve Fine-Grained Facial Representation
In this paper, at first, the impact of ImageNet pre-training on fine-grained Facial Emotion Recognition (FER) is investigated which shows that when enough augmentations on images are applied, training from scratch provides better result than fine-tuning on ImageNet pre-training. Next, we propose a method to improve fine-grained and in-the-wild FER, called Hybrid Multi-Task Learning (HMTL). HMTL uses Self-Supervised Learning (SSL) as an auxiliary task during classical Supervised Learning (SL) in the form of Multi-Task Learning (MTL). Leveraging SSL during training can gain additional information from images for the primary fine-grained SL task. We investigate how proposed HMTL can be used in the FER domain by designing two customized version of common pre-text task techniques, puzzling and in-painting. We achieve state-of-the-art results on the AffectNet benchmark via two types of HMTL, without utilizing pre-training on additional data. Experimental results on the common SSL pre-training and proposed HMTL demonstrate the difference and superiority of our work. However, HMTL is not only limited to FER domain. Experiments on two types of fine-grained facial tasks, i.e., head pose estimation and gender recognition, reveals the potential of using HMTL to improve fine-grained facial representation.
PDF AbstractDatasets
Results from the Paper
Ranked #4 on Facial Expression Recognition (FER) on CK+ (Accuracy (7 emotion) metric, using extra training data)
Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Uses Extra Training Data |
Benchmark |
---|---|---|---|---|---|---|---|
Facial Expression Recognition (FER) | AffectNet | SL + 20% train (B0) | Accuracy (8 emotion) | 52.46 | # 30 | ||
Facial Expression Recognition (FER) | AffectNet | SL+ SSL puzzling + 20% train (B0) | Accuracy (8 emotion) | 54.98 | # 28 | ||
Facial Expression Recognition (FER) | AffectNet | SL+ SSL in-painting-pl + 20% train (B0) | Accuracy (8 emotion) | 55.36 | # 27 | ||
Facial Expression Recognition (FER) | AffectNet | SL (B0) | Accuracy (8 emotion) | 60.34 | # 18 | ||
Facial Expression Recognition (FER) | AffectNet | SL (B2) | Accuracy (8 emotion) | 60.35 | # 17 | ||
Facial Expression Recognition (FER) | AffectNet | SL + SSL puzzling (B0) | Accuracy (8 emotion) | 61.09 | # 14 | ||
Facial Expression Recognition (FER) | AffectNet | SL + SSL puzzling (B2) | Accuracy (8 emotion) | 61.32 | # 12 | ||
Facial Expression Recognition (FER) | AffectNet | SL + SSL in-panting-pl (B0) | Accuracy (8 emotion) | 61.72 | # 10 | ||
Facial Expression Recognition (FER) | CK+ | Nonlinear eval on SL + SSL puzzling (B0) | Accuracy (7 emotion) | 98.23 | # 4 |