Recovering the Unbiased Scene Graphs from the Biased Ones

5 Jul 2021  ·  Meng-Jiun Chiou, Henghui Ding, Hanshu Yan, Changhu Wang, Roger Zimmermann, Jiashi Feng ·

Given input images, scene graph generation (SGG) aims to produce comprehensive, graphical representations describing visual relationships among salient objects. Recently, more efforts have been paid to the long tail problem in SGG; however, the imbalance in the fraction of missing labels of different classes, or reporting bias, exacerbating the long tail is rarely considered and cannot be solved by the existing debiasing methods. In this paper we show that, due to the missing labels, SGG can be viewed as a "Learning from Positive and Unlabeled data" (PU learning) problem, where the reporting bias can be removed by recovering the unbiased probabilities from the biased ones by utilizing label frequencies, i.e., the per-class fraction of labeled, positive examples in all the positive examples. To obtain accurate label frequency estimates, we propose Dynamic Label Frequency Estimation (DLFE) to take advantage of training-time data augmentation and average over multiple training iterations to introduce more valid examples. Extensive experiments show that DLFE is more effective in estimating label frequencies than a naive variant of the traditional estimate, and DLFE significantly alleviates the long tail and achieves state-of-the-art debiasing performance on the VG dataset. We also show qualitatively that SGG models with DLFE produce prominently more balanced and unbiased scene graphs.

PDF Abstract

Datasets


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Unbiased Scene Graph Generation Visual Genome DLFE (MOTIFS-ResNeXt-101-FPN backbone; PredCls mode) ng-mR@20 30.0 # 1
mR@20 22.1 # 2
Scene Graph Generation Visual Genome DLFE Recall@50 25.4 # 8
Unbiased Scene Graph Generation Visual Genome DLFE (VCTree-ResNeXt-101-FPN backbone; SGDet mode) ng-mR@20 11.8 # 12
mR@20 8.6 # 19
Unbiased Scene Graph Generation Visual Genome DLFE (MOTIFS-ResNeXt-101-FPN backbone; SGDet mode) ng-mR@20 11.7 # 13
mR@20 8.6 # 19
Unbiased Scene Graph Generation Visual Genome DLFE (VCTree-ResNeXt-101-FPN backbone; SGCls mode) ng-mR@20 21.6 # 5
mR@20 15.8 # 10
Unbiased Scene Graph Generation Visual Genome DLFE (MOTIFS-ResNeXt-101-FPN backbone; SGCls mode) ng-mR@20 17.6 # 8
mR@20 12.8 # 12
Unbiased Scene Graph Generation Visual Genome DLFE (VCTree-ResNeXt-101-FPN backbone; PredCls mode) ng-mR@20 29.1 # 2
mR@20 20.8 # 5

Methods


No methods listed for this paper. Add relevant methods here