Fine-Grained Scene Graph Generation with Data Transfer

22 Mar 2022  ·  Ao Zhang, Yuan YAO, Qianyu Chen, Wei Ji, Zhiyuan Liu, Maosong Sun, Tat-Seng Chua ·

Scene graph generation (SGG) is designed to extract (subject, predicate, object) triplets in images. Recent works have made a steady progress on SGG, and provide useful tools for high-level vision and language understanding. However, due to the data distribution problems including long-tail distribution and semantic ambiguity, the predictions of current SGG models tend to collapse to several frequent but uninformative predicates (e.g., on, at), which limits practical application of these models in downstream tasks. To deal with the problems above, we propose a novel Internal and External Data Transfer (IETrans) method, which can be applied in a plug-and-play fashion and expanded to large SGG with 1,807 predicate classes. Our IETrans tries to relieve the data distribution problem by automatically creating an enhanced dataset that provides more sufficient and coherent annotations for all predicates. By training on the enhanced dataset, a Neural Motif model doubles the macro performance while maintaining competitive micro performance. The code and data are publicly available at https://github.com/waxnkw/IETrans-SGG.pytorch.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Predicate Classification Visual Genome IETrans R@100 50.5 # 1
R@20 40.3 # 1
R@50 48.6 # 1
mean Recall @20 28.9 # 2
Scene Graph Detection Visual Genome IETrans R@100 27.2 # 2
R@20 17.5 # 2
R@50 23.5 # 2
Scene Graph Classification Visual Genome IETrans R@100 30.2 # 2
R@20 26.1 # 2
R@50 29.4 # 2
Scene Graph Generation Visual Genome IETrans Recall@50 23.5 # 9
Recall@100 27.2 # 2
mean Recall @100 18.0 # 1
Unbiased Scene Graph Generation Visual Genome IETrans (MOTIFS-ResNeXt-101-FPN backbone; PredCls mode) ng-mR@20 36.0 # 1
mR@20 28.9 # 2
Unbiased Scene Graph Generation Visual Genome IETrans (MOTIFS-ResNeXt-101-FPN backbone; SGCls mode) ng-mR@20 21.8 # 6
mR@20 17.5 # 10
Unbiased Scene Graph Generation Visual Genome IETrans (MOTIFS-ResNeXt-101-FPN backbone; SGDet mode) ng-mR@20 13.4 # 12
mR@20 10.9 # 18

Methods


No methods listed for this paper. Add relevant methods here