Tackling the Challenges in Scene Graph Generation with Local-to-Global Interactions

16 Jun 2021  ·  Sangmin Woo, Junhyug Noh, Kangil Kim ·

In this work, we seek new insights into the underlying challenges of the Scene Graph Generation (SGG) task. Quantitative and qualitative analysis of the Visual Genome dataset implies -- 1) Ambiguity: even if inter-object relationship contains the same object (or predicate), they may not be visually or semantically similar, 2) Asymmetry: despite the nature of the relationship that embodied the direction, it was not well addressed in previous studies, and 3) Higher-order contexts: leveraging the identities of certain graph elements can help to generate accurate scene graphs. Motivated by the analysis, we design a novel SGG framework, Local-to-Global Interaction Networks (LOGIN). Locally, interactions extract the essence between three instances of subject, object, and background, while baking direction awareness into the network by explicitly constraining the input order of subject and object. Globally, interactions encode the contexts between every graph component (i.e., nodes and edges). Finally, Attract & Repel loss is utilized to fine-tune the distribution of predicate embeddings. By design, our framework enables predicting the scene graph in a bottom-up manner, leveraging the possible complementariness. To quantify how much LOGIN is aware of relational direction, a new diagnostic task called Bidirectional Relationship Classification (BRC) is also proposed. Experimental results demonstrate that LOGIN can successfully distinguish relational direction than existing methods (in BRC task), while showing state-of-the-art results on the Visual Genome benchmark (in SGG task).

PDF Abstract


Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Predicate Classification Visual Genome LOGIN Recall@20 61.1 # 1
Recall@50 66.6 # 1
Recall@100 68.7 # 1
Scene Graph Classification Visual Genome LOGIN Recall@20 34.5 # 1
Recall@50 38.8 # 1
Recall@100 40.5 # 1
Bidirectional Relationship Classification Visual Genome LOGIN pair-wise Recall@2 8.6 # 1
pair-wise Recall@4 13.1 # 1
pair-wise Recall@8 17.6 # 1
pair-wise Recall@16 21.1 # 1
Scene Graph Generation Visual Genome LOGIN Recall@50 28.2 # 4
Recall@20 22.2 # 1
Recall@100 31.4 # 1