Collaborative Transformers for Grounded Situation Recognition

CVPR 2022  ·  Junhyeong Cho, Youngseok Yoon, Suha Kwak ·

Grounded situation recognition is the task of predicting the main activity, entities playing certain roles within the activity, and bounding-box groundings of the entities in the given image. To effectively deal with this challenging task, we introduce a novel approach where the two processes for activity classification and entity estimation are interactive and complementary. To implement this idea, we propose Collaborative Glance-Gaze TransFormer (CoFormer) that consists of two modules: Glance transformer for activity classification and Gaze transformer for entity estimation. Glance transformer predicts the main activity with the help of Gaze transformer that analyzes entities and their relations, while Gaze transformer estimates the grounded entities by focusing only on the entities relevant to the activity predicted by Glance transformer. Our CoFormer achieves the state of the art in all evaluation metrics on the SWiG dataset. Training code and model weights are available at https://github.com/jhcho99/CoFormer.

PDF Abstract CVPR 2022 PDF CVPR 2022 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Situation Recognition imSitu CoFormer Top-1 Verb 44.66 # 1
Top-1 Verb & Value 35.98 # 1
Top-5 Verbs 73.31 # 1
Top-5 Verbs & Value 57.76 # 1
Grounded Situation Recognition SWiG CoFormer Top-1 Verb 44.66 # 1
Top-1 Verb & Value 35.98 # 1
Top-1 Verb & Grounded-Value 29.05 # 2
Top-5 Verbs 73.31 # 1
Top-5 Verbs & Value 57.76 # 1
Top-5 Verbs & Grounded-Value 46.25 # 1

Methods