Discovering Human-Object Interaction Concepts via Self-Compositional Learning

27 Mar 2022  ·  Zhi Hou, Baosheng Yu, DaCheng Tao ·

A comprehensive understanding of human-object interaction (HOI) requires detecting not only a small portion of predefined HOI concepts (or categories) but also other reasonable HOI concepts, while current approaches usually fail to explore a huge portion of unknown HOI concepts (i.e., unknown but reasonable combinations of verbs and objects). In this paper, 1) we introduce a novel and challenging task for a comprehensive HOI understanding, which is termed as HOI Concept Discovery; and 2) we devise a self-compositional learning framework (or SCL) for HOI concept discovery. Specifically, we maintain an online updated concept confidence matrix during training: 1) we assign pseudo-labels for all composite HOI instances according to the concept confidence matrix for self-training; and 2) we update the concept confidence matrix using the predictions of all composite HOI instances. Therefore, the proposed method enables the learning on both known and unknown HOI concepts. We perform extensive experiments on several popular HOI datasets to demonstrate the effectiveness of the proposed method for HOI concept discovery, object affordance recognition and HOI detection. For example, the proposed self-compositional learning framework significantly improves the performance of 1) HOI concept discovery by over 10% on HICO-DET and over 3% on V-COCO, respectively; 2) object affordance recognition by over 9% mAP on MS-COCO and HICO-DET; and 3) rare-first and non-rare-first unknown HOI detection relatively over 30% and 20%, respectively. Code is publicly available at

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Human-Object Interaction Concept Discovery HICO-DET SCL Unknown (AP) 33.58 # 1
Affordance Recognition HICO-DET SCL HICO 82.47 # 1
COCO-Val2017 72.08 # 1
Object365 57.53 # 1
Novel classes 18.55 # 1
Affordance Recognition HICO-DET(Unknown Concepts) SCL COCO-Val2017 56.19 # 1
Obj365 46.32 # 1
HICO 64.50 # 1
Novel Classes 18.55 # 1


No methods listed for this paper. Add relevant methods here