Semantic2Graph: Graph-based Multi-modal Feature Fusion for Action Segmentation in Videos

13 Sep 2022  ·  Junbin Zhang, Pei-Hsuan Tsai, Meng-Hsun Tsai ·

Video action segmentation have been widely applied in many fields. Most previous studies employed video-based vision models for this purpose. However, they often rely on a large receptive field, LSTM or Transformer methods to capture long-term dependencies within videos, leading to significant computational resource requirements. To address this challenge, graph-based model was proposed. However, previous graph-based models are less accurate. Hence, this study introduces a graph-structured approach named Semantic2Graph, to model long-term dependencies in videos, thereby reducing computational costs and raise the accuracy. We construct a graph structure of video at the frame-level. Temporal edges are utilized to model the temporal relations and action order within videos. Additionally, we have designed positive and negative semantic edges, accompanied by corresponding edge weights, to capture both long-term and short-term semantic relationships in video actions. Node attributes encompass a rich set of multi-modal features extracted from video content, graph structures, and label text, encompassing visual, structural, and semantic cues. To synthesize this multi-modal information effectively, we employ a graph neural network (GNN) model to fuse multi-modal features for node action label classification. Experimental results demonstrate that Semantic2Graph outperforms state-of-the-art methods in terms of performance, particularly on benchmark datasets such as GTEA and 50Salads. Multiple ablation experiments further validate the effectiveness of semantic features in enhancing model performance. Notably, the inclusion of semantic edges in Semantic2Graph allows for the cost-effective capture of long-term dependencies, affirming its utility in addressing the challenges posed by computational resource constraints in video-based vision models.

PDF Abstract


Results from the Paper

 Ranked #1 on Action Segmentation on GTEA (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Action Segmentation 50 Salads Semantic2Graph F1@10% 91.5 # 2
Edit 89.1 # 1
Acc 88.6 # 5
F1@25% 90.2 # 2
F1@50% 87.3 # 2
Action Segmentation GTEA Semantic2Graph F1@10% 95.7 # 1
F1@50% 91.3 # 1
Acc 89.8 # 1
Edit 92.0 # 3
F1@25% 94.2 # 1


No methods listed for this paper. Add relevant methods here