STVGFormer: Spatio-Temporal Video Grounding with Static-Dynamic Cross-Modal Understanding

6 Jul 2022  ·  Zihang Lin, Chaolei Tan, Jian-Fang Hu, Zhi Jin, Tiancai Ye, Wei-Shi Zheng ·

In this technical report, we introduce our solution to human-centric spatio-temporal video grounding task. We propose a concise and effective framework named STVGFormer, which models spatiotemporal visual-linguistic dependencies with a static branch and a dynamic branch. The static branch performs cross-modal understanding in a single frame and learns to localize the target object spatially according to intra-frame visual cues like object appearances. The dynamic branch performs cross-modal understanding across multiple frames. It learns to predict the starting and ending time of the target moment according to dynamic visual cues like motions. Both the static and dynamic branches are designed as cross-modal transformers. We further design a novel static-dynamic interaction block to enable the static and dynamic branches to transfer useful and complementary information from each other, which is shown to be effective to improve the prediction on hard cases. Our proposed method achieved 39.6% vIoU and won the first place in the HC-STVG track of the 4th Person in Context Challenge.

PDF Abstract


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Spatio-Temporal Video Grounding HC-STVG2 STVGFormer Val m_vIoU 38.7 # 1
Val vIoU@0.3 65.5 # 1
Val vIoU@0.5 33.8 # 1


No methods listed for this paper. Add relevant methods here