Semi-Supervised Temporal Action Detection with Proposal-Free Masking

14 Jul 2022  ·  Sauradip Nag, Xiatian Zhu, Yi-Zhe Song, Tao Xiang ·

Existing temporal action detection (TAD) methods rely on a large number of training data with segment-level annotations. Collecting and annotating such a training set is thus highly expensive and unscalable. Semi-supervised TAD (SS-TAD) alleviates this problem by leveraging unlabeled videos freely available at scale. However, SS-TAD is also a much more challenging problem than supervised TAD, and consequently much under-studied. Prior SS-TAD methods directly combine an existing proposal-based TAD method and a SSL method. Due to their sequential localization (e.g, proposal generation) and classification design, they are prone to proposal error propagation. To overcome this limitation, in this work we propose a novel Semi-supervised Temporal action detection model based on PropOsal-free Temporal mask (SPOT) with a parallel localization (mask generation) and classification architecture. Such a novel design effectively eliminates the dependence between localization and classification by cutting off the route for error propagation in-between. We further introduce an interaction mechanism between classification and localization for prediction refinement, and a new pretext task for self-supervised model pre-training. Extensive experiments on two standard benchmarks show that our SPOT outperforms state-of-the-art alternatives, often by a large margin. The PyTorch implementation of SPOT is available at

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Semi-Supervised Action Detection ActivityNet-1.3 SPOT ( 60 % labeled, IoU thresh=0.5) mAP 52.8 # 1
Semi-Supervised Action Detection THUMOS' 14 SPOT ( 10 % labeled, IoU thresh=0.3) mAP 58.9 # 1


No methods listed for this paper. Add relevant methods here