Revisiting Foreground and Background Separation in Weakly-supervised Temporal Action Localization: A Clustering-based Approach

Weakly-supervised temporal action localization aims to localize action instances in videos with only video-level action labels. Existing methods mainly embrace a localization-by-classification pipeline that optimizes the snippet-level prediction with a video classification loss. However, this formulation suffers from the discrepancy between classification and detection, resulting in inaccurate separation of foreground and background (F\&B) snippets. To alleviate this problem, we propose to explore the underlying structure among the snippets by resorting to unsupervised snippet clustering, rather than heavily relying on the video classification loss. Specifically, we propose a novel clustering-based F\&B separation algorithm. It comprises two core components: a snippet clustering component that groups the snippets into multiple latent clusters and a cluster classification component that further classifies the cluster as foreground or background. As there are no ground-truth labels to train these two components, we introduce a unified self-labeling mechanism based on optimal transport to produce high-quality pseudo-labels that match several plausible prior distributions. This ensures that the cluster assignments of the snippets can be accurately associated with their F\&B labels, thereby boosting the F\&B separation. We evaluate our method on three benchmarks: THUMOS14, ActivityNet v1.2 and v1.3. Our method achieves promising performance on all three benchmarks while being significantly more lightweight than previous methods. Code is available at

PDF Abstract ICCV 2023 PDF ICCV 2023 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Weakly Supervised Action Localization ActivityNet-1.2 CASE mAP@0.5 43.8 # 3
Mean mAP 27.9 # 1
Weakly Supervised Action Localization ActivityNet-1.3 CASE mAP@0.5 43.2 # 2
mAP@0.5:0.95 26.8 # 2
Weakly Supervised Action Localization THUMOS 2014 CASE mAP@0.1:0.7 46.2 # 6
mAP@0.1:0.5 57.1 # 6
Weakly Supervised Action Localization THUMOS 2014 CASE + Zhou et al. mAP@0.1:0.7 49.2 # 4


No methods listed for this paper. Add relevant methods here