Alleviating Over-segmentation Errors by Detecting Action Boundaries

14 Jul 2020  ·  Yuchi Ishikawa, Seito Kasai, Yoshimitsu Aoki, Hirokatsu Kataoka ·

We propose an effective framework for the temporal action segmentation task, namely an Action Segment Refinement Framework (ASRF). Our model architecture consists of a long-term feature extractor and two branches: the Action Segmentation Branch (ASB) and the Boundary Regression Branch (BRB). The long-term feature extractor provides shared features for the two branches with a wide temporal receptive field. The ASB classifies video frames with action classes, while the BRB regresses the action boundary probabilities. The action boundaries predicted by the BRB refine the output from the ASB, which results in a significant performance improvement. Our contributions are three-fold: (i) We propose a framework for temporal action segmentation, the ASRF, which divides temporal action segmentation into frame-wise action classification and action boundary regression. Our framework refines frame-level hypotheses of action classes using predicted action boundaries. (ii) We propose a loss function for smoothing the transition of action probabilities, and analyze combinations of various loss functions for temporal action segmentation. (iii) Our framework outperforms state-of-the-art methods on three challenging datasets, offering an improvement of up to 13.7% in terms of segmental edit distance and up to 16.1% in terms of segmental F1 score. Our code will be publicly available soon.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Action Segmentation 50 Salads ASRF F1@10% 84.9 # 16
Edit 79.3 # 15
Acc 84.5 # 15
F1@25% 83.5 # 15
F1@50% 77.3 # 14
Action Segmentation Breakfast ASRF F1@10% 74.3 # 17
F1@50% 56.1 # 17
Acc 67.6 # 24
Edit 72.4 # 18
F1@25% 68.9 # 18
Action Segmentation GTEA ASRF F1@10% 89.4 # 16
F1@50% 79.8 # 10
Acc 77.3 # 21
Edit 83.7 # 20
F1@25% 87.8 # 16

Methods


No methods listed for this paper. Add relevant methods here