GateHUB: Gated History Unit with Background Suppression for Online Action Detection

CVPR 2022  ·  Junwen Chen, Gaurav Mittal, Ye Yu, Yu Kong, Mei Chen ·

Online action detection is the task of predicting the action as soon as it happens in a streaming video. A major challenge is that the model does not have access to the future and has to solely rely on the history, i.e., the frames observed so far, to make predictions. It is therefore important to accentuate parts of the history that are more informative to the prediction of the current frame. We present GateHUB, Gated History Unit with Background Suppression, that comprises a novel position-guided gated cross-attention mechanism to enhance or suppress parts of the history as per how informative they are for current frame prediction. GateHUB further proposes Future-augmented History (FaH) to make history features more informative by using subsequently observed frames when available. In a single unified framework, GateHUB integrates the transformer's ability of long-range temporal modeling and the recurrent model's capacity to selectively encode relevant information. GateHUB also introduces a background suppression objective to further mitigate false positive background frames that closely resemble the action frames. Extensive validation on three benchmark datasets, THUMOS, TVSeries, and HDD, demonstrates that GateHUB significantly outperforms all existing methods and is also more efficient than the existing best work. Furthermore, a flow-free version of GateHUB is able to achieve higher or close accuracy at 2.8x higher frame rate compared to all existing methods that require both RGB and optical flow information for prediction.

PDF Abstract CVPR 2022 PDF CVPR 2022 Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Online Action Detection THUMOS'14 GateHUB (RGB only) mAP 66.5 # 6
Online Action Detection THUMOS'14 GateHUB(TimeSformer) mAP 72.5 # 1
Online Action Detection THUMOS'14 GateHUB mAP 70.7 # 2
Online Action Detection TVSeries GateHUB mCAP 89.6 # 1


No methods listed for this paper. Add relevant methods here