You Only Watch Once: A Unified CNN Architecture for Real-Time Spatiotemporal Action Localization

15 Nov 2019  ·  Okan Köpüklü, Xiangyu Wei, Gerhard Rigoll ·

Spatiotemporal action localization requires the incorporation of two sources of information into the designed architecture: (1) temporal information from the previous frames and (2) spatial information from the key frame. Current state-of-the-art approaches usually extract these information with separate networks and use an extra mechanism for fusion to get detections. In this work, we present YOWO, a unified CNN architecture for real-time spatiotemporal action localization in video streams. YOWO is a single-stage architecture with two branches to extract temporal and spatial information concurrently and predict bounding boxes and action probabilities directly from video clips in one evaluation. Since the whole architecture is unified, it can be optimized end-to-end. The YOWO architecture is fast providing 34 frames-per-second on 16-frames input clips and 62 frames-per-second on 8-frames input clips, which is currently the fastest state-of-the-art architecture on spatiotemporal action localization task. Remarkably, YOWO outperforms the previous state-of-the art results on J-HMDB-21 and UCF101-24 with an impressive improvement of ~3% and ~12%, respectively. Moreover, YOWO is the first and only single-stage architecture that provides competitive results on AVA dataset. We make our code and pretrained models publicly available.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Action Recognition In Videos AVA v2.1 YOWO+LFB* mAP (Val) 19.2 # 1
Action Recognition In Videos AVA v2.2 YOWO+LFB* mAP (Val) 20.2 # 1
Temporal Action Localization J-HMDB-21 YOWO (16-frame) Frame-mAP 74.4 # 1
Video-mAP 0.2 87.8 # 1
Video-mAP 0.5 85.7 # 1
Video-mAP 0.75 58.1 # 1
Temporal Action Localization UCF101-24 YOWO (16-frame) Frame-mAP 87.2 # 1
Video-mAP 0.5 48.8 # 2


No methods listed for this paper. Add relevant methods here