End-to-end Temporal Action Detection with Transformer

18 Jun 2021  ·  Xiaolong Liu, Qimeng Wang, Yao Hu, Xu Tang, Shiwei Zhang, Song Bai, Xiang Bai ·

Temporal action detection (TAD) aims to determine the semantic label and the temporal interval of every action instance in an untrimmed video. It is a fundamental and challenging task in video understanding. Previous methods tackle this task with complicated pipelines. They often need to train multiple networks and involve hand-designed operations, such as non-maximal suppression and anchor generation, which limit the flexibility and prevent end-to-end learning. In this paper, we propose an end-to-end Transformer-based method for TAD, termed TadTR. Given a small set of learnable embeddings called action queries, TadTR adaptively extracts temporal context information from the video for each query and directly predicts action instances with the context. To adapt Transformer to TAD, we propose three improvements to enhance its locality awareness. The core is a temporal deformable attention module that selectively attends to a sparse set of key snippets in a video. A segment refinement mechanism and an actionness regression head are designed to refine the boundaries and confidence of the predicted instances, respectively. With such a simple pipeline, TadTR requires lower computation cost than previous detectors, while preserving remarkable performance. As a self-contained detector, it achieves state-of-the-art performance on THUMOS14 (56.7% mAP) and HACS Segments (32.09% mAP). Combined with an extra action classifier, it obtains 36.75% mAP on ActivityNet-1.3. Code is available at https://github.com/xlliu7/TadTR.

PDF Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Temporal Action Localization ActivityNet-1.3 TadTR (TSP features) mAP IOU@0.5 53.62 # 7
mAP 36.75 # 6
mAP IOU@0.75 37.52 # 4
mAP IOU@0.95 10.56 # 2
Temporal Action Localization HACS TadTR (I3D RGB) Average-mAP 32.09 # 1
mAP@0.5 47.14 # 1
mAP@0.75 32.11 # 1
mAP@0.95 10.94 # 1
Temporal Action Localization THUMOS’14 TadTR mAP IOU@0.5 60.1 # 2
mAP IOU@0.3 74.8 # 2
mAP IOU@0.4 69.1 # 2
mAP IOU@0.6 46.6 # 4
mAP IOU@0.7 32.8 # 5
Avg mAP (0.3:0.7) 56.7 # 2