TSM: Temporal Shift Module for Efficient Video Understanding

ICCV 2019  ·  Ji Lin, Chuang Gan, Song Han ·

The explosive growth in video streaming gives rise to challenges on performing video understanding at high accuracy and low computation cost. Conventional 2D CNNs are computationally cheap but cannot capture temporal relationships; 3D CNN based methods can achieve good performance but are computationally intensive, making it expensive to deploy. In this paper, we propose a generic and effective Temporal Shift Module (TSM) that enjoys both high efficiency and high performance. Specifically, it can achieve the performance of 3D CNN but maintain 2D CNN's complexity. TSM shifts part of the channels along the temporal dimension; thus facilitate information exchanged among neighboring frames. It can be inserted into 2D CNNs to achieve temporal modeling at zero computation and zero parameters. We also extended TSM to online setting, which enables real-time low-latency online video recognition and video object detection. TSM is accurate and efficient: it ranks the first place on the Something-Something leaderboard upon publication; on Jetson Nano and Galaxy Note8, it achieves a low latency of 13ms and 35ms for online video recognition. The code is available at: https://github.com/mit-han-lab/temporal-shift-module.

PDF Abstract ICCV 2019 PDF ICCV 2019 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
3D Action Recognition Assembly101 TSM Actions Top-1 35.27 # 2
Verbs Top-1 58.27 # 6
Object Top-1 47.45 # 2
Video Object Detection ImageNet VID Online TSM MAP 76.3 # 24
Action Classification Kinetics-400 TSM Acc@1 74.7 # 150
Action Recognition Something-Something V1 TSMEn Top 1 Accuracy 49.7 # 52
Top 5 Accuracy 78.5 # 29
Action Recognition Something-Something V1 TSM Top 1 Accuracy 47.2 # 63
Top 5 Accuracy 77.1 # 33
Action Recognition Something-Something V1 TSM (RGB + Flow) Top 1 Accuracy 50.7 # 48
Action Recognition Something-Something V2 TSM (RGB + Flow) Top-1 Accuracy 66.6 # 72
Top-5 Accuracy 91.3 # 38

Methods


No methods listed for this paper. Add relevant methods here