Is Space-Time Attention All You Need for Video Understanding?

9 Feb 2021  ·  Gedas Bertasius, Heng Wang, Lorenzo Torresani ·

We present a convolution-free approach to video classification built exclusively on self-attention over space and time. Our method, named "TimeSformer," adapts the standard Transformer architecture to video by enabling spatiotemporal feature learning directly from a sequence of frame-level patches. Our experimental study compares different self-attention schemes and suggests that "divided attention," where temporal attention and spatial attention are separately applied within each block, leads to the best video classification accuracy among the design choices considered. Despite the radically new design, TimeSformer achieves state-of-the-art results on several action recognition benchmarks, including the best reported accuracy on Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks, our model is faster to train, it can achieve dramatically higher test efficiency (at a small drop in accuracy), and it can also be applied to much longer video clips (over one minute long). Code and models are available at:

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Action Recognition Diving-48 TimeSformer Accuracy 75 # 12
Action Recognition Diving-48 TimeSformer-HR Accuracy 78 # 10
Action Recognition Diving-48 TimeSformer-L Accuracy 81 # 9
Video Question Answering Howto100M-QA TimeSformer Accuracy 62.1 # 1
Action Classification Kinetics-400 TimeSformer Acc@1 78 # 117
Acc@5 93.7 # 84
Action Classification Kinetics-400 TimeSformer-HR Acc@1 79.7 # 95
Acc@5 94.4 # 68
Action Classification Kinetics-400 TimeSformer-L (ImageNet-21k pretrain) Acc@1 80.7 # 81
Acc@5 94.7 # 58
FLOPs (G) x views 7140x3 # 1
Parameters (M) 121.4 # 23
Action Recognition Something-Something V2 TimeSformer Top-1 Accuracy 59.5 # 106
Action Recognition Something-Something V2 TimeSformer-HR Top-1 Accuracy 62.5 # 95
Action Recognition Something-Something V2 TimeSformer-L Top-1 Accuracy 62.3 # 97
Anomaly Detection UBnormal TimeSformer AUC 68.5% # 3
RBDC 0.04 # 3
TBDC 0.05 # 3