Video Classification with Channel-Separated Convolutional Networks

ICCV 2019  ยท  Du Tran, Heng Wang, Lorenzo Torresani, Matt Feiszli ยท

Group convolution has been shown to offer great computational savings in various 2D convolutional architectures for image classification. It is natural to ask: 1) if group convolution can help to alleviate the high computational cost of video classification networks; 2) what factors matter the most in 3D group convolutional networks; and 3) what are good computation/accuracy trade-offs with 3D group convolutional networks. This paper studies the effects of different design choices in 3D group convolutional networks for video classification. We empirically demonstrate that the amount of channel interactions plays an important role in the accuracy of 3D group convolutional networks. Our experiments suggest two main findings. First, it is a good practice to factorize 3D convolutions by separating channel interactions and spatiotemporal interactions as this leads to improved accuracy and lower computational cost. Second, 3D channel-separated convolutions provide a form of regularization, yielding lower training accuracy but higher test accuracy compared to 3D convolutions. These two empirical findings lead us to design an architecture -- Channel-Separated Convolutional Network (CSN) -- which is simple, efficient, yet accurate. On Sports1M, Kinetics, and Something-Something, our CSNs are comparable with or better than the state-of-the-art while being 2-3 times more efficient.

PDF Abstract ICCV 2019 PDF ICCV 2019 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Action Classification Kinetics-400 ip-CSN-152 Acc@1 77.8 # 128
Acc@5 92.8 # 102
Action Classification Kinetics-400 ip-CSN-152 (IG-65M pretraining) Acc@1 82.5 # 71
Acc@5 95.3 # 48
Action Classification Kinetics-400 ir-CSN-152 (IG-65M pretraining) Acc@1 82.6 # 70
Action Classification Kinetics-400 R[2+1]D-152 (IG-65M pretraining) Acc@1 81.3 # 80
Acc@5 95.1 # 53
Action Classification Kinetics-400 ip-CSN-152 (Sports-1M pretraining) Acc@1 79.2 # 110
Acc@5 93.8 # 83
Action Recognition Something-Something V1 ir-CSN-101 Top 1 Accuracy 48.4 # 61
Action Recognition Something-Something V1 ir-CSN-152 (IG-65M pretraining) Top 1 Accuracy 52.1 # 44
Action Recognition Something-Something V1 ip-CSN-152 (IG-65M pretraining) Top 1 Accuracy 53.3 # 38
Action Recognition Something-Something V1 ir-CSN-152 Top 1 Accuracy 49.3 # 58
Action Recognition Something-Something V1 R(2+1)D-152 (IG-65M pretraining) Top 1 Accuracy 51.6 # 48
Action Recognition Sports-1M ip-CSN-101 (RGB) Video hit@1 74.9 # 2
Video hit@5 92.6 # 2
Action Recognition Sports-1M ip-CSN-152 (RGB) Video hit@1 75.5 # 1
Video hit@5 92.8 # 1

Methods