D3D: Distilled 3D Networks for Video Action Recognition

19 Dec 2018  ·  Jonathan C. Stroud, David A. Ross, Chen Sun, Jia Deng, Rahul Sukthankar ·

State-of-the-art methods for video action recognition commonly use an ensemble of two networks: the spatial stream, which takes RGB frames as input, and the temporal stream, which takes optical flow as input. In recent work, both of these streams consist of 3D Convolutional Neural Networks, which apply spatiotemporal filters to the video clip before performing classification. Conceptually, the temporal filters should allow the spatial stream to learn motion representations, making the temporal stream redundant. However, we still see significant benefits in action recognition performance by including an entirely separate temporal stream, indicating that the spatial stream is "missing" some of the signal captured by the temporal stream. In this work, we first investigate whether motion representations are indeed missing in the spatial stream of 3D CNNs. Second, we demonstrate that these motion representations can be improved by distillation, by tuning the spatial stream to predict the outputs of the temporal stream, effectively combining both models into a single stream. Finally, we show that our Distilled 3D Network (D3D) achieves performance on par with two-stream approaches, using only a single model and with no need to compute optical flow.

PDF Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Action Recognition AVA v2.1 D3D (ResNet RPN, Kinetics-400 pretraining) mAP (Val) 23 # 10
Action Recognition HMDB-51 D3D (Kinetics-400 pretraining) Average accuracy of 3 splits 78.7 # 24
Action Recognition HMDB-51 D3D + D3D Average accuracy of 3 splits 80.5 # 20
Action Recognition HMDB-51 D3D (Kinetics-600 pretraining) Average accuracy of 3 splits 79.3 # 22
Action Classification Kinetics-400 D3D+S3D-G (RGB + RGB) Acc@1 76.5 # 118
Action Classification Kinetics-400 D3D (RGB) Acc@1 75.9 # 124
Action Classification Kinetics-600 D3D+S3D-G Top-1 Accuracy 79.1 # 50
Action Classification Kinetics-600 D3D Top-1 Accuracy 77.9 # 53
Action Recognition UCF101 D3D (Kinetics-600 pretraining) 3-fold Accuracy 97.1 # 18
Action Recognition UCF101 D3D + D3D 3-fold Accuracy 97.6 # 13
Action Recognition UCF101 D3D (Kinetics-400 pretraining) 3-fold Accuracy 97 # 20


No methods listed for this paper. Add relevant methods here