Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

CVPR 2017  ·  Joao Carreira, Andrew Zisserman ·

The paucity of videos in current action classification datasets (UCF-101 and HMDB-51) has made it difficult to identify good video architectures, as most methods obtain similar performance on existing small-scale benchmarks. This paper re-evaluates state-of-the-art architectures in light of the new Kinetics Human Action Video dataset. Kinetics has two orders of magnitude more data, with 400 human action classes and over 400 clips per class, and is collected from realistic, challenging YouTube videos. We provide an analysis on how current architectures fare on the task of action classification on this dataset and how much performance improves on the smaller benchmark datasets after pre-training on Kinetics. We also introduce a new Two-Stream Inflated 3D ConvNet (I3D) that is based on 2D ConvNet inflation: filters and pooling kernels of very deep image classification ConvNets are expanded into 3D, making it possible to learn seamless spatio-temporal feature extractors from video while leveraging successful ImageNet architecture designs and even their parameters. We show that, after pre-training on Kinetics, I3D models considerably improve upon the state-of-the-art in action classification, reaching 80.9% on HMDB-51 and 98.0% on UCF-101.

PDF Abstract CVPR 2017 PDF CVPR 2017 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Video Object Tracking CATER I3D-50 + LSTM Top 1 Accuracy 60.2 # 8
Top 5 Accuracy 81.8 # 7
L1 1.2 # 7
Action Recognition HMDB-51 Flow-I3D (Imagenet+Kinetics pre-training) Average accuracy of 3 splits 77.1 # 29
Action Recognition HMDB-51 Two-stream I3D Average accuracy of 3 splits 80.9 # 15
Action Recognition HMDB-51 Flow-I3D (Kinetics pre-training) Average accuracy of 3 splits 77.3 # 28
Action Recognition HMDB-51 RGB-I3D (Kinetics pre-training) Average accuracy of 3 splits 74.3 # 38
Action Recognition HMDB-51 Two-Stream I3D (Imagenet+Kinetics pre-training) Average accuracy of 3 splits 80.7 # 17
Action Recognition HMDB-51 RGB-I3D (Imagenet+Kinetics pre-training) Average accuracy of 3 splits 74.8 # 35
Skeleton Based Action Recognition J-HMDB I3D Accuracy (RGB+pose) 84.1 # 4
Action Classification Moments in Time I3D Top 1 Accuracy 29.51% # 21
Top 5 Accuracy 56.06% # 12
Action Classification Toyota Smarthome dataset I3D CS 53.4 # 9
CV1 34.9 # 5
CV2 45.1 # 6
Action Recognition UCF101 RGB-I3D (Imagenet+Kinetics pre-training) 3-fold Accuracy 95.6 # 35
Action Recognition UCF101 RGB-I3D (Kinetics pre-training) 3-fold Accuracy 95.1 # 39
Action Recognition UCF101 Two-Stream I3D (Imagenet+Kinetics pre-training) 3-fold Accuracy 98.0 # 5
Action Recognition UCF101 Two-Stream I3D (Kinetics pre-training) 3-fold Accuracy 97.8 # 6
Action Recognition UCF101 Flow-I3D (Imagenet+Kinetics pre-training) 3-fold Accuracy 96.7 # 24
Action Recognition UCF101 Flow-I3D (Kinetics pre-training) 3-fold Accuracy 96.5 # 26
Action Recognition UCF101 Two-stream I3D 3-fold Accuracy 93.4 # 51
Hand Gesture Recognition VIVA Hand Gestures Dataset I3D Accuracy 83.1 # 2

Results from Other Papers


Task Dataset Model Metric Name Metric Value Rank Uses Extra
Training Data
Source Paper Compare
Action Classification Charades I3D MAP 32.9 # 30
Hand Gesture Recognition EgoGesture I3D Accuracy 92.78 # 4
Action Classification Kinetics-400 I3D Acc@1 71.1 # 126
Acc@5 89.3 # 94
Semantic Object Interaction Classification VLOG I3D MAP 39.7 # 3

Methods


No methods listed for this paper. Add relevant methods here