Self-supervising Action Recognition by Statistical Moment and Subspace Descriptors

14 Jan 2020  ·  Lei Wang, Piotr Koniusz ·

In this paper, we build on a concept of self-supervision by taking RGB frames as input to learn to predict both action concepts and auxiliary descriptors e.g., object descriptors. So-called hallucination streams are trained to predict auxiliary cues, simultaneously fed into classification layers, and then hallucinated at the testing stage to aid network. We design and hallucinate two descriptors, one leveraging four popular object detectors applied to training videos, and the other leveraging image- and video-level saliency detectors. The first descriptor encodes the detector- and ImageNet-wise class prediction scores, confidence scores, and spatial locations of bounding boxes and frame indexes to capture the spatio-temporal distribution of features per video. Another descriptor encodes spatio-angular gradient distributions of saliency maps and intensity patterns. Inspired by the characteristic function of the probability distribution, we capture four statistical moments on the above intermediate descriptors. As numbers of coefficients in the mean, covariance, coskewness and cokurtotsis grow linearly, quadratically, cubically and quartically w.r.t. the dimension of feature vectors, we describe the covariance matrix by its leading n' eigenvectors (so-called subspace) and we capture skewness/kurtosis rather than costly coskewness/cokurtosis. We obtain state of the art on five popular datasets such as Charades and EPIC-Kitchens.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Action Classification Charades DEEP-HAL with ODF+SDF (AssembleNet++) MAP 62.29 # 3
Action Classification Charades DEEP-HAL with ODF+SDF (I3D) MAP 50.16 # 8
Egocentric Activity Recognition EPIC-KITCHENS-55 DEEP-HAL with ODF+SDF (AssembleNet++) Actions Top-1 (S2) 27.3 # 1
Actions Top-1 (S1) 35.8 # 1
Action Recognition HMDB-51 DEEP-HAL with ODF+SDF (I3D) Average accuracy of 3 splits 87.56 # 1
Scene Recognition YUP++ DEEP-HAL with ODF+SDF (I3D) Accuracy (%) 94.4 # 1


No methods listed for this paper. Add relevant methods here