High-order Tensor Pooling with Attention for Action Recognition

11 Oct 2021  ·  Lei Wang, Ke Sun, Piotr Koniusz ·

We aim at capturing high-order statistics of feature vectors formed by a neural network, and propose end-to-end second- and higher-order pooling to form a tensor descriptor. Tensor descriptors require a robust similarity measure due to low numbers of aggregated vectors and the burstiness phenomenon, when a given feature appears more/less frequently than statistically expected. The Heat Diffusion Process (HDP) on a graph Laplacian is closely related to the Eigenvalue Power Normalization (EPN) of the covariance/autocorrelation matrix, whose inverse forms a loopy graph Laplacian. We show that the HDP and the EPN play the same role, i.e., to boost or dampen the magnitude of the eigenspectrum thus preventing the burstiness. We equip higher-order tensors with EPN which acts as a spectral detector of higher-order occurrences to prevent burstiness. We also prove that for a tensor of order r built from d dimensional feature descriptors, such a detector gives the likelihood if at least one higher-order occurrence is 'projected' into one of binom(d,r) subspaces represented by the tensor; thus forming a tensor power normalization metric endowed with binom(d,r) such 'detectors'. For experimental contributions, we apply several second- and higher-order pooling variants to action recognition, provide previously not presented comparisons of such pooling variants, and show state-of-the-art results on HMDB-51, YUP++ and MPII Cooking Activities.

PDF Abstract

Datasets


Results from the Paper


Ranked #2 on Scene Recognition on YUP++ (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Benchmark
Action Recognition HMDB-51 SO+MaxExp+IDT Average accuracy of 3 splits 85.70 # 5
Action Recognition HMDB-51 TO+MaxExp+IDT Average accuracy of 3 splits 87.21 # 3
Scene Recognition YUP++ TO+MaxExp+IDT Accuracy (%) 93.1 # 2
Scene Recognition YUP++ SO+MaxExp+IDT Accuracy (%) 92.5 # 4

Methods