Video Representation Learning by Dense Predictive Coding

10 Sep 2019  ·  Tengda Han, Weidi Xie, Andrew Zisserman ·

The objective of this paper is self-supervised learning of spatio-temporal embeddings from video, suitable for human action recognition. We make three contributions: First, we introduce the Dense Predictive Coding (DPC) framework for self-supervised representation learning on videos. This learns a dense encoding of spatio-temporal blocks by recurrently predicting future representations; Second, we propose a curriculum training scheme to predict further into the future with progressively less temporal context. This encourages the model to only encode slowly varying spatial-temporal signals, therefore leading to semantic representations; Third, we evaluate the approach by first training the DPC model on the Kinetics-400 dataset with self-supervised learning, and then finetuning the representation on a downstream task, i.e. action recognition. With single stream (RGB only), DPC pretrained representations achieve state-of-the-art self-supervised performance on both UCF101(75.7% top1 acc) and HMDB51(35.7% top1 acc), outperforming all previous learning methods by a significant margin, and approaching the performance of a baseline pre-trained on ImageNet.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Self-Supervised Action Recognition HMDB51 DPC (Modified 3D ResNet-18) Top-1 Accuracy 34.5 # 37
Pre-Training Dataset Kinetics400 # 1
Frozen false # 1
Self-Supervised Action Recognition HMDB51 DPC (Modified 3D Resnet-34) Top-1 Accuracy 35.7 # 36
Pre-Training Dataset Kinetics400 # 1
Frozen false # 1
Self-Supervised Action Recognition UCF101 DPC (Modified 3D Resnet-34) 3-fold Accuracy 75.7 # 28
Pre-Training Dataset Kinetics400 # 1
Frozen false # 1
Self-Supervised Action Recognition UCF101 DPC (3D ResNet-18, Split 1) 3-fold Accuracy 60.6 # 41
Pre-Training Dataset UCF101 # 1
Frozen false # 1
Self-Supervised Action Recognition UCF101 DPC (3D ResNet-18) 3-fold Accuracy 68.2 # 34
Pre-Training Dataset Kinetics400 # 1
Frozen false # 1

Methods