Video Cloze Procedure for Self-Supervised Spatio-Temporal Learning

2 Jan 2020  ·  Dezhao Luo, Chang Liu, Yu Zhou, Dongbao Yang, Can Ma, Qixiang Ye, Weiping Wang ·

We propose a novel self-supervised method, referred to as Video Cloze Procedure (VCP), to learn rich spatial-temporal representations. VCP first generates "blanks" by withholding video clips and then creates "options" by applying spatio-temporal operations on the withheld clips. Finally, it fills the blanks with "options" and learns representations by predicting the categories of operations applied on the clips. VCP can act as either a proxy task or a target task in self-supervised learning. As a proxy task, it converts rich self-supervised representations into video clip operations (options), which enhances the flexibility and reduces the complexity of representation learning. As a target task, it can assess learned representation models in a uniform and interpretable manner. With VCP, we train spatial-temporal representation models (3D-CNNs) and apply such models on action recognition and video retrieval tasks. Experiments on commonly used benchmarks show that the trained models outperform the state-of-the-art self-supervised models with significant margins.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Self-supervised Video Retrieval HMDB51 VCP (R3D) Top-1 7.6 # 8
Self-Supervised Action Recognition HMDB51 VCP (R3D) Top-1 Accuracy 31.5 # 38
Pre-Training Dataset UCF101 # 1
Frozen false # 1
Self-Supervised Action Recognition UCF101 VCP (R3D) 3-fold Accuracy 66 # 34
Pre-Training Dataset UCF101 # 1
Frozen false # 1
Self-supervised Video Retrieval UCF101 VCP (R3D) Top-1 18.6 # 10


No methods listed for this paper. Add relevant methods here