Spatiotemporal Contrastive Video Representation Learning

We present a self-supervised Contrastive Video Representation Learning (CVRL) method to learn spatiotemporal visual representations from unlabeled videos. Our representations are learned using a contrastive loss, where two augmented clips from the same short video are pulled together in the embedding space, while clips from different videos are pushed away... We study what makes for good data augmentations for video self-supervised learning and find that both spatial and temporal information are crucial. We carefully design data augmentations involving spatial and temporal cues. Concretely, we propose a temporally consistent spatial augmentation method to impose strong spatial augmentations on each frame of the video while maintaining the temporal consistency across frames. We also propose a sampling-based temporal augmentation method to avoid overly enforcing invariance on clips that are distant in time. On Kinetics-600, a linear classifier trained on the representations learned by CVRL achieves 70.4% top-1 accuracy with a 3D-ResNet-50 (R3D-50) backbone, outperforming ImageNet supervised pre-training by 15.7% and SimCLR unsupervised pre-training by 18.8% using the same inflated R3D-50. The performance of CVRL can be further improved to 72.9% with a larger R3D-152 (2x filters) backbone, significantly closing the gap between unsupervised and supervised video representation learning. Our code and models will be available at https://github.com/tensorflow/models/tree/master/official/. read more

PDF Abstract CVPR 2021 PDF CVPR 2021 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Self-Supervised Action Recognition HMDB51 CVRL (R3D-152 2x; K600) Top-1 Accuracy 69.9 # 3
Pre-Training Dataset Kinetics600 # 1
Frozen false # 1
Self-Supervised Action Recognition HMDB51 CVRL (R3D-50; K400) Top-1 Accuracy 66.7 # 6
Pre-Training Dataset Kinetics400 # 1
Frozen false # 1
Self-Supervised Action Recognition HMDB51 CVRL (R3D-50; K600) Top-1 Accuracy 68.0 # 5
Pre-Training Dataset Kinetics600 # 1
Frozen false # 1
Self-Supervised Action Recognition HMDB51 (finetuned) CVRL (R3D-152 2x; K600) Top-1 Accuracy 69.9 # 3
Pretraining Dataset K600 # 1
Self-Supervised Action Recognition HMDB51 (finetuned) CVRL (R3D-50; K400) Top-1 Accuracy 66.7 # 7
Pretraining Dataset K400 # 1
Self-Supervised Action Recognition Kinetics-400 CVRL (R3D-152 2x; K600 pretrain) Top-1 accuracy % 71.6 # 1
Self-Supervised Action Recognition Kinetics-400 CVRL (R3D-101) Top-1 accuracy % 67.6 # 2
Self-Supervised Action Recognition Kinetics-400 CVRL (R3D-50) Top-1 accuracy % 66.1 # 3
Self-Supervised Action Recognition Kinetics-600 CVRL (R3D-152 2x) Top-1 Accuracy 72.9 # 1
Self-Supervised Action Recognition Kinetics-600 CVRL (R3D-101) Top-1 Accuracy 71.6 # 2
Self-Supervised Action Recognition UCF101 CVRL (R3D-50; K400) 3-fold Accuracy 92.2 # 9
Pre-Training Dataset Kinetics400 # 1
Frozen false # 1
Self-Supervised Action Recognition UCF101 CVRL (R3D-50; K600) 3-fold Accuracy 93.4 # 6
Pre-Training Dataset Kinetics600 # 1
Frozen false # 1
Self-Supervised Action Recognition UCF101 CVRL (R3D-152 2x; K600) 3-fold Accuracy 93.9 # 4
Pre-Training Dataset Kinetics600 # 1
Frozen false # 1
Self-Supervised Action Recognition UCF101 (finetuned) CVRL (R3D-50; K400) 3-fold Accuracy 92.2 # 6
Pretrain K400 # 1
Self-Supervised Action Recognition UCF101 (finetuned) CVRL (R3D-152 2x; K600) 3-fold Accuracy 93.9 # 3
Pretrain K600 # 1

Results from Other Papers


Task Dataset Model Metric Name Metric Value Rank Uses Extra
Training Data
Source Paper Compare
Self-Supervised Action Recognition HMDB51 (finetuned) CVRL (R3D-50; K600) Top-1 Accuracy 68.0 # 5
Pretraining Dataset K600 # 1
Self-Supervised Action Recognition Kinetics-600 CVRL (R3D-50) Top-1 Accuracy 70.4 # 4
Self-Supervised Action Recognition UCF101 (finetuned) CVRL (R3D-50; K600) 3-fold Accuracy 93.4 # 5
Pretrain K600 # 1

Methods