ViC-MAE: Self-Supervised Representation Learning from Images and Video with Contrastive Masked Autoencoders

21 Mar 2023  ·  Jefferson Hernandez, Ruben Villegas, Vicente Ordonez ·

We propose ViC-MAE, a model that combines both Masked AutoEncoders (MAE) and contrastive learning. ViC-MAE is trained using a global featured obtained by pooling the local representations learned under an MAE reconstruction loss and leveraging this representation under a contrastive objective across images and video frames. We show that visual representations learned under ViC-MAE generalize well to both video and image classification tasks. Particularly, ViC-MAE obtains state-of-the-art transfer learning performance from video to images on Imagenet-1k compared to the recently proposed OmniMAE by achieving a top-1 accuracy of 86% (+1.3% absolute improvement) when trained on the same data and 87.1% (+2.4% absolute improvement) when training on extra data. At the same time ViC-MAE outperforms most other methods on video benchmarks by obtaining 75.9% top-1 accuracy on the challenging Something something-v2 video benchmark . When training on videos and images from a diverse combination of datasets, our method maintains a balanced transfer-learning performance between video and image classification benchmarks, coming only as a close second to the best supervised method.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Image Classification ImageNet ViC-MAE (ViT-L) Top 1 Accuracy 85% # 254
Action Classification Kinetics-400 ViC-MAE (ViT-L) Acc@1 85.1 # 52
Image Classification Places365 ViC-MAE (ViT-L) Top 1 Accuracy 59.5% # 4
Action Recognition Something-Something V2 ViC-MAE (ViT-L) Top-1 Accuracy 73.7 # 17

Methods