Omni-sourced Webly-supervised Learning for Video Recognition

We introduce OmniSource, a novel framework for leveraging web data to train video recognition models. OmniSource overcomes the barriers between data formats, such as images, short videos, and long untrimmed videos for webly-supervised learning. First, data samples with multiple formats, curated by task-specific data collection and automatically filtered by a teacher model, are transformed into a unified form. Then a joint-training strategy is proposed to deal with the domain gaps between multiple data sources and formats in webly-supervised learning. Several good practices, including data balancing, resampling, and cross-dataset mixup are adopted in joint training. Experiments show that by utilizing data from multiple sources and formats, OmniSource is more data-efficient in training. With only 3.5M images and 800K minutes videos crawled from the internet without human labeling (less than 2% of prior works), our models learned with OmniSource improve Top-1 accuracy of 2D- and 3D-ConvNet baseline models by 3.0% and 3.9%, respectively, on the Kinetics-400 benchmark. With OmniSource, we establish new records with different pretraining strategies for video recognition. Our best models achieve 80.4%, 80.5%, and 83.6 Top-1 accuracies on the Kinetics-400 benchmark respectively for training-from-scratch, ImageNet pre-training and IG-65M pre-training.

PDF Abstract ECCV 2020 PDF ECCV 2020 Abstract

Results from the Paper

Ranked #2 on Action Recognition on UCF101 (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Action Recognition HMDB-51 OmniSource (SlowOnly-8x8-R101-RGB + I3D Flow) Average accuracy of 3 splits 83.8 # 8
Action Classification Kinetics-400 OmniSource SlowOnly R101 8x8 (Scratch) Acc@1 80.4 # 48
Acc@5 94.4 # 41
Action Classification Kinetics-400 OmniSource SlowOnly R101 8x8(ImageNet pretrain) Acc@1 80.5 # 44
Acc@5 94.4 # 41
Action Classification Kinetics-400 OmniSource irCSN-152 (IG-Kinetics-65M pretrain) Acc@1 83.6 # 23
Action Recognition UCF101 OmniSource (SlowOnly-8x8-R101-RGB + I3D-Flow) 3-fold Accuracy 98.6 # 2