Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples

Few-shot classification refers to learning a classifier for new classes given only a few examples. While a plethora of models have emerged to tackle it, we find the procedure and datasets that are used to assess their progress lacking... To address this limitation, we propose Meta-Dataset: a new benchmark for training and evaluating models that is large-scale, consists of diverse datasets, and presents more realistic tasks. We experiment with popular baselines and meta-learners on Meta-Dataset, along with a competitive method that we propose. We analyze performance as a function of various characteristics of test tasks and examine the models' ability to leverage diverse training sources for improving their generalization. We also propose a new set of baselines for quantifying the benefit of meta-learning in Meta-Dataset. Our extensive experimentation has uncovered important research challenges and we hope to inspire work in these directions. read more

PDF Abstract ICLR 2020 PDF ICLR 2020 Abstract

Datasets


Introduced in the Paper:

Meta-Dataset

Used in the Paper:

COCO CUB-200-2011 miniImageNet Omniglot
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Few-Shot Image Classification Meta-Dataset fo-Proto-MAML Accuracy 63.428 # 8
Few-Shot Image Classification Meta-Dataset k-NN Accuracy 54.319 # 13
Few-Shot Image Classification Meta-Dataset Finetune Accuracy 58.758 # 10
Few-Shot Image Classification Meta-Dataset Rank k-NN Mean Rank 10.85 # 12
Few-Shot Image Classification Meta-Dataset Rank Finetune Mean Rank 8.7 # 9
Few-Shot Image Classification Meta-Dataset Rank fo-Proto-MAML Mean Rank 6.65 # 7

Methods


No methods listed for this paper. Add relevant methods here