SIMPLE: Specialized Model-Sample Matching for Domain Generalization

In domain generalization (DG), most existing methods aspire to fine-tune a specific pretrained model through novel DG algorithms. In this paper, we propose an alternative direction, i.e., to efficiently leverage a pool of pretrained models without fine-tuning. Through extensive empirical and theoretical evidence, we demonstrate that (1) pretrained models have possessed generalization to some extent while there is no single best pretrained model across all distribution shifts, and (2) out-of-distribution (OOD) generalization error depends on the fitness between the pretrained model and unseen test distributions. This analysis motivates us to incorporate diverse pretrained models and to dispatch the best matched models for each OOD sample by means of recommendation techniques. To this end, we propose SIMPLE, a specialized model-sample matching method for domain generalization. First, the predictions of pretrained models are adapted to the target domain by a linear label space transformation. A matching network aware of model specialty is then proposed to dynamically recommend proper pretrained models to predict each test sample. The experiments on DomainBed show that our method achieves significant performance improvements (up to 12.2% for individual dataset and 3.9% on average) compared to state-of-the-art (SOTA) methods and further achieves 6.1% gain via enlarging the pretrained model pool. Moreover, our method is highly efficient and achieves more than 1000 times training speedup compared to the conventional DG methods with fine-tuning a pretrained model. Code and supplemental materials are available at https://seqml.github.io/simple.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Domain Generalization DomainNet SIMPLE Average Accuracy 49.2 # 14
Domain Generalization DomainNet SIMPLE+ Average Accuracy 61.9 # 4
Domain Generalization Office-Home SIMPLE+ Average Accuracy 87.7 # 3
Domain Generalization Office-Home SIMPLE Average Accuracy 84.6 # 6
Domain Generalization PACS SIMPLE+ Average Accuracy 99.0 # 1
Domain Generalization PACS SIMPLE Average Accuracy 88.6 # 19
Domain Generalization TerraIncognita SIMPLE Average Accuracy 57.6 # 7
Domain Generalization TerraIncognita SIMPLE+ Average Accuracy 59.0 # 5
Domain Generalization VLCS SIMPLE Average Accuracy 79.9 # 15
Domain Generalization VLCS SIMPLE+ Average Accuracy 82.7 # 6

Methods


AWARE โ€ข Test