Connecting Low-Loss Subspace for Personalized Federated Learning

16 Sep 2021  ·  Seok-Ju Hahn, Minwoo Jeong, Junghye Lee ·

Due to the curse of statistical heterogeneity across clients, adopting a personalized federated learning method has become an essential choice for the successful deployment of federated learning-based services. Among diverse branches of personalization techniques, a model mixture-based personalization method is preferred as each client has their own personalized model as a result of federated learning. It usually requires a local model and a federated model, but this approach is either limited to partial parameter exchange or requires additional local updates, each of which is helpless to novel clients and burdensome to the client's computational capacity. As the existence of a connected subspace containing diverse low-loss solutions between two or more independent deep networks has been discovered, we combined this interesting property with the model mixture-based personalized federated learning method for improved performance of personalization. We proposed SuPerFed, a personalized federated learning method that induces an explicit connection between the optima of the local and the federated model in weight space for boosting each other. Through extensive experiments on several benchmark datasets, we demonstrated that our method achieves consistent gains in both personalization performance and robustness to problematic scenarios possible in realistic services.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Personalized Federated Learning CIFAR-10 SuPerFed-MM ACC@1-50Clients 94.05 # 1
ACC@1-100Clients 93.25 # 1
ACC@1-500Clients 90.81 # 1
Personalized Federated Learning CIFAR-10 SuPerFed-LM ACC@1-50Clients 93.88 # 2
ACC@1-100Clients 93.20 # 2
ACC@1-500Clients 89.63 # 2
Personalized Federated Learning CIFAR-100 SuPerFed-LM ACC@5-100Clients 62.50 # 1
Personalized Federated Learning CIFAR-100 SuPerFed-MM ACC@5-100Clients 60.14 # 2
Personalized Federated Learning FEMNIST SuPerFed-MM Acc@1 85.20 # 1
Acc@5 99.16 # 1
Personalized Federated Learning FEMNIST SuPerFed-LM Acc@1 83.36 # 2
Acc@5 98.81 # 2
Personalized Federated Learning MNIST SuPerFed-LM ACC@1-50Clients 99.48 # 1
ACC@1-100Clients 99.31 # 2
ACC@1-500Clients 98.83 # 2
Personalized Federated Learning MNIST SuPerFed-MM ACC@1-100Clients 99.38 # 1
ACC@1-500Clients 99.24 # 1
Personalized Federated Learning Shakespeare SuPerFed-LM Acc@1 54.52 # 1
Acc@5 83.97 # 2
Personalized Federated Learning Shakespeare SuPerFed-MM Acc@1 54.52 # 1
Acc@5 84.27 # 1
Personalized Federated Learning Tiny ImageNet SuPerFed-LM ACC@5-200Clients 49.29 # 2
Personalized Federated Learning Tiny ImageNet SuPerFed-MM ACC@5-200Clients 50.07 # 1