Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and Tasks

1 code implementation • 2 Oct 2024 • Edan Kinderman, Itay Hubara, Haggai Maron, Daniel Soudry

Many recent methods aim to merge neural networks (NNs) with identical architectures trained on different tasks to obtain a single multi-task model.

Knowledge Distillation