Search Results for author:
Found 5 papers, 3 papers with code
A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning
The availability of performant pre-trained models has led to a proliferation of fine-tuned expert models that are specialized to a particular domain or task.
Learning to Route Among Specialized Experts for Zero-Shot Generalization
Unlike past methods that learn to route among specialized models, PHATGOOSE explores the possibility that zero-shot generalization will be improved if different experts can be adaptively chosen for each token and at each layer in the model.
Git-Theta: A Git Extension for Collaborative Development of Machine Learning Models
Currently, most machine learning models are trained by centralized teams and are rarely updated.
Soft Merging of Experts with Adaptive Routing
To address this issue, we introduce Soft Merging of Experts with Adaptive Routing (SMEAR), which avoids discrete routing by using a single "merged" expert constructed via a weighted average of all of the experts' parameters.
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
ICL incurs substantial computational, memory, and storage costs because it involves processing all of the training examples every time a prediction is made.
Ranked #1 on
Few-Shot Text Classification
on RAFT
