Search Results for author:
Found 3 papers, 1 papers with code
Llama 3 Meets MoE: Efficient Upcycling
Scaling large language models (LLMs) significantly improves performance but comes with prohibitive computational costs.
Ranked #2 on
Multi-task Language Understanding
on MMLU
(using extra training data)
Upcycling Large Language Models into Mixture of Experts
Upcycling pre-trained dense language models into sparse mixture-of-experts (MoE) models is an efficient approach to increase the model capacity of already trained models.
Gradient Sparification for Asynchronous Distributed Training
We let workers download model difference, instead of the global model, from the server, and the model difference information is also sparsified so that the information exchanged overhead is reduced by sparsifying the dual-way communication between the server and workers.
