Search Results for author:
Found 7 papers, 5 papers with code
Toward Inference-optimal Mixture-of-Expert Large Language Models
Like dense models, training MoEs requires answering the same question: given a training budget, what is the optimal allocation on the model size and number of tokens?
LLM360: Towards Fully Transparent Open-Source LLMs
The recent surge in open-source Large Language Models (LLMs), such as LLaMA, Falcon, and Mistral, provides diverse options for AI practitioners and researchers.
RedCoast: A Lightweight Tool to Automate Distributed Training of LLMs on Any GPU/TPUs
In this work, we present RedCoast(Redco), a lightweight and user-friendly tool crafted to automate distributed training and inference for LLMs, as well as to simplify ML pipeline development.
LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset
Studying how people interact with large language models (LLMs) in real-world scenarios is increasingly important due to their widespread use in various applications.
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Evaluating large language model (LLM) based chat assistants is challenging due to their broad capabilities and the inadequacy of existing benchmarks in measuring human preferences.
Ranked #3 on
Long-Context Understanding
on Ada-LEval (TSort)
On Optimizing the Communication of Model Parallelism
This pattern emerges when the two paradigms of model parallelism - intra-operator and inter-operator parallelism - are combined to support large models on large clusters.
Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning
Existing model-parallel training systems either require users to manually create a parallelization plan or automatically generate one from a limited space of model parallelism configurations.
