Search Results for author:
Found 5 papers, 3 papers with code
Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs
Ultimately, our experimental findings demonstrate that a 300B MoE LLM can be effectively trained on lower-performance devices while achieving comparable performance to models of a similar scale, including dense and MoE models.
EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models
Local SGD methods have been proposed to address these issues, but their effectiveness remains limited to small-scale training due to additional memory overhead and lack of concerns on efficiency and stability.
Couler: Unified Machine Learning Workflow Optimization in Cloud
This variety poses a challenge for end-users in terms of mastering different engine APIs.
mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs
In this paper, we present mLoRA, a parallelism-efficient fine-tuning system designed for training multiple LoRA across GPUs and machines.
DLRover-RM: Resource Optimization for Deep Recommendation Models Training in the Cloud
To overcome them, we introduce DLRover-RM, an elastic training framework for DLRMs designed to increase resource utilization and handle the instability of a cloud environment.
