Search Results for author:
Found 7 papers, 3 papers with code
Nonuniform-Tensor-Parallelism: Mitigating GPU failure impact for Scaled-up LLM Training
We also propose a rack-design with improved electrical and thermal capabilities in order to sustain power-boosting of scale-up domains that have experienced failures; combined with NTP, this can allow the DP replica with the reduced TP degree (i. e., with failed GPUs) to keep up with the others, thereby achieving near-zero throughput loss for large-scale LLM training.
PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Training
Unfortunately, PP model training can use GPUs inefficiently, especially at large scale, due to idle GPU time caused by pipeline bubbles, which are often 15-30% and can exceed 60% of the training job's GPU allocation.
GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism
GraphPipe partitions a DNN into a graph of stages, optimizes micro-batch schedules for these stages, and parallelizes DNN training using the discovered GPP strategies.
Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning
Some recent schedulers choose job resources for users, but do so without awareness of how DL training can be re-optimized to better utilize the provided resources.
Accelerating Deep Learning by Focusing on the Biggest Losers
This paper introduces Selective-Backprop, a technique that accelerates the training of deep neural networks (DNNs) by prioritizing examples with high loss at each iteration.
MLSys: The New Frontier of Machine Learning Systems
Machine learning (ML) techniques are enjoying rapidly increasing adoption.
MLtuner: System Support for Automatic Machine Learning Tuning
MLtuner automatically tunes settings for training tunables (such as the learning rate, the momentum, the mini-batch size, and the data staleness bound) that have a significant impact on large-scale machine learning (ML) performance.
