Search Results for author:
Found 3 papers, 2 papers with code
Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimization
Various parallelism, such as data, tensor, and pipeline parallelism, along with memory optimizations like activation checkpointing, redundancy elimination, and offloading, have been proposed to accelerate distributed training for Large Language Models.
Seesaw: High-throughput LLM Inference via Model Re-sharding
To improve the efficiency of distributed large language model (LLM) inference, various parallelization strategies, such as tensor and pipeline parallelism, have been proposed.
Tempo: Accelerating Transformer-Based Model Training through Memory Footprint Reduction
We implement Tempo and evaluate the throughput, memory usage, and accuracy/loss on the BERT Large pre-training task.
Cannot find the paper you are looking for? You can
Submit a new open access paper.
