Search Results for author:
Found 5 papers, 2 papers with code
TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training
By stacking training optimizations, we demonstrate accelerations of 65. 08% with 1D parallelism at the 128-GPU scale (Llama 3. 1 8B), an additional 12. 59% with 2D parallelism at the 256-GPU scale (Llama 3. 1 70B), and an additional 30% with 3D parallelism at the 512-GPU scale (Llama 3. 1 405B) on NVIDIA H100 GPUs over optimized baselines.
PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
It is widely acknowledged that large models have the potential to deliver superior performance across a broad range of domains.
Supporting Very Large Models using Automatic Dataflow Graph Partitioning
This paper presents Tofu, a system that partitions very large DNN models across multiple GPU devices to reduce per-GPU memory footprint.
Unifying Data, Model and Hybrid Parallelism in Deep Learning via Tensor Tiling
We present this automatic tiling in a new system, SoyBean, that can act as a backend for Tensorflow, MXNet, and others.
Get More With Less: Near Real-Time Image Clustering on Mobile Phones
In this paper, we show that by combining the computing power distributed over a number of phones, judicious optimization choices, and contextual information it is possible to execute the end-to-end pipeline entirely on the phones at the edge of the network, efficiently.
