Search Results for author:
Found 8 papers, 4 papers with code
Towards Universal Performance Modeling for Machine Learning Training on Multi-GPU Platforms
Characterizing and predicting the training performance of modern machine learning (ML) workloads on compute systems with compute and communication spread between CPUs, GPUs, and network devices is not only the key to optimization and planning but also a complex goal to achieve.
Chakra: Advancing Performance Benchmarking and Co-design using Standardized Execution Traces
Benchmarking and co-design are essential for driving optimizations and innovation around ML models, ML software, and next-generation hardware.
Pre-train and Search: Efficient Embedding Table Sharding with Pre-trained Neural Cost Models
In this work, we explore a "pre-train, and search" paradigm for efficient sharding.
Mystique: Enabling Accurate and Scalable Generation of Production AI Benchmarks
We evaluate our methodology on several production AI models, and show that benchmarks generated with Mystique closely resemble original AI models, both in execution time and system-level metrics.
DreamShard: Generalizable Embedding Table Placement for Recommender Systems
Although prior work has explored learning-based approaches for the device placement of computational graphs, embedding table placement remains to be a challenging problem because of 1) the operation fusion of embedding tables, and 2) the generalizability requirement on unseen placement tasks with different numbers of tables and/or devices.
AutoShard: Automated Embedding Table Sharding for Recommender Systems
This is a significant design challenge of distributed systems named embedding table sharding, i. e., how we should partition the embedding tables to balance the costs across devices, which is a non-trivial task because 1) it is hard to efficiently and precisely measure the cost, and 2) the partition problem is known to be NP-hard.
Building a Performance Model for Deep Learning Recommendation Model Training on GPUs
We show that our general performance model not only achieves low prediction error on DLRM, which has highly customized configurations and is dominated by multiple factors but also yields comparable accuracy on other compute-bound ML models targeted by most previous methods.
Alternate Model Growth and Pruning for Efficient Training of Recommendation Systems
Our method leverages structured sparsification to reduce computational cost without hurting the model capacity at the end of offline training so that a full-size model is available in the recurring training stage to learn new data in real-time.
