Search Results for author:
Found 12 papers, 2 papers with code
Chakra: Advancing Performance Benchmarking and Co-design using Standardized Execution Traces
Benchmarking and co-design are essential for driving optimizations and innovation around ML models, ML software, and next-generation hardware.
ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale
In this paper, we extend the open-source ASTRA-sim infrastructure and endow it with the capabilities to model state-of-the-art and emerging distributed training models and platforms.
Mystique: Enabling Accurate and Scalable Generation of Production AI Benchmarks
We evaluate our methodology on several production AI models, and show that benchmarks generated with Mystique closely resemble original AI models, both in execution time and system-level metrics.
Impact of RoCE Congestion Control Policies on Distributed Training of DNNs
Our results indicate that previously proposed RoCE congestion control schemes have little impact on the end-to-end performance of training workloads, motivating the necessity of designing an optimized, yet low-overhead, congestion control scheme based on the characteristics of distributed training platforms and workloads.
Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Models
Distributed training is a solution to reduce DNN training time by splitting the task across multiple NPUs (e. g., GPU/TPU).
Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models
Deep learning recommendation models (DLRMs) are used across many business-critical services at Facebook and are the single largest AI application in terms of infrastructure demand in its data-centers.
Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems
Large-scale training is important to ensure high performance and accuracy of machine-learning models.
Distributed, Parallel, and Cluster Computing 68T05, 68M10 H.3.3; I.2.6; C.2.1
Automatic Model Parallelism for Deep Neural Networks with Compiler and Hardware Support
We model the computation and communication costs of a dataflow graph that embodies the neural network training process and then, partition the graph using heuristics in such a manner that the communication between compute devices is minimal and we have a good load balance.
Mixed Precision Training of Convolutional Neural Networks using Integer Operations
The state-of-the-art (SOTA) for mixed precision training is dominated by variants of low precision floating point operations, and in particular, FP16 accumulating into FP32 Micikevicius et al. (2017).
On Scale-out Deep Learning Training for Cloud and HPC
The exponential growth in use of large deep neural networks has accelerated the need for training these deep neural networks in hours or even minutes.
Deep Learning at 15PF: Supervised and Semi-Supervised Classification for Scientific Data
This paper presents the first, 15-PetaFLOP Deep Learning system for solving scientific pattern classification problems on contemporary HPC architectures.
Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
We design and implement a distributed multinode synchronous SGD algorithm, without altering hyper parameters, or compressing data, or altering algorithmic behavior.
