Search Results for author:
Found 15 papers, 8 papers with code
Computron: Serving Distributed Deep Learning Models with Model Parallel Swapping
In anticipation of workloads that involve serving many of such large models to handle different tasks, we develop Computron, a system that uses memory swapping to serve multiple distributed models on a shared GPU cluster.
Distributed-Memory Sparse Kernels for Machine Learning
Further, we give two communication-eliding strategies to reduce costs further for FusedMM kernels: either reusing the replication of an input dense matrix for the SDDMM and SpMM in sequence, or fusing the local SDDMM and SpMM kernels.
CoSA: Scheduling by Constrained Optimization for Spatial Accelerators
Recent advances in Deep Neural Networks (DNNs) have led to active development of specialized DNN accelerators, many of which feature a large number of processing elements laid out spatially, together with a multi-level memory hierarchy and flexible interconnect.
Avoiding Communication in Logistic Regression
Stochastic gradient descent (SGD) is one of the most widely used optimization methods for solving various machine learning problems.
Training EfficientNets at Supercomputer Scale: 83% ImageNet Top-1 Accuracy in One Hour
EfficientNets are a family of state-of-the-art image classification models based on efficiently scaled convolutional neural networks.
The Limit of the Batch Size
For the first time we scale the batch size on ImageNet to at least a magnitude larger than all previous work, and provide detailed studies on the performance of many state-of-the-art optimization schemes under this setting.
Auto-Precision Scaling for Distributed Deep Learning
Our experimental results show that for many applications, APS can train state-of-the-art models by 8-bit gradients with no or only a tiny accuracy loss (<0. 05%).
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
In this paper, we first study a principled layerwise adaptation strategy to accelerate training of deep neural networks using large mini-batches.
Ranked #11 on
Question Answering
on SQuAD1.1 dev
(F1 metric)
Large-Batch Training for LSTM and Beyond
LEGW enables Sqrt Scaling scheme to be useful in practice and as a result we achieve much better results than the Linear Scaling learning rate scheme.
Avoiding Synchronization in First-Order Methods for Sparse Convex Optimization
Parallel computing has played an important role in speeding up convex optimization methods for big data analytics and large-scale machine learning (ML).
Avoiding Communication in Proximal Methods for Convex Optimization Problems
We formulate the algorithm for two different optimization methods on the Lasso problem and show that the latency cost is reduced by a factor of k while bandwidth and floating-point operation costs remain the same.
ImageNet Training in Minutes
If we can make full use of the supercomputer for DNN training, we should be able to finish the 90-epoch ResNet-50 training in one minute.
Asynchronous Parallel Greedy Coordinate Descent
n this paper, we propose and study an Asynchronous parallel Greedy Coordinate Descent (Asy-GCD) algorithm for minimizing a smooth function with bounded constraints.
Matrix Factorization at Scale: a Comparison of Scientific Data Analytics in Spark and C+MPI Using Three Case Studies
We explore the trade-offs of performing linear algebra using Apache Spark, compared to traditional C and MPI implementations on HPC platforms.
Distributed, Parallel, and Cluster Computing G.1.3; C.2.4
Communication-Optimal Parallel Algorithm for Strassen's Matrix Multiplication
We obtain a new parallel algorithm that is based on Strassen's fast matrix multiplication and minimizes communication.
Data Structures and Algorithms Computational Complexity Distributed, Parallel, and Cluster Computing Numerical Analysis Combinatorics Numerical Analysis 68W40, 68W10 F.2.1
