Search Results for author:
Found 15 papers, 11 papers with code
LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization
The immense sizes of LLMs have led to very high resource demand and cost for running the models.
MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
Training LLMs at this scale brings unprecedented challenges to training efficiency and stability.
CDMPP: A Device-Model Agnostic Framework for Latency Prediction of Tensor Programs
Considering the large space of DNN models and devices that impede direct profiling of all combinations, recent efforts focus on building a predictor to model the performance of DNN models on different devices.
LEMON: Lossless model expansion
Our empirical results demonstrate that LEMON reduces computational costs by 56. 7% for Vision Transformers and 33. 2% for BERT when compared to training from scratch.
ByteComp: Revisiting Gradient Compression in Distributed Training
It first designs a decision tree abstraction to express all the compression strategies and develops empirical models to timeline tensor computation, communication, and compression to enable ByteComp to derive the intricate interactions among tensors.
dPRO: A Generic Profiling and Optimization System for Expediting Distributed DNN Training
Distributed training using multiple devices (e. g., GPUs) has been widely adopted for learning DNN models over large datasets.
Compressed Communication for Distributed Training: Adaptive Methods and System
Recently, there has been a growing interest in using gradient compression to reduce the communication overhead of the distributed training.
CSER: Communication-efficient SGD with Error Reset
The scalability of Distributed Stochastic Gradient Descent (SGD) is today limited by communication bottlenecks.
Accelerated Large Batch Optimization of BERT Pretraining in 54 minutes
Using the proposed LANS method and the learning rate scheme, we scaled up the mini-batch sizes to 96K and 33K in phases 1 and 2 of BERT pretraining, respectively.
Is Network the Bottleneck of Distributed Training?
As such, we advocate that the real challenge of distributed training is for the network community to develop high-performance network transport to fully utilize the network capacity and achieve linear scale-out.
ResNeSt: Split-Attention Networks
It is well known that featuremap attention and multi-path representation are important for visual recognition.
Ranked #8 on
Instance Segmentation
on COCO test-dev
(APM metric)
Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates
When scaling distributed training, the communication overhead is often the bottleneck.
VisDrone-DET2019: The Vision Meets Drone Object Detection in Image Challenge Results
Results of 33 object detection algorithms are presented.
GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing
We present GluonCV and GluonNLP, the deep learning toolkits for computer vision and natural language processing based on Apache MXNet (incubating).
Dynamic Mini-batch SGD for Elastic Distributed Training: Learning in the Limbo of Resources
One difficulty we observe is that the noise in the stochastic momentum estimation is accumulated over time and will have delayed effects when the batch size changes.
