Search Results for author:
Found 12 papers, 6 papers with code
RedSync : Reducing Synchronization Traffic for Distributed Deep Learning
Data parallelism has become a dominant method to scale Deep Neural Network (DNN) training across multiple nodes.
swCaffe: a Parallel Framework for Accelerating Deep Learning Applications on Sunway TaihuLight
Second, we propose a set of optimization strategies for redesigning a variety of neural network layers based on Caffe.
TurboTransformers: An Efficient GPU Serving System For Transformer Models
This paper designed a transformer serving system called TurboTransformers, which consists of a computing runtime and a serving framework to solve the above challenges.
PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management
PatrickStar uses the CPU-GPU heterogeneous memory space to store the model data.
Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training
The success of Transformer models has pushed the deep learning model scale to billions of parameters.
FastFold: Reducing AlphaFold Training Time from 11 Days to 67 Hours
In this work, we present FastFold, an efficient implementation of AlphaFold for both training and inference.
A Frequency-aware Software Cache for Large Recommendation System Embeddings
Deep learning recommendation models (DLRMs) have been widely applied in Internet companies.
EnergonAI: An Inference System for 10-100 Billion Parameter Transformer Models
Although the AI community has expanded the model scale to the trillion parameter level, the practical deployment of 10-100 billion parameter models is still uncertain due to the latency, throughput, and memory constraints.
Elixir: Train a Large Language Model on a Small GPU Cluster
To reduce GPU memory usage, memory partitioning, and memory offloading have been proposed.
Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Models
To address these challenges, we introduce a system that can jointly optimize distributed execution and gradient checkpointing plans.
AutoChunk: Automated Activation Chunk for Memory-Efficient Long Sequence Inference
The experiments demonstrate that AutoChunk can reduce over 80\% of activation memory while maintaining speed loss within 10%, extend max sequence length by 3. 2x to 11. 7x, and outperform state-of-the-art methods by a large margin.
RoCBert: Robust Chinese Bert with Multimodal Contrastive Pretraining
Large-scale pretrained language models have achieved SOTA results on NLP tasks.
