Search Results for author:
Found 11 papers, 5 papers with code
RoCBert: Robust Chinese Bert with Multimodal Contrastive Pretraining
Large-scale pretrained language models have achieved SOTA results on NLP tasks.
Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Models
To address these challenges, we introduce a system that can jointly optimize distributed execution and gradient checkpointing plans.
Elixir: Train a Large Language Model on a Small GPU Cluster
People who are inaccessible to a large number of GPUs resort to heterogeneous training systems for storing model parameters in CPU memory.
EnergonAI: An Inference System for 10-100 Billion Parameter Transformer Models
Although the AI community has expanded the model scale to the trillion parameter level, the practical deployment of 10-100 billion parameter models is still uncertain due to the latency, throughput, and memory constraints.
A Frequency-aware Software Cache for Large Recommendation System Embeddings
Deep learning recommendation models (DLRMs) have been widely applied in Internet companies.
FastFold: Reducing AlphaFold Training Time from 11 Days to 67 Hours
In this work, we present FastFold, an efficient implementation of AlphaFold for both training and inference.
Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training
The success of Transformer models has pushed the deep learning model scale to billions of parameters.
PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management
PatrickStar uses the CPU-GPU heterogeneous memory space to store the model data.
TurboTransformers: An Efficient GPU Serving System For Transformer Models
This paper designed a transformer serving system called TurboTransformers, which consists of a computing runtime and a serving framework to solve the above challenges.
swCaffe: a Parallel Framework for Accelerating Deep Learning Applications on Sunway TaihuLight
Second, we propose a set of optimization strategies for redesigning a variety of neural network layers based on Caffe.
RedSync : Reducing Synchronization Traffic for Distributed Deep Learning
Data parallelism has become a dominant method to scale Deep Neural Network (DNN) training across multiple nodes.
