Search Results for author:
Found 20 papers, 6 papers with code
Don't Waste Your Bits! Squeeze Activations and Gradients for Deep Neural Networks via TinyScript
Recent years have witnessed intensive research interests on training deep neural networks (DNNs) more efficiently by quantization-based compression methods, which facilitate DNNs training in two ways: (1) activations are quantized to shrink the memory consumption, and (2) gradients are quantized to decrease the communication cost.
Galvatron: An Automatic Distributed System for Efficient Foundation Model Training
Galvatron is a distributed system for efficiently training large-scale Foundation Models.
ByteScale: Efficient Scaling of LLM Training with a 2048K Context Length on More Than 12,000 GPUs
The mismatch between data heterogeneity and static mesh causes redundant communication and imbalanced computation, degrading the training efficiency.
Demystifying Workload Imbalances in Large Transformer Model Training over Variable-length Sequences
To optimize large Transformer model training, efficient parallel computing and advanced data management are essential.
FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelism
Furthermore, we implement our method in a high-performance system that supports adaptive parallelization in distributed LLM training.
Gradual Learning: Optimizing Fine-Tuning with Partially Mastered Knowledge in Large Language Models
During the pretraining phase, large language models (LLMs) acquire vast amounts of knowledge from extensive text corpora.
Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learning
Recently, using large language models (LLMs) for data augmentation has led to considerable improvements in unsupervised sentence embedding models.
Retrofitting Temporal Graph Neural Networks with Transformer
We also adapt Transformer codebase to train TF-TGN efficiently with multiple GPUs.
Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Scheduling
Recent foundation models are capable of handling multiple tasks and multiple data modalities with the unified base model structure and several specialized model components.
MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training
By leveraging fine-grained activation memory management, MEMO facilitates efficient training of 7B LLM with 1 million sequence length on just 8 A800 GPUs, achieving an MFU of 52. 30%.
PQCache: Product Quantization-based KVCache for Long Context LLM Inference
During the prefilling phase, we apply PQ to tokens' keys for each LLM layer and head.
Retrieval-Augmented Generation for AI-Generated Content: A Survey
We first classify RAG foundations according to how the retriever augments the generator, distilling the fundamental abstractions of the augmentation methodologies for various retrievers and generators.
Generative and Contrastive Paradigms Are Complementary for Graph Self-Supervised Learning
For graph self-supervised learning (GSSL), masked autoencoder (MAE) follows the generative paradigm and learns to reconstruct masked graph edges or node features.
Improving Automatic Parallel Training via Balanced Memory Workload Optimization
Transformer models have emerged as the leading approach for achieving state-of-the-art performance across various application domains, serving as the foundation for advanced large-scale deep learning (DL) models.
Accelerating Text-to-Image Editing via Cache-Enabled Sparse Diffusion Inference
The key intuition behind our approach is to utilize the semantic mapping between the minor modifications on the input text and the affected regions on the output image.
Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent
Recent years have witnessed the unprecedented achievements of large-scale pre-trained models, especially the Transformer models.
Towards Communication-efficient Vertical Federated Learning Training via Cache-enabled Local Updates
Vertical federated learning (VFL) is an emerging paradigm that allows different parties (e. g., organizations or enterprises) to collaboratively build machine learning models with privacy protection.
BlindFL: Vertical Federated Machine Learning without Peeking into Your Data
First, to address the functionality of VFL models, we propose the federated source layers to unite the data from different parties.
K-Core Decomposition on Super Large Graphs with Limited Resources
For example, the distributed K-core decomposition algorithm can scale to a large graph with 136 billion edges without losing correctness with our divide-and-conquer technique.
An Experimental Evaluation of Large Scale GBDT Systems
Gradient boosting decision tree (GBDT) is a widely-used machine learning algorithm in both data analytic competitions and real-world industrial applications.
