Search Results for author:
Found 13 papers, 7 papers with code
Goku: Flow Based Video Generative Foundation Models
This paper introduces Goku, a state-of-the-art family of joint image-and-video generation models leveraging rectified flow Transformers to achieve industry-leading performance.
HybridFlow: A Flexible and Efficient RLHF Framework
Traditional RL can be modeled as a dataflow, where each node represents computation of a neural network (NN) and each edge denotes data dependencies between the NNs.
Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation
Multimodal large language models (MLLMs) have extended the success of large language models (LLMs) to multiple data types, such as image, text and audio, achieving significant performance in various domains, including multimodal translation, visual question answering and content generation.
ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development
In production, different LFMs are trained with various frameworks and storage backends, depending on model sizes and training scales.
QSync: Quantization-Minimized Synchronous Distributed Training Across Hybrid Devices
A number of production deep learning clusters have attempted to explore inference hardware for DNN training, at the off-peak serving hours with many inference GPUs idling.
LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization
The immense sizes of LLMs have led to very high resource demand and cost for running the models.
MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
Training LLMs at this scale brings unprecedented challenges to training efficiency and stability.
CDMPP: A Device-Model Agnostic Framework for Latency Prediction of Tensor Programs
Considering the large space of DNN models and devices that impede direct profiling of all combinations, recent efforts focus on building a predictor to model the performance of DNN models on different devices.
dPRO: A Generic Profiling and Optimization System for Expediting Distributed DNN Training
Distributed training using multiple devices (e. g., GPUs) has been widely adopted for learning DNN models over large datasets.
BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessing
Extensive experiments on various GNN models and large graph datasets show that BGL significantly outperforms existing GNN training systems by 20. 68x on average.
DL2: A Deep Learning-driven Scheduler for Deep Learning Clusters
DL2 is a DL-driven scheduler for DL clusters, targeting global training job expedition by dynamically resizing resources allocated to jobs.
Optimus: An Efficient Dynamic Resource Scheduler for Deep Learning Clusters
A deep learning training job is resource-intensive and time-consuming.
Online Job Scheduling in Distributed Machine Learning Clusters
In a shared cluster handling multiple training jobs, a fundamental issue is how to efficiently schedule jobs and set the number of concurrent workers to run for each job, such that server resources are maximally utilized and model training can be completed in time.
Distributed, Parallel, and Cluster Computing
