Search Results for author:
Found 5 papers, 3 papers with code
MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism
Combined with distinct model parallelism for each module, MegaScale-Infer effectively hides communication overhead and maximizes GPU utilization.
Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts
The inter-device communication of a MoE layer can occupy 47% time of the entire model execution with popular models and frameworks.
ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
By evaluating ShadowKV on a broad range of benchmarks, including RULER, LongBench, and Needle In A Haystack, and models like Llama-3. 1-8B, Llama-3-8B-1M, GLM-4-9B-1M, Yi-9B-200K, Phi-3-Mini-128K, and Qwen2-7B-128K, we demonstrate that it can support up to 6$\times$ larger batch sizes and boost throughput by up to 3. 04$\times$ on an A100 GPU without sacrificing accuracy, even surpassing the performance achievable with infinite batch size under the assumption of infinite GPU memory.
FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion
Overall, it can achieve up to 1. 24x speedups for training over Megatron-LM on a cluster of 128 GPUs with various GPU generations and interconnects, and up to 1. 66x and 1. 30x speedups for prefill and decoding inference over vLLM on a cluster with 8 GPUs with various GPU generations and interconnects.
NGEMM: Optimizing GEMM for Deep Learning via Compiler-based Techniques
Various approaches have been developed by leveraging techniques such as vectorization and memory layout to improve the performance of integer GEMM.
