Search Results for author:
Found 10 papers, 6 papers with code
SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding
Low-latency decoding for large language models (LLMs) is crucial for applications like chatbots and code assistants, yet generating long outputs remains slow in single-query settings.
MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production
We present MegaScale-MoE, a production system tailored for the efficient training of large-scale mixture-of-experts (MoE) models.
MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design
Mixture-of-Experts (MoE) models face deployment challenges due to their large parameter counts and computational demands.
MoQa: Rethinking MoE Quantization with Multi-stage Data-model Distribution Awareness
However, their analysis is designed for dense LLMs and relies on the simple one-model-all-data mapping, which is unsuitable for MoEs.
Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts
The inter-device communication of a MoE layer can occupy 47% time of the entire model execution with popular models and frameworks.
ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
By evaluating ShadowKV on a broad range of benchmarks, including RULER, LongBench, and Needle In A Haystack, and models like Llama-3. 1-8B, Llama-3-8B-1M, GLM-4-9B-1M, Yi-9B-200K, Phi-3-Mini-128K, and Qwen2-7B-128K, we demonstrate that it can support up to 6$\times$ larger batch sizes and boost throughput by up to 3. 04$\times$ on an A100 GPU without sacrificing accuracy, even surpassing the performance achievable with infinite batch size under the assumption of infinite GPU memory.
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation.
vMCU: Coordinated Memory Management and Kernel Optimization for DNN Inference on MCUs
The key idea is to virtualize the limited memory of MCU as a large memory pool.
Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
To maximize LLMs' serving throughput, we introduce Atom, a low-bit quantization method that achieves high throughput improvements with negligible accuracy loss.
HASCO: Towards Agile HArdware and Software CO-design for Tensor Computation
Second, the overall design space composed of HW/SW partitioning, hardware optimization, and software optimization is huge.
