Search Results for author:
Found 7 papers, 3 papers with code
TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval
To address these challenges, we propose TeleRAG, an efficient inference system that reduces RAG latency with minimal GPU memory requirements.
Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs
Long-context models are essential for many applications but face inefficiencies in loading large KV caches during decoding.
BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching
Offline batch inference, which leverages the flexibility of request batching to achieve higher throughput and lower costs, is becoming more popular for latency-insensitive applications.
Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
By only loading the Top-K critical KV cache pages for attention, Quest significantly speeds up self-attention without sacrificing accuracy.
Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models
Large Language Models (LLMs) with the Mixture-of-Experts (MoE) architectures have shown promising performance on various tasks.
Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
To maximize LLMs' serving throughput, we introduce Atom, a low-bit quantization method that achieves high throughput improvements with negligible accuracy loss.
Practical Algorithms for Learning Near-Isometric Linear Embeddings
Motivated by non-negative matrix factorization, we reformulate our problem into a Frobenius norm minimization problem, which is solved by the Alternating Direction Method of Multipliers (ADMM) and develop an algorithm, FroMax.
