Search Results for author:
Found 12 papers, 4 papers with code
MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs
MoE-Lightning can achieve up to 10. 3x higher throughput than state-of-the-art offloading-enabled LLM inference systems for Mixtral 8x7B on a single T4 GPU (16GB).
UNetMamba: An Efficient UNet-Like Mamba for Semantic Segmentation of High-Resolution Remote Sensing Images
Semantic segmentation of high-resolution remote sensing images is vital in downstream applications such as land-cover mapping, urban planning and disaster assessment. Existing Transformer-based methods suffer from the constraint between accuracy and efficiency, while the recently proposed Mamba is renowned for being efficient.
Optimizing Speculative Decoding for Serving Large Language Models Using Goodput
SmartSpec dynamically determines the best speculation length for each request (from 0, i. e., no speculation, to many tokens) -- hence the associated speculative execution costs -- based on a new metric called goodput, which characterizes the current observed load of the entire system and the speculation accuracy.
Towards Clinical AI Fairness: Filling Gaps in the Puzzle
The ethical integration of Artificial Intelligence (AI) in healthcare necessitates addressing fairness-a concept that is highly context-specific across medical fields.
Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity
Based on this analysis, we introduce M\'elange, a GPU allocation framework that navigates these diverse LLM service characteristics and heterogeneous GPU option space to automatically and efficiently derive the minimal-cost GPU allocation for a given LLM service.
Computing in the Era of Large Generative Models: From Cloud-Native to AI-Native
In this paper, we investigate the intersection of large generative AI models and cloud-native computing architectures.
Learned Best-Effort LLM Serving
Many applications must provide low-latency LLM service to users or risk unacceptable user experience.
QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources
Large Language Models (LLMs) have showcased remarkable impacts across a wide spectrum of natural language processing tasks.
Online Speculative Decoding
Adapting to query distribution mitigates the shifts between the training distribution of the draft model and the query distribution, enabling the draft model to more accurately predict the target model's outputs.
An Evaluation of Memory Optimization Methods for Training Neural Networks
To address the challenge, this paper summarizes the scenarios in which MOMs prove advantageous for model training.
GACT: Activation Compressed Training for Generic Network Architectures
Training large neural network (NN) models requires extensive memory resources, and Activation Compressed Training (ACT) is a promising approach to reduce training memory footprint.
Long-run User Value Optimization in Recommender Systems through Content Creation Modeling
Content recommender systems are generally adept at maximizing immediate user satisfaction but to optimize for the \textit{long-run} user value, we need more statistically sophisticated solutions than off-the-shelf simple recommender algorithms.
