Search Results for author:
Found 6 papers, 2 papers with code
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
However, batching multiple requests leads to an interleaving of prefill and decode iterations which makes it challenging to achieve both high throughput and low latency.
SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
SARATHI employs chunked-prefills, which splits a prefill request into equal sized chunks, and decode-maximal batching, which constructs a batch using a single prefill chunk and populates the remaining slots with decodes.
Synergy: Resource Sensitive DNN Scheduling in Multi-Tenant Clusters
Unfortunately, these schedulers do not consider the impact of a job's sensitivity to allocation of CPU, memory, and storage resources.
Memory Optimization for Deep Networks
Deep learning is slowly, but steadily, hitting a memory bottleneck.
Analyzing and Mitigating Data Stalls in DNN Training
We analyze nine different models across three tasks and four datasets while varying factors such as the amount of memory, number of CPU threads, storage device, GPU generation etc on servers that are a part of a large production cluster at Microsoft.
RECIPE : Converting Concurrent DRAM Indexes to Persistent-Memory Indexes
We present Recipe, a principled approach for converting concurrent DRAM indexes into crash-consistent indexes for persistent memory (PM).
Distributed, Parallel, and Cluster Computing Databases Data Structures and Algorithms
