Search Results for author:
Found 2 papers, 1 papers with code
APIServe: Efficient API Support for Large-Language Model Inferencing
Large language models are increasingly integrated with external tools and APIs like ChatGPT plugins to extend their capability beyond language-centric tasks.
SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification
Our evaluation shows that SpecInfer outperforms existing LLM serving systems by 1. 5-2. 8x for distributed LLM inference and by 2. 6-3. 5x for offloading-based LLM inference, while preserving the same generative performance.
Cannot find the paper you are looking for? You can
Submit a new open access paper.
