Unifying KV Cache Compression for Large Language Models with LeanKV

no code implementations • 4 Dec 2024 • Yanqi Zhang, Yuwei Hu, Runyuan Zhao, John C. S. Lui, Haibo Chen

Large language models (LLMs) demonstrate exceptional performance but incur high serving costs due to substantial memory demands, with the key-value (KV) cache being a primary bottleneck.

Quantization

ChipExpert: The Open-Source Integrated-Circuit-Design-Specific Large Language Model

no code implementations • 26 Jul 2024 • Ning Xu, Zhaoyang Zhang, Lei Qi, Wensuo Wang, Chao Zhang, Zihao Ren, Huaiyuan Zhang, Xin Cheng, Yanqi Zhang, Zhichao Liu, Qingwen Wei, Shiyang Wu, Lanlan Yang, Qianfeng Lu, Yiqun Ma, Mengyao Zhao, Junbo Liu, Yufan Song, Xin Geng, Jun Yang

Finally, to mitigate the hallucinations of ChipExpert, we have developed a Retrieval-Augmented Generation (RAG) system, based on the IC design knowledge base.

Language Modeling Language Modelling +1

Analytically-Driven Resource Management for Cloud-Native Microservices

no code implementations • 5 Jan 2024 • Yanqi Zhang, Zhuangzhuang Zhou, Sameh Elnikety, Christina Delimitrou

Resource management for cloud-native microservices has attracted a lot of recent attention.

Management

Sinan: Data-Driven, QoS-Aware Cluster Management for Microservices

no code implementations • 27 May 2021 • Yanqi Zhang, Weizhe Hua, Zhuangzhuang Zhou, Edward Suh, Christina Delimitrou

Cloud applications are increasingly shifting from large monolithic services, to large numbers of loosely-coupled, specialized microservices.

Management

Leveraging Deep Learning to Improve the Performance Predictability of Cloud Microservices

no code implementations • 2 May 2019 • Yu Gan, Yanqi Zhang, Kelvin Hu, Dailun Cheng, Yuan He, Meghna Pancholi, Christina Delimitrou

We show that Seer correctly anticipates QoS violations 91% of the time, and avoids the QoS violation to begin with in 84% of cases.