P/D-Serve: Serving Disaggregated Large Language Model at Scale

no code implementations • 15 Aug 2024 • Yibo Jin, Tao Wang, Huimin Lin, Mingyang Song, Peiyang Li, Yipeng Ma, Yicheng Shan, Zhengfan Yuan, Cailong Li, Yajing Sun, Tiandeng Wu, Xing Chu, Ruizhi Huan, Li Ma, Xiao You, Wenting Zhou, Yunpeng Ye, Wen Liu, Xiangkun Xu, Yongsheng Zhang, Tiantian Dong, Jiawei Zhu, Zhe Wang, Xijian Ju, Jianxun Song, Haoliang Cheng, Xiaojing Li, Jiandong Ding, Hefei Guo, Zhengyong Zhang

To overcome previous problems, this paper proposes an end-to-end system P/D-Serve, complying with the paradigm of MLOps (machine learning operations), which models end-to-end (E2E) P/D performance and enables: 1) fine-grained P/D organization, mapping the service with RoCE (RDMA over converged ethernet) as needed, to facilitate similar processing and dynamic adjustments on P/D ratios; 2) on-demand forwarding upon rejections for idle prefill, decoupling the scheduler from regular inaccurate reports and local queues, to avoid timeouts in prefill; and 3) efficient KVCache transfer via optimized D2D access.

Language Modeling Language Modelling +1

Can LSH (Locality-Sensitive Hashing) Be Replaced by Neural Network?

no code implementations • 15 Oct 2023 • Renyang Liu, Jun Zhao, Xing Chu, Yu Liang, Wei Zhou, Jing He

With the rapid development of GPU (Graphics Processing Unit) technologies and neural networks, we can explore more appropriate data structures and algorithms.

Robust Event Triggering Control for Lateral Dynamics of Intelligent Vehicles with Designable Inter-event Times

no code implementations • 14 Mar 2022 • Xing Chu, Zhi Liu, Lei Mao, Xin Jin, Zhaoxia Peng, Guoguang Wen

In this brief, an improved event-triggered update mechanism (ETM) for the linear quadratic regulator is proposed to solve the lateral motion control problem of intelligent vehicle under bounded disturbances.