Search Results for author:
Found 5 papers, 4 papers with code
CacheBlend: Fast Large Language Model Serving with Cached Knowledge Fusion
To speed up the prefill of the long LLM inputs, one can pre-compute the KV cache of a text and re-use the KV cache when the context is reused as the prefix of another LLM input.
White-box Compiler Fuzzing Empowered by Large Language Models
Nonetheless, prompting LLMs with compiler source-code information remains a missing piece of research in compiler testing.
CacheGen: KV Cache Compression and Streaming for Fast Language Model Serving
Compared to the recent systems that reuse the KV cache, CacheGen reduces the KV cache size by 3. 5-4. 3x and the total delay in fetching and processing contexts by 3. 2-3. 7x while having negligible impact on the LLM response quality in accuracy or perplexity.
A pruning method based on the dissimilarity of angle among channels and filters
Firstly, we train a sparse model by GL penalty, and impose an angle dissimilarity constraint on the channels and filters of convolutional network to obtain a more sparse structure.
Neural Network Panning: Screening the Optimal Sparse Network Before Training
Pruning on neural networks before training not only compresses the original models, but also accelerates the network training phase, which has substantial application value.
