Search Results for author:
Found 6 papers, 3 papers with code
InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU
To enable efficient and practical long-context utilization, we introduce InfiniteHiP, a novel, and practical LLM inference framework that accelerates processing by dynamically eliminating irrelevant context tokens through a modular hierarchical token pruning algorithm.
Training-Free Exponential Context Extension via Cascading KV Cache
The transformer's context window is vital for tasks such as few-shot learning and conditional generation as it preserves previous tokens for active memory.
A Training-free Sub-quadratic Cost Transformer Model Serving Framework With Hierarchically Pruned Attention
In addition to improving the time complexity of the attention mechanism, we further optimize GPU memory usage by implementing KV cache offloading, which stores only $O(\log T)$ tokens on the GPU while maintaining similar decoding throughput.
SEA: Sparse Linear Attention with Estimated Attention Mask
SEA estimates the attention matrix with linear complexity via kernel-based linear attention, then subsequently creates a sparse attention matrix with a top-k selection to perform a sparse attention operation.
K-MHaS: A Multi-label Hate Speech Detection Dataset in Korean Online News Comment
Online hate speech detection has become an important issue due to the growth of online content, but resources in languages other than English are extremely limited.
Minimax Risk in Estimating Kink Threshold and Testing Continuity
We derive a risk lower bound in estimating the threshold parameter without knowing whether the threshold regression model is continuous or not.
