通过智能分块与层级索引，让大模型长文本推理快3.6倍。
LycheeCluster: Efficient Long-Context Inference with Structure-Aware Chunking and Hierarchical KV Indexing
- 按语义边界分块，用三角不等式构建分层索引
- 检索时间从线性变对数，提速3.6倍且性能几乎不变
- 适合需要高效长文本生成的场景，如文档问答
注意力机制的二次复杂度和键值（KV）缓存的巨大内存开销，给处理长上下文的大语言模型带来严重计算与内存挑战。现有基于检索的方法常因固定大小分块破坏语义完整性，并存在低效的线性扫描问题。本文提出LycheeCluster，一种新型的高效KV缓存管理方法。该方法通过边界感知的分块策略保持局部语义连贯性，并基于三角不等式构建递归分层索引。这一设计将缓存检索从线性扫描转变为理论上有界、对数时间的剪枝过程，同时采用懒更新策略支持高效的流式生成。实验表明，LycheeCluster在端到端推理中实现最高3.6倍的速度提升，模型性能下降可忽略，优于当前最先进方法（如Quest、ClusterKV）。代码与内核将在发表后开源。
原文摘要 · Abstract (English)
The quadratic complexity of the attention mechanism and the substantial memory footprint of the Key-Value (KV) cache present severe computational and memory challenges for Large Language Models (LLMs) processing long contexts. Existing retrieval-based methods often compromise semantic integrity through fixed-size chunking and suffer from inefficient linear scanning. In this paper, we propose LycheeCluster, a novel method for efficient KV cache management. LycheeCluster preserves local semantic coherence via boundary-aware chunking and constructs a recursive hierarchical index rooted in the triangle inequality. This design transforms cache retrieval from a linear scan into a theoretically bounded, logarithmic-time pruning process, while a lazy update strategy supports efficient streaming generation. Experiments demonstrate that LycheeCluster achieves up to a 3.6x end-to-end inference speedup with negligible degradation in model performance, outperforming state-of-the-art KV cache management methods (e.g., Quest, ClusterKV). We will release our code and kernels after publication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。