CHESS通过分层语义选择，用1%缓存实现长文本推理质量超越全缓存。
CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inference
- 分层动态选择相关词元，保持上下文连贯性
- 仅用1%缓存即达全缓存效果，吞吐量提升4.56倍
- 适合追求低延迟长文本生成的部署场景
长上下文大模型推理需在低延迟下保持精度，但解码过程随上下文增长主要受限于键值缓存（KV cache）。现有剪枝方法多忽略步骤相关性和局部语义，导致质量下降；且不规则访问与选择开销使其实际加速有限。为此，我们提出 extbf{CHESS}，一个算法-系统协同设计的KV缓存管理方案。算法上，引入上下文感知的分层选择策略，动态重构当前解码所需的连贯上下文；系统上，粗粒度选择避免昂贵数据移动，充分实现理论稀疏性带来的实际加速。大量实验表明，CHESS 仅使用 extbf{1 ext extpercent} 的 KV 缓存即可超越全缓存性能，实现低延迟稳定推理，吞吐量最高提升 extbf{4.56$ imes$}，始终优于其他强基线。代码已公开于 exttt{https://anonymous.4open.science/r/CHESS/}。
原文摘要 · Abstract (English)
Long-context LLMs demand accurate inference at low latency, yet decoding becomes primarily constrained by KV cache as context grows. Prior pruning methods are largely context-agnostic: their token selection ignores step-wise relevance and local semantics, which undermines quality. Moreover, their irregular accesses and selection overheads yield only limited wall-clock speedups. To address this, we propose \textbf{CHESS}, an \textit{algorithm-system co-design} KV-cache management system. Algorithmically, CHESS introduces a context-aware, hierarchical selection policy that dynamically reconstructs a coherent context for the current decoding. System-wise, coarse granularity selection eliminates expensive data movement, fully realizing practical acceleration from theoretical sparsity. Extensive evaluations demonstrate that CHESS surpasses Full-KV quality using only \textbf{1\%} of the KV cache, delivers low-latency stable inference with up to \textbf{4.56$\times$} higher throughput, and consistently outperforms other strong baselines. Code is available at \href{https://anonymous.4open.science/r/CHESS-9958/}{https://anonymous.4open.science/r/CHESS/}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。