提出Nexus采样，在固定内存下更好保留重要上下文。
Forget Without Compromise: Nexus Sampling for Streaming KV-Cache Eviction Under Fixed Budgets
- 用迭代注意力发现关键连接词，替代直接选Top-K
- 80%缓存淘汰率下，长文本任务误差仅1%以内
- 适合需要长时间记忆的推理场景
长上下文和代理型大模型工作负载使KV缓存超出任何固定内存预算，迫使推理系统在连续推理流中每一步都永久淘汰标记。现有方法均采用相同模板：基于每步直接注意力得分进行确定性Top-K选择，将一个低于阈值的步骤转化为不可逆判决，永久删除那些难以从噪声中识别出的重要标记。为此，我们提出Nexus Sampling，一种无需训练的淘汰方法，结合Nexus评分（对直接注意力进行迭代遍历以揭示桥梁标记）与加权水库采样（以包含概率替代确定性Top-K）。理论上，我们证明Nexus Sampling在长期存活微妙重要标记方面优于确定性Top-K。实验上，在80% KV缓存淘汰率下，Nexus Sampling在LongBench上与密集注意力差距不足1%，并在检索密集任务中超越Top-K基线，单序列缓存内存可减少至原有1/10。
原文摘要 · Abstract (English)
Long-context and agentic LLM workloads push the KV cache past any fixed memory budget, forcing the inference stack to permanently evict tokens at every step of a continuous-inference stream. Existing methods all share the same template, a per-step direct-attention score followed by deterministic top-$K$ selection, which converts a single below-cutoff step into an irreversible verdict and permanently erases any subtly important token that direct attention cannot single out from noise. To address this challenge, we propose Nexus Sampling, a training-free eviction method that pairs Nexus scoring, an iterative walk over direct attention that surfaces bridge tokens, with weighted reservoir sampling, which retains tokens with inclusion probability in place of deterministic top-$K$. Theoretically, we show that Nexus Sampling dominates deterministic top-$K$ in long-run survival of subtly important tokens. Empirically, at 80% KV cache eviction, Nexus Sampling matches dense attention within 1% on LongBench while outperforming top-$K$ baselines on retrieval-heavy tasks, with up to 10x smaller per-sequence cache memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。