用固定滞后平滑视角重看测试时记忆,发现测量比累积更准
Eviction as Estimation: A Fixed-Lag Smoothing View of Test-Time Memory, and When Measuring Beats Accumulating
- 将记忆淘汰视为隐藏信号估计问题,引入固定滞后平滑机制
- 在特定场景下,测量已用记忆远优于累积注意力,小内存表现如大内存
- 自然文本中测量优势消失,适用于需精准判断重复使用的场景
具有有限工作内存的语言模型必须反复决定保留哪些信息。现有方法在项目到达时立即决策:从过去(StreamingLLM、H2O)或基于对未来的猜测(SnapKV)。本文将该决策重构为对隐藏信号(项目是否会被重用)的估计问题,将现有方法置于以提交延迟 $H$ 为轴的连续谱上:在线滤波器和学习预测器在 $H=0$ 提交,而 Belady 的离线最优解则在完整未来已知时达到。中间缺失的区间——固定滞后平滑——等待有限步数,观察正确近未来预测所关注的项目后才最终决定。这种“测量”机制将不可观测的未来请求转化为模型自身可读的信息。我们提出无需训练的策略 RMM,是 H2O 的严格推广,当测量均匀时即退化为 H2O。在内生性重用且时间分离的受控环境中,测量显著优于累积注意力,小内存表现接近大内存。但在第三方基准测试中(使用 NVIDIA KVPress 对比 SnapKV、H2O、StreamingLLM),RMM 在单轮问答中与 H2O 相当,在流式多轮任务中落后于两者。原因在于:自然文本中模型对多数词已准确,按正确性加权注意力影响微弱,导致测量效果退化为累积注意力,除非重用明显且内生,而标准基准未充分激发此类条件。本文贡献在于框架构建与诚实评估边界,非新SOTA。
原文摘要 · Abstract (English)
A language model with a bounded working memory must repeatedly decide which stored items to keep. Every deployed method decides the moment an item arrives, from the past (StreamingLLM, H2O) or from a guess about the future (SnapKV). We recast the choice as an estimation problem on a hidden signal, whether an item will be reused, placing existing methods on one axis, the commit lag $H$: online filters and learned predictors commit at $H=0$, while Belady's offline optimum sits where the whole future is known. The missing regime in between, fixed-lag smoothing, waits a bounded number of steps, observes which items a correct near-future prediction attended to, and only then commits. This measurement, demonstrated utility, turns Belady's unobservable future request into something we read off the model itself. We instantiate it as a training-free policy, RMM, a strict generalization of H2O that reduces to it exactly when the measurement is uniform. In controlled settings where reuse is endogenous and separated in time, demonstrated utility identifies used memory far better than accumulated attention, and a small bounded memory behaves like a much larger one. But on independent third-party benchmarks, run inside NVIDIA's KVPress harness against its own SnapKV, H2O, and StreamingLLM implementations, the advantage mostly disappears: RMM is on par with H2O for single-turn question answering and loses to both H2O and SnapKV in a streaming multi-turn setting. The cause is simple: on natural text the model is correct about most tokens, so weighting attention by correctness barely changes it, and demonstrated utility collapses onto accumulated attention unless reuse is sharp and endogenous, which standard benchmarks do not exercise. Our contribution is the framework and an honest map of when measuring beats accumulating, not a new state of the art.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。