改进大模型推理的缓存机制，有效减少冗余结构信息干扰。
Adaptive Filtering of the KV Cache: Diagnosing and Correcting Structural-Role Bias in LLM Inference
- 基于角色条件分配，动态过滤无意义的结构符号
- 在5%缓存预算下将准确率从0%提升至88%
- 无需重训练，适合资源受限的部署场景
基于注意力的键值缓存淘汰策略（如H2O）通过累积注意力能量对令牌排序并保留权重最高的部分。在嵌套JSON等结构密集型输入中，该机制会过度保留非内容角色（如分隔符、空白），其能量比内容角色高一个数量级，导致结构键令牌被保留速率是答案值令牌的1.8倍，使5%缓存预算下的精确匹配准确率从88%暴跌至0%。反事实实验表明抑制键令牌是最优可部署滤波器。本文提出的无需重训练的角色条件分配方法，在子20%缓存预算下修复了63%-98%的H2O性能差距；在更高预算时，可小幅匹配或超越全缓存精度——仅需15MB线性角色探测器提供标签，推理开销极低，但与解析器级下游准确率的匹配仍待解决。
原文摘要 · Abstract (English)
Attention-based KV cache eviction (H2O and its descendants) compresses the memory-constrained state of a long-context model by ranking tokens on accumulated attention mass, treated here as signal energy, and keeping the heaviest. On schema-dense input streams such as nested JSON, this score acts as a non-stationary filter that disproportionately retains noise: a non-content sink role (delimiters or whitespace) carries an order of magnitude more energy than any content role, and structural KEY tokens are over-retained at roughly 1.8x the rate of the answer-carrying VALUE tokens, collapsing exact-match accuracy from 88% to 0% at a 5% budget as the signal-to-noise ratio of the retained state degrades. A counterfactual experiment establishes that suppressing KEY tokens is the best deployable filter. Our retraining-free, role-conditional allocation over SnapKV's windowed score, governed by a single tuned hyperparameter, closes 63-98% of the H2O gap at sub-20% budgets and, at higher budgets, modestly matches or exceeds full-cache accuracy -- a small, seed-sensitive denoising effect (borderline significant at B=0.50; not distinguishable from zero at B=0.30 over four seeds). A 15 MB linear role probe supplies these labels at negligible inference cost, though matching parser-level downstream accuracy remains open.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。