arXiv:2608.18098cs.CLcs.AI2026-08

提出新型键值缓存机制,让对话系统更懂话题变化

Fractional Decay KV-Cache: Ownership-Aware Memory Management for Improved Inference Relevancy in Dialog Systems

  • 用双通道评分机制区分重要性和时效性,动态管理缓存
  • 在五类对话中提升6.7%准确率,话题切换速度提升3.6倍
  • 无需额外计算开销,适合部署在普通设备上

键值(KV)缓存对基于Transformer的对话系统高效自回归推理至关重要,但现有策略对所有缓存项一视同仁或使用粗粒度淘汰规则,无法随对话主题演变自适应调整。本文提出分数衰减键值缓存(FD-KVC),为每个缓存的键值对维护双通道评分机制:累计注意力通道追踪整体重要性(类似H2O),时序衰减与强化学习式更新的时效相关通道则反映近期相关性。两者结合使FD-KVC既能保留历史关键信息,又能快速响应话题转移。由归属损失函数驱动的自适应学习率确保收敛且无振荡。FD-KVC完全运行在CPU上,开销极低。在五个多样化的多轮对话场景(每场景600个对话)中,相比当前最优的重热点基线H2O,FD-KVC在复合后期对齐指标上提升+6.7%,话题突变场景提升+127%,渐进演化提升+87%,混合话题提升+30%。其话题适应速度比H2O快3.6倍,达到最高话题多样性(80.6%)。消融实验证实各组件贡献。

原文摘要 · Abstract (English)

Key-value (KV) caching is essential for efficient autoregressive inference in transformer based dialog systems, yet existing strategies treat all cached entries uniformly or apply coarse eviction heuristics that fail to adapt as dialog topics evolve. We propose Fractional Decay KV-Cache (FD-KVC), a novel algorithm that maintains a dual-channel scoring mechanism for each cached KV pair: a cumulative attention channel that tracks aggregate importance (akin to H2O), and a recency-weighted relevance channel governed by temporal decay and reinforcement-inspired updates. The combination enables FD-KVC to both preserve historically important tokens and rapidly adapt when dialog topics shift. An adaptive learning rate driven by an ownership loss function ensures convergence without oscillation. FD-KVC operates entirely on CPU with negligible overhead. Across five diverse multi-turn dialog scenarios with 600 dialogs each, FD-KVC outperforms H2O, the state-of-the-art heavy-hitter baseline, by +6.7% on composite late-turn alignment, with improvements of +127% on topic-shift, +87% on gradual evolution, and +30% on mixed-topic dialogs. FD-KVC adapts to new topics 3.6X faster than H2O and achieves the highest topic diversity (80.6%) across all methods. Ablation studies confirm the contribution of each component.

对话系统缓存优化推理加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。