用动态注意力机制压缩长文本生成的键值缓存，提升质量不增加延迟。
Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation
- 基于动量的时序注意力聚合，持续追踪令牌重要性变化
- 长文本生成任务中生成保真度提升2.3%-3.2%，延迟不变
- 适合追求长文本生成质量的模型部署场景
键值（KV）缓存仍是大语言模型在长文本生成任务中部署的主要瓶颈。以往方法对预填充和解码阶段的缓存采用统一压缩策略，但压缩预填充缓存会破坏关键上下文，影响性能。尽管保留预填充缓存至关重要，解码阶段的压缩仍研究不足，现有方法依赖固定的时效窗口或瞬时注意力。我们分析了注意力动态，发现关键令牌在长时程内持续获得关注，而局部推理则表现为短时爆发。静态启发式方法无法捕捉此行为，导致重要令牌过早淘汰或过时信息滞留。为此，我们提出Moment-KV，一种基于动量驱动的解码时KV缓存压缩方法。该方法将令牌重要性建模为连续演化的状态，通过带衰减的注意力聚合，同时捕获长期影响与近期相关性。实验表明，Moment-KV在长文本生成任务中显著提升了生成保真度（2.3-3.2%），同时保持解码延迟不变。
原文摘要 · Abstract (English)
Key-Value (KV) cache remains a major bottleneck for deploying Large Language Models (LLMs) in long-generation tasks. Prior work often applies uniform compression across both prefill and decoding caches, but compressing the prefill cache degrades performance by corrupting critical context. While preserving the prefill cache is essential, decoding-phase compression remains underexplored, with existing methods relying on rigid recency windows or instantaneous attention. Our analysis of attention dynamics reveals strong temporal patterns: critical tokens receive sustained attention over long horizons, while local reasoning involves short-lived bursts. Static heuristics fail to capture this behavior, leading to premature eviction of important tokens or retention of stale ones. We propose Moment-KV, a decoding-time KV cache compression method based on momentum-driven temporal attention aggregation. Our method models token importance as a continuously evolving state, where attention is aggregated with decay, capturing both long-term influence and recent relevance. Experiments show that Moment-KV significantly improves generation fidelity in long-generation tasks (2.3-3.2 %) while maintaining decoding latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。