通过回溯修正注意力输出，提升长文本生成效率与准确率
Retrospective Sparse Attention for Efficient Long-Context Generation
- 回溯式稀疏注意力：用后续新内容修正过去注意力结果
- 相比现有方法，有效缓存暴露提升1.6倍，准确率最高提高21.9%
- 适合长文本生成、代码推理等需高精度上下文的任务
大语言模型在推理、代码生成和多轮对话等长上下文任务中应用日益广泛。然而，推理过程受限于键值（KV）缓存，其内存占用随序列长度线性增长，成为每步解码的延迟瓶颈。现有KV缓存压缩方法虽能识别并加载关键少数词元，但主要针对输入上下文，未能解决长期解码中累积的注意力误差。本文提出RetroAttention，一种新型的KV缓存更新技术，通过利用后续解码步骤中新到达的键值条目，回溯性地修正过去的注意力输出。通过维护轻量级输出缓存，RetroAttention使过往查询能高效补充更多上下文信息，仅带来极小延迟开销。该方法打破固定注意力输出范式，实现对先前近似结果的持续修正。大量实验表明，RetroAttention在多个长文本生成基准上均优于当前最优（SOTA）KV压缩方法，有效KV暴露度提升达1.6倍，准确率最高提升21.9%。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly deployed in long-context tasks such as reasoning, code generation, and multi-turn dialogue. However, inference over extended contexts is bottlenecked by the Key-Value (KV) cache, whose memory footprint grows linearly with sequence length and dominates latency at each decoding step. While recent KV cache compression methods identify and load important few tokens, they focus predominantly on input contexts and fail to address the cumulative attention errors that arise during long decoding. In this paper, we introduce RetroAttention, a novel KV cache update technique that retrospectively revises past attention outputs using newly arrived KV entries from subsequent decoding steps. By maintaining a lightweight output cache, RetroAttention enables past queries to be efficiently supplemented with more contexts, while incurring minimal latency overhead. This breaks the fixed-attention-output paradigm and allows continual correction of prior approximations. Extensive experiments on long-generation benchmarks show that RetroAttention consistently outperforms state-of-the-art (SOTA) KV compression methods, increasing effective KV exposure by up to 1.6$\times$ and accuracy by up to 21.9\%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。