通过周期性刷新缓存，解决长序列生成中的误差累积问题。
Rectified Sparse Attention
- 用块稀疏注意力结合固定间隔的密集刷新机制
- 256K序列下实现2.42倍端到端加速，几乎无质量损失
- 适合需要高效长文本生成的场景，如数学推理与检索
大语言模型的高效长序列生成是一个关键挑战。尽管近期稀疏解码方法提升了效率，但存在KV缓存错位问题，导致近似误差累积并降低生成质量。本文提出修正稀疏注意力（ReSA），将块稀疏注意力与周期性密集修正相结合。通过在固定间隔使用密集前向传播刷新KV缓存，ReSA有效控制误差积累，并保持与预训练分布的一致性。在数学推理、语言建模和检索任务上的实验表明，ReSA实现了近乎无损的生成质量，同时显著提升效率。特别地，在256K序列长度解码下，端到端速度最高提升2.42倍，为可扩展的长上下文推理提供了实用方案。代码已开源：https://aka.ms/ReSA-LM。
原文摘要 · Abstract (English)
Efficient long-sequence generation is a critical challenge for Large Language Models. While recent sparse decoding methods improve efficiency, they suffer from KV cache misalignment, where approximation errors accumulate and degrade generation quality. In this work, we propose Rectified Sparse Attention (ReSA), a simple yet effective method that combines block-sparse attention with periodic dense rectification. By refreshing the KV cache at fixed intervals using a dense forward pass, ReSA bounds error accumulation and preserves alignment with the pretraining distribution. Experiments across math reasoning, language modeling, and retrieval tasks demonstrate that ReSA achieves near-lossless generation quality with significantly improved efficiency. Notably, ReSA delivers up to 2.42$\times$ end-to-end speedup under decoding at 256K sequence length, making it a practical solution for scalable long-context inference. Code is available at https://aka.ms/ReSA-LM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。