稀疏注意力在长文本生成中反而更慢，新方法可减少90%耗 token
Lil: Less is Less When Applying Post-Training Sparse-Attention Algorithms in Long-Decode Stage
- 发现稀疏注意力导致信息丢失，反而生成更长序列
- 提出早停机制，最多降低90%令牌消耗，精度下降<2%
- 适合需要高效推理的长文本生成任务
大型语言模型在复杂任务中表现强大，部署规模日益扩大，对推理效率要求提高。以往工作将推理分为预填充和解码阶段，其中解码阶段主导总延迟。为降低解码阶段的时间与内存开销，研究者引入稀疏注意力算法。本文通过实证与理论分析发现，稀疏注意力可能产生悖论：信息丢失常导致生成序列显著变长，我们称之为“少即是少”（Less is Less, Lil）。为缓解此问题，我们提出一种早停算法，能在稀疏解码过程中检测信息损失超过增益的阈值。该算法在多个推理密集型基准上，最多减少90%的令牌消耗，同时精度下降不足2%。
原文摘要 · Abstract (English)
Large language models (LLMs) demonstrate strong capabilities across a wide range of complex tasks and are increasingly deployed at scale, placing significant demands on inference efficiency. Prior work typically decomposes inference into prefill and decode stages, with the decode stage dominating total latency. To reduce time and memory complexity in the decode stage, a line of work introduces sparse-attention algorithms. In this paper, we show, both empirically and theoretically, that sparse attention can paradoxically increase end-to-end complexity: information loss often induces significantly longer sequences, a phenomenon we term ``Less is Less'' (Lil). To mitigate the Lil problem, we propose an early-stopping algorithm that detects the threshold where information loss exceeds information gain during sparse decoding. Our early-stopping algorithm reduces token consumption by up to 90% with a marginal accuracy degradation of less than 2% across reasoning-intensive benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。