arXiv:2508.07101cs.CLcs.AI2025-08

通过统一跨头选词,实现快速推理且不降准确率

Less Is More: Fast and Accurate Reasoning with Cross-Head Unified Sparse Attention

  • 用全局稳定的选词机制替代逐头独立选择
  • 在多个基准上保持准确率,仅需原计算量的30%~50%注意力
  • 无需重训练,适合长文本推理场景

大型推理模型通过测试时扩展获得强性能,但短提示下的长序列生成带来巨大计算开销。稀疏注意力虽能降低延迟与内存占用,但现有方法常因选择误差在长生成过程中累积而降低准确率,或需昂贵重训练。本文提出 LessIsMore,一种无需训练的稀疏注意力机制,用于长时序推理。核心洞察是:推理中关键标记的全局性与稳定性——关键令牌在多头间共享且随解码步骤保持稳定。基于此,LessIsMore 强制跨头统一选词,并通过稳定最近窗口保留近期上下文,形成可跨层复用的全局一致令牌集。在多个模型族和挑战性推理基准上,LessIsMore 在显著减少注意力计算的同时,匹配或超越原有准确率。经内核级优化,其端到端解码速度最高提升1.6倍,稀疏注意力计算速度最高提升1.72倍;长上下文实验进一步验证了方法的通用性。代码已开源。

原文摘要 · Abstract (English)

Large reasoning models achieve strong performance through test-time scaling, but this incurs substantial computational overhead due to long decoding from short prompts. While sparse attention can reduce latency and memory usage, existing methods often degrade reasoning accuracy because selection errors accumulate over long generation horizons, or require costly retraining. We introduce LessIsMore, a training-free sparse attention mechanism for long-horizon reasoning. Our key insight is that token importance in reasoning is global and stable: critical tokens are largely shared across attention heads and remain stable over decoding steps. Guided by this structure, LessIsMore enforces cross-head unified token selection and preserves recent context via a stable recency window, yielding a globally consistent token set that can be reused across layers. Across multiple model families and challenging reasoning benchmarks, LessIsMore matches or improves accuracy while attending to substantially fewer tokens. With kernel-level optimizations, LessIsMore achieves up to $1.6\times$ end-to-end decoding speedup and up to $1.72\times$ faster sparse attention computation, with additional long-context results demonstrating the generality of our approach. Code is available at \href{https://github.com/DerrickYLJ/LessIsMore}{https://github.com/DerrickYLJ/LessIsMore}.

稀疏注意力推理加速长文本生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。