arXiv:2510.01643cs.LG2025-10被引 2

提出无需限制条件的快速注意力机制,提升大模型效率与适用性。

Support Basis: Fast Attention Beyond Bounded Entries

  • 基于查询/键矩阵的亚高斯特性,分稀疏与稠密部分分别计算
  • 实现亚二次时间复杂度,误差逼近现有最优方法
  • 适用于真实大模型,支持多阈值设置,适合追求高效推理的研究者

大型语言模型在各类任务中表现优异,但其softmax注意力的二次复杂度仍是可扩展性的主要瓶颈。Alman和Song(NeurIPS 2023a;NeurIPS 2024a)提出了亚二次时间的注意力推理与训练算法,但依赖于严格的有界输入假设。我们发现该假设在现代大模型中极少成立,严重限制了其应用。本文提出支持基分解(support-basis decomposition),一种无需有界假设的精确且高效的注意力推理与训练方法。实验证明,查询与键矩阵的元素呈现亚高斯行为。我们据此对稀疏部分进行精确计算,对稠密部分采用多项式逼近。理论证明:本方法在无严格假设下仍可达到亚二次运行时间,并保持与先前工作相当的近似误差;实验验证了其计算效率与下游任务性能。进一步推广至多阈值设置,消除所有分布假设,首次为多项式注意力的实证成功提供理论依据。此外,我们表明软最大注意力可由多个多项式注意力近似,$\\\\\\_p$ 误差显著更小。

原文摘要 · Abstract (English)

Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks. However, the quadratic complexity of softmax attention remains a central bottleneck that limits their scalability. Alman and Song (NeurIPS 2023a; NeurIPS 2024a) proposed sub-quadratic time algorithms for attention inference and training, respectively, but they rely on the restrictive bounded-entry assumption. We show that this assumption rarely holds in practice, which significantly limits their applicability to modern LLMs. In this paper, we introduce support-basis decomposition, a new technique for accurate and efficient attention inference and training without the bounded-entry assumption. We empirically show that the entries of the query and key matrices exhibit sub-Gaussian behavior. Leveraging this widely observed property, we perform exact computation on sparse components and polynomial approximation on dense components. Without relying on restrictive assumptions, we theoretically show that our algorithm achieves sub-quadratic runtime while matching the approximation error of prior work, and we empirically validate its computational efficiency and downstream task performance. We further generalize our method to a multi-threshold setting that eliminates all distributional assumptions, providing the first theoretical justification for the empirical success of polynomial attention. Moreover, we show that softmax attention can be closely approximated by multiple polynomial attentions with significantly smaller $\ell_p$ error.

注意力机制大模型优化亚二次复杂度多项式逼近

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。