arXiv:2606.28560cs.CLcs.LG2026-06

用斐波那契间距+静态调度,稀疏注意力在长序列上表现远超密集注意力。

Depth-Staggered Fibonacci Spacing for Sparse Attention: Static Schedules Beat Learned Dilation and Extrapolate Where Dense Attention Fails

  • 每层用固定递增的间隔选择关注位置,不需学习参数。
  • 训练长度内略差于稠密模型,但推理长度达4倍时仍稳定,而稠密模型崩溃。
  • 适合需要长上下文处理的任务,如超长文本生成与理解。

我们研究稀疏自注意力机制,其中每个查询关注一个密集局部窗口和一组斐波那契间距的偏移量,每层有一个标量α控制间距压缩或扩展。在21个语言模型(60M参数、512维隐状态、16层、426M训练词)上,比较四种α设置方式:固定值、逐层学习、静态线性递增、互质重排递增,以及一个匹配可达范围的2的幂次控制。三个结果突出:第一,静态逐层递增优于固定和学习的α,且与基础模型无关——同一递增策略使2的幂次基线超越固定斐波那契,并达到学习型斐波那契水平;第二,逐层学习无效:未超越静态调度,且推理延迟增加约五倍;第三,也是最关键的,所有稀疏变体在四倍训练长度下仍保持性能几乎不变,而匹配的稠密基线崩溃(4倍长度时困惑度上升201%),这归因于固定偏移仅查询训练中见过的相对位置。同时报告两个负面结果:训练长度下最优稀疏模型困惑度比稠密基线高约26%,且增益在上下文各位置均匀分布,而非集中在长距离。

原文摘要 · Abstract (English)

We study sparse self-attention in which each query attends to a dense local window plus a set of Fibonacci-spaced offsets, with a per-layer scalar alpha that compresses or expands the spacing. Across 21 language models trained under one matched recipe (60M parameters, 512 hidden, 16 layers, 426M tokens), we compare four ways of setting alpha across depth: fixed, per-layer learned, a static linear stagger, and a coprime (anti-gridding) reassignment of that stagger, together with a reach-matched power-of-2 control. Three results stand out. First, a static per-layer stagger improves perplexity over both fixed and learned alpha, and the gain is base-agnostic: applying the same stagger to a power-of-2 base lifts it above fixed Fibonacci and to parity with learned Fibonacci attention. Second, learning per layer is inert: it does not beat the static schedule and costs roughly five times the inference latency. Third, and most consequential, all sparse variants extrapolate to four times their training length with little or no degradation, whereas a recipe-matched dense baseline collapses (perplexity rises by 201% at 4x length); we attribute this to fixed-offset attention only ever querying relative positions seen during training. We also report two honest negatives: at training length the best sparse model has about 26% higher perplexity than the dense baseline, and the staggering gain is uniform across context positions rather than concentrated at long range.

稀疏注意力长序列建模斐波那契间距可扩展性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。