arXiv:2605.13473cs.LGcs.CL2026-05被引 1

用在线预处理提升线性注意力的上下文记忆能力,效果显著且不增加计算负担。

OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention

论文配图:OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention
图 1 · 摘自论文原文
  • 引入动态对角预处理,按特征维度自适应调整更新步长。
  • 340M模型上上下文召回率提升32%,13亿参数时召回残差降低39%。
  • 保持原有并行结构,适合大规模语言模型高效部署。

线性注意力与状态空间模型提供了常数内存的softmax注意力替代方案,但通常在上下文关联回忆方面表现不佳。Delta Rule通过一步在线梯度下降写入每个标记来缓解此问题,但其步长依赖于单一标量门控,忽略了内部目标的特征级曲率。我们提出在线缩放的DeltaNet(OSDN),在标量门控基础上引入通过超梯度反馈在线更新的对角预处理器。关键的是,这种右预处理在代数上等价于对写侧键的每特征缩放,使得OSDN严格保持DeltaNet硬件友好的分块并行流水线,且不产生高维状态开销。理论上,通过利用内部回归损失的精确二次结构,我们建立了相对于右牛顿比较器的超几何收敛,并证明了算法对齐的局部残差收缩界。为应对非平稳上下文,进一步引入自适应预处理遗忘(APF)以动态刷新过时校准。实验表明,OSDN在不同规模下表现优异:在3.4亿参数规模下,相比DeltaNet,JRT风格的上下文召回率提升32%;扩展至13亿参数时,召回残差比降低39%,同时在通用下游任务(如困惑度和LongBench)上保持一致性能,证明该在线预处理机制可在十亿参数级别有效迁移与放大。

原文摘要 · Abstract (English)

Linear attention and state-space models offer constant-memory alternatives to softmax attention, but often struggle with in-context associative recall. The Delta Rule mitigates this by writing each token via one step of online gradient descent. However, its step size relies on a single scalar gate that ignores the feature-wise curvature of the inner objective. We propose Online Scaled DeltaNet (OSDN), which augments the scalar gate with a diagonal preconditioner updated online via hypergradient feedback. Crucially, this right-preconditioning is algebraically equivalent to a per-feature scaling of the write-side key. This equivalence allows OSDN to strictly preserve the hardware-friendly chunkwise parallel pipeline of DeltaNet without incurring high-dimensional state overhead. Theoretically, by exploiting the exact-quadratic structure of the inner regression loss, we establish super-geometric convergence against a right-Newton comparator and prove an algorithm-aligned token-local residual contraction bound. To handle non-stationary contexts, we further introduce Adaptive Preconditioner Forgetting (APF) to dynamically refresh stale calibration. Empirically, OSDN demonstrates strong performance across scales. At the 340M-parameter scale, OSDN improves JRT-style in-context recall by 32% over DeltaNet. Scaling to 1.3B parameters, it achieves a 39% reduction in the recall residual ratio while maintaining parity on general downstream tasks (e.g., perplexity and LongBench) -- demonstrating that our online-preconditioning mechanism effectively transfers and amplifies at the billion-parameter scale.

线性注意力在线优化预处理上下文记忆

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。