arXiv:2608.27763cs.LGcs.CL2026-08

提出快速权重注意力机制,实现持续学习中的高效在线更新与长序列外推。

Fast Weight Attention for Continual Learning

  • 基于快速权重记忆设计可微分的在线学习规则,支持局部自回归推理。
  • 在变位数加法任务中显著提升序列长度外推能力,语言建模性能保持竞争力。
  • 适用于需要持续学习、长期记忆与可控遗忘的序列建模场景。

循环快速权重记忆与选择性状态空间模型将不断扩展的上下文压缩为固定大小的循环状态,使状态转移成为一种在线学习规则。本文研究该规则在读写后自回归语义下的表现。针对此处考虑的前缀预测目标,步骤 $t$ 时揭示的局部快速记忆样本为前缀对齐对 $(oldsymbol{x}_t, oldsymbol{y}_t) = (oldsymbol{ ho}(oldsymbol{k}_{t-1}), oldsymbol{v}_t)$。常见的同步关联 $(oldsymbol{ ho}(oldsymbol{k}_t), oldsymbol{v}_t)$ 仍保持因果性,但优化不同的内部目标。我们推导了平方误差回归与负内积目标的一阶归一化更新。回归族包括 Falcon-1(标量 NLMS 更新)、Falcon-2(每列扩展)和 Falcon-3(滑动窗口小批量更新);对应的内积变体为 Falcon-1A/Falcon-2A/Falcon-3A。提供循环、掩码并行和分块并行形式,以及数值稳定的正衰减重归一化。代表性变体在语言建模中表现良好,并在变位数加法任务上提升长度外推性能。该框架分离了时间对齐、可塑性、遗忘与有限回放,在循环序列模型中具有清晰机制解耦。

原文摘要 · Abstract (English)

Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state, making the state transition an online learning rule. We study this rule under read-after-write autoregressive semantics. For the prefix-prediction objective considered here, the local fast-memory example revealed at step $t$ is the prefix-aligned pair $(\mathbf{x}_t,\mathbf{y}_t)=(ϕ(\mathbf{k}_{t-1}),\mathbf{v}_t)$. The common same-step association $(ϕ(\mathbf{k}_t),\mathbf{v}_t)$ remains causal, but optimizes a different internal objective. We derive normalized first-order updates for squared-error regression and negative inner-product objectives. The regression family comprises Falcon-1 (a scalar NLMS update), Falcon-2 (its per-column extension), and Falcon-3 (a sliding-window mini-batch update); Falcon-1A/Falcon-2A/Falcon-3A are the corresponding inner-product variants. We provide recurrent, masked-parallel, and chunk-parallel forms, together with numerically stable positive-decay renormalization. Representative variants remain competitive in language modeling and improve length extrapolation on variable-digit addition. This framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models.

持续学习快速权重序列建模在线学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。