线性循环网络通过新结构实现单次前向传播的梯度更新。
Learning in the Recurrent State: Gradient Descent with Linear Recurrent Networks
- 设计带乘法读出和滑窗交叉注意力的对角状态，支持梯度下降式学习
- 单次前向传播即可完成小批量梯度更新，准确复现理论预测参数
- 适用于长序列任务和语言建模，为上下文学习提供可验证的归纳偏置
线性循环网络（LRNN）具备线性时间序列建模能力，但标准循环更新无法直接暴露监督所需的梯度信息。我们提出一种充分的构造性归纳偏置：将对角循环状态与乘法读出及短滑窗交叉注意力自注意力更新结合。由此构建的梯度基循环上下文学习器（GRIL）可在单次前向传播中对特定任务的线性预测器执行小批量梯度下降。该设计还可扩展至多步更新与交叉熵分类，通过有限的MLP扩展实现非线性回归。实验表明，训练后的GRIL在合成上下文学习任务中准确恢复了构造预测的行为与参数；相同架构在长程基准（Long Range Arena）和语言建模任务中也展现出有效性能。这些结果将滑窗交叉注意力视为一种实用且可检验的归纳偏置，使LRNN能够通过类梯度更新实现上下文学习。
原文摘要 · Abstract (English)
Linear recurrent networks (LRNNs) offer linear-time sequence modeling, but standard recurrent updates do not directly expose the supervised products needed for in-context gradient descent. We propose a sufficient constructive inductive bias for LRNNs: equip a diagonal recurrent state with multiplicative readout and a short sliding-window cross-product self-attention update. The resulting architecture, Gradient-based Recurrent In-context Learner (GRIL), can implement minibatch gradient descent on a task-specific linear predictor during a single forward pass. The same design extends to multi-step updates and cross-entropy classification, with a limited MLP-based extension to non-linear regression. Empirically, trained GRILs recover the behavior and parameters predicted by the construction on synthetic ICL tasks, and the same architectural bias yields useful performance on Long Range Arena and language modelling. These results present windowed cross-product self-attention as a practical, testable inductive bias for LRNNs that learn in context through gradient-descent-like updates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。