门控机制让RNN的梯度更新随时间延迟和方向变化,提升训练稳定性。
Time-Scale Coupling Between States and Parameters in Recurrent Neural Networks
- 门控通过耦合状态与参数空间动态,改变梯度传播路径
- 实验验证门控产生滞后依赖的更新速率,并聚焦梯度至低维子空间
- 适合研究RNN优化机制或设计更高效门控结构的研究者
我们发现,即使使用固定的全局学习率,递归神经网络(RNN)中的门控机制也会引发滞后依赖且方向相关的有效学习率。这种现象源于状态空间的时间尺度(由门控参数化)与参数空间梯度下降动力学之间的耦合。通过对漏积分器和门控RNN推导精确雅可比矩阵并应用一阶展开,我们揭示了恒定标量或多维门控如何重塑梯度传播、调节有效步长,并在参数更新中引入各向异性。这些发现表明,门控不仅是信息流的过滤器,更是数据驱动的优化预处理器,与学习率调度、动量及自适应方法(如Adam)存在形式关联。实证模拟支持上述预测:在多个序列任务中,门控产生滞后依赖的有效学习率,并将梯度流集中于低维子空间,其各向异性程度可媲美甚至超过Adam。值得注意的是,门控与优化器驱动的自适应性分别塑造信用分配的不同方面:门控使状态空间传输对齐损失相关方向,而优化器调整参数空间更新尺度。总体而言,该工作从统一的动力系统视角阐明了门控如何耦合状态演化与参数更新,解释了为何门控架构在实践中具有鲁棒的可训练性。
原文摘要 · Abstract (English)
We show that gating mechanisms in recurrent neural networks (RNNs) induce lag-dependent and direction-dependent effective learning rates, even when training uses a fixed, global step size. This behavior arises from a coupling between state-space time-scales (parametrized by the gates) and parameter-space dynamics during gradient descent. By deriving exact Jacobians for leaky-integrator and gated RNNs and applying a first-order expansion, we make explicit how constant, scalar, and multi-dimensional gates reshape gradient propagation, modulate effective step sizes, and introduce anisotropy in parameter updates. These findings reveal that gates act not only as filters of information flow, but also as data-driven preconditioners of optimization, with formal connections to learning-rate schedules, momentum, and adaptive methods such as Adam. Empirical simulations corroborate these predictions: across several sequence tasks, gates produce lag-dependent effective learning rates and concentrate gradient flow into low-dimensional subspaces, matching or exceeding the anisotropic structure induced by Adam. Notably, gating and optimizer-driven adaptivity shape complementary aspects of credit assignment: gates align state-space transport with loss-relevant directions, while optimizers rescale parameter-space updates. Overall, this work provides a unified dynamical systems perspective on how gating couples state evolution with parameter updates, clarifying why gated architectures achieve robust trainability in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。