揭示闭环环境下RNN学习路径差异及其内在机制
Learning Dynamics of RNNs in Closed-Loop Environments
- 建立线性RNN在闭环环境中的数学理论框架
- 发现闭环训练受短期策略优化与长期稳定性双重目标驱动
- 适用于神经科学启发的运动控制任务，对脑计算建模有启示
在神经科学启发的任务中，循环神经网络（RNN）是强大的脑计算模型。然而，传统训练多采用开环、监督式设置，而真实世界的学习发生在闭环环境中。本文建立了线性RNN在闭环情境下的学习动力学数学理论。我们首先证明，相同结构的RNN在开环与闭环模式下训练，其学习轨迹显著不同。通过解析分析闭环情形，我们揭示了学习过程分阶段演化，对应训练损失的变化。特别地，闭环RNN的学习动态由两个竞争目标共同决定：短期策略改进与代理-环境交互的长期稳定性。最后，我们将该框架应用于一个真实的运动控制任务，验证其广泛适用性。结果强调了在生物合理设定中建模闭环动力学的重要性。
原文摘要 · Abstract (English)
Recurrent neural networks (RNNs) trained on neuroscience-inspired tasks offer powerful models of brain computation. However, typical training paradigms rely on open-loop, supervised settings, whereas real-world learning unfolds in closed-loop environments. Here, we develop a mathematical theory describing the learning dynamics of linear RNNs trained in closed-loop contexts. We first demonstrate that two otherwise identical RNNs, trained in either closed- or open-loop modes, follow markedly different learning trajectories. To probe this divergence, we analytically characterize the closed-loop case, revealing distinct stages aligned with the evolution of the training loss. Specifically, we show that the learning dynamics of closed-loop RNNs, in contrast to open-loop ones, are governed by an interplay between two competing objectives: short-term policy improvement and long-term stability of the agent-environment interaction. Finally, we apply our framework to a realistic motor control task, highlighting its broader applicability. Taken together, our results underscore the importance of modeling closed-loop dynamics in a biologically plausible setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。