发现RNN保持持续振荡的条件，为设计有记忆能力的时序模型提供新思路。
Generative System Dynamics in Recurrent Neural Networks
- 通过偏对称权重矩阵和双曲正切类激活函数，实现稳定振荡
- 非线性激活函数可维持极限环并提升数值积分稳定性
- 适合研究长期依赖建模与动态记忆机制的开发者参考
本研究探讨了具有非线性激活函数的循环神经网络（RNN）在连续时间下的动态特性。目标是识别RNN在不收敛至静态固定点的情况下产生持续振荡的条件。研究发现，偏对称权重矩阵是在线性与非线性配置下实现稳定极限环的关键。进一步证明，具有双曲正切特性的激活函数（奇函数、有界、连续）能保持状态空间中的运动不变性，从而维持振荡动态。数值模拟显示，非线性激活函数不仅可维持极限环，还能增强系统积分过程的数值稳定性，缓解通常与前向欧拉法相关的不稳定性问题。该分析的实验结果揭示了设计能够捕捉复杂时序依赖关系的神经架构的实际考量，即提升递归模型记忆能力的策略。
原文摘要 · Abstract (English)
In this study, we investigate the continuous time dynamics of Recurrent Neural Networks (RNNs), focusing on systems with nonlinear activation functions. The objective of this work is to identify conditions under which RNNs exhibit perpetual oscillatory behavior, without converging to static fixed points. We establish that skew-symmetric weight matrices are fundamental to enable stable limit cycles in both linear and nonlinear configurations. We further demonstrate that hyperbolic tangent-like activation functions (odd, bounded, and continuous) preserve these oscillatory dynamics by ensuring motion invariants in state space. Numerical simulations showcase how nonlinear activation functions not only maintain limit cycles, but also enhance the numerical stability of the system integration process, mitigating those instabilities that are commonly associated with the forward Euler method. The experimental results of this analysis highlight practical considerations for designing neural architectures capable of capturing complex temporal dependencies, i.e., strategies for enhancing memorization skills in recurrent models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。