固定极点RNN在数据少时比可学习极点更高效稳定,适合实时在线训练。
When Learning Hurts: Fixed-Pole RNN for Real-Time Online Training
- 固定递归极点,只训练读出层,降低优化难度。
- 数据有限时,固定极点网络收敛更快,性能更优。
- 适合资源受限的实时在线学习场景。
循环神经网络(RNN)可视为离散时间状态空间模型,其状态演化对应由前馈权重和递归极点控制的无限冲激响应(IIR)滤波操作。尽管理论上可通过时间反向传播(BPTT)联合优化所有参数,但这种联合学习带来巨大计算开销,在训练数据有限时通常不切实际。回声状态网络(ESNs)通过固定递归动态、仅训练线性读出层,实现了高效稳定的在线适应。本文从理论与实证角度分析发现,在数据受限的实时学习场景中,学习递归极点无法带来实际收益。分析表明,极点学习使优化问题高度非凸,需更多样本与迭代才能收敛。实验显示,对于复数数据,梯度下降常陷入长期平台期,先进优化器改善有限。相反,固定极点架构即使在数据有限时也能产生稳定且条件良好的状态表示。数值结果表明,固定极点网络以更低训练复杂度实现更优性能,更适合在线实时任务。
原文摘要 · Abstract (English)
Recurrent neural networks (RNNs) can be interpreted as discrete-time state-space models, where the state evolution corresponds to an infinite-impulse-response (IIR) filtering operation governed by both feedforward weights and recurrent poles. While, in principle, all parameters including pole locations can be optimized via backpropagation through time (BPTT), such joint learning incurs substantial computational overhead and is often impractical for applications with limited training data. Echo state networks (ESNs) mitigate this limitation by fixing the recurrent dynamics and training only a linear readout, enabling efficient and stable online adaptation. In this work, we analytically and empirically examine why learning recurrent poles does not provide tangible benefits in data-constrained, real-time learning scenarios. Our analysis shows that pole learning renders the weight optimization problem highly non-convex, requiring significantly more training samples and iterations for gradient-based methods to converge to meaningful solutions. Empirically, we observe that for complex-valued data, gradient descent frequently exhibits prolonged plateaus, and advanced optimizers offer limited improvement. In contrast, fixed-pole architectures induce stable and well-conditioned state representations even with limited training data. Numerical results demonstrate that fixed-pole networks achieve superior performance with lower training complexity, making them more suitable for online real-time tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。