不用反向传播训练的循环神经网络,提升训练速度并保持精度。
Koopman-informed recurrent neural networks
- 用随机特征+科伊曼算子理论构造网络权重,无需梯度优化。
- 在混沌系统预测等任务中,训练速度显著快于传统方法。
- 适合追求高效训练的时序建模与动力系统研究者。
循环神经网络在时间依赖问题中表现优异,但在动力系统建模中,基于反向传播的时间训练易出现梯度爆炸或消失。本文提出科伊曼启发的循环神经网络,通过随机特征网络与科伊曼算子理论结合,直接构造网络权重与偏置,无需梯度方法。单个循环模块的隐藏参数随机采样,外层权重由扩展动态模式分解构建。该方法缓解了反向传播带来的常见问题,并使网络可借助科伊曼理论进行分析。在时间序列、混沌系统预测、控制问题及真实数据上的实验表明,该模型在预测精度相当的前提下,训练时间显著优于常用梯度方法。
原文摘要 · Abstract (English)
Recurrent neural networks are a successful neural architecture for many time-dependent problems, including time series analysis, forecasting, and modeling of dynamical systems. In the context of dynamical systems, training with backpropagation through time can lead to challenges arising from exploding or vanishing gradients. In this contribution, we introduce Koopman-informed recurrent neural networks, a computational approach to construct all weights and biases of a recurrent neural network without using gradient-based methods. The approach is based on a combination of random feature networks and Koopman operator theory for dynamical systems. The hidden parameters of a single recurrent block are sampled at random, while the outer weights are constructed using extended dynamic mode decomposition. This approach alleviates some problems with backpropagation commonly related to recurrent networks. The connection to Koopman operator theory also allows us to start using results in this area to analyze recurrent neural networks. In computational experiments on time series, forecasting for chaotic dynamical systems, control problems, and on real-world data, we observe that with comparable forecasting accuracy, the training time of the Koopman-informed recurrent neural networks is significantly improved when compared to models trained with commonly used gradient-based methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。