通过残差正交连接提升深度回声状态网络的长期记忆能力
Deep Residual Echo State Networks: exploring residual orthogonal connections in untrained Recurrent Neural Networks
- 用分层残差结构连接未训练的递归层，增强时间建模能力
- 在多个时序任务上优于传统浅层与深层回声状态网络
- 适合需要长序列预测且注重计算效率的研究者
回声状态网络（ESN）是储备计算（RC）框架中一类无需训练的循环神经网络，以快速高效的训练著称。然而，传统ESN在处理长期信息时表现不佳。本文提出一种基于时间残差连接的新型深度未训练RNN——深度残差回声状态网络（DeepResESN）。通过构建多层未训练的残差递归结构，显著提升了记忆容量与长期时序建模能力。研究了随机生成与固定结构等不同正交配置对网络动态的影响，并从数学上推导出确保系统稳定的充分必要条件。实验证明，该方法在多种时序任务中持续优于传统的浅层和深层储备计算模型。整体而言，DeepResESN为设计具有更好预测精度的分层ESN提供了新路径，在不牺牲储备计算原有计算优势的前提下，有效提升长序列建模性能。
原文摘要 · Abstract (English)
Echo State Networks (ESNs) are a particular type of untrained Recurrent Neural Networks (RNNs) within the Reservoir Computing (RC) framework, popular for their fast and efficient learning. However, traditional ESNs often struggle with long-term information processing. In this paper, we introduce a novel class of deep untrained RNNs based on temporal residual connections, called Deep Residual Echo State Networks (DeepResESNs). We show that leveraging a hierarchy of untrained residual recurrent layers significantly boosts memory capacity and long-term temporal modeling. For the temporal residual connections, we consider different orthogonal configurations, including randomly generated and fixed-structure, and study their effect on network dynamics. A thorough mathematical analysis outlines necessary and sufficient conditions to ensure stable dynamics within DeepResESN. Empirically, the proposed approach consistently outperforms traditional shallow and deep RC on a variety of time series tasks. Overall, DeepResESN offers a promising approach for designing hierarchical ESNs with better prediction accuracy on long sequences, without sacrificing the computational advantages that make RC attractive.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。