将递归网络计算搬进频域,实现高效长序列建模。
Frequency Domain Reservoir Computing

- 在频域中设计递归网络,避免传统循环的高昂计算开销。
- 训练与推理复杂度降至O(N),支持大规模模型部署。
- 性能媲美前沿模型,适合长时序预测与复杂动态建模任务。
尽管变换器的二次序列长度瓶颈促使循环模型复兴,但捕捉复杂动态仍需兼顾高效训练与高表达力的隐状态架构。回声状态网络(ESNs)通过固定循环权重绕过时间反向传播,实现闭式训练。然而,复杂任务所需的大规模储备池导致状态更新存在$/mathcal{O}(N^2)$瓶颈,限制了其与现代循环模型的规模匹配。为此,我们提出频域回声状态计算(FRESCO),一种完全在频域运行的ESN架构,通过避免域转换开销,实现密集非线性递归更新的$/mathcal{O}(N)$复杂度。通过新颖的维度零填充输入嵌入、压缩的 extit{FDh}读出层及原生频域非线性,FRESCO显著降低训练与推理的计算成本和能耗。此外,其在记忆基准、序列分类与多变量长期预测任务上达到当前最优性能,为密集循环架构提供了可扩展路径。
原文摘要 · Abstract (English)
While the quadratic sequence-length bottleneck of transformers has fueled a resurgence in recurrent models, effectively capturing complex dynamics requires architectures that balance efficient training with highly expressive latent states. Echo State Networks (ESNs) offer a compelling approach by utilizing fixed recurrent weights to circumvent backpropagation through time, enabling a closed-form training solution. However, achieving the expressivity needed for complex tasks demands large reservoirs, exposing an $\mathcal{O}(N^2)$ state-update bottleneck that prevents ESNs from matching the scale of contemporary recurrent models. To address this limitation, we introduce Frequency Domain Reservoir Computing (FRESCO), an ESN architecture operating entirely in the frequency domain while avoiding domain-shift overheads to achieve $\mathcal{O}(N)$ complexity for dense, non-linear recurrent updates. By employing a novel dimensional zero-padding input embedding, a packed \FDh readout, and a natively applied frequency-domain non-linearity, FRESCO drastically reduces computational costs and energy consumption of training and inference. Furthermore, FRESCO matches the state-of-the-art predictive performance on memory benchmarks, sequential classification, and multivariate long-horizon forecasting, offering a scalable path forward for dense recurrent architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。