提出一种新型神经网络框架,解决传统循环网络训练难的问题。
Reservoir Computing: A New Paradigm for Neural Networks
- 用随机固定权重的动态网络作为信息存储池,简化训练过程
- 在自然语言处理、生物计算等领域已成功应用
- 适合需要快速建模动态系统的科研人员参考
本文综述了储层计算(Reservoir Computing)的发展历程与应用前景。尽管人工神经网络早在20世纪40年代就已出现,但因难以实际应用而一度停滞。随着计算能力提升,深度神经网络在自然语言处理等任务中取得突破。然而,循环神经网络(RNN)由于存在梯度消失、收敛慢等问题,限制了其广泛应用。储层计算通过使用随机固定的非线性动态系统作为‘储层’,仅训练读出层,大幅降低训练难度。该方法兼具理论严谨性和计算高效性,已在自然语言处理、计算生物学、机器人学及物理学等多个领域实现成功应用。本文回顾了前馈与循环神经网络的发展脉络,系统阐述储层计算的原理与模型,并梳理近年来跨学科应用的研究成果。
原文摘要 · Abstract (English)
A Literature Review of Reservoir Computing. Even before Artificial Intelligence was its own field of computational science, humanity has tried to mimic the activity of the human brain. In the early 1940s the first artificial neuron models were created as purely mathematical concepts. Over the years, ideas from neuroscience and computer science were used to develop the modern Neural Network. The interest in these models rose quickly but fell when they failed to be successfully applied to practical applications, and rose again in the late 2000s with the drastic increase in computing power, notably in the field of natural language processing, for example with the state-of-the-art speech recognizer making heavy use of deep neural networks. Recurrent Neural Networks (RNNs), a class of neural networks with cycles in the network, exacerbates the difficulties of traditional neural nets. Slow convergence limiting the use to small networks, and difficulty to train through gradient-descent methods because of the recurrent dynamics have hindered research on RNNs, yet their biological plausibility and their capability to model dynamical systems over simple functions makes then interesting for computational researchers. Reservoir Computing emerges as a solution to these problems that RNNs traditionally face. Promising to be both theoretically sound and computationally fast, Reservoir Computing has already been applied successfully to numerous fields: natural language processing, computational biology and neuroscience, robotics, even physics. This survey will explore the history and appeal of both traditional feed-forward and recurrent neural networks, before describing the theory and models of this new reservoir computing paradigm. Finally recent papers using reservoir computing in a variety of scientific fields will be reviewed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。