随机非线性循环网络的记忆容量可任意变化，与性能无关。
Memory Capacity of Nonlinear Recurrent Networks: Is it Informative?
- 通过输入过程尺度决定非线性RNN的记忆容量范围
- 记忆容量在理论上下界间连续变化，无法区分模型性能
- 揭示现有记忆容量定义对实际应用无意义
线性循环神经网络（RNN）的总记忆容量（MC）已被证明等于相应Kalman可控性矩阵的秩，且在连接权和输入权从正则分布中随机抽取时几乎必然达到最大值。这一事实引发了该指标在处理随机信号时区分能力的质疑。本文表明，随机非线性RNN的记忆容量可在已知上下界之间任意取值，仅取决于输入过程的尺度。这证实了线性和非线性情况下现有记忆容量定义均无实际价值。
原文摘要 · Abstract (English)
The total memory capacity (MC) of linear recurrent neural networks (RNNs) has been proven to be equal to the rank of the corresponding Kalman controllability matrix, and it is almost surely maximal for connectivity and input weight matrices drawn from regular distributions. This fact questions the usefulness of this metric in distinguishing the performance of linear RNNs in the processing of stochastic signals. This work shows that the MC of random nonlinear RNNs yields arbitrary values within established upper and lower bounds depending exclusively on the scale of the input process. This confirms that the existing definition of MC in linear and nonlinear cases has no practical value.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。