深度让RNN记忆更强、表达更复杂，尤其在多层交互下效果显著。
On the Role of Depth in the Expressivity of RNNs
- 通过多层递归结构增强记忆容量和输入变换能力。
- 深度提升使RNN的表达力随参数增长而高效提升。
- 适用于需要长期依赖建模的任务，如时序分析与生成。
前馈神经网络中深度带来的优势众所周知：多层线性变换与非线性激活的组合可实现复杂计算。尽管类似效应预期存在于循环神经网络（RNNs）中，但深度如何与循环机制共同影响其表达能力仍不明确。本文形式化证明，深度能以较少参数高效提升RNN的记忆容量，从而既增强复杂输入转换能力，又改善历史信息保留。我们进一步将分析扩展至2RNNs——一种引入输入与隐藏状态乘积交互的RNN推广模型。与仅靠非线性激活保持线性的传统RNN不同，2RNNs可实现多项式变换，其最大次数随深度增加而上升。我们还证明，乘积交互通常无法被分层非线性替代。最后，我们在合成与真实世界任务上验证了这些发现。
原文摘要 · Abstract (English)
The benefits of depth in feedforward neural networks are well known: composing multiple layers of linear transformations with nonlinear activations enables complex computations. While similar effects are expected in recurrent neural networks (RNNs), it remains unclear how depth interacts with recurrence to shape expressive power. Here, we formally show that depth increases RNNs' memory capacity efficiently with respect to the number of parameters, thus enhancing expressivity both by enabling more complex input transformations and improving the retention of past information. We broaden our analysis to 2RNNs, a generalization of RNNs with multiplicative interactions between inputs and hidden states. Unlike RNNs, which remain linear without nonlinear activations, 2RNNs perform polynomial transformations whose maximal degree grows with depth. We further show that multiplicative interactions cannot, in general, be replaced by layerwise nonlinearities. Finally, we validate these insights empirically on synthetic and real-world tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。