发现经典初始化在长序列下会失效，提出改进方案防止隐藏状态爆炸
Revisiting Glorot Initialization for Long-Range Linear Recurrences
- 针对长序列设计更稳定的权重初始化方法
- 理论证明序列长度达√n时原方法即不稳
- 适合研究RNN长期依赖的学者与工程师
恰当的初始化对循环神经网络（RNN）至关重要，尤其在长序列推理任务中，重复应用同一权重矩阵会导致信号消失或爆炸。当前线性递归常用Glorot初始化，其设计基于无限宽度、固定长度的理想假设，不适用于处理长序列的RNN。本文证明，Glorot初始化实际存在不稳定性：谱半径微小正偏差会随时间放大，导致隐藏状态爆炸。理论分析表明，当序列长度 t = O(√n)（n为隐藏层维度）时，即可引发不稳定。为此，我们提出一种简单、维度感知的重缩放策略，将谱半径略调至1以下，有效防止信号快速爆炸或衰减。结果表明，标准初始化在长序列场景下可能失效，亟需专门的稳定递归初始化理论。
原文摘要 · Abstract (English)
Proper initialization is critical for Recurrent Neural Networks (RNNs), particularly in long-range reasoning tasks, where repeated application of the same weight matrix can cause vanishing or exploding signals. A common baseline for linear recurrences is Glorot initialization, designed to ensure stable signal propagation--but derived under the infinite-width, fixed-length regime--an unrealistic setting for RNNs processing long sequences. In this work, we show that Glorot initialization is in fact unstable: small positive deviations in the spectral radius are amplified through time and cause the hidden state to explode. Our theoretical analysis demonstrates that sequences of length $t = O(\sqrt{n})$, where $n$ is the hidden width, are sufficient to induce instability. To address this, we propose a simple, dimension-aware rescaling of Glorot that shifts the spectral radius slightly below one, preventing rapid signal explosion or decay. These results suggest that standard initialization schemes may break down in the long-sequence regime, motivating a separate line of theory for stable recurrent initialization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。