发现小规模LSTM在最佳训练阶段呈现临界动力学，与大脑神经活动相似。
Towards Critical Branching Mechanism in Recurrent Neural Networks
- 通过分析LSTM隐藏层动态，发现小模型在最优训练期具自相似级联特征。
- 小模型分支参数趋近1，符合临界态统计；大模型则保持亚临界状态。
- 提出混合分支过程模型，解释长期相关性与噪声的共存机制，适合神经动力学研究者。
临界性被认为是生物神经系统的组织原则，但在人工神经网络中的起源和意义仍不明确。我们分析了经过训练的长短期记忆（LSTM）网络的隐藏状态动态，发现小型网络在接近其最优训练阶段时表现出尺度无关的级联统计特征，分支参数接近1，表明其处于近临界状态，而大型模型则始终处于亚临界状态。为解释亚临界分支与稳健 $1/f^β$ 噪声并存的现象，我们引入了一种混合分支过程框架，将异质分支动态与长程时间相关性联系起来。结果表明，LSTM中的类临界行为是一种涌现的、容量依赖的动力学状态。
原文摘要 · Abstract (English)
Criticality has been proposed as a key organizing principle in biological neural systems, yet its origin and relevance in artificial neural networks remain unclear. We analyze hidden-state dynamics in trained long short-term memory (LSTM) networks and show that small networks near their optimal training epochs (steps) exhibit scale-free avalanche statistics and branching parameters close to unity, indicative of near-critical dynamics, while larger models remain subcritical. To explain the coexistence of subcritical branching with robust $1/f^β$ noise, we introduce a mixture branching process framework that links heterogeneous branching dynamics to long-range temporal correlations. These results identify critical-like behavior in LSTMs as an emergent, capacity-dependent dynamical regime.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。