发现神经ODE中激活饱和会导致动态特性崩溃，限制模型表达能力。
Activation Saturation and Floquet Spectrum Collapse in Neural ODEs
- 激活饱和使神经ODE的输入雅可比矩阵被抑制，导致系统失去复杂动态
- 随着饱和加深，弗洛凯指数全趋零，无法实现强收缩或混沌敏感性
- 该现象是结构性限制，与训练好坏无关，适用于tanh等常见激活函数
我们证明，在具有饱和激活函数（如tanh、sigmoid）的自主神经微分方程（Neural ODE） $\dot{h}=f_θ(h)$中，若其MLP结构中有 $q$ 层隐藏层在区域 $U$ 上满足 $|σ'|\leδ$，则输入雅可比范数被抑制为 $ orm{Df_θ(x)}\le C(U)$；对满足 $ ext{sup}_x|σ'(x)|\le 1$ 的激活函数（如tanh、sigmoid），该界简化为 $C_Wδ^q$。这导致任意 $T$-周期轨道 $γ\subset U$ 上的每个弗洛凯（李雅普诺夫）指数均落入区间 $[-C(U), C(U)]$。当饱和加深（$δ\to 0$）时，弗洛凯谱完全坍缩：所有指数趋近于零，严重限制了系统产生强收缩或混沌敏感性的能力。该障碍为结构性限制，独立于训练质量，在推理阶段即生效。作为次要贡献，对于满足 $σ'>0$ 的激活函数，提出一种加权谱分解，获得更紧的界 $ ilde{C}(U)\le C(U)$，且该改进在流层面随 $T$ 指数放大。所有结论均通过斯图亚特-兰道振子进行数值验证；理论界解释了为何tanh-NODE在莫里斯-莱卡尔神经元模型上出现经验失败。
原文摘要 · Abstract (English)
We prove that activation saturation imposes a structural dynamical limitation on autonomous Neural ODEs $\dot{h}=f_θ(h)$ with saturating activations ($\tanh$, sigmoid, etc.): if $q$ hidden layers of the MLP $f_θ$ satisfy $|σ'|\leδ$ on a region~$U$, the input Jacobian is attenuated as $\norm{Df_θ(x)}\le C(U)$ (for activations with $\sup_{x}|σ'(x)|\le 1$, e.g.\ $\tanh$ and sigmoid, this reduces to $C_Wδ^q$), forcing every Floquet (Lyapunov) exponen along any $T$-periodic orbit $γ\subset U$ into the interval $[-C(U),\;C(U)]$. This is a collapse of the Floquet spectrum: as saturation deepens ($δ\to 0$), all exponents are driven to zero, limiting both strong contraction and chaotic sensitivity. The obstruction is structural -- it constrains the learned vector field at inference time, independent of training quality. As a secondary contribution, for activations with $σ'>0$, a saturation-weighted spectral factorisation yields a refined bound $\widetilde{C}(U)\le C(U)$ whose improvement is amplified exponentially in~$T$ at the flow level. All results are numerically illustrated on the Stuart--Landau oscillator; the bounds provide a theoretical explanation for the empirically observed failure of $\tanh$-NODEs on the Morris--Lecar neuron model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。