arXiv:2607.14576cs.LGstat.ML2026-07

提出深度残差网络的稳定性判据,仅凭输入幅度指数即可判断训练是否稳定。

Sharp Stability Threshold and Certification for Designing Stable Residual Architectures

  • 以输入幅度指数为标准,给出残差块稳定性的精确阈值
  • 指数不超过1时训练稳定,超过1则必然发散
  • 适用于无需归一化层的模型设计验证,尤其适合架构搜索

我们提出深度残差网络的子线性增长原理——对每个残差块速度场的输入幅度指数设定严格阈值:‖v(x, t)‖ ≤ c‖x‖^q + b,其中 q ∈ [0, 1]。通过常微分方程理论和最优控制分析,证明 q = 1 是全局稳定性的临界点:当 q ≤ 1 时存在全局前向流,而 q > 1 时速度场必然发散;最优控制视角进一步揭示,训练最优解在 q > 1 时会爆炸,而在 q ≤ 1 时天然安全。因此,q ≤ 1 是稳定训练的充要条件。该准则可解释层归一化等结构的稳定作用,并定义了前向动力学、伴随敏感性和架构组合皆可控的函数空间。通过五种基本构建操作下的指数运算法则,可在架构原语层面高效验证每层 q_k ≤ 1,避免试错。一个无参改进将超临界 Mamba 块从 q = 5 降至 q = 1,无需层归一化。Mamba 和 PatchTST 的实验表明,只要满足 q ≤ 1,训练即稳定,说明稳定性取决于输入指数而非归一化层的存在。

原文摘要 · Abstract (English)

We propose \emph{the sublinear-growth principle} for deep residual architectures -- a sharp stability threshold on the input-magnitude exponent of every residual block's velocity field: $$\|v(x, t)\| \leq c\,\|x\|^q + b, \qquad q \in [0, 1].$$ The threshold $q = 1$ is established via two independent arguments. Classical ODE theory gives a global forward flow on $[0, T]$ at $q \le 1$ and exhibits divergent velocity fields at any $q > 1$. The optimal-control analysis, via the Hamilton-Jacobi-Bellman equation, sharpens this to a selection statement: the training optimum is bang-bang on the boundary of the admissible class, so the optimum at $q > 1$ blows up while the optimum at $q \le 1$ is safe by construction. The exponent criterion $q \le 1$ is thereby a necessary and sufficient condition for stable training. It clarifies architectural placements that ensure the stability of training and inference, explaining, for instance, the stabilizing role of layer normalization. The sublinear-growth velocity fields form \emph{the right function space} on which forward dynamics, adjoint sensitivity, and architectural composition are all well-controlled. An arithmetic of input-magnitude exponents under the five operations that build residual blocks enables efficient certification of $q_k \le 1$ at the level of architectural primitives, in place of ad hoc trial and error in the search for stable neural architectural designs. A parameter-free modification reduces the supercritical Mamba block from $q = 5$ to $q = 1$ without layer normalization, demonstrating this point. Experiments on Mamba and PatchTST confirm that the $q \le 1$ variants train stably: the criterion is the input-magnitude exponent, not the presence of a normalization layer.

残差网络稳定性分析架构设计数学建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。