用哈密顿-雅可比理论精确解释深度学习训练与推理的本质。
The Hamilton-Jacobi Theory of Deep Learning

- 将神经网络训练视为求解黏性哈密顿-雅可比初值问题的搜索过程。
- 揭示了ReLU、Sigmoid等激活函数的统一数学根源,且在深层宽网络中误差趋近于零。
- 为残差网络、Transformer等提供统一的偏微分方程视角,适合理论研究者深入理解模型机制。
本文精确地将神经网络训练识别为对哈密顿-雅可比初值问题的搜索:每一步梯度更新选择一个黏性哈密顿-雅可比方程的初始数据,其霍普夫-科尔传播器最符合观测;推理时,输入是空间点,初始条件已编码于权重中。该对应关系在log-sum-exp层下完全成立,ReLU、Sigmoid、SiLU和GELU均为同一对象的极限、梯度或矩,且在深度与宽度上精确组合,有限深度下有可量化误差,但在联合极限下趋于零。该框架对残差网络、Transformer及循环网络(RNN、LSTM、SSM)均具结构性,各自离散化同一类方程,并带有命名明确、可量化的逼近误差。单一变形参数ε统一了四种视角(网络、热带代数、黏性PDE、凸优化),构成在Lipschitz条件下封闭的交换图。定量推论包括:固定t时最小最大泛化率$O(n^{-1/(d+2)})$;对抗鲁棒性由ε控制;反向传播即残差网络哈密顿系统的伴随方程(庞特里亚金极值原理);缩放指数与数据内在维度一致,通过PDE数值积分实现;以及闭式$O(N)$影响函数(softmax归因权重π_j),其熵景观随ε增加发生折叠分歧,合并归因区域。
原文摘要 · Abstract (English)
In this paper, training a neural network is identified, exactly, as a search through Hamilton--Jacobi initial-value problems: each gradient step selects the initial data of a viscous Hamilton--Jacobi equation whose Hopf--Cole propagator best fits the observations; at inference, the input is the spatial point at which that solution is evaluated and the initial condition is already encoded in the weights. The correspondence is exact for log-sum-exp layers, with ReLU, sigmoid, SiLU, and GELU each an exact limit, gradient, or moment of the same object, and exact in composition across depth and width, with a quantified error at finite depth that vanishes in the joint limit. It is structural for residual networks, transformers, and recurrent networks (RNNs, LSTMs, SSMs), each discretizing the same class of equations, at a named and quantified approximation error. A single deformation parameter $\varepsilon$ unifies all four perspectives (network, tropical algebra, viscous PDE, convex optimization) in a commutative diagram closed under Lipschitz conditions. Quantitative consequences include: the minimax optimal generalization rate $O(n^{-1/(d+2)})$ for fixed $t$; adversarial robustness controlled by $\varepsilon$; backpropagation as the co-state equation of the Hamiltonian system for residual networks (Pontryagin Maximum Principle); scaling exponents consistent with data intrinsic dimension via PDE quadrature; and a closed-form $O(N)$ influence function (softmax attribution weights $π_j$) whose entropy landscape undergoes fold bifurcations as $\varepsilon$ increases, each merging attribution basins.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。