揭示交叉熵优化中步长安全性的几何限制，解释为何大步长会引发训练崩溃。
Ghosts of Softmax: Complex Singularities That Limit Safe Step Sizes in Cross-Entropy
- 通过分析软最大函数的复数奇点，推导出步长的安全边界公式。
- 实验证明当归一化步长超过1时，所有模型均出现训练崩溃。
- 该机制适用于任何架构，可用来设计抗梯度突增的优化控制器。
交叉熵训练的优化分析依赖于损失函数的局部泰勒近似来预测步长是否下降。但该近似仅在真实损失沿更新方向的泰勒收敛半径内有效，而该半径由最近的复数奇点决定。对于交叉熵，软最大函数 $F=\sum_j \exp(z_j)$ 存在复数零点——即“软最大幽灵”——导致损失出现对数奇点并限制收敛半径。在对数几率线性化假设下，我们推导出闭式表达式：二分类情形下精确半径为 $ρ^*=\sqrt{δ^2+ π^2}/Δ_a$；多分类情形下得到下界 $ρ_a=π/Δ_a$，其中 $Δ_a=\max_k a_k-\min_k a_k$ 为方向导数的跨度。该下界只需一次雅可比-向量乘积，揭示了脆弱步长的根源：靠近决策边界且对方向敏感的样本会缩小半径。归一化步长 $r=τ/ρ_a$ 可区分安全与危险更新。在六种架构和多种方向上，$r<1$ 时无失败，$r≥1$ 时立即崩溃。温度缩放验证机制：将步长归一化后，临界阈值标准差从0.992降至0.164。强制 $τ≤ρ_a$ 的控制器在梯度突增达10,000倍时仍稳定，而梯度裁剪仍失效。这些结果揭示了基于泰勒收敛而非海森曲率的交叉熵优化几何约束。
原文摘要 · Abstract (English)
Optimization analyses for cross-entropy training rely on local Taylor models of the loss to predict whether a proposed step will decrease the objective. These surrogates are reliable only inside the Taylor convergence radius of the true loss along the update direction. That radius is set not by real-line curvature alone but by the nearest complex singularity. For cross-entropy, the softmax partition function $F=\sum_j \exp(z_j)$ has complex zeros -- ``ghosts of softmax'' -- that induce logarithmic singularities in the loss and cap this radius. To make this geometry usable, we derive closed-form expressions under logit linearization along the proposed update direction. In the binary case, the exact radius is $ρ^*=\sqrt{δ^2+ π^2}/Δ_a$. In the multiclass case, we obtain the lower bound $ρ_a=π/Δ_a$, where $Δ_a=\max_k a_k-\min_k a_k$ is the spread of directional logit derivatives $a_k=\nabla z_k\cdot v$. This bound costs one Jacobian-vector product and reveals what makes a step fragile: samples that are both near a decision flip and highly sensitive to the proposed direction tighten the radius. The normalized step size $r=τ/ρ_a$ separates safe from dangerous updates. Across six tested architectures and multiple step directions, no model fails for $r<1$, yet collapse appears once $r\ge 1$. Temperature scaling confirms the mechanism: normalizing by $ρ_a$ shrinks the onset-threshold spread from standard deviation $0.992$ to $0.164$. A controller that enforces $τ\leρ_a$ survives learning-rate spikes up to $10{,} 000\times$ in our tests, where gradient clipping still collapses. Together, these results identify a geometric constraint on cross-entropy optimization that operates through Taylor convergence rather than Hessian curvature.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。