提出快速高斯-牛顿法,加速多分类交叉熵优化
Fast Gauss-Newton for Multiclass Cross-Entropy

- 将软最大值交叉熵的曲率分解为真对余项与同类间耦合项
- 保留真对余项,忽略同类间耦合,实现高效曲率近似
- 适用于大类别数场景,尤其在竞争集中或正则强时表现优
在多分类软最大值交叉熵中,完整的广义高斯-牛顿(GGN)曲率通过软最大值协方差耦合所有输出逻辑值,导致曲率向量乘积随类别数增长而难以扩展。我们证明标准多分类GGN可被精确分解为真对余项与半正定同类间协方差项。快速高斯-牛顿(FGN)保留第一项并舍弃第二项,得到一个半正定的曲率下界近似,且在二分类时完全等价。该推导基于软最大值交叉熵的精确真对余标量边界表示:损失与梯度不变,近似仅作用于曲率层级。利用FGN曲率结构,阻尼更新可转化为每小批量样本对应一行的白化行空间系统。通过共轭梯度法以矩阵无关方式求解,仅需标量边界映射的雅可比-向量积与向量-雅可比积。针对固定特征多分类头的机制实验与评估验证了分解预测:当竞争质量集中或阻尼较大时,FGN最接近完整软最大值GGN;随着被舍弃的同类间协方差增大,偏差随之增加。
原文摘要 · Abstract (English)
In multiclass softmax cross-entropy, the full generalized Gauss-Newton (GGN) curvature couples all output logits through the softmax covariance, making curvature-vector products harder to scale as the number of classes grows. We show that the standard multiclass GGN can be decomposed exactly into a true-vs-rest term and a positive semidefinite within-competitor covariance term. Fast Gauss-Newton (FGN) retains the first term and drops the second, yielding a positive semidefinite under-approximation of the multiclass GGN that is exact for binary classification. The derivation uses an exact true-vs-rest scalar-margin representation of softmax cross-entropy: the loss and gradient are unchanged, and the approximation enters only at the curvature level. Exploiting the FGN curvature structure, the damped update can be written as an equivalent whitened row-space system with one row per mini-batch example. We solve this system matrix-free by conjugate gradient using Jacobian-vector and vector-Jacobian products of the scalar margin map. Targeted mechanism experiments and an evaluation on a fixed-feature multiclass head support the predictions from the decomposition: FGN stays closest to the full softmax GGN when competitor mass is concentrated or damping is large, and deviates as the dropped within-competitor covariance grows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。