提出统一的反向传播对偶算子，让深度学习优化与参数缩放无关。
Unit-Consistent (UC) Adjoint for GSD and Backprop in Deep Learning Applications
- 用单位一致对偶替代欧氏转置，统一处理各网络组件的反向传播
- 在任意参数缩放下保持优化轨迹不变，避免初始权重影响
- 适用于所有正齐次网络，适合追求优化稳定性的研究者
由线性映射和正齐次非线性（如ReLU）构成的深度神经网络具有基本的规范对称性：网络函数在节点级对角缩放下保持不变。然而，标准梯度下降并不满足该对称性，导致优化轨迹严重依赖于任意的参数化方式。已有工作提出了针对正齐次网络的缩放不变优化方案（如路径相关或路径空间更新）。本文贡献是互补性的：我们在反向传播的伴随/优化几何层面提出不变性要求，提供一种简洁、操作层面的通用方法，可统一应用于各类网络组件和优化器状态。通过将欧氏转置替换为单位一致（UC）对偶，推导出符合UC规范一致性的最速下降与反向传播算法。
原文摘要 · Abstract (English)
Deep neural networks constructed from linear maps and positively homogeneous nonlinearities (e.g., ReLU) possess a fundamental gauge symmetry: the network function is invariant to node-wise diagonal rescalings. However, standard gradient descent is not equivariant to this symmetry, causing optimization trajectories to depend heavily on arbitrary parameterizations. Prior work has proposed rescaling-invariant optimization schemes for positively homogeneous networks (e.g., path-based or path-space updates). Our contribution is complementary: we formulate the invariance requirement at the level of the backward adjoint/optimization geometry, which provides a simple, operator-level recipe that can be applied uniformly across network components and optimizer state. By replacing the Euclidean transpose with a Unit-Consistent (UC) adjoint, we derive UC gauge-consistent steepest descent and backprogation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。