arXiv:2605.19856cs.LGcs.AI2026-05

提出无需批归一化的反向梯度缩放方法,让深层网络训练更稳定。

StableGrad: Backward Scale Control without Batch Normalization

论文配图:StableGrad: Backward Scale Control without Batch Normalization
图 1 · 摘自论文原文
  • 在优化器层面调整梯度尺度,不改动前向模型结构。
  • 在无批归一化情况下,使深层PINN和ResNet等模型训练不崩溃。
  • 适合物理信息神经网络等禁止批归一化的场景使用。

训练超深神经网络需控制深度方向上的数值传播。若无此控制,激活值和梯度可能消失、爆炸或进入不稳定状态导致优化失败。现代架构常通过批归一化、残差连接等归一化层缓解此问题,但这些方法并非总适用。在物理信息神经网络(PINNs)中,网络表示连续物理场,其输入导数定义训练目标,批依赖归一化会引入非局部依赖,影响预测场及其导数的准确性。本文提出StableGrad,一种优化器级别的尺度控制机制,在不修改前向模型的前提下纠正逐层权重梯度不平衡。由于归一化仅在反向传播后、优化器更新前应用,网络输出、导数及物理残差保持不变。我们分析了该重缩放诱导的有效训练动态,并在深层PINNs上评估StableGrad,以无批归一化的卷积网络作为诊断压力测试。在PINN基准测试中,StableGrad提升了同深度模型的解精度,使更深模型在标准优化下更可靠。在ResNet和EfficientNet上,移除批归一化通常导致训练崩溃,而StableGrad可在不引入其他架构变化的情况下稳定优化。结果表明,优化器层面的梯度尺度控制可为前向归一化不可用或不理想时提供实用替代方案。

原文摘要 · Abstract (English)

Training very deep neural networks requires controlling the propagation of magnitudes across depth. Without such control, activations and gradients may vanish, explode, or enter unstable regimes that make optimization fail. Modern architectures often mitigate this problem through Batch Normalization, residual connections, or other normalization layers, which repeatedly re-scale or bypass intermediate representations. However, these mechanisms are not always appropriate. In Physics-Informed Neural Networks (PINNs), the network represents a continuous physical field and its input derivatives define the training objective, making batch-dependent normalization problematic because it can introduce non-local dependencies into the predicted field and its derivatives. We propose StableGrad, an optimizer-level scale-control mechanism that corrects layer-wise weight-gradient imbalances without modifying the forward model. Because the normalization is applied only after backpropagation and before the optimizer update, the network output, its derivatives, and the physical residual remain unchanged. We analyze the effective training dynamics induced by this rescaling and evaluate StableGrad on deep PINNs as the target application, with BatchNorm-free convolutional networks serving as a diagnostic stress test. On PINN benchmarks, StableGrad improves matched-depth solution accuracy and makes deeper models more reliable under standard optimization. On ResNet and EfficientNet architectures, where removing Batch Normalization normally leads to training collapse, StableGrad stabilizes optimization without introducing any other architectural change. These results show that optimizer-level control of weight-gradient scale can provide a practical alternative when forward normalization is unavailable or undesirable.

神经网络梯度控制PINN优化器

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。