arXiv:2605.29494cs.LG2026-05

通过自适应扰动梯度,提升分类模型的泛化能力。

Gradient Perturbation: Learning to Perturb Gradients for Adaptive Training

  • 设计可学习的梯度扰动机制,按类别动态调整梯度强度。
  • 在平衡、长尾及噪声标签数据上均显著优于现有方法。
  • 理论证明扰动范围与泛化能力相关,适合各类分类任务优化。

深度神经网络训练包含前向传播(从特征到logits再到损失)和反向传播(从损失到梯度再到参数更新)。尽管前向链上的扰动(如特征、logits、标签扰动)已被广泛研究,但反向链的梯度扰动却缺乏系统性探讨。本文建立统一的梯度扰动框架,揭示了已有方法如锐度感知最小化(SAM)、梯度裁剪和梯度噪声注入均可视为特定形式的梯度扰动。受最近提出的Logit Perturbation Learning(LPL)启发,我们提出:增强某类别的梯度范数相当于正向增强(促进学习),减弱则为负向增强(抑制过拟合)。基于此,提出学习梯度扰动(LPG),在类别级别自适应地扰动logits层梯度,实现类别感知训练。进一步通过PAC-Bayesian分析建立了梯度扰动界与泛化保证之间的理论联系。在平衡分类、长尾分类和噪声标签学习任务上的实验表明,LPG始终优于现有方法,且可作为即插即用模块与其他方法结合使用。

原文摘要 · Abstract (English)

Deep neural network training involves both forward propagation (from features through logits to loss) and backward propagation (from loss through gradients to parameter updates). While perturbations along the forward chain, including feature perturbation, logit perturbation, and label perturbation, have been extensively studied, the backward chain's gradient perturbation has received little systematic investigation. In this paper, we establish a unified framework for gradient perturbation, revealing that existing methods such as Sharpness-Aware Minimization (SAM), gradient clipping, and gradient noise injection can all be interpreted as imposing specific forms of gradient perturbation. Analogous to the recently proposed Logit Perturbation Learning (LPL), we conjecture that amplifying the gradient norm for a class acts as positive augmentation (enhancing learning), while dampening it acts as negative augmentation (suppressing overfitting). Based on these observations, we propose Learning to Perturb Gradients (LPG), which adaptively perturbs logit-level gradients at the class level to achieve category-aware training. We also establish theoretical connections between gradient perturbation bounds and generalization guarantees via PAC-Bayesian analysis. Experiments on balanced classification, long-tail classification, and noisy label learning demonstrate that LPG consistently outperforms existing methods and can be combined with them as a plug-in module.

梯度扰动类别感知泛化提升自适应训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。