通过约束梯度方向抑制噪声标签记忆，提升模型泛化能力
Orthogonal Gradient Constraints Shape Noisy-Label Memorization Dynamics
- 在优化器更新中移除梯度与权重的平行分量，实现几何正则化
- 小数据下显著提升测试准确率，减少对错误标签的拟合
- 适合研究学习动态机制，但效果依赖数据规模
现代神经网络能够拟合损坏的训练标签，使噪声标签学习成为研究记忆驱动过拟合的有用场景。现有正则化方法多作用于目标函数、网络结构或数据分布；本文则提出一种对优化器更新本身的几何干预——OrthoGrad，即移除每个权重梯度中与当前权重向量平行的分量。在小数据下的MNIST图像分类任务中，OrthoGrad对CNN显著提升测试准确率，并减少对错误标签的拟合。基于权重范数和梯度-权重余弦相似性的机制诊断表明，该投影在原始梯度存在显著径向分量时效果最强，而在大数据情况下梯度已近似正交，效果减弱。在CIFAR-10上的ResNet-18实验进一步显示，该方法可改变记忆轨迹，但无法完全阻止最终的噪声标签记忆。结果支持正交更新约束作为研究学习动态的有效诊断工具，同时表明OrthoGrad具有场景依赖性，非普适正则化方法。
原文摘要 · Abstract (English)
Modern neural networks can fit corrupted training labels, making noisy-label learning a useful setting for studying memorization-driven overfitting. Most regularization methods modify the objective, architecture, or data distribution; here we instead study a geometric intervention on the optimizer update itself. We evaluate OrthoGrad, which removes the component of each weight gradient parallel to the current weight vector, in noisy-label image classification. On MNIST with small-data regimes, OrthoGrad improves test accuracy most clearly for CNNs while reducing corrupted-label fitting. Mechanism diagnostics based on weight norms and gradient-weight cosine similarity suggest that the projection has the strongest effect when the raw gradient contains a nontrivial radial component, and becomes weaker in larger-data regimes where gradients are already nearly orthogonal to weights. Additional CIFAR-10 ResNet-18 experiments show that the method can alter memorization trajectories but does not prevent eventual noisy-label memorization. These results support orthogonal update constraints as a useful diagnostic for studying learning dynamics, while showing that OrthoGrad is regime-dependent rather than universally regularizing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。