arXiv:2607.16261cs.LGcs.AI2026-07

提出新优化方法,减少单样本损失波动,提升模型泛化能力。

Reducing Per-Sample Interference in Stochastic Optimization

  • 将参数更新建模为最小化单样本干扰的优化问题
  • 仅在最后一层优化,计算量低且效果稳定
  • 兼容SGD、AdamW,适合大模型和大规模数据

现代优化器将当前小批量的梯度与历史优化状态(如动量或自适应矩)结合。虽然有效,但这种做法可能导致个别样本的损失被主动增大,我们称之为单样本干扰。本文提出将参数更新重新定义为显式最小化该干扰的优化问题。由于精确求解计算开销过大,我们设计了一个高效近似:将问题维度降至小批量大小,并仅在最后一层进行优化,从而克服内存与速度瓶颈。关键发现是,该层足以捕捉网络整体的核心二阶统计特征。所提近似可无缝集成至SGD、AdamW等标准优化器中,通过少量对GPU友好的迭代即可求解。方法具有良好的扩展性,随着模型规模或输入增长,相对计算开销反而下降。图像分类基准实验验证了其有效降低单样本干扰并提升泛化性能。

原文摘要 · Abstract (English)

Modern optimizers combine gradients from the current mini-batch with historical optimization state, such as momentum or adaptive moments. While effective, this standard practice can produce parameter updates that actively increase the loss of individual samples. We term this phenomenon per-sample interference and propose redefining the parameter update as an optimization problem that explicitly minimizes it. Because the exact formulation of the problem is computationally prohibitive, we introduce a highly efficient surrogate. By reducing the problem's dimensionality to the batch size and restricting the optimization to the last linear layer, we overcome memory and speed bottlenecks. This strategy hinges on our unexpected finding that this layer alone can reliably capture core second-order statistics of the full network. The resulting surrogate problem integrates readily into standard optimizers like SGD and AdamW, and can be solved using a small number of GPU-friendly iterations. Crucially, the method exhibits favorable scaling properties, as the relative computational overhead shrinks as the model size or input grows. Experiments on image classification benchmarks confirm reduced per-sample interference and improved generalization.

优化器深度学习泛化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。