arXiv:2602.01777cs.LGcs.AI2026-02

用收缩估计提升高维梯度精度,显著改善大批次训练效果

Stein-Rule Shrinkage for Stochastic Gradient Estimation in High Dimensions

  • 基于Stein规则对批量梯度进行自适应收缩,向历史动量稳定估计收敛
  • 在高维噪声下均匀优于标准梯度,且在平方误差损失下为极小极大最优
  • 可无缝集成到Adam中,仅需极少计算开销,适合深度网络的高维层

随机梯度方法是大规模学习的核心,但其将小批量梯度视为无偏估计,在高维情形下已被经典决策理论证明为不可接受。本文将梯度计算建模为高维估计问题,提出基于Stein规则的收缩框架。构造的梯度估计器自适应地将噪声小批量梯度收缩至由历史动量导出的稳定估计。收缩强度通过在线估计梯度噪声方差确定,利用自适应优化器中的统计信息。在高斯噪声模型下,该估计器在平方误差损失下均匀优于标准随机梯度,并达到极小极大最优。将其融入Adam优化器得到SR-Adam,计算开销几乎可忽略。在CIFAR10和CIFAR100上,多种输入噪声水平下的大批次实验均显示性能持续优于Adam。消融实验表明,收益主要来自对高维卷积层选择性收缩,全参数无差别收缩反而降低性能。结果表明,经典收缩原理为深度学习中的随机梯度估计提供了严谨改进路径。

原文摘要 · Abstract (English)

Stochastic gradient methods are central to large-scale learning, but they treat mini-batch gradients as unbiased estimators, which classical decision theory shows are inadmissible in high dimensions. We formulate gradient computation as a high-dimensional estimation problem and introduce a framework based on Stein-rule shrinkage. We construct a gradient estimator that adaptively contracts noisy mini-batch gradients toward a stable estimator derived from historical momentum. The shrinkage intensity is determined in a data-driven manner using an online estimate of gradient noise variance, leveraging statistics from adaptive optimizers. Under a Gaussian noise model, we show our estimator uniformly dominates the standard stochastic gradient under squared error loss and is minimax-optimal. We incorporate this into the Adam optimizer, yielding SR-Adam, a practical algorithm with negligible computational cost. Empirical evaluations on CIFAR10 and CIFAR100 across multiple levels of input noise show consistent improvements over Adam in the large-batch regime. Ablation studies indicate that gains arise primarily from selectively applying shrinkage to high-dimensional convolutional layers, while indiscriminate shrinkage across all parameters degrades performance. These results illustrate that classical shrinkage principles provide a principled approach to improving stochastic gradient estimation in deep learning.

优化器梯度估计高维统计收缩估计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。