arXiv:2605.00064cs.LG2026-05

提出可预测的历史自适应噪声,提升SGD泛化界对动态优化路径的刻画能力。

Information-Theoretic Generalization Bounds for Stochastic Gradient Descent with Predictable Virtual Noise

  • 引入依赖历史的可预测虚拟噪声,使扰动协方差随优化过程动态调整
  • 在条件独立假设下,泛化误差界中引入累积噪声协方差的敏感性惩罚项
  • 适用于自适应优化器(如Adam)的理论分析,适合研究者和算法设计者

信息论泛化界通过学习参数与训练数据间的互信息来分析随机优化的泛化误差。虚拟扰动分析在证明中加入辅助高斯噪声,使互信息可计算,同时不改变实际SGD轨迹。然而,现有界通常要求扰动协方差固定且与优化历史无关,难以捕捉梯度统计、预条件器、曲率代理等路径相关几何结构。本文提出可预测的历史自适应虚拟扰动:每步扰动协方差可依赖过去真实优化历史,但不可依赖当前或未来随机性。该可预测性支持条件高斯相对熵论证,导出带有自适应几何的SGD泛化界。新界将固定灵敏度与梯度偏差项替换为条件自适应形式,包含由累积扰动协方差带来的输出灵敏度惩罚,并在条件无偏下将偏差项简化为条件方差。由于自适应协方差可能依赖数据,本文将局部高斯平滑与全局参考核比较分离。最终界包含一个协方差比较代价,衡量使用非实际自适应协方差作为参考几何的KL成本。在确定性、公开或前缀可观察等同步条件下,可还原固定噪声型界。该框架既涵盖固定各向同性和几何感知界,又将虚拟扰动分析扩展至历史依赖的SGD,无需修改算法。

原文摘要 · Abstract (English)

Information-theoretic generalization bounds analyze stochastic optimization by relating expected generalization error to the mutual information between learned parameters and training data. Virtual perturbation analyses of SGD add auxiliary Gaussian noise only in the proof, making mutual information tractable while leaving the actual SGD trajectory unchanged. Existing bounds, however, typically require perturbation covariances to be fixed independently of the optimization history, limiting their ability to represent geometries induced by moving gradient statistics, preconditioners, curvature proxies, and other pathwise information. We introduce predictable history-adaptive virtual perturbations, where the perturbation covariance at each iteration may depend on the past real SGD history but not on current or future randomness. This predictability enables a conditional Gaussian relative-entropy argument and yields generalization bounds for SGD with adaptive virtual-noise geometry. The bounds replace fixed sensitivity and gradient-deviation terms with conditional adaptive counterparts, include an output-sensitivity penalty from accumulated perturbation covariance, and reduce the deviation term to a conditional variance only under conditional unbiasedness. Since adaptive covariances may be data-dependent, we separate local Gaussian smoothing from global reference-kernel comparison. The resulting bound includes a covariance-comparison cost measuring the KL price of using an admissible reference geometry different from the actual adaptive covariance. Fixed-noise-style bounds are recovered under admissible synchronization, such as deterministic, public, or prefix-observable covariance rules. The framework recovers fixed isotropic and geometry-aware bounds as special cases while extending virtual perturbation analysis to history-dependent SGD without modifying the algorithm.

泛化界SGD信息论自适应优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。