研究离散参数下梯度下降的泛化误差与稳定性,发现确定性舍入会恶化性能。
Uniform Stability and Generalization Error of GD and SGD on Fixed-Point Parameters
- 在离散参数空间中分析GD和SGD的稳定性与泛化误差,考虑确定性或随机舍入更新。
- 确定性舍入使GD泛化误差从O(T/n)恶化到O(T/√n),且稳定性下界为Ω(T)。
- 随机舍入可避免此问题,尤其在高维时仍能保持非平凡稳定性,适合关注量化模型的读者。
我们分析了在离散参数空间中,梯度下降(GD)和随机梯度下降(SGD)的泛化误差、统一稳定性及统一论证稳定性,其中每次更新包含确定性或随机舍入。对于凸、Lipschitz且光滑的损失函数,确定性舍入会使GD的泛化误差从O(T/n)恶化至O(T/√n),并建立了匹配的下界。我们进一步证明,此时GD的统一稳定性达到Ω(T),表明基于稳定性的泛化界在此设定下无效。相比之下,使用确定性舍入的SGD对相同损失函数具有非平凡的统一稳定性保证,其依赖关系与实值情形不同:一维情况下为O(T/n),高维时为O(T²/n),且界限紧致。此外,我们发现随机舍入可能引入随维度增长的泛化误差,这一现象在标准实值优化及确定性舍入情形中均不存在。最后,我们为随机舍入方案提供了统一论证稳定性的上界,并在损失可表示为坐标独立函数之和时证明其紧致性。
原文摘要 · Abstract (English)
We analyze generalization error, uniform stability, and uniform argument stability of gradient descent (GD) and stochastic gradient descent (SGD) over discrete parameter spaces, where each update involves deterministic or stochastic rounding. We show that deterministic rounding degrades the generalization error of GD on convex, Lipschitz, and smooth loss functions, increasing the rate from $O(T/n)$ to $O(T/\sqrt{n})$, and establish matching lower bounds. We further prove that uniform stability of GD becomes $Ω(T)$, showing that stability-based generalization bounds are vacuous in this setting. In contrast, for the same losses, stochastic gradient descent with deterministic rounding admits nontrivial uniform stability guarantees, which differ qualitatively from the real-valued case and exhibit distinct dependencies on the number of iterations and the dimension: we prove tight bounds $O(T/n)$ for one dimension and $O(T^2/n)$ for higher dimensions. We also show that stochastic rounding can introduce generalization error that increases with the dimension; such a phenomenon is absent in standard real-valued optimization and in the deterministic rounding case. Finally, we provide upper bounds on uniform argument stability for stochastic rounding schemes and show that these bounds are tight when the loss can be represented as a sum of coordinate-wise functions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。