arXiv:2507.06775cs.LGmath.AT2025-07被引 1

提出新框架,用可计算的复杂度指标绑定随机优化的泛化误差。

Stability, Complexity and Data-Dependent Worst-Case Generalization Bounds

  • 引入随机集稳定性概念,适配随机优化生成的数据依赖集合。
  • 泛化误差上界由稳定参数与可计算复杂度量决定,避免难算的互信息项。
  • 实验验证边界紧致性,适合关注理论保障的算法研究者。

为随机优化算法提供泛化保证仍是学习理论中的关键挑战。近期研究发现优化轨迹的几何特性影响泛化性能,提出基于内在维度或拓扑复杂度的最坏情况泛化界,这些量与实际泛化误差有经验相关性。然而,多数方法涉及难以计算的互信息项,阻碍深入理解。另一些工作基于算法稳定性,但使用的组合型几何量又难以计算。本文结合可计算的复杂度度量与避免难算量的框架,提出适用于随机优化生成数据依赖随机集的‘随机集稳定性’概念。在该框架下,我们证明最坏情况泛化误差可被随机集稳定性参数及数据与算法相关的可计算复杂度量所控制。此外,本框架改进了现有拓扑泛化界,无需依赖互信息项即可恢复以往复杂度概念。通过一系列实际场景实验,我们验证了理论的有效性,评估了拓扑复杂度与稳定性之间的相互作用及边界的紧致性。

原文摘要 · Abstract (English)

Providing generalization guarantees for stochastic optimization algorithms remains a key challenge in learning theory. Recently, numerous works demonstrated the impact of the geometric properties of optimization trajectories on generalization performance. These works propose worst-case generalization bounds in terms of various notions of intrinsic dimension and/or topological complexity, which were found to empirically correlate with the generalization error. However, most of these approaches involve intractable mutual information terms, which limit a full understanding of the bounds. In contrast, some authors built on algorithmic stability to obtain worst-case bounds involving geometric quantities of a combinatorial nature, which are impractical to compute. In this paper, we address these limitations by combining empirically relevant complexity measures with a framework that avoids intractable quantities. To this end, we introduce the concept of \emph{random set stability}, tailored for the data-dependent random sets produced by stochastic optimization algorithms. Within this framework, we show that the worst-case generalization error can be bounded in terms of (i) the random set stability parameter and (ii) empirically relevant, data- and algorithm-dependent complexity measures of the random set. Moreover, our framework improves existing topological generalization bounds by recovering previous complexity notions without relying on mutual information terms. Through a series of experiments in practically relevant settings, we validate our theory by evaluating the tightness of our bounds and the interplay between topological complexity and stability.

泛化分析随机优化稳定性拓扑复杂度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。