揭示了算法稳定性的随机性需求本质,给出可计算的量化边界。
The Role of Randomness in Stability
- 通过弱稳定性推导强稳定性,建立随机性复杂度的统一框架。
- 证明有限Littlestone维数等价于随机性复杂度有界,且最坏对数增长。
- 解决了一个关于列表可复现性的重要开放问题,适用于学习理论研究者。
稳定性是学习与统计中的核心性质,指算法在相似数据集上的输出不发生显著变化。已有基本结论表明,足够稳定的算法(如满足隐私保护的算法)必须是随机的。本文探讨:实现算法稳定性需要多少随机性?针对两个重要稳定性概念——可复现性(同一分布下多次运行输出一致)和差分隐私(邻近数据集输出分布相似),我们建立‘弱到强’的提升定理:任务的随机性复杂度由任意确定性算法的最优复现概率决定,该弱指标被普遍限制在1/2(Chase et al. FOCS 2023)。基于此,我们刻画了PAC学习的随机性复杂度:一个学习类具有有界随机性复杂度当且仅当其具有有限Littlestone维数,且最坏情况下随学习器超额误差对数增长。这解决了Chase等(STOC 2024)提出的关于误差依赖型‘列表可复现性’的表征问题。
原文摘要 · Abstract (English)
Stability is a central property in learning and statistics promising the output of an algorithm $A$ does not change substantially when applied to similar datasets $S$ and $S'$. It is an elementary fact that any sufficiently stable algorithm (e.g.\ one returning the same result with high probability, satisfying privacy guarantees, etc.) must be randomized. This raises a natural question: can we quantify how much randomness is needed for algorithmic stability? We study the randomness complexity of two influential notions of stability in learning: replicability, which promises $A$ usually outputs the same result when run over samples from the same distribution (and shared random coins), and differential privacy, which promises the output distribution of $A$ remains similar under neighboring datasets. The randomness complexity of these notions was studied recently in (Dixon et al. ICML 2024) and (Cannone et al. ITCS 2024) for basic $d$-dimensional tasks (e.g. estimating the bias of $d$ coins), but little is known about the measures more generally or in complex settings like classification. Toward this end, we prove a `weak-to-strong' boosting theorem for stability: the randomness complexity of a task $M$ (either under replicability or DP) is tightly controlled by the best replication probability of any deterministic algorithm solving the task, a weak measure called `global stability' that is universally capped at $\frac{1}{2}$ (Chase et al. FOCS 2023). Using this, we characterize the randomness complexity of PAC Learning: a class has bounded randomness complexity iff it has finite Littlestone dimension, and moreover scales at worst logarithmically in the excess error of the learner. This resolves a question of (Chase et al. STOC 2024) who asked for such a characterization in the equivalent language of (error-dependent) `list-replicability'.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。