arXiv:2502.09496cs.LGstat.ML2025-02NeurIPS被引 5

提出高效算法,解决小误差下元学习的理论瓶颈。

On Agnostic PAC Learning in the Small Error Regime

  • 通过聚合ERM分类器设计新学习算法
  • 在τ≈d/m时达到接近最优的误差上界
  • 对小误差场景下的学习复杂度给出紧致结论

经典PAC模型中,经验风险最小化(ERM)在可实现情形下次优,但在不可实现情形下最优。这源于非可实现分布本身更难学习——即使扣除最优假设的误差τ,仍存在额外损失。近期工作将τ纳入误差项,提出更精细的分析框架,并在τ > d/m时证明了误差下界紧致。本文进一步解决了τ ≈ d/m时的下界问题,构造出误差为c·τ + O(√(τ(d + log(1/δ))/m) + (d + log(1/δ))/m)的学习算法,其中常数c ≤ 2.1,从而在该情形下匹配已知下界。该算法计算高效,基于对ERM分类器的精心聚合,推进了相关开放问题。尚待解决的是:能否将常数降至1,彻底确定元学习的复杂性。

原文摘要 · Abstract (English)

Binary classification in the classic PAC model exhibits a curious phenomenon: Empirical Risk Minimization (ERM) learners are suboptimal in the realizable case yet optimal in the agnostic case. Roughly speaking, this owes itself to the fact that non-realizable distributions $\mathcal{D}$ are simply more difficult to learn than realizable distributions -- even when one discounts a learner's error by $\mathrm{err}(h^*_{\mathcal{D}})$, the error of the best hypothesis in $\mathcal{H}$ for $\mathcal{D}$. Thus, optimal agnostic learners are permitted to incur excess error on (easier-to-learn) distributions $\mathcal{D}$ for which $τ= \mathrm{err}(h^*_{\mathcal{D}})$ is small. Recent work of Hanneke, Larsen, and Zhivotovskiy (FOCS `24) addresses this shortcoming by including $τ$ itself as a parameter in the agnostic error term. In this more fine-grained model, they demonstrate tightness of the error lower bound $τ+ Ω\left(\sqrt{\frac{τ(d + \log(1 / δ))}{m}} + \frac{d + \log(1 / δ)}{m} \right)$ in a regime where $τ> d/m$, and leave open the question of whether there may be a higher lower bound when $τ\approx d/m$, with $d$ denoting $\mathrm{VC}(\mathcal{H})$. In this work, we resolve this question by exhibiting a learner which achieves error $c \cdot τ+ O \left(\sqrt{\frac{τ(d + \log(1 / δ))}{m}} + \frac{d + \log(1 / δ)}{m} \right)$ for a constant $c \leq 2.1$, thus matching the lower bound when $τ\approx d/m$. Further, our learner is computationally efficient and is based upon careful aggregations of ERM classifiers, making progress on two other questions of Hanneke, Larsen, and Zhivotovskiy (FOCS `24). We leave open the interesting question of whether our approach can be refined to lower the constant from 2.1 to 1, which would completely settle the complexity of agnostic learning.

PAC学习理论机器学习误差分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。