提出主动多分布学习新算法,揭示其标签复杂度理论极限。
Towards Fundamental Limits for Active Multi-distribution Learning
- 设计新算法,结合最大分歧系数与VC维优化主动学习
- 实证可实现下界达最优,且广义情形中关键项不可省略
- 适用于公平性、鲁棒性等需多分布保障的场景
多分布学习将无差错可能近似正确(PAC)学习扩展至考虑一组共 $k$ 个分布 $\{D_i\}_{i o[k]}$,并以最坏分布下的分类误差衡量性能。该问题因在协作学习、公平性与鲁棒性中的应用而受到广泛关注。尽管被动多分布学习的样本复杂度已较完整,主动学习的研究仍匮乏,现有算法的最优性未知。本文提出新的主动多分布学习算法,并建立分布依赖与分布无关设置下的改进标签复杂度上下界。具体地,在近可实现情形下,我们证明实现在可实现设置下上界为 $ ilde{O}igl(θ_{ ext{max}}(d+k) ext{ln}rac{1}{ ext{ε}}igr)$,在广义情形下为 $ ilde{O}igl(θ_{ ext{max}}(d+k)( ext{ln}rac{1}{ ext{ε}}+rac{ν^2}{ ext{ε}^2})+rac{kν}{ ext{ε}^2}igr)$,其中 $θ_{ ext{max}}$ 为 $k$ 个分布中最大分歧系数,$d$ 为假设类的VC维,$ν$ 为最优假设的多分布误差,$ε$ 为目标过失误差。此外,我们证明实现在可实现情形下的界为信息论最优,且广义情形中 $kν/ε^2$ 项对恰当学习器是根本性的。还建立了被动多分布学习的实例依赖样本复杂度界,平滑衔接可实现与广义情形,或具独立价值。
原文摘要 · Abstract (English)
Multi-distribution learning extends agnostic Probably Approximately Correct (PAC) learning to the setting in which a family of $k$ distributions, $\{D_i\}_{i\in[k]}$, is considered and a classifier's performance is measured by its error under the worst distribution. This problem has attracted a lot of recent interests due to its applications in collaborative learning, fairness, and robustness. Despite a rather complete picture of sample complexity of passive multi-distribution learning, research on active multi-distribution learning remains scarce, with algorithms whose optimality remaining unknown. In this paper, we develop new algorithms for active multi-distribution learning and establish improved label complexity upper and lower bounds, in distribution-dependent and distribution-free settings. Specifically, in the near-realizable setting we prove an upper bound of $\widetilde{O}\Bigl(θ_{\max}(d+k)\ln\frac{1}{\varepsilon}\Bigr)$ and $\widetilde{O}\Bigl(θ_{\max}(d+k)\Bigl(\ln\frac{1}{\varepsilon}+\frac{ν^2}{\varepsilon^2}\Bigr)+\frac{kν}{\varepsilon^2}\Bigr)$ in the realizable and agnostic settings respectively, where $θ_{\max}$ is the maximum disagreement coefficient among the $k$ distributions, $d$ is the VC dimension of the hypothesis class, $ν$ is the multi-distribution error of the best hypothesis, and $\varepsilon$ is the target excess error. Moreover, we show that the bound in the realizable setting is information-theoretically optimal and that the $kν/\varepsilon^2$ term in the agnostic setting is fundamental for proper learners. We also establish instance-dependent sample complexity bound for passive multidistribution learning that smoothly interpolates between realizable and agnostic regimes~\citep{blum2017collaborative,zhang2024optimal}, which may be of independent interest.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。