用简单集成方法突破多分类学习的正则性障碍,提升样本效率。
Understanding Aggregations of Proper Learners in Multiclass Classification
- 通过推广二分类最优学习器构造多分类集成方法,实现更优样本复杂度。
- 对图维度有限的类别,集成方法样本复杂度达 $O\left(\frac{d_G + \ln(1 / δ)}ε\right)$,优于ERM。
- 首次证明:某些可学习类无法被任意有限个正则学习器集成逼近,揭示局限性。
多分类学习存在正则性障碍:某些可学习类无法被任何正则学习器学习。虽然二分类在最优学习中也面临类似障碍,但近期进展表明可通过正则学习器的集成实现最优。本文研究此类集成在多分类中的适用性。对于图维度 $d_G$ 有限的类别,我们证明经适当推广的Hanneke、Larsen及Aden-Ali等人的最优二分类学习器,可实现样本复杂度 $O\left(\frac{d_G + \ln(1 / δ)}ε\right)$,显著优于ERM。同时给出下界:对某些图维度为 $d_G$ 的类别,多数投票式集成需 $Ω\left( \frac{d_G + \ln(1 / δ)}ε\right)$ 样本。此外,单个ERM在这些类别上需 $Ω\left(\frac{d_G \ ln(1 / ε) + \ln(1 / δ)}ε\right)$ 样本,超出Daniely等人(2015)下界一个 $\ln(1 / ε)$ 因子。针对一般情况(仅DS维度有限,图维度可能无限),我们反例证明:存在可学习类,无法被任意有限个正则学习器的聚合以常数误差逼近。
原文摘要 · Abstract (English)
Multiclass learnability is known to exhibit a properness barrier: there are learnable classes which cannot be learned by any proper learner. Binary classification faces no such barrier for learnability, but a similar one for optimal learning, which can in general only be achieved by improper learners. Fortunately, recent advances in binary classification have demonstrated that this requirement can be satisfied using aggregations of proper learners, some of which are strikingly simple. This raises a natural question: to what extent can simple aggregations of proper learners overcome the properness barrier in multiclass classification? We give a positive answer to this question for classes which have finite Graph dimension, $d_G$. Namely, we demonstrate that the optimal binary learners of Hanneke, Larsen, and Aden-Ali et al. (appropriately generalized to the multiclass setting) achieve sample complexity $O\left(\frac{d_G + \ln(1 / δ)}ε\right)$. This forms a strict improvement upon the sample complexity of ERM. We complement this with a lower bound demonstrating that for certain classes of Graph dimension $d_G$, majorities of ERM learners require $Ω\left( \frac{d_G + \ln(1 / δ)}ε\right)$ samples. Furthermore, we show that a single ERM requires $Ω\left(\frac{d_G \ln(1 / ε) + \ln(1 / δ)}ε\right)$ samples on such classes, exceeding the lower bound of Daniely et al. (2015) by a factor of $\ln(1 / ε)$. For multiclass learning in full generality -- i.e., for classes of finite DS dimension but possibly infinite Graph dimension -- we give a strong refutation to these learning strategies, by exhibiting a learnable class which cannot be learned to constant error by any aggregation of a finite number of proper learners.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。