揭示多分类学习的深层限制:无法通过扩大假设类实现纯学习,且正则化不通用。
Algorithmic Principles For Multiclass Learning Are Hard To Come By: Limits of Regularization and Proper Learning
- 证明纯学习无法通过扩大假设类解决多分类可学习性问题
- 发现纯学习可能需训练误差达到亚线性阶,且该阶数不可再低
- 说明正则化不能覆盖所有可学习问题,特别是结构风险最小化失效场景
统计学习理论中两个核心问题是:哪些预测问题可学习?如何学习?前者已有组合维度等优雅答案,后者却仍不明确。现有通用多分类学习方法依赖指数级大一包含结构的复杂方向,而常见的算法原则如纯学习和正则化仍未被充分理解。本文研究学习能否简化为纯学习(可能在更大假设类中)以及正则化是否能统一多分类学习。结果表明两者均不成立:首先,存在可学习的多分类问题无法嵌入任何纯可学习类;其次,纯学习可能需要训练误差达到o(m),且任意亚线性尺度a_m=o(m)对某些问题都是必要的;第三,存在可通过纯学习解决的问题无法用结构风险最小化(SRM)学习,也有可学习问题无法被局部正则化器捕捉。文章还给出了SRM可学习性的两个充分条件,并通过揭示偏好可积性刻画了其表示性。
原文摘要 · Abstract (English)
Two of the most fundamental questions in statistical learning theory are the following: which prediction problems are learnable, and how should they be learned? For the former, elegant answers often take the form of combinatorial dimensions. The latter question, however, has proved considerably more elusive: all known general-purpose multiclass learners rely on intricate orientations of exponentially large one-inclusion structures, and familiar algorithmic principles such as proper learning and regularization remain poorly understood. Motivated by prior work, we ask whether learning reduces to proper learning---possibly over a larger hypothesis class---and whether proper or improper multiclass learning can ultimately be captured by suitable regularizers. Our primary results answer both questions negatively, resolving three open problems from prior work. First, we exhibit a learnable multiclass problem that cannot be embedded in any properly learnable class, meaning learning cannot be reduced to proper learning by enlarging the hypothesis class. Second, we demonstrate that proper learning can require training error and characterize this phenomenon precisely: every properly learnable class admits a proper learner making $o(m)$ errors on samples of size $m$, but every prescribed sublinear scale $a_m=o(m)$ is necessary for some properly learnable problem. Third, regularization is not a general learner: we exhibit a properly learnable class that cannot be learned by any Structural Risk Minimization (SRM) learner, and a learnable class that cannot be learned by any local regularizer. We complement these impossibility results with a positive theory that gives two sufficient conditions for SRM learnability and characterizes SRM representability through integrability of revealed preferences.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。