arXiv:2411.10784cs.LGstat.ML2024-11被引 2

揭示分类问题在欧氏空间中的维度复杂度,发现参数量可能指数级大于VC维。

On Reductions and Representations of Learning Problems in Euclidean Spaces

  • 用拓扑与凸性结合的新定理,分析分类问题映射到凸优化的最低维度需求。
  • 证明某些情况下所需维度D需指数级大于概念类的VC维d,即使还原稍非平凡。
  • 提出新维度复杂度指标,适用于带随机性的学习任务,启发后续研究方向。

许多实际预测算法将输入表示为欧氏空间中的向量,并用实值代理损失替代离散的0/1分类损失,将分类任务转化为随机凸优化(SCO)。本文研究此类还原在关键资源(如维度、随机性)下的表达能力。我们建立了将一个VC维为d的概念类还原为R^D中SCO问题所需的最小欧氏维度D的理论边界,形式化了VC维作为学习该类所需参数数量的直观理解。为此,我们发展了一种推广的Borsuk-Ulam定理,融合经典拓扑方法与凸性分析。令人惊讶的是,在某些情况下,维度D必须指数级大于VC维d,即便还原仅略微非平凡。同时,我们展示了通过随机初始化等技术,某些自然分类任务可在更小维度中实现,从而解决Kamath、Montasser和Srebro(COLT 2020)提出的开放问题。我们的成果引入了维度复杂度的新变体(也称sign-rank),包括一种近似版sign-rank及捕捉还原至SCO所需最小维度的变体。我们还提出若干未来研究方向。

原文摘要 · Abstract (English)

Many practical prediction algorithms represent inputs in Euclidean space and replace the discrete 0/1 classification loss with a real-valued surrogate loss, effectively reducing classification tasks to stochastic optimization. In this paper, we investigate the expressivity of such reductions in terms of key resources, including dimension and the role of randomness. We establish bounds on the minimum Euclidean dimension $D$ needed to reduce a concept class with VC dimension $d$ to a Stochastic Convex Optimization (SCO) problem in $\mathbb{R}^D$, formally addressing the intuitive interpretation of the VC dimension as the number of parameters needed to learn the class. To achieve this, we develop a generalization of the Borsuk-Ulam Theorem that combines the classical topological approach with convexity considerations. Perhaps surprisingly, we show that, in some cases, the number of parameters $D$ must be exponentially larger than the VC dimension $d$, even if the reduction is only slightly non-trivial. We also present natural classification tasks that can be represented in much smaller dimensions by leveraging randomness, as seen in techniques like random initialization. This result resolves an open question posed by Kamath, Montasser, and Srebro (COLT 2020). Our findings introduce new variants of \emph{dimension complexity} (also known as \emph{sign-rank}), a well-studied parameter in learning and complexity theory. Specifically, we define an approximate version of sign-rank and another variant that captures the minimum dimension required for a reduction to SCO. We also propose several open questions and directions for future research.

学习理论维度复杂度凸优化拓扑学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。