arXiv:2505.21475cs.LGcs.DS2025-05NeurIPS被引 8

提出高效算法学习高维多指标模型,且对对抗噪声鲁棒。

Algorithms and SQ Lower Bounds for Robustly Learning Real-valued Multi-index Models

  • 设计基于统计查询的通用学习算法,处理带噪声的实值多指标模型。
  • 在对抗噪声下复杂度为 $d^{O(m)}2^{ ext{poly}(K/ε)}$,与理论下界接近。
  • 适用于光滑齐次的ReLU网络,摆脱以往依赖网络规模的指数开销。

研究在高斯分布下学习实值多指标模型(K-MIM)的复杂性。K-MIM 是输入投影到 K 维子空间后的函数。本文给出一类广义 MIM 的 PAC 学习算法,即使存在对抗标签噪声也有效。考虑有界变差的 MIM,其任意子空间上的投影存在不超过 m 次的区分性矩。在对抗噪声下,算法复杂度为 $d^{O(m)}2^{ ext{poly}(K/ε)}$;在可实现及独立噪声情形下,复杂度为 $d^{O(m)}2^{ ext{poly}(K)}(1/ε)^{O(K)}$。进一步证明:若某子空间不存在 m 次区分性矩,则任何统计查询学习器需 $d^{Ω(m)}$ 复杂度。作为应用,首次给出正齐次 L-Lipschitz K-MIM 的高效学习算法,复杂度为 $ ext{poly}(d)2^{ ext{poly}(KL/ε)}$,从而获得无需依赖网络规模的 Lipschitz 齐次 ReLU 网络学习算法,消除先前工作中的指数依赖。

原文摘要 · Abstract (English)

We study the complexity of learning real-valued Multi-Index Models (MIMs) under the Gaussian distribution. A $K$-MIM is a function $f:\mathbb{R}^d\to \mathbb{R}$ that depends only on the projection of its input onto a $K$-dimensional subspace. We give a general algorithm for PAC learning a broad class of MIMs with respect to the square loss, even in the presence of adversarial label noise. Moreover, we establish a nearly matching Statistical Query (SQ) lower bound, providing evidence that the complexity of our algorithm is qualitatively optimal as a function of the dimension. Specifically, we consider the class of bounded variation MIMs with the property that degree at most $m$ distinguishing moments exist with respect to projections onto any subspace. In the presence of adversarial label noise, the complexity of our learning algorithm is $d^{O(m)}2^{\mathrm{poly}(K/ε)}$. For the realizable and independent noise settings, our algorithm incurs complexity $d^{O(m)}2^{\mathrm{poly}(K)}(1/ε)^{O(K)}$. To complement our upper bound, we show that if for some subspace degree-$m$ distinguishing moments do not exist, then any SQ learner for the corresponding class of MIMs requires complexity $d^{Ω(m)}$. As an application, we give the first efficient learner for the class of positive-homogeneous $L$-Lipschitz $K$-MIMs. The resulting algorithm has complexity $\mathrm{poly}(d) 2^{\mathrm{poly}(KL/ε)}$. This gives a new PAC learning algorithm for Lipschitz homogeneous ReLU networks with complexity independent of the network size, removing the exponential dependence incurred in prior work.

机器学习多指标模型鲁棒学习统计查询

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。