噪声敏感指数决定高维模型中统计与计算的鸿沟大小
A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Models
- 用噪声敏感指数统一刻画模型的计算难易程度
- 指数越小,学习越难,且在多类模型中普遍存在此规律
- 适合研究高维学习理论或算法瓶颈的学者参考
理解何时学习在统计上可行但计算上困难,是高维统计的核心挑战。本文研究单索引和多索引模型中的此类问题,这两类函数广泛用于检验机器学习方法发现高维数据特征的能力。我们的核心贡献是证明:噪声敏感指数(NSE)——一个由激活函数决定的简单量——控制着这些模型中统计到计算鸿沟的存在与大小。首先,在具有大加性噪声的单索引模型中,计算瓶颈的出现完全由NSE决定。其次,该指数同样控制大型可分离多索引模型在专业化过渡中的统计-计算差距,此时各分量变得可学习。最后,在层级多索引模型中,NSE决定了不同方向被顺序学习的最优计算速率。综上,我们的结果将NSE确立为连接噪声鲁棒性、计算难度与特征专化的统一性质。
原文摘要 · Abstract (English)
Understanding when learning is statistically possible yet computationally hard is a central challenge in high-dimensional statistics. In this work, we investigate this question in the context of single- and multi-index models, classes of functions widely studied as benchmarks to probe the ability of machine learning methods to discover features in high-dimensional data. Our main contribution is to show that a Noise Sensitivity Exponent (NSE) - a simple quantity determined by the activation function - governs the existence and magnitude of statistical-to-computational gaps within a broad regime of these models. We first establish that, in single-index models with large additive noise, the onset of a computational bottleneck is fully characterized by the NSE. We then demonstrate that the same exponent controls a statistical-computational gap in the specialization transition of large separable multi-index models, where individual components become learnable. Finally, in hierarchical multi-index models, we show that the NSE governs the optimal computational rate in which different directions are sequentially learned. Taken together, our results identify the NSE as a unifying property linking noise robustness, computational hardness, and feature specialization in high-dimensional learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。