输入标签相关性决定随机特征模型是线性还是非线性,影响性能表现。
Input-Label Correlation Governs a Linear-to-Nonlinear Transition in Random Features under Spiked Covariance
- 通过分析输入-标签相关性与数据协方差结构的交互,揭示模型从线性到非线性的相变机制。
- 在相关性高于阈值时,高阶非线性项保留,模型获得明显性能提升;低于阈值则退化为线性模型。
- 理论给出明确边界,适用于理解真实数据中随机特征模型为何优于线性方法。
随机特征模型(RFMs)是具有随机初始化固定第一层和可训练线性读出的两层网络,是最简单的非线性预测器之一。先前在比例高维极限下的渐近分析表明,在各向同性数据下,RFMs会退化为带噪声的线性模型,无法超越岭回归等经典线性方法。然而,RFMs在结构化真实数据上常表现更优。本文表明,这种矛盾由相关性驱动的相变所解释:在刺状协方差设计下,非各向同性与输入-标签相关性的相互作用决定了RFM是否表现为有效线性预测器或实现真正的非线性优势。我们建立了一个普适性原理,将RFM泛化误差等价于一个带噪声的多项式模型。该多项式的有效阶数(即激活函数中存活的Hermite阶数)由输入-标签相关性强度决定,从而在相关性-刺状幅度平面上划出明确边界。低于此边界时,RFM退化为线性代理,可能弱于强线性基线;高于此边界时,高阶项持续存在,实现清晰的非线性优势。数值模拟与真实数据实验验证了理论,并刻画了这两个区域间的过渡行为。
原文摘要 · Abstract (English)
Random feature models (RFMs), two-layer networks with a randomly initialized fixed first layer and a trained linear readout, are among the simplest nonlinear predictors. Prior asymptotic analyses in the proportional high-dimensional regime show that, under isotropic data, RFMs reduce to noisy linear models and offer no advantage over classical linear methods such as ridge regression. Yet RFMs frequently outperform linear baselines on structured real data. We show that this tension is explained by a correlation-driven phase transition: under spiked-covariance designs, the interaction between anisotropy and input-label correlation determines whether the RFM behaves as an effectively linear predictor or exhibits genuinely nonlinear gains. Concretely, we establish a universality principle under anisotropy and characterize the RFM generalization error via an equivalent noisy polynomial model. The effective degree of this polynomial, equivalently, which Hermite orders of the activation survive, is governed by the strength of input-label correlation, yielding an explicit boundary in the correlation-spike-magnitude plane. Below the boundary, the RFM collapses to a linear surrogate and can underperform strong linear baselines; above it, higher-order terms persist and the RFM achieves a clear nonlinear advantage. Numerical simulations and real-data experiments corroborate the theory and delineate the transition between these two regimes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。