研究非同分布数据下随机特征岭回归的泛化性能，揭示方差异质性对模型表现的影响。
High-dimensional ridge regression with random features for non-identically distributed data with a variance profile
- 引入行列依赖的方差轮廓模型，刻画数据异质性
- 推导出训练与测试风险的渐近等价式，精度高
- 发现小正则化时出现双下降现象，适用于复杂数据场景
随机特征岭回归通常在同分布高维设定下分析，即样本 $x_i=Σ^{1/2}x_i'$，其中 $x_i'$ 独立同分布且共享协方差矩阵 $Σ$。本文突破此框架，通过行依赖的方差轮廓模型研究非同分布数据，其中训练与测试协方差矩阵分别为 $Σ_i=\diag(γ_{i1}^2,…,γ_{ip}^2)$ 与 $\widetildeΣ_i=\diag(\tildeγ_{i1}^2,…,\tildeγ_{ip}^2)$。主要贡献是当 $n$、$p$、$m$ 按比例增长时，推导出岭回归随机特征模型的训练与测试风险的渐近等价表达式。第一类等价式结合线性加混沌近似与交通概率论证获得；第二类为确定性结果，源自算子值自由概率中的对角合并思想。数值实验验证其精确性，并揭示异质方差轮廓（如受 MNIST 启发的混合型）如何影响泛化能力，尤其在小岭参数下表现出双下降行为。
原文摘要 · Abstract (English)
Random feature ridge regression is often analyzed in the high-dimensional regime under the homogeneous sampling model $x_i=Σ^{1/2}x_i'$, where the vectors $x_i'$ have iid entries and the same covariance matrix $Σ$ is shared by all samples. In this paper, we move beyond this setting and study non-identically distributed data through a variance-profile model in which the training and test covariates have row-dependent diagonal covariance matrices $Σ_i=\diag(γ_{i1}^2,\ldots,γ_{ip}^2)$ and $\widetildeΣ_i=\diag(\tildeγ_{i1}^2,\ldots,\tildeγ_{ip}^2)$. Our main contribution is the derivation of asymptotic equivalents for the training and test risks of ridge regression with random features when $n$, $p$, and $m$ grow proportionally. The first set of equivalents is obtained by combining the linear-plus-chaos approximation with traffic-probability arguments, whereas the second set is deterministic and follows from operator-valued free probability through an amalgamation-over-the-diagonal argument. These equivalents are sharp in numerical experiments. They also reveal how heterogeneous variance profiles, including mixture-type profiles inspired by MNIST, can modify generalization and exhibit double-descent behavior when the ridge parameter is small.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。