研究相关数据下线性回归的渐近行为，揭示依赖结构对正则化和过拟合的影响。
Asymptotics of Linear Regression with Linearly Dependent Data
- 用时空协方差过程建模相关特征，分析高维比例下岭回归表现
- 证明高斯普适性：非高斯特征可用高斯替代，误差由谱特性决定
- 揭示依赖数据中的最优正则化与双下降现象，适合机器学习理论研究者
本文研究非高斯协变量具有线性依赖结构时线性回归的渐近性质，突破独立性假设。通过使用具有时空协方差的随机过程建模协变量，在样本数与特征维数同比例增长的高维比例情形下分析岭回归性能。证明了高斯普适性定理：在保持均值与协方差不变的前提下，非高斯协变量可被高斯向量替代，从而利用随机矩阵论工具精确刻画估计误差。估计误差由涉及时空协方差矩阵谱特性的固定点方程表征，支持高效计算。进一步研究了依赖数据下的最优正则化、过参数化及双下降现象。仿真验证了理论预测，阐明了相关性如何影响估计误差与正则化参数选择。
原文摘要 · Abstract (English)
In this paper we study the asymptotics of linear regression in settings with non-Gaussian covariates where the covariates exhibit a linear dependency structure, departing from the standard assumption of independence. We model the covariates using stochastic processes with spatio-temporal covariance and analyze the performance of ridge regression in the high-dimensional proportional regime, where the number of samples and feature dimensions grow proportionally. A Gaussian universality theorem is proven, demonstrating that the asymptotics are invariant under replacing the non-Gaussian covariates with Gaussian vectors preserving mean and covariance, for which tools from random matrix theory can be used to derive precise characterizations of the estimation error. The estimation error is characterized by a fixed-point equation involving the spectral properties of the spatio-temporal covariance matrices, enabling efficient computation. We then study optimal regularization, overparameterization, and the double descent phenomenon in the context of dependent data. Simulations validate our theoretical predictions, shedding light on how dependencies influence estimation error and the choice of regularization parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。