用普通SGD学习带方向性的数据，发现模型能自动适应数据结构。
Learning a Single Index Model from Anisotropic Data with vanilla Stochastic Gradient Descent
- 用基础SGD训练单指数模型，无需额外估计协方差。
- 证明了普通SGD能自动适应数据的协方差结构。
- 提出基于协方差决定的有效维度，降低样本需求。
我们研究了在各向异性高斯输入下，通过基础随机梯度下降（vanilla SGD）训练单指数模型（SIM）的学习动态。尽管各向同性情况已被广泛研究，但各向异性情形的关注较少，且协方差矩阵对学习过程的影响仍不明确。例如，Mousavi-Hosseini等人（2023b）提出的球形SGD需单独估计数据协方差，从而简化了其影响。本文分析了在各向异性输入下基础SGD的动态行为，证明其能自动适应数据的协方差结构。基于此，我们引入由协方差结构决定的有效维度，推导出样本复杂度的上下界。
原文摘要 · Abstract (English)
We investigate the problem of learning a Single Index Model (SIM)- a popular model for studying the ability of neural networks to learn features - from anisotropic Gaussian inputs by training a neuron using vanilla Stochastic Gradient Descent (SGD). While the isotropic case has been extensively studied, the anisotropic case has received less attention and the impact of the covariance matrix on the learning dynamics remains unclear. For instance, Mousavi-Hosseini et al. (2023b) proposed a spherical SGD that requires a separate estimation of the data covariance matrix, thereby oversimplifying the influence of covariance. In this study, we analyze the learning dynamics of vanilla SGD under the SIM with anisotropic input data, demonstrating that vanilla SGD automatically adapts to the data's covariance structure. Leveraging these results, we derive upper and lower bounds on the sample complexity using a notion of effective dimension that is determined by the structure of the covariance matrix instead of the input data dimension.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。