通过数据重复使用，线性回归模型在数据有限时能实现更优的误差下降规律。
Improved Scaling Laws in Linear Regression via Data Reuse
- 用多轮随机梯度下降重复利用数据，提升模型泛化能力。
- 理论证明误差随模型规模与迭代次数呈幂律下降，优于单轮训练。
- 适合数据稀缺场景下优化模型性能的研究者参考。
神经网络缩放定律表明，大规模语言模型在线训练时测试误差随模型和数据规模增加呈多项式下降。但当新数据耗尽时，该趋势难以为继。本文研究发现，在线性回归中通过数据重用可改善现有缩放规律。具体而言，针对特征经压缩的 $N$ 组数据，对 $M$ 维线性模型使用多轮随机梯度下降（multi-pass SGD），假设数据协方差具有指数为 $a$ 的幂律谱，真实参数先验具有与之对齐的指数为 $b-a$（$a > b > 1$）的幂律谱，我们证明多轮 SGD 的测试误差为 $Θ(M^{1-b} + L^{(1-b)/a})$，其中 $L esim N^{a/b}$ 为迭代次数。而单轮 SGD 仅能达到 $Θ(M^{1-b} + N^{(1-b)/a})$（参见 Lin 等, 2024）。这表明在数据受限情况下，通过增加迭代次数（即 $L>N$）可获得更优的缩放规律。数值模拟验证了理论结果。
原文摘要 · Abstract (English)
Neural scaling laws suggest that the test error of large language models trained online decreases polynomially as the model size and data size increase. However, such scaling can be unsustainable when running out of new data. In this work, we show that data reuse can improve existing scaling laws in linear regression. Specifically, we derive sharp test error bounds on $M$-dimensional linear models trained by multi-pass stochastic gradient descent (multi-pass SGD) on $N$ data with sketched features. Assuming that the data covariance has a power-law spectrum of degree $a$, and that the true parameter follows a prior with an aligned power-law spectrum of degree $b-a$ (with $a > b > 1$), we show that multi-pass SGD achieves a test error of $Θ(M^{1-b} + L^{(1-b)/a})$, where $L \lesssim N^{a/b}$ is the number of iterations. In the same setting, one-pass SGD only attains a test error of $Θ(M^{1-b} + N^{(1-b)/a})$ (see e.g., Lin et al., 2024). This suggests an improved scaling law via data reuse (i.e., choosing $L>N$) in data-constrained regimes. Numerical simulations are also provided to verify our theoretical findings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。