arXiv:2511.13421cs.LGstat.ML2025-11被引 11

数据重复训练时,大样本能多重复几次才不浪费,理论揭示了这一上限。

Larger Datasets Can Be Repeated More: A Theoretical Analysis of Multi-Epoch Scaling in Linear Regression

  • 用有效重用率衡量重复训练的收益,推导出其随数据量和轮次变化的规律。
  • 小轮次时收益线性增长,大轮次后增益趋缓,上限与数据量呈对数关系。
  • 解释了为何大模型训练中多轮次效果随数据规模变化,适合关注训练效率的研究者。

尽管大规模语言模型在单遍训练下的数据缩放规律已被广泛研究,但在数据有限且重复训练多轮的情况下,其规律仍不明确。本文针对线性回归中的随机梯度下降(SGD),理论上分析了重复使用同一数据集对数据缩放规律的影响。核心问题是:若仅单遍训练,需多大数据集才能达到 $K$ 轮训练 $N$ 大小数据集的效果?我们定义了数据的有效重用率 $E(K, N)$,即在单遍训练下为达到相同测试损失所需的数据放大倍数。分析表明:当 $K$ 较小时,$E(K, N) o K$,即每轮带来线性收益;随着 $K$ 增大,$E(K, N)$ 在问题相关值上趋于饱和,且随 $N$ 增长(强凸情况下为 $Θ(/log N)$)。该结果修正了近期实证研究(Muennighoff et al., 2023)的结论,指出 $E(K, N) o K$ 的最大 $K$ 取决于数据规模与分布,强调未来缩放律研究必须同时建模数据大小与重用策略。

原文摘要 · Abstract (English)

While data scaling laws of large language models (LLMs) have been widely examined in the one-pass regime with massive corpora, their form under limited data and repeated epochs remains largely unexplored. This paper presents a theoretical analysis of how a common workaround, training for multiple epochs on the same dataset, reshapes the data scaling laws in linear regression. Concretely, we ask: to match the performance of training on a dataset of size $N$ for $K$ epochs, how much larger must a dataset be if the model is trained for only one pass? We quantify this using the \textit{effective reuse rate} of the data, $E(K, N)$, which we define as the multiplicative factor by which the dataset must grow under one-pass training to achieve the same test loss as $K$-epoch training. Our analysis precisely characterizes the scaling behavior of $E(K, N)$ for SGD in linear regression under either strong convexity or Zipf-distributed data: (1) When $K$ is small, we prove that $E(K, N) \approx K$, indicating that every new epoch yields a linear gain; (2) As $K$ increases, $E(K, N)$ plateaus at a problem-dependent value that grows with $N$ ($Θ(\log N)$ for the strongly-convex case), implying that larger datasets can be repeated more times before the marginal benefit vanishes. These theoretical findings point out a neglected factor in a recent empirical study (Muennighoff et al. (2023)), which claimed that training LLMs for up to $4$ epochs results in negligible loss differences compared to using fresh data at each step, \textit{i.e.}, $E(K, N) \approx K$ for $K \le 4$ in our notation. Supported by further empirical validation with LLMs, our results reveal that the maximum $K$ value for which $E(K, N) \approx K$ in fact depends on the data size and distribution, and underscore the need to explicitly model both factors in future studies of scaling laws with data reuse.

线性回归数据缩放重复训练优化理论

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。