在数据有限时，重新训练反而有害，应优先扩大模型规模。
Prescriptive Scaling Laws for Data Constrained Training
- 用附加过拟合项建模重复数据带来的损失
- 发现过度重复训练会降低性能，应转向增大模型容量
- 揭示强权重衰减可减少过拟合，适合数据受限场景
训练算力正快速超越高质量数据的供应，使核心挑战从算力分配转向如何从有限数据中获取最大价值。现有广泛采用的Chinchilla缩放定律假设每个训练标记都是唯一的，这限制了其在数据受限情况下的指导能力。我们通过引入简单的加性过拟合惩罚项来建模重复带来的额外损失，发现该模型能准确描述模型行为。新提出的缩放定律给出了全新的算力最优分配建议：超过一定阈值后，继续重复训练反而有害，算力应优先用于提升模型容量。实验证明，遵循该定律推荐配置可在数据受限条件下显著提升性能。由于该定律仅用一个参数表征过拟合，可直接跨配置比较。以案例研究显示，强权重衰减（λ=1.0）使该系数降低约70%，为近期发现‘数据受限时最优权重衰减比常规大一个数量级’提供了缩放律解释。
原文摘要 · Abstract (English)
Training compute is increasingly outpacing the availability of high-quality data. This shifts the central challenge from optimal compute allocation to extracting maximum value from limited data. The widely adopted Chinchilla scaling law assumes every training token is unique. This limits its ability to guide pretraining decisions in data-constrained regimes. We model the excess loss under repetition with a simple additive overfitting penalty and find that it accurately describes model behavior. Our scaling law yields qualitatively new compute-optimal allocation advice. Beyond a point, further repetition is counterproductive and compute is better spent on model capacity. We show that following our law's recommended configuration improves performance in data-constrained regimes. Finally, because our one-parameter form isolates overfitting in a single coefficient, it enables direct comparison across training configurations. As a case study, we show that strong weight decay ($λ=1.0$) reduces this coefficient by approximately 70%, providing a scaling-law explanation for recent findings that optimal weight decay in data-constrained regimes is an order of magnitude larger than standard practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。