arXiv:2606.07597cs.LGcs.AI2026-06

数据重复率变化导致小规模实验失效,控制重复率可显著提升混合训练效果。

Repetition Mismatch: Why Data Mixture Experiments Don't Scale and How to Fix Them

  • 通过匹配目标重复率的采样方法,解决高质量数据稀缺下的混合实验偏差。
  • 仅用1/16数据量的受控实验,误差低于0.05,未控实验误差达0.75。
  • 适用于大规模模型训练的数据混合优化,尤其适合资源受限场景。

预训练数据混合常通过小规模实验推断大预算下的最优配置。当高质量数据稀缺需重复时,这种外推常失败,根源在于重复率随训练预算变化,导致小规模实验无法预测最优混合比例。我们提出一种匹配目标重复率的子采样方法,可消除该偏差。在两源混合(有限高质量数据+网络爬取)场景中,仅使用1/16目标词元量的重复率受控实验,即可将757M参数模型的混合误差降至0.05,而未控实验误差高达0.75;后者需3~4次实验,消耗44%~94%的总词元预算。三源混合下,虽需多个受控实验,但仅两个即能收敛最优混合,优于依赖完整两源实验的基线方法。结果表明,重复动态是决定小规模实验泛化能力的关键因素,而非训练规模本身。更广泛地,数据重复应被视为混合优化中的首要变量,而非数据稀缺的副作用。

原文摘要 · Abstract (English)

Pre-training data mixtures are commonly tuned by running small-scale experiments and extrapolating to the target training budget. When high-quality data is scarce and must be repeated, this extrapolation frequently fails, but the source of the failure has not been isolated. We show that a primary culprit is a repetition mismatch: because high-quality datasets are small, their repetition rate changes as the training budget grows, shifting the optimal mixture in ways that small-scale proxy experiments do not anticipate. A subsampling procedure that matches the target repetition rate controls for this effect. In a two-source setting combining limited high-quality data with web crawl, a single repetition-controlled experiment using only 1/16 of the target tokens recovers a mixture within 0.05 of the optimum for a 757M parameter model, compared to an error of 0.75 without repetition control. Achieving comparable accuracy without repetition control requires three to four horizons, consuming 44 to 94% of the target token budget. With three data sources, the larger mixture space requires more than a single experiment to constrain, but the approach remains effective: at the 757M scale, just two repetition-controlled horizons recover the optimal mixture, outperforming baselines that instead require the full two-source experiments to construct. Our results reveal that repetition dynamics, not scale alone, shape whether small-scale mixture experiments generalize. More broadly, they suggest that data repetition deserves treatment as a first-class variable in mixture optimization, rather than an inconvenient side effect of limited data.

数据混合训练优化重复率控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。