提出新数据调度策略,让高质量数据在训练中发挥最大效用。
How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Laws

- 基于质量感知的函数尺度律,推导出最优数据与批次调度方案。
- 在150亿参数模型上,准确率提升最高达+4.23(GSM8K)。
- 适合中段训练阶段优化数据使用,尤其适用于数学与代码任务。
高质量数据在大语言模型训练中稀缺,但如何结合优化动态调度仍缺乏理论指导。本文将函数尺度律扩展至随时间变化的数据质量,推导出特征空间回归模型下的渐近最优联合数据质量与批量调度方案。结果揭示两种训练阶段:在噪声受限阶段,较小批量可将更清洁数据转化为更多有效信号;在信号受限阶段,延迟引入高质量数据可抑制终末噪声而不损失信号积累。这解释了传统衰减调度与课程学习流程的冲突。据此提出中段训练策略Drop-Stable-Rampup:在质量转折点降低批量以积累信号,随后回升以压制噪声。在150亿参数MoE模型上,使用1080亿条通用领域专有数据进行中段训练,该方法相比Warmup-Stable-Decay提升平均准确率+1.70,较Cosine衰减提升+2.98,其中在GSM8K上提升+4.23,在MATH上提升+2.80。在公开数学与代码混合数据集上,优于所有已报告基准,在6亿参数稠密模型上整体均值提升+3.27,在相同MoE架构上提升+5.25。
原文摘要 · Abstract (English)
High-quality data is scarce in large language model (LLM) training, yet how to schedule its use with optimization dynamics lacks theoretical guidance. We extend functional scaling laws with time-varying data quality and derive asymptotically optimal joint data-quality and batch-size schedules within a feature-space regression model. The solution reveals two regimes and dual uses of high-quality data: in the noise-limited regime, a smaller batch converts cleaner data into more signal at comparable noise; in the signal-limited regime, late placement suppresses terminal noise without sacrificing signal accumulation. This explains why conventional decay schedules can conflict with curriculum-style pipelines. Motivated by the theoretical structure, we propose Drop-Stable-Rampup for LLM midtraining: drop the batch size at the quality transition, keep it low to accumulate signal, then ramp up to suppress noise. On a 15B MoE model midtrained on 108B tokens of general-domain proprietary data, Drop-Stable-Rampup improves average accuracy over Warmup-Stable-Decay by +1.70 and Cosine-decay by +2.98, including +4.23 on GSM8K and +2.80 on MATH. On a public math-and-code mixture, it leads all reported STEM, mathematics, and code benchmarks, improving the overall mean over the strongest baseline by +3.27 on a 600M dense model and +5.25 on the same MoE architecture.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。