用大模型估算数据价值，自动优化训练集组合。
Optimizing Pretraining Data Mixtures with LLM-Estimated Utility
- 基于令牌数量和多样性设计简单高效的数据混合策略。
- 相比人工调参提速最高达10.6倍，计算开销降低约200倍。
- 适合大规模模型训练中需要快速筛选高质量数据的场景。
大型语言模型的性能随高质量训练数据量增加而提升。然而，利用更大数据集需在质量、数量和来源多样性间权衡。在计算与数据受限条件下评估九种基线方法后发现，基于令牌数量的启发式方法优于人工和学习型混合方案，表明仅考虑数据规模与多样性的简单策略效果出人意料地好。基于此，我们提出两种互补方法：UtiliMax通过引入小规模消融实验的效用估计，扩展了基于令牌的启发式方法，相较人工基线实现最高10.6倍加速；Model Estimated Data Utility（MEDU）则利用大模型从少量样本中估计数据效用，达到与消融法相当性能的同时，计算需求降低约200倍。两者共同构建了一个自动化、计算高效的新型数据混合框架，适用于多种训练场景。
原文摘要 · Abstract (English)
Large Language Models improve with increasing amounts of high-quality training data. However, leveraging larger datasets requires balancing quality, quantity, and diversity across sources. After evaluating nine baseline methods under both compute- and data-constrained scenarios, we find token-count heuristics outperform manual and learned mixes, indicating that simple approaches accounting for dataset size and diversity are surprisingly effective. Building on this insight, we propose two complementary approaches: UtiliMax, which extends token-based heuristics by incorporating utility estimates from reduced-scale ablations, achieving up to a 10.6x speedup over manual baselines; and Model Estimated Data Utility (MEDU), which leverages LLMs to estimate data utility from small samples, matching ablation-based performance while reducing computational requirements by $\sim$200x. Together, these approaches establish a new framework for automated, compute-efficient data mixing that is robust across training regimes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。