用小模型自动精细化处理每条训练数据，效果超越人工规则。
Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale
- 将数据清洗转化为编程任务，为每条数据生成定制化操作
- 在多个基准上提升超2%，7B模型性能接近200B训练量的模型
- 适合追求高效预训练与领域持续学习的研究者
大语言模型预训练长期依赖人工专家设计规则提升语料质量，但这些规则缺乏对单个样本的针对性，且难以规模化应用。本文提出Programming Every Example（ProX），让仅0.3B参数的小模型承担数据精炼任务，将数据优化视为编程问题，可对每个样本生成并执行细粒度操作（如字符串标准化）。实验表明，使用ProX处理的数据预训练的模型，在多个下游任务中表现优于原始数据或传统筛选方法，平均提升超2%。该方法适用于不同规模模型与语料（包括C4、RedPajama-V2、FineWeb等），在领域持续预训练中表现尤为突出：无需专门设计，基于OpenWebMath的模型经ProX优化后，相比Mistral-7B提升7.6%，Llama-2-7B提升14.6%，CodeLlama-7B提升20.3%，仅用100亿参数量即达到类似Llemma-7B（2000亿参数）的效果。进一步分析显示，ProX显著降低训练所需算力，为高效预训练提供新路径。项目已开源，包含超5000亿条语料、模型及完整训练细节。
原文摘要 · Abstract (English)
Large language model pre-training has traditionally relied on human experts to craft heuristics for improving the corpora quality, resulting in numerous rules developed to date. However, these rules lack the flexibility to address the unique characteristics of individual example effectively. Meanwhile, applying tailored rules to every example is impractical for human experts. In this paper, we demonstrate that even small language models, with as few as 0.3B parameters, can exhibit substantial data refining capabilities comparable to those of human experts. We introduce Programming Every Example (ProX), a novel framework that treats data refinement as a programming task, enabling models to refine corpora by generating and executing fine-grained operations, such as string normalization, for each individual example at scale. Experimental results show that models pre-trained on ProX-curated data outperform either original data or data filtered by other selection methods by more than 2% across various downstream benchmarks. Its effectiveness spans various model sizes and pre-training corpora, including C4, RedPajama-V2, FineWeb, FineWeb-Edu, and DCLM. Furthermore, ProX exhibits significant potential in domain-specific continual pre-training: without domain specific design, models trained on OpenWebMath refined by ProX outperform human-crafted rule-based methods, improving average accuracy by 7.6% over Mistral-7B, with 14.6% for Llama-2-7B and 20.3% for CodeLlama-7B, all within 10B tokens to be comparable to models like Llemma-7B trained on 200B tokens. Further analysis highlights that ProX significantly saves training FLOPs, offering a promising path for efficient LLM pre-training. We are open-sourcing ProX with >500B corpus, models, and sharing all training and implementation details for reproducible research and future innovation. Code: https://github.com/GAIR-NLP/ProX
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。