为每条训练数据定制数据清洗流程，提升大模型预训练效果。
DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data
- 根据每个数据片段的特点动态决定丢弃、保留或清洗，灵活调整处理策略。
- 在11个基准测试中平均性能超越单一处理方法，7B模型训练效果更稳定。
- 适合需要高质量预训练数据的模型研发团队，尤其关注计算效率优化者。
大语言模型的预训练数据处理对下游性能至关重要。然而，现有方法通常在语料库或领域层面设定固定处理策略，并对所有样本统一应用，缺乏对单个样本需求的适应性。我们提出DataOrchestra框架，统一多种处理操作，为每个样本生成个性化的处理流水线。给定一段预训练数据，调度器决定是否丢弃、保留或清洗；若需清洗，则选择一个或多个下游操作（从程序化编辑到不同形式的LLM重写），并为每一步生成具体指令，由对应工具模型执行。我们使用DataOrchestra处理网络数据，从0.5B到7B规模的模型从零开始预训练，在11个基准测试中均实现稳定平均提升。该方法在数学持续预训练中也表现优异，优于更强的基线方法，同时通过跳过不必要的下游操作降低了处理计算开销。
原文摘要 · Abstract (English)
Pretraining data processing is critical to the downstream performance of Large Language Models (LLMs). However, many existing approaches define a fixed processing strategy at the corpus or domain level and apply it uniformly to many examples, without adapting to the needs of each example. We propose DataOrchestra, a framework that unifies different processing operations and orchestrates an example-specific pipeline for each example. Given a chunk of pretraining data, an orchestrator decides whether to drop, untouch, or clean it. For a chunk to be cleaned, it selects one or more downstream operations, ranging from programmatic editing to different forms of LLM-based rewriting. For each rewriting step, it further generates a concrete instruction, which is executed by the corresponding downstream tool model. We pretrain models from 0.5B to 7B from scratch on web data processed by DataOrchestra and observe stable average gains over individual data-processing methods across 11 benchmarks. DataOrchestra is also effective for math continued pretraining and outperforms stronger processing baselines, while reducing processing compute by skipping unnecessary downstream operations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。