AI自动演化数据清洗策略,提升大模型预训练数据质量。
Data Darwinism Part II: DataEvolve -- AI can Autonomously Evolve Pretraining Data Curation
- 用迭代优化替代人工设计,让数据策略在闭环中自主进化。
- 生成5040亿token高质量数据集,跨18项基准平均分44.13,优于现有方案。
- 适合大规模预训练数据构建者,尤其关注知识密集型任务性能提升。
Data Darwinism(Part I)提出了数据处理的十级层次结构,表明更强的数据处理能释放更大价值。但该工作依赖人工设计单一类别策略,而现代预训练语料涵盖数百个异构类别,跨领域与内容类型,手动设计已不可行。本文提出DataEvolve框架,实现策略的自动化演化:针对每类数据,通过识别质量问题、生成候选策略、采样执行、评估效果并跨代优化,形成闭环进化。过程中积累问题经验池与策略表现池。应用于包含6720亿token的Nemotron-CC中的8个类别,经每类30轮迭代,生成5040亿token的Darwin-CC数据集。在5000亿token上训练30亿参数模型,其性能比原始数据提升3.96分,18项基准平均得分44.13,超越DCLM、Ultra-FineWeb和FineWeb-Edu,尤其在知识密集型任务如MMLU上优势显著。分析显示,演化策略收敛于以清洗为核心的路径:针对性降噪与格式归一化,同时保持领域特性,呼应了Part I中L4(生成精炼)原则。消融实验确认迭代演化不可或缺:优化策略相较次优方案提升2.93分,证明进化式策略设计对预训练规模数据治理具有可行性与必要性。
原文摘要 · Abstract (English)
Data Darwinism (Part I) established a ten-level hierarchy for data processing, showing that stronger processing can unlock greater data value. However, that work relied on manually designed strategies for a single category. Modern pretraining corpora comprise hundreds of heterogeneous categories spanning domains and content types, each demanding specialized treatment. At this scale, manual strategy design becomes prohibitive. This raises a key question: can strategies evolve in an automated way? We introduce DataEvolve, a framework that enables strategies to evolve through iterative optimization rather than manual design. For each data category, DataEvolve operates in a closed evolutionary loop: it identifies quality issues, generates candidate strategies, executes them on sampled data, evaluates results, and refines approaches across generations. The process accumulates knowledge through an experience pool of discovered issues and a strategy pool tracking performance across iterations. Applied to 8 categories spanning 672B tokens from Nemotron-CC, DataEvolve produces Darwin-CC, a 504B-token dataset with strategies evolved through 30 iterations per category. Training 3B models on 500B tokens, Darwin-CC outperforms raw data (+3.96 points) and achieves a 44.13 average score across 18 benchmarks, surpassing DCLM, Ultra-FineWeb, and FineWeb-Edu, with strong gains on knowledge-intensive tasks such as MMLU. Analysis shows evolved strategies converge on cleaning-focused approaches: targeted noise removal and format normalization with domain-aware preservation, echoing the L4 (Generative Refinement) principles from Part I. Ablation studies confirm iterative evolution is essential: optimized strategies outperform suboptimal ones by 2.93 points, establishing evolutionary strategy design as feasible and necessary for pretraining-scale data curation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。