自动优化语言模型预训练数据混合比例，提升性能并发现最优数据构成。
Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training
- 基于语义聚类与迭代搜索，自动发现最佳预训练数据组合。
- 在4000亿token上训练的10亿参数模型超越Llama-3.2-1B 2.0%。
- 可针对特定领域（如社会科学）提升5%性能，适合高效预训练研究。
预训练数据集通常来自网络内容，缺乏明确领域划分。例如，常用数据集Common Crawl无显式领域标签，而手动标注的The Pile等数据集则需大量人力。因此，确定最优预训练数据混合仍具挑战性，尽管其对预训练性能有显著影响。为此，我们提出基于聚类的迭代数据混合自举框架Nemotron-CLIMB，可在预训练设置中自动发现、评估并优化数据混合。具体而言，该方法将大规模数据集嵌入语义空间并聚类，再通过小型代理模型和预测器迭代搜索最优混合。持续在4000亿个标记上训练的10亿参数模型，性能超越当前最先进模型Llama-3.2-1B达2.0%。此外，针对特定领域（如社会科学）优化可比随机采样提升5%。我们还推出Nemotron-ClimbLab——一个经筛选的1.2万亿标记语料库，含20个聚类，作为研究平台；以及专为高效预训练设计的4000亿标记紧凑数据集Nemotron-ClimbMix，同等标记预算下表现更优。我们分析了最终数据混合，揭示了最优数据混合的特征。相关数据公开于：https://research.nvidia.com/labs/lpr/climb/
原文摘要 · Abstract (English)
Pre-training datasets are typically collected from web content and lack inherent domain divisions. For instance, widely used datasets like Common Crawl do not include explicit domain labels, while manually curating labeled datasets such as The Pile is labor-intensive. Consequently, identifying an optimal pre-training data mixture remains a challenging problem, despite its significant benefits for pre-training performance. To address these challenges, we propose CLustering-based Iterative Data Mixture Bootstrapping (Nemotron-CLIMB), an automated framework that discovers, evaluates, and refines data mixtures in a pre-training setting. Specifically, Nemotron-CLIMB embeds and clusters large-scale datasets in a semantic space and then iteratively searches for optimal mixtures using a smaller proxy model and a predictor. When continuously trained on 400B tokens with this mixture, our 1B model exceeds the state-of-the-art Llama-3.2-1B by 2.0%. Moreover, we observe that optimizing for a specific domain (e.g., Social Sciences) yields a 5% improvement over random sampling. Finally, we introduce Nemotron-ClimbLab, a filtered 1.2-trillion-token corpus with 20 clusters as a research playground, and Nemotron-ClimbMix, a compact yet powerful 400-billion-token dataset designed for efficient pre-training that delivers superior performance under an equal token budget. We analyze the final data mixture, elucidating the characteristics of an optimal data mixture. Our data is available at: https://research.nvidia.com/labs/lpr/climb/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。