用新方法从网页数据中提取高质量长文本，训练效果超Llama 3.1。
Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset
- 组合分类器集成与合成重写，减少人工规则过滤
- 6.3万亿真实唯一词元，比DCLM多4倍，性能相当
- 训练15万亿词元的80亿参数模型，超越Llama 3.1
近期英语Common Crawl数据集如FineWeb-Edu和DCLM通过模型驱动的严格过滤取得显著基准提升，但删除了90%的数据，不利于长序列训练（如Llama 3.1所需的15T词元）。本文提出结合分类器集成、合成数据重写及降低对启发式规则依赖的新方法，在保持高精度的同时大幅保留数据量。训练80亿参数模型1万亿词元时，使用我们高质量子集可使MMLU得分提高5.6分；完整6.3万亿词元数据集在MMLU上与DCLM相当，但包含四倍于DCLM的真实唯一词元。基于此数据集训练的80亿参数模型达15万亿词元（其中7.2万亿来自本数据集），在MMLU上领先Llama 3.1 5分，ARC-Challenge领先3.1分，十项任务平均分领先0.5分，实现长周期训练的最新水平。数据集地址：https://data.commoncrawl.org/contrib/Nemotron/Nemotron-CC/index.html
原文摘要 · Abstract (English)
Recent English Common Crawl datasets like FineWeb-Edu and DCLM achieved significant benchmark gains via aggressive model-based filtering, but at the cost of removing 90% of data. This limits their suitability for long token horizon training, such as 15T tokens for Llama 3.1. In this paper, we show how to achieve better trade-offs between accuracy and data quantity by a combination of classifier ensembling, synthetic data rephrasing, and reduced reliance on heuristic filters. When training 8B parameter models for 1T tokens, using a high-quality subset of our data improves MMLU by 5.6 over DCLM, demonstrating the efficacy of our methods for boosting accuracies over a relatively short token horizon. Furthermore, our full 6.3T token dataset matches DCLM on MMLU, but contains four times more unique real tokens than DCLM. This unlocks state-of-the-art training over a long token horizon: an 8B parameter model trained for 15T tokens, of which 7.2T came from our dataset, is better than the Llama 3.1 8B model: +5 on MMLU, +3.1 on ARC-Challenge, and +0.5 on average across ten diverse tasks. The dataset is available at https://data.commoncrawl.org/contrib/Nemotron/Nemotron-CC/index.html
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。