构建1330亿词元高质量数学预训练数据集，提升大模型推理能力。
Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset
- 用新型无领域依赖管道提取网页科学文本，保留数学结构
- 产出1330亿词元的Nemotron-CC-Math-3+，比现有开源数据多5.5倍
- 用该数据预训练模型，在数学和代码任务上显著超越基线
在高质量结构化数据（如数学与代码）上预训练大语言模型可显著增强其推理能力。然而，现有基于Common Crawl构建的数学数据集因提取规则脆弱、HTML转文本损失严重、难以保留数学结构而质量下降。本文提出Nemotron-CC-Math，一个大规模高质量数学语料库，通过创新的无领域依赖管道从Common Crawl中提取科学文本。该管道利用lynx进行布局感知渲染，并结合专用LLM清洗阶段，有效恢复各类格式（如MathJax、KaTeX、MathML）中的数学表达式，保持方程与代码块结构完整，去除冗余内容，统一符号为LaTeX表示并修正不一致处。我们构建了Nemotron-CC-Math-3+（133B tokens）和Nemotron-CC-Math-4+（52B tokens）。其中，Nemotron-CC-Math-4+不仅超越所有先前开源数学数据集（包括MegaMath、FineMath、OpenWebMath），且词元量达FineMath-4+的5.5倍。将其用于预训练Nemotron-T 8B模型后，相较于强基线，在MATH任务上提升4.8至12.6分，在MBPP+任务上提升4.6至14.3分，同时改善MMLU和MMLU-Stem等通用领域表现。本工作首次实现从噪声网络数据中可靠提取科学内容，推动数学、代码与通用推理性能提升，并确立开源数学预训练语料新基准。代码与数据已公开支持开源研究。
原文摘要 · Abstract (English)
Pretraining large language models (LLMs) on high-quality, structured data such as mathematics and code substantially enhances reasoning capabilities. However, existing math-focused datasets built from Common Crawl suffer from degraded quality due to brittle extraction heuristics, lossy HTML-to-text conversion, and the failure to reliably preserve mathematical structure. In this work, we introduce Nemotron-CC-Math, a large-scale, high-quality mathematical corpus constructed from Common Crawl using a novel, domain-agnostic pipeline specifically designed for robust scientific text extraction. Unlike previous efforts, our pipeline recovers math across various formats (e.g., MathJax, KaTeX, MathML) by leveraging layout-aware rendering with lynx and a targeted LLM-based cleaning stage. This approach preserves the structural integrity of equations and code blocks while removing boilerplate, standardizing notation into LaTeX representation, and correcting inconsistencies. We collected a large, high-quality math corpus, namely Nemotron-CC-Math-3+ (133B tokens) and Nemotron-CC-Math-4+ (52B tokens). Notably, Nemotron-CC-Math-4+ not only surpasses all prior open math datasets-including MegaMath, FineMath, and OpenWebMath-but also contains 5.5 times more tokens than FineMath-4+, which was previously the highest-quality math pretraining dataset. When used to pretrain a Nemotron-T 8B model, our corpus yields +4.8 to +12.6 gains on MATH and +4.6 to +14.3 gains on MBPP+ over strong baselines, while also improving general-domain performance on MMLU and MMLU-Stem. We present the first pipeline to reliably extract scientific content--including math--from noisy web-scale data, yielding measurable gains in math, code, and general reasoning, and setting a new state of the art among open math pretraining corpora. To support open-source efforts, we release our code and datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。