arXiv:2511.16397cs.CL2025-11被引 4

用模型代替规则提取网页,大幅提升结构化内容保留率

AICC: Parse HTML Finer, Make Models Better -- A 7.3T AI-Ready Corpus Built by a Model-Based HTML Parser

  • 用0.6亿参数模型将提取任务转为序列标注,理解语义并分步处理
  • 在7887页测试集上,代码和公式保留率达90%以上,远超传统方法
  • 构建7.3万亿词的多语言语料AICC,训练模型性能提升1.08个百分点

尽管网络数据质量对大语言模型至关重要,现有整理工作多聚焦过滤与去重,将HTML转文本视为固定预处理步骤。现有语料依赖如Trafilatura等基于启发式的提取器,难以保持文档结构,常破坏公式、代码和表格等结构化内容。我们提出,提升提取质量与激进过滤同样重要。引入MinerU-HTML,一种将内容提取重构为序列标注问题的新型流水线,由0.6B参数语言模型解决。该方法不依赖文本密度启发式,而是通过两阶段格式化流程,在转换为Markdown前显式分类语义元素。其模型驱动方式天然可扩展,而启发式方法改进空间有限。在包含7,887个标注网页的MainWebBench上,MinerU-HTML达到81.8% ROUGE-N F1,优于Trafilatura的63.6%,结构化元素保留率高达90.9%(代码块)和94.0%(公式)。基于此,我们构建了AICC(AI-ready Common Crawl),一个从两个Common Crawl快照生成的7.3万亿词多语言语料库。在控制实验中,使用相同过滤策略的AICC(62B tokens)相比Trafilatura提取的TfCC,在13个基准上平均准确率达50.8%,领先1.08个百分点,直接证明提取质量显著影响模型能力。AICC在关键基准上也优于RefinedWeb和FineWeb。我们公开发布MainWebBench、MinerU-HTML和AICC,表明HTML提取是语料构建中常被低估的关键环节。

原文摘要 · Abstract (English)

While web data quality is crucial for large language models, most curation efforts focus on filtering and deduplication,treating HTML-to-text extraction as a fixed pre-processing step. Existing web corpora rely on heuristic-based extractors like Trafilatura, which struggle to preserve document structure and frequently corrupt structured elements such as formulas, codes, and tables. We hypothesize that improving extraction quality can be as impactful as aggressive filtering strategies for downstream performance. We introduce MinerU-HTML, a novel extraction pipeline that reformulates content extraction as a sequence labeling problem solved by a 0.6B-parameter language model. Unlike text-density heuristics, MinerU-HTML leverages semantic understanding and employs a two-stage formatting pipeline that explicitly categorizes semantic elements before converting to Markdown. Crucially, its model-based approach is inherently scalable, whereas heuristic methods offer limited improvement pathways. On MainWebBench, our benchmark of 7,887 annotated web pages, MinerU-HTML achieves 81.8\% ROUGE-N F1 compared to Trafilatura's 63.6\%, with exceptional structured element preservation (90.9\% for code blocks, 94.0\% for formulas). Using MinerU-HTML, we construct AICC (AI-ready Common Crawl), a 7.3-trillion token multilingual corpus from two Common Crawl snapshots. In controlled pretraining experiments where AICC and Trafilatura-extracted TfCC undergo identical filtering, models trained on AICC (62B tokens) achieve 50.8\% average accuracy across 13 benchmarks, outperforming TfCC by 1.08pp-providing direct evidence that extraction quality significantly impacts model capabilities. AICC also surpasses RefinedWeb and FineWeb on key benchmarks. We publicly release MainWebBench, MinerU-HTML, and AICC, demonstrating that HTML extraction is a critical, often underestimated component of web corpus construction.

网页提取语料构建模型增强结构保留

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。