用多个提取器组合提升网页文本采集量，不影响模型性能
Beyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pretraining
- 不再用单一提取器，改用多提取器并集提高覆盖范围
- 文本量提升71%，在标准任务上性能保持不变
- 对表格和代码等结构化内容效果显著，提升最高达10个百分点
构建大规模语言模型预训练数据集时，从HTML中提取文本是关键的预处理步骤。然而，现有开源数据集普遍对所有网页使用单一固定提取器。本文研究发现，尽管不同提取器在标准语言理解任务上表现相似，但经过固定过滤流程后存活的网页差异显著。通过合并多个提取器的结果，可使DCLM-Baseline的文本 token 产量提升高达71%，同时保持基准性能。此外，对于表格和代码块等结构化内容，提取器的选择显著影响下游任务表现，在WikiTQ上差距可达10个百分点，在HumanEval上达3个百分点。
原文摘要 · Abstract (English)
One of the first pre-processing steps for constructing web-scale LLM pretraining datasets involves extracting text from HTML. Despite the immense diversity of web content, existing open-source datasets predominantly apply a single fixed extractor to all webpages. In this work, we investigate whether this practice leads to suboptimal coverage and utilization of Internet data. We first show that while different extractors may lead to similar model performance on standard language understanding tasks, the pages surviving a fixed filtering pipeline can differ substantially. This suggests a simple intervention: by taking a Union over different extractors, we can increase the token yield of DCLM-Baseline by up to 71% while maintaining benchmark performance. We further show that for structured content such as tables and code blocks, extractor choice can significantly impact downstream task performance, with differences of up to 10 percentage points (p.p.) on WikiTQ and 3 p.p. on HumanEval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。