用多维分类体系从低质量网页数据中挖出高价值内容,效果超越优质数据集。
Unlocking Latent Value: Taxonomy-Guided Recovery of High-Performing Data from Low-Tier Web Corpora

- 基于语义分类体系,新增时效性和文化特异性维度,捕捉传统评分忽略的高质量内容。
- 在中等质量数据上,推理、编程能力提升超12%,甚至超过顶级数据集表现。
- 通过高效筛选框架,仅用少量计算就找到最优过滤组合,适合大规模数据清洗场景。
主流预训练网页数据清洗流程将文档质量压缩为单一综合得分,系统性遗漏了评分权重不足维度的高价值内容。本文提出一种基于分类体系的框架,通过沿语义有意义的维度过滤,恢复被忽略的价值。基于ESSENTIAL-WEB分类体系,引入两个新维度:时效性与文化特异性,二者与已有维度的互信息(NMI)均较低。使用Qwen2.5 32B模型标注1400万文档,并蒸馏为0.5B轻量模型;为实现快速全库标注,额外训练73M多任务MLP,基于E5嵌入,实现50倍推理吞吐提升。针对过滤配置组合爆炸问题,设计高效双阶段框架:第一阶段小规模识别强信号维度;第二阶段从最优单维构建并评估合取/析取复合过滤器,以极低成本发现高性能配置。应用该框架于被降权的网页数据,分类筛选后的子集不仅优于未过滤基线,甚至超越最高质量层级。在中等质量数据上,最佳过滤方案使推理、编程、知识基准分别提升12.1%、9.5%、2.0%,推理性能超越未过滤顶级数据6.7%,编程超越13.7%。而来自低于典型生产阈值两个层级的数据,经筛选后推理与编程能力较其未过滤基线分别提升22.3%和19.5%,编程表现超越顶级数据。结果表明,大量潜在价值仍深埋于被忽视的网页数据中,多维度分类过滤是可解释且高效的解锁路径。
原文摘要 · Abstract (English)
Dominant web data curation pipelines for pretraining collapse document quality into a single composite score, systematically missing high-value content along dimensions the scorer underweights. We present a taxonomy-driven framework that recovers this value by filtering along semantically meaningful dimensions that composite scores fail to capture. First, building on the ESSENTIAL-WEB taxonomy, we introduce two novel dimensions: timeliness and cultural specificity, both of which show low pairwise NMI with existing ones. We annotate 14M documents using Qwen2.5 32B and distill into a lightweight 0.5B model. To enable rapid corpus-wide annotation, we additionally train a 73M multi-task MLP on E5 embeddings, achieving 50x inference throughput. Second, to navigate the combinatorial explosion of filter configurations, we introduce a compute-efficient two-pass framework: Pass 1 identifies the strongest dimension signals at small scale; Pass 2 constructs and evaluates conjunctive and disjunctive compound filters from the top performers - identifying high-performing configurations at a fraction of full scaling-law cost. Applying the selected filters to deprioritized web data, taxonomy-filtered subsets outperform their unfiltered baselines and even surpass the highest-quality tier. On mid-tier data, our best filter improves over its unfiltered baseline by 12.1% on reasoning, 9.5% on coding, and 2.0% on knowledge benchmarks, exceeding unfiltered top-tier data by 6.7% on reasoning and 13.7% on coding. Furthermore, filtered data from two tiers below the typical production threshold improves by 22.3% on reasoning and 19.5% on coding over its unfiltered baseline, surpassing top-tier data on coding benchmarks. These results establish that vast latent value remains locked in deprioritized web data, and that multi-dimensional taxonomy filtering is a principled, compute-efficient key to unlocking it.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。