用采样BPE分析中文网络数据污染，高效发现隐性垃圾内容。
Auditing Chinese Web-scale Corpora via Sampled BPE Token Statistics
- 采样小样本训练BPE，轻量级定位污染词元。
- 提速148.4倍、内存降35.8倍，误差仅4.25%。
- 适合关注中文LLM数据质量的研究者与工程师。
中文网络污染已影响大语言模型，亟需对上游中文语料库进行审计。然而，现有方法面临三大挑战：（1）语料规模庞大，全量扫描成本高；（2）以往分析粒度粗，难以暴露词元级污染；（3）中文网络污染具有隐性且快速演变特性。本文提出Sampled-BPE轻量级词元级审计流程，通过采样小样本并训练BPE分词器，有效识别污染词元。实验表明，该方法在显著降低运行时间和内存开销的同时保持可接受的估计精度：相较全量分析实现148.4×加速和35.8×内存减少，污染类别相对误差仅为4.25%。我们将该流程应用于11个公开中文语料库及2021至2026年间的6个中文Common Crawl快照，揭示了开放语料中污染分布广泛但不均，且网络内容污染程度高且随时间动态变化。此外，我们发布了包含66万+条记录的分层中文网络词元数据集，每条记录含网页上下文、类别和解释字段，并以树形结构组织，支持污染溯源与审查。
原文摘要 · Abstract (English)
Chinese web pollution has surfaced in LLMs, motivating audits of upstream Chinese corpora. However, auditing such corpora faces three challenges: (1) their web-scale size makes full scan costly; (2) prior analyses are often too coarse to expose token-level pollution; (3) Chinese web pollution is implicit and rapidly changing. We propose Sampled-BPE, a lightweight token-level auditing pipeline that sample a small subset and train BPE tokenizer to surface polluted tokens. Experiments show that Sampled-BPE preserves usable estimates while substantially reducing runtime and memory: a 148.4 $\times$ speedup and a 35.8 $\times$ memory reduction induce only 4.25% relative error for pollution categories. We apply the pipeline to 11 open Chinese corpora and 6 Chinese Common Crawl snapshots from 2021 to 2026. The audit reveals widespread but uneven pollution across open corpora, as well as highly polluted and temporally shifting Chinese web content. We further release a hierarchical Chinese web token dataset with 660k+ token records, each with web context, category, and explanation fields, organized as trees to support review and tracing of pollution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。