提出可扩展的子文档去重方法，更好保留低频短重复内容
Scalable Frequency- and Length-Aware Subdocument Deduplication for Large Language Model Pretraining
- 分段+哈希+分布式聚合识别重复组
- 按频率和长度动态分配保留副本数
- 适合大规模预训练数据清洗，提升模型性能
大规模预训练语料库包含大量重复内容。尽管文档级去重广泛使用，但子文档级冗余去除仍具挑战。在语料规模下，基于后缀数组的方法通常在分片内独立执行，导致跨分片重复无法检测，且保留行为对分片配置敏感。基于哈希的方法虽能全局精确计数，但常依赖固定保留策略，难以适应异构重复模式。本文提出一种可扩展的子文档去重框架，将重复检测与副本保留解耦。通过自然边界分段、归一化精确哈希和分布式聚合识别重复组，并应用显式的频率-长度感知保留策略，为每组动态分配副本预算：对低频或短重复保留更多副本，对高频或长重复更激进删除。在FineWeb-Edu和含代码的网络语料上实验表明，经本方法处理的数据训练的模型在评估设置中表现最佳。结果凸显了显式副本保留控制的重要性。
原文摘要 · Abstract (English)
Large-scale pretraining corpora contain substantial duplicate content. Although document-level deduplication is widely used, removing subdocument-level redundancy remains challenging. At corpus scale, suffix-array-based methods are commonly applied independently within shards, leaving cross-shard duplicates undetected and making the resulting retention behavior sensitive to the sharding configuration. Hash-based methods enable global exact duplicate counting, but often rely on fixed copy-retention policies that cannot accommodate heterogeneous repetition patterns. We propose a scalable subdocument deduplication framework that decouples duplicate detection from copy retention. It identifies duplicate groups through natural-boundary segmentation, normalized exact hashing, and distributed aggregation, and then applies an explicit frequency- and length-aware retention policy that allocates an adaptive copy budget to each group, retaining more copies of low-frequency or short repetitions while more aggressively deleting high-frequency or long ones. Experiments on FineWeb-Edu and a code-containing web corpus show that models trained on data processed by our method achieve the best overall performance among the evaluated settings. These results underscore the importance of explicit copy-retention control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。