利用多源重复内容筛选高质量多语言训练数据，提升效率与质量。
Mix, MinHash, and Match: Cross-Source Agreement for Multilingual Pretraining Datasets
- 通过跨源一致性筛选，用重复内容做免费质量过滤。
- 阿拉伯语数据集相对基线提升4.5%，土耳其语提升5.5%，最多增加4倍唯一词元。
- 无需额外计算，适合大规模多语言预训练数据构建者使用。
网络获取的多语言数据对大语言模型预训练至关重要，但爬取成本高，研究机构重复采集相同内容。我们发现，主要阿拉伯语语料库中超过40%的词元在不同来源间重复。本文提出利用这种冗余作为质量信号，构建高质量预训练语料库。核心思想是：被多个独立爬虫获取的内容更可能是高质量文本，这一信号无需额外计算，仅需标准去重操作即可获得。我们提出 MixMinMatch 方法，整合多个现有网页语料库，采用跨数据集 MinHash 去重，识别出被多个来源独立恢复的文档。该方法应用于阿拉伯语、土耳其语和印地语，生成的语料库质量达到或超过最佳单源基准，同时提供最多4倍的唯一词元。在阿拉伯语上，匹配子集相比 ArabicWeb24 提升4.5%；在土耳其语上，相比 FineWeb-2 提升5.5%。数据集已公开于：https://huggingface.co/collections/AdaMLLab/mixminmatch
原文摘要 · Abstract (English)
Multilingual data from the web is essential for LLM pretraining. Yet, scraping it is expensive, and research groups repeatedly crawl the same content. For example, we found that over 40\% of tokens across major Arabic web corpora are duplicated between sources. In this work, we propose to use this wasteful redundancy as a quality signal to create high-quality pretraining datasets. Our key insight is that cross-source agreement functions as a free, model-free quality filter: content retained by multiple independent pipelines is more likely to represent high-quality text. Crucially, this signal requires no additional computation beyond standard deduplication, which is already performed at scale when pretraining language models. So, we propose MixMinMatch, a method that combines multiple existing web corpora, performs cross-dataset MinHash deduplication, and identifies documents independently recovered by multiple sources. We apply MixMinMatch to Arabic, Turkish, and Hindi, producing corpora that match or exceed the quality of the best single-source baselines, while providing up to 4$\times$ more unique tokens. On Arabic, our matched subset achieves a 4.5\% relative improvement over ArabicWeb24, while on Turkish, we improve over FineWeb-2 by 5.5\%. We release the datasets at: https://huggingface.co/collections/AdaMLLab/mixminmatch
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。