用大模型能力自动筛选多语言数据，提升质量与效率。
Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models
- 将大模型的判断能力压缩为轻量级标注器，实现高效多语言过滤。
- 在35种语言上表现优于传统规则方法，数据保留率更高。
- 适合需要高质量多语言训练数据的研究者与开发者。
高质量的多语言训练数据对大规模语言模型的有效预训练至关重要。然而，当前可用的开源多语言数据集仍十分有限。现有先进数据集主要依赖启发式过滤方法，限制了其跨语言迁移能力和可扩展性。本文提出JQL，一种系统化的方法，可在大规模下高效构建多样且高质量的多语言数据，同时显著降低计算成本。JQL将大模型的标注能力提炼为基于预训练多语言嵌入的轻量级标注器，这些模型在未见语言和书写系统上也表现出稳健的跨语言性能。在35种语言上的实证评估表明，该标注流程显著优于当前主流的启发式过滤方法（如Fineweb2）。JQL不仅大幅提升了下游模型训练质量，还提高了数据保留率。本研究为多语言数据清洗提供了实用洞见与宝贵资源，推动了多语言数据集构建标准的提升。
原文摘要 · Abstract (English)
High-quality multilingual training data is essential for effectively pretraining large language models (LLMs). Yet, the availability of suitable open-source multilingual datasets remains limited. Existing state-of-the-art datasets mostly rely on heuristic filtering methods, restricting both their cross-lingual transferability and scalability. Here, we introduce JQL, a systematic approach that efficiently curates diverse and high-quality multilingual data at scale while significantly reducing computational demands. JQL distills LLMs' annotation capabilities into lightweight annotators based on pretrained multilingual embeddings. These models exhibit robust multilingual and cross-lingual performance, even for languages and scripts unseen during training. Evaluated empirically across 35 languages, the resulting annotation pipeline substantially outperforms current heuristic filtering methods like Fineweb2. JQL notably enhances downstream model training quality and increases data retention rates. Our research provides practical insights and valuable resources for multilingual data curation, raising the standards of multilingual dataset development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。