用模型筛选多语言数据，15%样本达到基准性能。
Enhancing Multilingual LLM Pretraining with Model-Based Data Selection
- 用Transformer与FastText模型筛选结构化、知识丰富的多语言数据。
- 仅用15%训练数据量，10亿参数模型在MMLU上达基线水平。
- 适用于资源少的语言，适合做多语种预训练的数据构建。
数据集构建已成为高性能大语言模型的基础。尽管英语和多语言数据集已有多种基于规则的过滤方法，但基于模型的过滤技术主要集中在英语。为解决非英语语言研究不足的问题，我们提出一种面向多语言数据集的模型驱动筛选框架，旨在识别多样、结构化且富含知识的样本。该方法强调透明性、简洁性和效率，采用基于Transformer与FastText的分类器，确保技术可广泛使用。我们在涵盖多种语言家族、文字系统及资源水平的FineWeb-2网页爬取数据集上进行了全面消融实验。训练一个10亿参数的Llama模型，分别使用700亿和1190亿个标记时，仅需15%的训练标记即可达到基线的MMLU得分，同时在其他基准测试中表现更优，并缓解多语言困境。结果表明该方法具有良好的泛化能力。据此，我们将其扩展至20种语言，并发布经过优化的预训练数据集。
原文摘要 · Abstract (English)
Dataset curation has become a basis for strong large language model (LLM) performance. While various rule-based filtering heuristics exist for English and multilingual datasets, model-based filtering techniques have primarily focused on English. To address the disparity stemming from limited research on non-English languages, we develop a model-based filtering framework for multilingual datasets that aims to identify a diverse set of structured and knowledge-rich samples. Our approach emphasizes transparency, simplicity, and efficiency, leveraging Transformer- and FastText-based classifiers to ensure the broad accessibility of our technique and data. We conduct comprehensive ablation studies on the FineWeb-2 web crawl dataset across diverse language families, scripts, and resource availability to demonstrate the effectiveness of our method. Training a 1B-parameter Llama model for 70B and 119B tokens, our approach can match the baseline MMLU score with as little as 15% of the training tokens, while also improving across other benchmarks and mitigating the curse of multilinguality. These findings provide strong evidence for the generalizability of our approach to other languages. As a result, we extend our framework to 20 languages for which we release the refined pretraining datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。