用好坏两种语言模型组合，高效过滤网络文本中的低质内容
Rethinking KenLM: Good and Bad Model Ensembles for Efficient Text Quality Filtering in Large Web Corpora
- 构建好数据与坏数据分别训练的双语料库模型组
- 相比传统方法，噪声内容减少且优质内容保留更完整
- 适合计算资源有限场景下大规模文本清洗
随着大语言模型对高质量数据需求增加，高效过滤大规模网络语料成为关键挑战。现有常用工具KenLM是一种基于n-gram的轻量级语言模型，可在CPU上运行，但传统训练仅使用高质量数据，未能显式学习低质量数据的语言模式。为此，本文提出一种集成方法，利用两个对立的KenLM：(i) Good KenLM，由高质量数据训练；(ii) Bad KenLM，由低质量数据训练。实验表明，该方法在显著减少噪声内容的同时，有效保留高质量内容，优于传统训练方式。结果表明，该方法在计算开销极小的前提下，可作为资源受限环境下的实用文本质量过滤方案。
原文摘要 · Abstract (English)
With the increasing demand for substantial amounts of high-quality data to train large language models (LLMs), efficiently filtering large web corpora has become a critical challenge. For this purpose, KenLM, a lightweight n-gram-based language model that operates on CPUs, is widely used. However, the traditional method of training KenLM utilizes only high-quality data and, consequently, does not explicitly learn the linguistic patterns of low-quality data. To address this issue, we propose an ensemble approach that leverages two contrasting KenLMs: (i) Good KenLM, trained on high-quality data; and (ii) Bad KenLM, trained on low-quality data. Experimental results demonstrate that our approach significantly reduces noisy content while preserving high-quality content compared to the traditional KenLM training method. This indicates that our method can be a practical solution with minimal computational overhead for resource-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。