用词频统计替代困惑度，快速高效筛选训练数据
Prior-based Noisy Text Data Filtering: Fast and Strong Alternative For Perplexity
- 基于语料库词频统计估算词先验，无需模型推理
- 在20个下游任务中平均表现最优，提速超1000倍
- 适用于代码、数学等符号语言，支持无监督多语言自适应
大型语言模型在海量网络语料上预训练，数据筛选对学习效率至关重要。尽管困惑度（PPL）过滤效果良好，但存在耗时长、对噪声或分布外样本可靠性差的问题。本文提出一种基于先验的过滤方法：利用语料级词频统计估算词项先验，借鉴语言学中关于词性角色与词汇密度的洞见。通过词先验的均值与标准差过滤文档，作为PPL的快速代理，无需模型推断。尽管方法简单，其在20个下游基准测试中平均表现最佳，时间成本较PPL过滤降低超过1000倍。我们进一步验证其在代码和数学等符号语言中的适用性，以及在无监督条件下对多语言语料的动态适应能力。
原文摘要 · Abstract (English)
As large language models (LLMs) are pretrained on massive web corpora, careful selection of data becomes essential to ensure effective and efficient learning. While perplexity (PPL)-based filtering has shown strong performance, it suffers from drawbacks: substantial time costs and inherent unreliability of the model when handling noisy or out-of-distribution samples. In this work, we propose a simple yet powerful alternative: a prior-based data filtering method that estimates token priors using corpus-level term frequency statistics, inspired by linguistic insights on word roles and lexical density. Our approach filters documents based on the mean and standard deviation of token priors, serving as a fast proxy to PPL while requiring no model inference. Despite its simplicity, the prior-based filter achieves the highest average performance across 20 downstream benchmarks, while reducing time cost by over 1000x compared to PPL-based filtering. We further demonstrate its applicability to symbolic languages such as code and math, and its dynamic adaptability to multilingual corpora without supervision
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。