CQF筛选器通过隐式过滤高质量数据提升模型性能，而非单纯依赖高质量数据。
Removing Noise, not Finding Gold: Quality Filtering for Large-Scale Pretraining
- 用分类器对大规模语料进行质量评分并筛选
- 选中的低质数据训练效果竟优于直接用高质量数据
- 揭示了筛选机制背后的隐式过滤效应，适合数据清洗研究者
大规模模型在包含混合质量文档的网络爬取数据集上预训练，因此数据过滤至关重要。一种流行方法是基于分类器的质量过滤（CQF），即训练一个二分类器区分预训练数据与一小部分高质量数据，为每篇文档赋予基于分类器得分的质量分数，并保留得分最高的部分。我们深入分析了CQF。结果显示，尽管CQF能提升下游任务表现，但并未改善在高质量数据集上的语言建模性能。更重要的是，我们发现使用CQF筛选后的数据训练，其效果甚至超过直接在足够大的高质量数据集上训练，这一发现尤为引人注目，因为近期大量资源被投入于扩充高质量数据。我们解释该悖论源于CQF同时对高质量和低质量数据进行了隐式过滤。最后，我们提出一种基于优化目标的数据质量定义，并证明可通过小规模代理实验可靠估计。总体而言，我们的结果揭示了CQF的作用机制，深化了对实践中广泛使用的数据选择方法的理解。
原文摘要 · Abstract (English)
Large-scale models are pretrained on massive web-crawled datasets containing documents of mixed quality, making data filtering essential. A popular method is Classifier-based Quality Filtering (CQF), which trains a binary classifier to distinguish between pretraining data and a small, high-quality set. It assigns each pretraining document a quality score defined as the classifier's score and retains only the top-scoring ones. We provide an in-depth analysis of CQF. We show that while CQF improves downstream task performance, it does not necessarily enhance language modeling on the high-quality set. Importantly, we find that training on CQF-selected data can outperform training directly on the high-quality set, even when the latter is sufficiently large. This finding alone is particularly striking, given the substantial effort and cost recently devoted to augmenting high-quality data. We explain this paradox by the fact that CQF implicitly filters the high-quality dataset as well as the low-quality one. Finally, we introduce an optimization-driven notion of data quality and demonstrate that it can be reliably estimated using small-scale proxy experiments. Altogether, our results both elucidate the mechanisms behind CQF and deepen our understanding of data selection methods widely used in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。