用大模型逐行过滤网页数据，提升训练质量与效率
FinerWeb-10BT: Refining Web Data with LLM-Based Line-Level Filtering
- 用GPT-4o mini标注2万文档的每行内容，生成9类低质标签
- 训练DeBERTa-v3模型对100亿词元数据进行自动化过滤
- 过滤后数据使模型更快达标，节省25%数据量仍更优
高质量数据对训练大语言模型至关重要。传统启发式过滤常遗漏低质文本或误删有价值内容。本文提出基于大模型的逐行过滤方法，使用GPT-4o mini对FineWeb中20,000份文档进行行级标注，生成描述性低质标签，并归纳为9类主类别。基于此构建数据集，训练DeBERTa-v3分类器，实现对100亿词元（10B-token）FineWeb子集的规模化过滤。通过在原始数据与过滤后数据上训练GPT-2模型验证效果，结果表明：过滤后模型在HellaSwag基准上准确率更高，且达到性能目标所需数据量最多减少25%。这证明该方法显著提升了数据质量和训练效率。研究公开了标注数据集FinerWeb-10BT及代码库，以支持后续研究。
原文摘要 · Abstract (English)
Data quality is crucial for training Large Language Models (LLMs). Traditional heuristic filters often miss low-quality text or mistakenly remove valuable content. In this paper, we introduce an LLM-based line-level filtering method to enhance training data quality. We use GPT-4o mini to label a 20,000-document sample from FineWeb at the line level, allowing the model to create descriptive labels for low-quality lines. These labels are grouped into nine main categories, and we train a DeBERTa-v3 classifier to scale the filtering to a 10B-token subset of FineWeb. To test the impact of our filtering, we train GPT-2 models on both the original and the filtered datasets. The results show that models trained on the filtered data achieve higher accuracy on the HellaSwag benchmark and reach their performance targets faster, even with up to 25\% less data. This demonstrates that LLM-based line-level filtering can significantly improve data quality and training efficiency for LLMs. We release our quality-annotated dataset, FinerWeb-10BT, and the codebase to support further work in this area.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。