改进了大模型预训练数据的过滤方法，保留更多有用内容
Beyond Line-Level Filtering for the Pretraining Corpora of LLMs
- 结合文本顺序分布优化去重和标点过滤
- 在多选题和问答任务上均提升性能
- 适合关注预训练数据质量的研究者
传统行级过滤技术（如行级去重和尾部标点过滤）虽常用，但可能误删有价值内容，影响下游表现。本文提出两种增强方法：模式感知行级去重（PLD）和模式感知尾部标点过滤（PTF），不仅考虑行级信号，还分析其在文档中的序列分布，从而保留原本可能被剔除的结构化重要信息。我们在英语和韩语环境下训练了参数量为10亿的小型语言模型，结果表明，所提方法在多个选择题基准测试中持续提升性能，并显著提高SQuAD v1和KorQuAD v1上的生成式问答准确率。
原文摘要 · Abstract (English)
While traditional line-level filtering techniques, such as line-level deduplication and trailing-punctuation filters, are commonly used, these basic methods can sometimes discard valuable content, negatively affecting downstream performance. In this paper, we introduce two methods-pattern-aware line-level deduplication (PLD) and pattern-aware trailing punctuation filtering (PTF)-by enhancing the conventional filtering techniques. Our approach not only considers line-level signals but also takes into account their sequential distribution across documents, enabling us to retain structurally important content that might otherwise be removed. We evaluate these proposed methods by training small language models (1 B parameters) in both English and Korean. The results demonstrate that our methods consistently improve performance on multiple-choice benchmarks and significantly enhance generative question-answering accuracy on both SQuAD v1 and KorQuAD v1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。