arXiv:2606.24998cs.LGcs.AI2026-06被引 1

重复数据会系统性损害语言模型性能,且影响随模型规模增长而加剧。

Internal Data Repetition Destroys Language Models

  • 通过计算等效损失量化重复数据的伤害,发现中等重复次数最损性能
  • 当重复文档占10%算力预算时,性能损失相当于少用67%算力训练
  • 该现象适用于各类模型,可由统计学上的记忆与泛化权衡解释

语言模型面临高质量训练数据枯竭问题,即使经过严格去重,仍存在重复。我们基于Chinchilla时代的缩放规律,引入计算等效增益与损失指标,重新评估重复的影响。结果表明:在固定重复数据算力投入下,评估损失在中等重复次数$\ ext{Rep}$处达到峰值;重复中等规模的子集若干次比重复大规模或小规模多次更损害性能。该峰值位置与模型规模呈幂律关系,说明最损重复量的增长速度超过算力增长。在精确文档重复设置下,若重复文档消耗10%的FLOPs预算,对于344M参数的Qwen3模型,其性能损失相当于无重复训练时仅使用67%算力的效果。这些现象并非语言模型特有,可通过一个包含冗余数据的线性回归模型进行解析模拟,揭示了记忆与泛化之间的统计权衡如何导致损失峰。研究为预训练语料中的重复问题提供了量化工具。

原文摘要 · Abstract (English)

Language models are running out of high-quality training data, and even aggressively deduplicated corpora retain some amount of repetition. Earlier controlled studies predated Chinchilla-style scaling laws and could only measure the cost of repetition indirectly. We revisit repetition in the Chinchilla era, using a fitted no-repetition scaling law to report Compute-Equivalent Gain and Compute-Equivalent Loss. We show that under this modernized paradigm, repetition damage is systematic in three ways. First, holding compute allocated to repeated data constant, eval loss peaks at an intermediate repeat count $\Rep$; repeating a moderately sized subset a moderate number of times damages performance more than repeating a large subset a few times or a small subset many times. Second, the location of this peak is well-fit by a power law in model size; this scaling law reveals that the most damaging number of repeated data grows more quickly than compute. Finally, when repeated documents consume 10\% of the FLOPs budget in a controlled exact-document repetition setting, the compute-equivalent loss can be large: on FineWeb-Edu-Dedup, the most damaging repeat count for a Qwen3-style 344M-parameter model at $\OT=1$ matches the loss of a no-repetition run using 67% of the FLOPs. We demonstrate that these phenomena are not language-model-specific, and can be analytically understood in a simple statistical model: a misspecified linear regression with verbatim duplicates reproduces the same qualitative loss peak, quantifying how such peaks can arise from a statistical tradeoff between memorization and generalization. Our findings add precision to the study of duplication in language models, allowing practitioners to quantify the wasted compute incurred by the presence and repeat structure of duplicates in pretraining corpora.

语言模型重复数据训练效率算力浪费

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。