训练数据含测试题的语义重复，会误导模型性能评估。
Soft Contamination Means Benchmarks Test Shallow Generalization
- 用语义嵌入检测训练数据中的非字符串重复内容
- 78%代码题、50%逻辑题存在语义重复，提升评测分数
- 模型在重复数据上微调后，对新题也表现更好
若大语言模型训练数据中混入了基准测试数据，会导致对模型分布外泛化能力的评估产生偏差。现有去重方法依赖n-gram匹配，难以发现语义重复——即内容等价但字符串不相近的文本。本文研究训练数据中语义重复（软污染）的影响。通过对Olmo3训练语料进行嵌入分析发现：1）污染仍广泛存在，如CodeForces有78%的问题存在语义重复，ZebraLogic有50%存在完全重复；2）将基准测试数据的语义重复纳入训练，可提升其在该基准上的表现；3）在基准数据的重复样本上微调后，模型在真正未见数据上同样表现提升。因此，近期基准性能提升可能被高估：部分来自真实能力提升，部分源于测试数据和有效测试数据不断进入训练集。
原文摘要 · Abstract (English)
If LLM training data is polluted with benchmark test data, then benchmark performance gives biased estimates of out-of-distribution (OOD) generalization. Typical decontamination filters use n-gram matching which fail to detect semantic duplicates: sentences with equivalent (or near-equivalent) content that are not close in string space. We study this soft contamination of training data by semantic duplicates. Among other experiments, we embed the Olmo3 training corpus and find that: 1) contamination remains widespread, e.g. we find semantic duplicates for 78% of CodeForces and exact duplicates for 50% of ZebraLogic problems; 2) including semantic duplicates of benchmark data in training does improve benchmark performance; and 3) when finetuning on duplicates of benchmark datapoints, performance also improves on truly-held-out datapoints from the same benchmark. We argue that recent benchmark gains are thus confounded: the prevalence of soft contamination means gains reflect both genuine capability improvements and the accumulation of test data and effective test data in growing training corpora.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。