提出评估时去污染方法,让大模型在不改数据的前提下避开记忆捷径。
When Benchmarks Leak: Inference-Time Decontamination for LLMs
- 评估时对输入嵌入施加小扰动,引导模型避开记忆路径。
- 在多个模型和基准上实现强去污染效果,干净样本性能损失极小。
- 适合关注评测公平性的研究人员与模型部署者使用。
基于基准的评估是对比大语言模型(LLMs)的通用标准。然而,测试集污染问题日益威胁其可靠性:测试样本或其近似版本泄露至训练数据,导致性能被人为夸大。现有方法主要分两类:一类在评估前识别并移除污染项,但会改变评估集,且在中度或严重污染下不可靠;另一类保留原始基准,但在推理时抑制污染行为,却常干扰正常推理,导致干净输入性能显著下降。本文提出DeconIEP,一种完全在推理阶段运行的去污染框架,通过在输入嵌入空间施加小而有界的扰动实现。该方法由一个相对较少污染的参考模型指导,学习实例自适应的扰动生成器,使被评估模型远离记忆驱动的捷径路径。在多个开源权重的大语言模型与基准上,实证结果表明,DeconIEP在保持良好泛化能力的同时,实现了强大的去污染效果,仅带来微小的良性性能损失。
原文摘要 · Abstract (English)
Benchmark-based evaluation is the de facto standard for comparing large language models (LLMs). However, its reliability is increasingly threatened by test set contamination, where test samples or their close variants leak into training data and artificially inflate reported performance. To address this issue, prior work has explored two main lines of mitigation. One line attempts to identify and remove contaminated benchmark items before evaluation, but this inevitably alters the evaluation set itself and becomes unreliable when contamination is moderate or severe. The other line preserves the benchmark and instead suppresses contaminated behavior at evaluation time; however, such interventions often interfere with normal inference and lead to noticeable performance degradation on clean inputs. We propose DeconIEP, a decontamination framework that operates entirely during evaluation by applying small, bounded perturbations in the input embedding space. Guided by a relatively less-contaminated reference model, DeconIEP learns an instance-adaptive perturbation generator that steers the evaluated model away from memorization-driven shortcut pathways. Across multiple open-weight LLMs and benchmarks, extensive empirical results show that DeconIEP achieves strong decontamination effectiveness while incurring only minimal degradation in benign utility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。