arXiv:2601.04895cs.AI2026-01被引 1

提出新方法DVD,精准识别大模型评估中的变体污染问题。

DVD: A Robust Method for Detecting Variant Contamination in Large Language Model Evaluation

  • 基于温度采样生成分布的方差变化检测污染样本
  • 在多个数据集和模型上显著优于现有方法
  • 适合关注评估可信度的研究者与开发者

大语言模型评估正面临一种新型干扰——变体污染:训练数据中包含与测试项语义等价但词法或句法不同的改写版本。这类变体不同于原文泄露,能绕过基于采样一致性或困惑度的检测机制,导致模型通过记忆而非推理获得高分。本文正式定义该问题,提出单样本检测器DVD(Detection via Variance of generation Distribution),通过建模温度采样下的局部输出分布来捕捉异常方差。核心发现:污染样本会引发‘记忆保持’与‘扰动漂移’状态交替,使低概率词的合成难度方差异常升高;未污染样本则保持平稳漂移。构建首个跨领域(Omni-MATH与SuperGPQA)变体污染基准,通过生成并筛选语义等价变体,模拟不同规模与架构模型(Qwen2.5、Llama3.1)的微调污染。实验表明,DVD在多种数据集与模型上持续超越困惑度、Min-k%++、编辑距离(CDD)及嵌入相似性等基线方法,且对超参数具有强鲁棒性。结果证明,生成分布方差是一种可靠且实用的变体污染指纹。

原文摘要 · Abstract (English)

Evaluating large language models (LLMs) is increasingly confounded by \emph{variant contamination}: the training corpus contains semantically equivalent yet lexically or syntactically altered versions of test items. Unlike verbatim leakage, these paraphrased or structurally transformed variants evade existing detectors based on sampling consistency or perplexity, thereby inflating benchmark scores via memorization rather than genuine reasoning. We formalize this problem and introduce \textbf{DVD} (\textbf{D}etection via \textbf{V}ariance of generation \textbf{D}istribution), a single-sample detector that models the local output distribution induced by temperature sampling. Our key insight is that contaminated items trigger alternation between a \emph{memory-adherence} state and a \emph{perturbation-drift} state, yielding abnormally high variance in the synthetic difficulty of low-probability tokens; uncontaminated items remain in drift with comparatively smooth variance. We construct the first benchmark for variant contamination across two domains Omni-MATH and SuperGPQA by generating and filtering semantically equivalent variants, and simulate contamination via fine-tuning models of different scales and architectures (Qwen2.5 and Llama3.1). Across datasets and models, \textbf{DVD} consistently outperforms perplexity-based, Min-$k$\%++, edit-distance (CDD), and embedding-similarity baselines, while exhibiting strong robustness to hyperparameters. Our results establish variance of the generation distribution as a principled and practical fingerprint for detecting variant contamination in LLM evaluation.

大模型评估变体污染检测方法生成分布

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。