通过输出分布峰值检测小模型数据污染，但效果有限。
No Memorization, No Detection: Output Distribution-Based Contamination Detection in Small Language Models
- 用输出分布尖锐度衡量模型是否记忆训练数据。
- 在多数情况下，该方法准确率仅达随机水平。
- 概率类方法（如困惑度）更可靠，适合小模型检测。
CDD（基于输出分布的污染检测）通过测量模型采样输出的尖锐程度来识别数据污染。我们在70M至410M参数的小型语言模型上，针对GSM8K、HumanEval和MATH数据集进行了受控污染实验，发现CDD的有效性取决于微调是否导致原文记忆。在大多数测试条件下，即使数据被明确污染且可被简单方法识别，CDD仍仅达到随机水平。我们证明，在所有超过随机性能的方法中，基于概率的策略（如困惑度和最小k%概率）均优于CDD，表明其基于尖锐度的思路在小型语言模型中不足以有效检测污染。代码已开源。
原文摘要 · Abstract (English)
CDD, or Contamination Detection via output Distribution, identifies data contamination by measuring the peakedness of a model's sampled outputs. We study the conditions under which this approach succeeds and fails on small language models ranging from 70M to 410M parameters. Using controlled contamination experiments on GSM8K, HumanEval, and MATH, we find that CDD's effectiveness depends critically on whether fine-tuning produces verbatim memorization. In the majority of conditions we test, CDD performs at chance level even when the data is verifiably contaminated and detectable by simpler methods. We show that probability-based methods, specifically perplexity and Min-k\% Prob, outperform CDD in all conditions where any method exceeds chance, suggesting that CDD's peakedness-based approach is insufficient for contamination detection in small language models. Our code is available at https://github.com/Sela-Omer/Contamination-Detection-Small-LM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。