通过自参照对比检测代码大模型的数据泄露，更准更稳。
SrDetection: A Self-Referential Framework for Data Leakage Detection in Code Large Language Models
- 用语义等价变体对比原样本，判断模型是否依赖训练数据
- 灰箱下F1提升21.52点，黑箱下提升14.46点，效果显著
- 适合评估代码模型时排查隐藏的数据泄露问题
评估代码大语言模型（Code LLMs）需可靠检测数据泄露——即模型在预训练中接触过评测数据，导致性能虚高。现有方法或需访问私有训练语料，或依赖脆弱启发式规则（如时间戳过滤），或使用外部参考集并设定难以泛化的阈值。为此，我们提出统一的自参照泄漏检测框架SrDetection，适用于灰箱（可获取模型输出概率）和黑箱（仅可获取模型输出）场景。SrDetection生成评测样本的语义等价变体，通过对比模型对原样本与变体的行为差异来检测泄漏，标记出原样本明显更易处理的情况。我们还构建了受控泄漏测试环境进行评估。在不同模型和训练阶段上，相比强基线，灰箱设置下平均F1提升21.52点，黑箱设置下提升14.46点，验证了其鲁棒且无需阈值的检测能力。最后，在15个主流Code LLM上对四个常用基准的灰箱研究揭示了超越以往重叠分析的、具有基准特异性的泄漏模式。
原文摘要 · Abstract (English)
Evaluating code large language models (Code LLMs) requires reliable detection of data leakage, where benchmark performance is artificially inflated by exposure to benchmark data during pre-training. Existing approaches either assume access to proprietary training corpora, rely on brittle heuristics such as timestamp filtering, or use external reference sets with manually tuned, non-generalizable thresholds. To address these limitations, we introduce \textbf{SrDetection}, a unified \textbf{s}elf-\textbf{r}eferential leakage detection framework for both gray-box (access to model logits) and black-box (access to model outputs) settings. SrDetection generates semantically equivalent variants of a benchmark sample and detects leakage by contrasting the model's behavior on the original versus its variants, flagging cases where the original is disproportionately easier for the model. We further design a controlled leakage detection testbed and evaluate SrDetection in this environment. Across different models and training stages, SrDetection improves average F1 by 21.52 points in the gray-box setting and 14.46 points in the black-box setting over strong baselines, demonstrating robust, threshold-independent leakage detection. Finally, a gray-box study of 15 widely used Code LLMs on four popular benchmarks reveals benchmark-specific leakage patterns beyond prior overlap-based analyses\footnote{\footnotesize Source code and data are available at https://github.com/SMinL/SrDetectionCode
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。