arXiv:2507.17389cs.SEcs.AI2025-07

测试七种检测代码模型训练数据的方法,发现它们在真实代码上效果参差不齐。

Investigating Training Data Detection in AI Coders

  • 构建了9000个函数级代码样本的标注数据集CodeSnitch
  • 在8个CodeLLM上测试,发现现有方法对代码的检测准确率普遍低于60%
  • 用四类代码克隆策略验证鲁棒性,揭示方法在结构化代码中的局限性

代码大语言模型(CodeLLMs)已成为现代软件工程的重要工具,但其输出偶尔会包含专有或敏感代码片段,引发训练数据非合规使用的隐私与知识产权风险。为确保合规部署,训练数据检测(TDD)至关重要。尽管现有TDD方法在自然语言中表现良好,但在代码数据上的有效性仍缺乏系统研究,尤其因代码具有结构化语法和不同于自然语言的相似性标准。为此,本文对七种先进TDD方法在源代码上的表现进行了全面实证研究,评估对象涵盖八个CodeLLM。为支持评估,我们构建了CodeSnitch——一个包含9000个代码样本的函数级基准数据集,覆盖三种编程语言,每个样本均明确标注是否来自训练数据。除原始评估外,我们设计了基于类型1至类型4代码克隆分类法的针对性变异策略,在三种不同场景下测试TDD方法的鲁棒性。本研究系统评估了当前代码TDD技术,并为未来开发更有效、更稳健的检测方法提供指导。

原文摘要 · Abstract (English)

Recent advances in code large language models (CodeLLMs) have made them indispensable tools in modern software engineering. However, these models occasionally produce outputs that contain proprietary or sensitive code snippets, raising concerns about potential non-compliant use of training data, and posing risks to privacy and intellectual property. To ensure responsible and compliant deployment of CodeLLMs, training data detection (TDD) has become a critical task. While recent TDD methods have shown promise in natural language settings, their effectiveness on code data remains largely underexplored. This gap is particularly important given code's structured syntax and distinct similarity criteria compared to natural language. To address this, we conduct a comprehensive empirical study of seven state-of-the-art TDD methods on source code data, evaluating their performance across eight CodeLLMs. To support this evaluation, we introduce CodeSnitch, a function-level benchmark dataset comprising 9,000 code samples in three programming languages, each explicitly labeled as either included or excluded from CodeLLM training. Beyond evaluation on the original CodeSnitch, we design targeted mutation strategies to test the robustness of TDD methods under three distinct settings. These mutation strategies are grounded in the well-established Type-1 to Type-4 code clone detection taxonomy. Our study provides a systematic assessment of current TDD techniques for code and offers insights to guide the development of more effective and robust detection methods in the future.

代码生成数据检测模型安全

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。