用对抗提示检测代码模型是否记住了训练数据,效果远超现有方法。
Effective Code Membership Inference for Code Completion Models via Adversarial Prompts
- 设计特定对抗提示诱导输出变化,捕捉模型记忆特征。
- 在APPS和HumanEval上AUC提升最高达102%。
- 方法通用性强,可跨模型、跨数据集迁移使用。
代码补全模型的成员推理攻击(MIAs)能有效评估隐私风险,判断某段代码是否曾出现在训练集中。现有黑盒与灰盒方法依赖昂贵的替代模型或人工规则,难以捕捉大规模代码语言模型的细微记忆模式。为此,我们提出针对代码补全模型的AdvPrompt-MIA方法,结合代码特异性对抗扰动与深度学习。核心思路是设计一系列对抗提示,引发目标模型输出差异;通过对比输出与真实补全结果,构建特征向量训练分类器,自动区分成员与非成员样本。该设计可更全面捕获记忆模式,实现精准成员推断。我们在Code Llama 7B等主流模型上,于APPS和HumanEval基准上进行综合评估,结果表明本方法持续优于现有最佳基线,最高实现AUC提升102%。此外,方法在不同模型与数据集间表现出强迁移能力,验证其实际应用价值与普适性。
原文摘要 · Abstract (English)
Membership inference attacks (MIAs) on code completion models offer an effective way to assess privacy risks by inferring whether a given code snippet was part of the training data. Existing black- and gray-box MIAs rely on expensive surrogate models or manually crafted heuristic rules, which limit their ability to capture the nuanced memorization patterns exhibited by over-parameterized code language models. To address these challenges, we propose AdvPrompt-MIA, a method specifically designed for code completion models, combining code-specific adversarial perturbations with deep learning. The core novelty of our method lies in designing a series of adversarial prompts that induce variations in the victim code model's output. By comparing these outputs with the ground-truth completion, we construct feature vectors to train a classifier that automatically distinguishes member from non-member samples. This design allows our method to capture richer memorization patterns and accurately infer training set membership. We conduct comprehensive evaluations on widely adopted models, such as Code Llama 7B, over the APPS and HumanEval benchmarks. The results show that our approach consistently outperforms state-of-the-art baselines, with AUC gains of up to 102%. In addition, our method exhibits strong transferability across different models and datasets, underscoring its practical utility and generalizability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。