arXiv:2502.05739cs.CRcs.AI2025-02被引 1

用机器遗忘技术减少代码大模型泄露敏感信息,保护隐私。

Mitigating Sensitive Information Leakage in LLMs4Code through Machine Unlearning

  • 提出专用基准测试,评估遗忘算法对代码模型的影响。
  • 遗忘后敏感信息直接泄露率下降超50%,代码生成性能保留91%以上。
  • 发现遗忘后仍存在间接泄露风险,需新方法同时应对两类泄露。

代码大语言模型(LLMs4Code)在代码生成任务中表现优异,但近期研究揭示其可能记忆并泄露训练数据中的敏感信息,带来严重隐私风险。本文首次系统性地开展机器遗忘在LLMs4Code中的实证研究。构建专用基准,包含:(i) 包含多种个人信息的合成遗忘集,(ii) 用于评估遗忘后代码生成能力的保留集。在三个主流开源模型(AIXCoder-7B、CodeLlama-7B、CodeQwen-7B)上评估三种代表性遗忘算法(GA、GA+GD、GA+KL)。实验表明,机器遗忘可显著降低直接记忆型泄露:平均泄漏率下降超过50%,同时保留超过91%的原始代码生成性能。进一步分析发现,遗忘后泄露模式从直接转向间接,揭示了尚未被充分关注的持续性风险。结果表明,机器遗忘是提升LLMs4Code隐私保护的有效方案,也凸显了未来需发展能同时抑制直接与间接泄露的技术。

原文摘要 · Abstract (English)

Large Language Models for Code (LLMs4Code) have achieved strong performance in code generation, but recent studies reveal that they may memorize and leak sensitive information contained in training data, posing serious privacy risks. To address this gap, this work presents the first comprehensive empirical study on applying machine unlearning to mitigate sensitive information leakage in LLMs4Code. We first construct a dedicated benchmark that includes: (i) a synthetic forget set containing diverse forms of personal information, and (ii) a retain set designed to evaluate whether code-generation capability is preserved after unlearning. Using this benchmark, we systematically assess three representative unlearning algorithms (GA, GA+GD, GA+KL) across three widely used open-source LLMs4Code models (AIXCoder-7B, CodeLlama-7B, CodeQwen-7B). Experimental results demonstrate that machine unlearning can substantially reduce direct memorization-based leakage: on average, the direct leak rate drops by more than 50% while retaining about over 91% of the original code-generation performance. Moreover, by analyzing post-unlearning outputs, we uncover a consistent shift from direct to indirect leakage, revealing an underexplored vulnerability that persists even when the target data has been successfully forgotten. Our findings show that machine unlearning is a feasible and effective solution for enhancing privacy protection in LLMs4Code, while also highlighting the need for future techniques capable of mitigating both direct and indirect leakage simultaneously.

大模型安全隐私保护机器遗忘代码生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。