提出一种高效安全的LLM删忆方法,可精准删除敏感信息且不损失模型性能。
Reinforcement Unlearning via Group Relative Policy Optimization
- 基于组相对策略优化,用内在奖励惩罚敏感内容提及
- 删忆效率提升46倍,流畅度提升5.48%,抗干扰能力提升12.02%
- 适合需合规删忆的工业级大模型部署场景
在预训练阶段,大语言模型会无意中记忆敏感或受版权保护的数据,这在欧盟《通用数据保护条例》(GDPR)和《欧盟人工智能法案》等法规下构成重大合规风险。满足这些要求需要可在不从头训练的前提下移除信息的技术。现有删忆方法常出现数据泄露、牺牲流畅性与鲁棒性,或依赖昂贵的外部奖励模型。本文提出PURGE(Policy Unlearning through Relative Group Erasure),一种基于组相对策略优化框架的新方法,将删忆问题建模为可验证任务。PURGE使用内在奖励信号,惩罚任何对禁止概念的提及,实现安全且一致的删忆。实验表明,相比最先进方法,PURGE每目标减少46倍的令牌使用量,同时使流畅度提升5.48%,对抗鲁棒性提高12.02%。在真实世界知识删忆(RWKU)基准测试中,PURGE达到11%的删忆效果,同时保留98%的原始实用性。结果表明,将大模型删忆视为可验证任务,可实现更可靠、高效、可扩展的遗忘,为删忆研究提供了兼具理论保障、安全性与部署效率的新方向。
原文摘要 · Abstract (English)
During pretraining, LLMs inadvertently memorize sensitive or copyrighted data, posing significant compliance challenges under legal frameworks like the GDPR and the EU AI Act. Fulfilling these mandates demands techniques that can remove information from a deployed model without retraining from scratch. Existing unlearning approaches attempt to address this need, but often leak the very data they aim to erase, sacrifice fluency and robustness, or depend on costly external reward models. We introduce PURGE (Policy Unlearning through Relative Group Erasure), a novel method grounded in the Group Relative Policy Optimization framework that formulates unlearning as a verifiable problem. PURGE uses an intrinsic reward signal that penalizes any mention of forbidden concepts, allowing safe and consistent unlearning. Our approach achieves up to x46 lower token usage per target than state-of-the-art methods, while improving fluency by +5.48% and adversarial robustness by +12.02% over the base model. Extensive evaluation on the Real World Knowledge Unlearning (RWKU) benchmark shows that PURGE reaches 11% unlearning effectiveness while preserving 98% of original utility. PURGE shows that framing LLM unlearning as a verifiable task enables more reliable, efficient, and scalable forgetting, suggesting a promising new direction for unlearning research that combines theoretical guarantees, improved safety, and practical deployment efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。