arXiv:2506.07171cs.CLcs.LG2025-06NeurIPS被引 13

用强化学习让大模型精准删掉敏感信息,不降性能还更自然。

RULE: Reinforcement UnLEarning Achieves Forget-Retain Pareto Optimality

  • 将删除信息建模为拒绝边界优化,用小样本和合成数据训练。
  • 仅用12%遗忘数据和8%合成数据,删除效果提升17.5%,输出更自然。
  • 适合需隐私保护的大模型部署,尤其关注安全与自然性的场景。

大规模语言模型(LLMs)在未经筛选的海量语料上训练,引发了对敏感、版权或非法内容的担忧。为此,研究者越来越关注模型遗忘——在不重新训练且不损害整体能力的前提下,有选择地移除特定信息。然而,现有方法通常依赖大规模遗忘与保留数据集,常导致回应不自然、泛化差或性能严重下降。本文提出强化遗忘(RULE),将遗忘任务建模为拒绝边界优化问题。RULE仅使用少量遗忘数据(12%)和合成边界查询,通过可验证的奖励函数,鼓励模型对敏感问题拒绝响应,同时保持对合法输入的有用回答。理论与实证均证明,RULE可在不牺牲通用能力的情况下实现精准遗忘。实验表明,在仅12%遗忘数据和8%合成数据条件下,其遗忘效果优于基线最多17.5%,自然度提升16.3%,并实现了遗忘-保留帕累托最优。尤为关键的是,RULE还提升了输出自然度,加快训练效率,并展现出强泛化能力,能将拒绝行为推广至语义相关但未见过的查询。

原文摘要 · Abstract (English)

The widespread deployment of Large Language Models (LLMs) trained on massive, uncurated corpora has raised growing concerns about the inclusion of sensitive, copyrighted, or illegal content. This has led to increasing interest in LLM unlearning: the task of selectively removing specific information from a model without retraining from scratch or degrading overall utility. However, existing methods often rely on large-scale forget and retain datasets, and suffer from unnatural responses, poor generalization, or catastrophic utility loss. In this work, we propose Reinforcement UnLearning (RULE), an efficient framework that formulates unlearning as a refusal boundary optimization problem. RULE is trained with a small portion of the forget set and synthesized boundary queries, using a verifiable reward function that encourages safe refusal on forget--related queries while preserving helpful responses on permissible inputs. We provide both theoretical and empirical evidence demonstrating the effectiveness of RULE in achieving targeted unlearning without compromising model utility. Experimental results show that, with only $12%$ forget set and $8%$ synthesized boundary data, RULE outperforms existing baselines by up to $17.5%$ forget quality and $16.3%$ naturalness response while maintaining general utility, achieving forget--retain Pareto optimality. Remarkably, we further observe that RULE improves the naturalness of model outputs, enhances training efficiency, and exhibits strong generalization ability, generalizing refusal behavior to semantically related but unseen queries.

模型遗忘强化学习大模型安全

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。