改进奖励设计,让模型更快忘记特定知识。
Beyond Binary Rewards: A Comparative Study of Reward Design for Reinforcement Unlearning

- 拆解奖励的可验证性与稀疏性,设计非二值奖励
- 新奖励使遗忘速度提升3倍,效果相当且保持模型能力
- 适合关注高效隐私合规的AI研发人员
机器遗忘旨在不重新训练的前提下,从已训练的语言模型中选择性移除特定知识,这在GDPR和欧盟《人工智能法案》等隐私法规下日益重要。近期研究将遗忘问题重构为可验证奖励的强化学习(RLVR)任务,其中奖励直接基于模型输出计算。然而现有方法依赖稀疏的二值奖励,仅指示是否避免了禁止内容,学习信号微弱,限制收敛速度。本文在强化遗忘(RUL)框架下研究奖励设计对效率的影响,提出一种系统性的奖励分解方法,解耦可验证性与稀疏性,并设计两种新奖励:基于禁止概念出现次数的指数惩罚奖励,以及受PageRank启发、按语义重要性加权的惩罚奖励。在真实世界知识遗忘(RWKU)基准上的实验表明,两种新奖励均显著优于二值设定,在达到相似遗忘效果的同时,速度最高提升3倍,并保持模型通用性能。结果表明,奖励设计是推动遗忘效率的关键因素,为可扩展、高效的机器遗忘提供了可行路径。
原文摘要 · Abstract (English)
Machine unlearning seeks to selectively remove specific knowledge from trained language models without full retraining, a growing necessity under privacy regulations such as GDPR and the EU AI Act. Recent work has reformulated unlearning as a Reinforcement Learning with Verifiable Rewards (RLVR) problem, where models are optimized against verifiable rewards computed directly from their outputs. However, existing methods rely on sparse binary rewards that provide minimal learning signal, indicating only whether forbidden content was avoided, and limiting convergence speed. In this paper, we study how reward design affects unlearning efficiency within the Reinforcement Unlearning (RUL) framework. We introduce a principled reward decomposition framework that decouples verifiability from sparsity, and propose two new reward functions: an exponential reward that provides graded penalties based on the count of forbidden-concept occurrences, and a PageRank inspired reward that weights penalties by semantic importance. We conduct experiments on the Real World Knowledge Unlearning (RWKU) benchmark, demonstrating that both rewards consistently outperform the binary setting, while reaching similar forgetting performance up to $3\times$ faster and preserving general model utility. Our results show that reward design is a key driver of unlearning efficiency offering a practical path toward scalable and efficient machine unlearning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。