通过推理引导实现可解释的LLM知识清除,精准去除非目标知识且不损伤模型能力。
Explainable LLM Unlearning Through Reasoning
- 设计基于推理的清除目标,指导模型精确删除特定知识。
- 在多个基准上优于基线,清除效果更可靠,通用能力保留更好。
- 适合关注AI安全、版权与隐私的开发者和研究者使用。
大语言模型(LLM)的去学习对于缓解预训练模型中的安全、版权和隐私问题至关重要。相比偏好对齐,去学习能更明确地移除由特定去学习数据集定义的不良知识。以往工作采用梯度上升(GA)及其变体实现去学习,但其非针对性导致通用能力意外下降、知识清除不彻底以及生成内容不连贯等问题。我们认为这些问题源于缺乏对‘应清除什么’和‘如何清除’的显式指导。为此,我们提出一种新的去学习目标——基于推理的去学习目标,该目标同时满足指定的去学习范围和去学习后的响应要求。在此基础上,我们提出目标推理去学习(TRU),利用基于推理的去学习目标作为指导。通过将目标与交叉熵监督损失结合,并引入基于GA的损失,使模型在学习推理能力的同时实现精准知识删除,同时保留无关能力。我们在多个基准和不同LLM骨干网络上评估TRU,结果表明其在实现更可靠的去学习的同时有效保持了通用能力。此外,由于通过推理目标学习到的推理能力,TRU在多种攻击场景下表现出更强鲁棒性。总体而言,我们的研究确立了增强推理的去学习作为一种可信赖且可解释的LLM去学习实用范式。
原文摘要 · Abstract (English)
LLM unlearning is essential for mitigating safety, copyright, and privacy concerns in pre-trained large language models (LLMs). Compared to preference alignment, it offers a more explicit way by removing undesirable knowledge characterized by specific unlearning datasets. In previous works, gradient ascent (GA) and its variants have shown promise for implementing unlearning, yet their untargeted nature results in unintended degradation of general capabilities, incomplete removal of knowledge, and the generation of incoherent responses, among many others. We argue that these issues stem from the absence of explicit guidance on what and how models should unlearn. To fill this gap, we introduce a novel unlearning target, reasoning-based unlearning target, which satisfies both the specified unlearning scope and the specified post-unlearning response. Building on this, we propose targeted reasoning unlearning (TRU), which leverages reasoning-based unlearning target as guidance. We employ the target using a cross-entropy supervised loss combined with a GA-based loss, enabling the model to learn reasoning ability for precise knowledge removal while preserving unrelated abilities. We evaluate TRU against strong baselines across multiple benchmarks and LLM backbones, and find that it achieves more reliable unlearning while preserving general capabilities. Moreover, TRU exhibits superior robustness under diverse attack scenarios, stemming from the reasoning ability learned through reasoning-based targets. Overall, our study establishes reasoning-augmented unlearning as a practical paradigm for reliable and explainable LLM unlearning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。