让大模型忘记特定知识,同时保持推理能力
Selective Forgetting for Large Reasoning Models
- 用多模型+检索增强分析推理链,定位需删除内容
- 替换敏感段落为结构保留的无害占位符
- 新损失函数平衡遗忘与推理完整性,适合医疗等敏感场景
大型推理模型(LRMs)在生成最终答案前会输出结构化的思维链(CoTs),这使其极易因中间推理步骤泄露敏感信息。训练数据中对版权或隐私内容的过度记忆已引发伦理与法律问题。为此,选择性遗忘(又称机器遗忘)成为解决该问题的潜在方案。然而,现有方法多针对最终答案进行遗忘,可能导致模型整体推理能力下降;直接对整个思维链执行遗忘,亦可能削弱通用推理能力。核心挑战在于:精准移除目标知识的同时,保留通用推理能力。本文提出一种新型LRM遗忘框架,可选择性删除敏感推理成分,同时维持通用推理能力。该方法利用多个具备检索增强生成(RAG)的LLM分析思维链痕迹,识别需遗忘的片段,并以保持逻辑结构的无害占位符替代。我们还引入一种新的特征替换遗忘损失,能同时抑制遗忘内容的生成概率,并强化结构合理的替代内容。在合成数据集和医疗数据集上的大量实验验证了所提方法的有效性。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) generate structured chains of thought (CoTs) before producing final answers, making them especially vulnerable to knowledge leakage through intermediate reasoning steps. Yet, the memorization of sensitive information in the training data such as copyrighted and private content has led to ethical and legal concerns. To address these issues, selective forgetting (also known as machine unlearning) has emerged as a potential remedy for LRMs. However, existing unlearning methods primarily target final answers and may degrade the overall reasoning ability of LRMs after forgetting. Additionally, directly applying unlearning on the entire CoTs could degrade the general reasoning capabilities. The key challenge for LRM unlearning lies in achieving precise unlearning of targeted knowledge while preserving the integrity of general reasoning capabilities. To bridge this gap, we in this paper propose a novel LRM unlearning framework that selectively removes sensitive reasoning components while preserving general reasoning capabilities. Our approach leverages multiple LLMs with retrieval-augmented generation (RAG) to analyze CoT traces, identify forget-relevant segments, and replace them with benign placeholders that maintain logical structure. We also introduce a new feature replacement unlearning loss for LRMs, which can simultaneously suppress the probability of generating forgotten content while reinforcing structurally valid replacements. Extensive experiments on both synthetic and medical datasets verify the desired properties of our proposed method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。