让推理模型忘记敏感信息,连中间思考过程都清干净。
Reasoning Model Unlearning: Forgetting Traces, Not Just Answers, While Preserving Reasoning Skills
- 提出新方法 R²MU,专门清除推理过程中的敏感痕迹。
- 实验显示能有效降低推理路径泄露风险,同时保持推理能力。
- 适合关注大模型安全与可撤销性的研究者和开发者。
大推理模型(LRMs)通过测试时计算实现强链式思维(CoT)生成,但这也带来了新的安全风险。本文首次系统研究了在LRM背景下机器遗忘问题。传统遗忘算法仅删除最终答案,却无法清除中间推理步骤中的敏感信息。为此,我们提出推理感知的表征误导遗忘方法(R²MU),能有效抑制敏感推理痕迹并防止相关答案生成,同时保留模型的推理能力。实验在DeepSeek-R1-Distill-LLaMA-8B和DeepSeek-R1-Distill-Qwen-14B等先进模型上验证,R²MU显著降低了推理路径中的敏感信息泄露,且在安全与推理基准上均表现优异。
原文摘要 · Abstract (English)
Recent advances in large reasoning models (LRMs) have enabled strong chain-of-thought (CoT) generation through test-time computation. While these multi-step reasoning capabilities represent a major milestone in language model performance, they also introduce new safety risks. In this work, we present the first systematic study to revisit the problem of machine unlearning in the context of LRMs. Machine unlearning refers to the process of removing the influence of sensitive, harmful, or undesired data or knowledge from a trained model without full retraining. We show that conventional unlearning algorithms, originally designed for non-reasoning models, are inadequate for LRMs. In particular, even when final answers are successfully erased, sensitive information often persists within the intermediate reasoning steps, i.e., CoT trajectories. To address this challenge, we extend conventional unlearning and propose Reasoning-aware Representation Misdirection for Unlearning ($R^2MU$), a novel method that effectively suppresses sensitive reasoning traces and prevents the generation of associated final answers, while preserving the model's reasoning ability. Our experiments demonstrate that $R^2MU$ significantly reduces sensitive information leakage within reasoning traces and achieves strong performance across both safety and reasoning benchmarks, evaluated on state-of-the-art models such as DeepSeek-R1-Distill-LLaMA-8B and DeepSeek-R1-Distill-Qwen-14B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。