提出深度遗忘新范式，让大模型不仅删除事实，还切断其推理链条。
Evaluating Deep Unlearning in Large Language Models
- 设计深度遗忘任务，要求消除目标事实及其逻辑推导路径。
- 在MQuAKE和Eval-DU数据集上，现有方法要么遗忘不彻底，要么误删无关知识。
- 提出三类新指标，评估遗忘效果、召回率与模型可用性，适合安全可信研究者参考。
机器遗忘已成为构建安全可信模型的重要环节。以往大语言模型的事实遗忘研究多聚焦于稳健移除特定事实，却常忽略其与其它知识的逻辑推导关系。本文提出新的深度遗忘设定：不仅要移除目标事实，还需防止该事实通过模型保留的知识与逻辑推理被重新推导出。为此，我们提出三项新指标：Success-DU与Recall用于衡量遗忘有效性，Accuracy用于评估剩余模型的实用性。为基准测试，我们利用现有真实世界知识数据集MQuAKE（提供单步推理实例），并构建全新半合成数据集Eval-DU，支持多步真实推理链。实验表明，当前方法在深度遗忘上表现不佳：或未能彻底遗忘，或过度删除无关事实。结果提示需针对性开发算法以实现大语言模型的鲁棒深度遗忘。
原文摘要 · Abstract (English)
Machine unlearning has emerged as an important component in developing safe and trustworthy models. Prior work on fact unlearning in LLMs has mostly focused on removing a specified target fact robustly, but often overlooks its deductive connections to other knowledge. We propose a new setting for fact unlearning, deep unlearning, where the goal is not only to remove a target fact but also to prevent it from being deduced via retained knowledge in the LLM and logical reasoning. We propose three novel metrics: Success-DU and Recall to measure unlearning efficacy, and Accuracy to measure the remainder model utility. To benchmark this setting, we leverage both (1) an existing real-world knowledge dataset, MQuAKE, that provides one-step deduction instances, and (2) newly construct a novel semi-synthetic dataset, Eval-DU, that allows multiple steps of realistic deductions among synthetic facts. Experiments reveal that current methods struggle with deep unlearning: they either fail to deeply unlearn, or excessively remove unrelated facts. Our results suggest that targeted algorithms may have to be developed for robust/deep fact unlearning in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。