提出新评估框架，检测大模型是否真忘了知识。
Do LLMs Really Forget? Evaluating Unlearning with Knowledge Correlation and Confidence Awareness
- 用知识图谱+置信度建模事实关联，更贴近真实知识结构。
- 发现现有方法高估遗忘效果，部分知识仍隐性留存。
- 用大模型作裁判，提升评估可靠性，适合研究遗忘机制者。
机器遗忘技术旨在缓解大语言模型（LLMs）中的意外记忆问题。然而，现有方法多聚焦于删除孤立事实，忽视了知识内部潜在的推断依赖关系以及知识的非确定性特征。因此，看似已被遗忘的事实可能通过相关知识隐性保留。为此，我们提出一种基于知识图谱与置信度评分的遗忘评估框架，更准确捕捉现实世界知识的隐式结构。我们进一步设计基于推理的评估协议，利用强大LLM作为评判者，通过推理提取的知识子图判断遗忘成效。这些LLM裁判采用精心设计的提示并经人工评估校准，确保其可信性与稳定性。在新构建基准上的大量实验表明，该框架能提供更真实、严谨的遗忘性能评估。此外，我们的研究发现，当前评估策略普遍高估遗忘效果。代码已公开于 https://github.com/Graph-COM/Knowledge_Unlearning.git。
原文摘要 · Abstract (English)
Machine unlearning techniques aim to mitigate unintended memorization in large language models (LLMs). However, existing approaches predominantly focus on the explicit removal of isolated facts, often overlooking latent inferential dependencies and the non-deterministic nature of knowledge within LLMs. Consequently, facts presumed forgotten may persist implicitly through correlated information. To address these challenges, we propose a knowledge unlearning evaluation framework that more accurately captures the implicit structure of real-world knowledge by representing relevant factual contexts as knowledge graphs with associated confidence scores. We further develop an inference-based evaluation protocol leveraging powerful LLMs as judges; these judges reason over the extracted knowledge subgraph to determine unlearning success. Our LLM judges utilize carefully designed prompts and are calibrated against human evaluations to ensure their trustworthiness and stability. Extensive experiments on our newly constructed benchmark demonstrate that our framework provides a more realistic and rigorous assessment of unlearning performance. Moreover, our findings reveal that current evaluation strategies tend to overestimate unlearning effectiveness. Our code is publicly available at https://github.com/Graph-COM/Knowledge_Unlearning.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。