提出因果审计框架,精准定位语言模型遗忘的残留来源。
Auditing Forgetting in Limited Memory Language Models

- 通过三类数据库干预,分离出参数泄漏、检索纠错与近邻干扰三种机制。
- 删除后残留率最高达13.6%,主要来自近邻检索而非模型参数记忆。
- 模型本身几乎无残留,未删知识存活由数据库设计决定。
有限记忆语言模型(LMLMs)将事实知识外置于数据库,实现无需重训练的删除式遗忘。现有评估仅统计删除后的整体正确率,无法区分被删事实是否通过参数记忆、替代检索路径或邻近检索伪影残留。本文提出因果审计框架,在推理时固定模型,动态切换数据库状态为完整(FULL)、开启删除(DEL-ON)和关闭删除(DEL-OFF)三种干预。该框架将删除后行为分解为参数泄漏量 L(f)、检索增强正确率 R(f) 及基于推理轨迹的检索伪影率。在十三个数据库(含四种对抗性拓扑:Base, Alias, Noise, Collision)上执行12,228次别名闭包删除,覆盖六种提示格式。结果显示,所有变体与提示风格下参数泄漏接近于零,模型在无检索时极少返回被删答案;残存部分均源于检索图结构:检索增强正确率与伪影率在四舍五入范围内一致,说明删除后正确率主要由近邻检索重建。残存量从公开数据库的0.7%到最极端对抗变体的13.6%不等,且提示形式无法独立控制残留程度。结论表明,此类LMLM与删除流程中,遗忘边界主要由数据库管理员定义,而非模型自身。
原文摘要 · Abstract (English)
Limited Memory Language Models (LMLMs) externalize factual knowledge to a database to enable deletion-based unlearning without retraining. Existing evaluations measure post-deletion correctness in aggregate and cannot tell whether a deleted fact persists through residual parametric memory, alternative retrieval paths, or near-neighbor retrieval artifacts. We propose a causal auditing framework that holds the model fixed and varies the database state at inference time across three interventions: FULL, DEL-ON, and DEL-OFF. The framework decomposes post-deletion behavior into parametric leakage L(f), retrieval-mediated correctness R(f), and a retrieval artifact rate grounded in the inference-time retrieval trace. We apply it to 12,228 alias-closure deletions across thirteen databases, including four adversarial topologies (Base, Alias, Noise, Collision) we construct in three domains, and six prompt formulations. Parametric leakage is near zero in every variant and every prompt style: the model rarely returns the deleted answer in the absence of retrieval. The residual that does survive lives in the retrieval graph: retrieval-mediated correctness and the retrieval artifact rate match within rounding everywhere, so post-deletion correctness is, in our audit, predominantly reconstituted from near-neighbor retrieval. This residual ranges from 0.7% on the released LMLM database to 13.6% on the most adversarial variant, and prompt formulation does not independently control how much of a deleted fact survives. These results suggest that, for this class of LMLM and deletion procedure, the unlearning boundary is drawn primarily by the database administrator rather than by the model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。