arXiv:2607.27539cs.LGcs.CL2026-07被引 1

提出无需训练即可实现可审计删除的预训练模型记忆管理方法

Subtract, Transport, or Replay? Auditable Deletion from Language-Model Memory

论文配图:Subtract, Transport, or Replay? Auditable Deletion from Language-Model Memory
图 1 · 摘自论文原文
  • 通过检查点重放实现无损记忆恢复,验证删除可审计性
  • 在1B~12B模型上实现基线匹配的准入率,4B模型仅1.85%困惑度损耗
  • 不依赖参数更新或注意力迁移,适合隐私敏感场景使用

精确删除持久化语言模型中的记忆,取决于记录的影响在后续计算中是否仍可定位。针对测试的收据接口,原生Kimi Delta Attention(KDA)结果为负面:语料池化的原始循环贡献变化达12%-49%,经衰减账本修正后仍保持8%-49%。原生省略也改变了后续转移与写入项及其他活跃缓存。冻结输入传输在固定输入控制下成功;改变后的项使原生省略超出测试收据类别。检查点重放提供评估的重新计算路径;最终逻辑值残差为零,且全部80个审计KDA数组均验证跨声明检查点表面的恢复。互补结果具有建设性:我们无需注意力迁移、低秩恢复、蒸馏、适配器或语言模型参数更新,将支持向量记忆融入冻结的Gemma 3。前缀质量保持与每前缀一个盒子方案实现4B模型上基线匹配的准入率,困惑度增加1.85%。在1B和4B模型上,验证删除与条件保留键重校准的差异最大不超过1.3e-10的下一个词KL;4B模型上的行为攻击达到从未存储或随机基线水平。在1B、4B和12B模型中,4B检查点唯一结合了基线匹配准入与低开销。本文两项贡献为:对原生KDA测试收据类别的负面结果,以及一种面向可定位预训练记忆的免训练构造。

原文摘要 · Abstract (English)

Exact deletion from persistent language-model memory depends on whether a record's effect remains addressable after later computation. Native Kimi Delta Attention (KDA) gives a negative result for the tested receipt interface: the corpus-pooled raw recurrent contribution changes by 12-49% with the suffix and remains 8-49% after a decay-ledger correction. Native omission also changes later transition and write terms and other active caches. Frozen-input transport succeeds on its fixed-input control; the changed terms place native omission outside the tested receipt classes. Checkpoint replay supplies the evaluated recomputation path; zero residual on final logits and all 80 audited KDA arrays verifies restoration across the declared checkpoint surface. The complementary result is constructive. We retrofit support-vector memory into frozen Gemma 3 without attention transfer, low-rank recovery, distillation, adapters, or language-model parameter updates. Prefix-mass preservation and one box per prefix solve give base-matched admission at 4B with 1.85% perplexity overhead. At 1B and 4B, verified deletion agrees with its conditional retained-key refit within 1.3e-10 maximum next-token KL; behavioral attacks at 4B reach never-stored or chance baselines. Across 1B, 4B, and 12B, the 4B checkpoint uniquely combines base-matched admission with low overhead. The paper's two contributions are a negative result for native KDA's tested receipt classes and a positive training-free construction for addressable pretrained memory.

可审计删除记忆管理预训练模型隐私保护

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。