提出精确遗忘的运输准则,发现多数方法无法完全删除记录。
Exact Record Omission in Delta Attention: A Transport Criterion, Its Cost, and a Replay Certificate
- 用运输凭证机制实现记录删除,仅需一次固定开销
- 4096个后续标记后仍残留约4.5%状态印记
- 重放检查点是唯一能精确遗忘的方法
当用户要求遗忘某条记录时,验证标准是:记忆状态是否与从未存储该记录时一致。独立编码的行可直接删除;循环记忆将记录融合进动态状态。一种希望是携带凭证:保存记录到来时产生的差异,持续传递后续更新,再减去它,使删除代价恒定。我们证明,只有当记录引发的后续变化净抵消时,运输凭证才能实现精确遗忘。在发布的480亿参数Kimi线性混合模型上测试发现,4096个后续标记后,记录仍留下约4.5%状态范数的痕迹,所有测试凭证类均无法清除;重算一半后缀也仅减少不到一半差距,而每标记所需的凭证日志成本在88标记后超过完整检查点。该写入规则分类在Mamba-2、Falcon-H1和RWKV-7上同样成立,预测结果提前记录。从记录前的检查点恢复并重放剩余后缀,在所有检测数组中均能精确匹配从未存储的状态。在混合后缀扫描中,屏蔽记录注意力行虽使采样恢复接近未存储基准,但循环印记仍存,审计者仍可重建参考并检测。评估方法中,检查点重放实现精确遗忘,工作量正比于重放后缀长度。
原文摘要 · Abstract (English)
When a user asks an assistant to forget a record, the test is whether the memory now matches the state it would hold if the record had never been stored. Independently encoded rows can be removed directly; a recurrent memory folds records into an evolving state. One hope is a receipt: save the difference the record made when it arrived, carry it forward through later updates, and subtract it, so that deletion costs one fixed-size edit no matter how long the conversation runs. We show that a transported receipt reaches exact omission if and only if the changes the record induces in later updates cancel out on net, and we measure whether they do on the released 48B Kimi Linear hybrid. They do not: after 4,096 further tokens the record still leaves an imprint of about 4.5% of the state norm that none of the tested receipt classes removes, recomputing half the suffix closes less than half the gap, and the per-token log a receipt needs costs more than a full checkpoint after 88 tokens. The same write-rule classification held on Mamba-2, Falcon-H1, and RWKV-7 with predictions recorded before the runs. Restoring a checkpoint from before the record and replaying the surviving suffix matches the never-stored state exactly on every array we check. In the hybrid suffix sweep, masking the record's attention rows brings sampled recovery close to the never-stored floor even though the recurrent imprint remains, and an auditor who rebuilds the reference can still detect it. Among the evaluated methods, checkpoint replay achieves exact omission, with work proportional to the replayed suffix.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。