揭示二阶优化器在机器遗忘中的记忆残留现象
Shape of Memory: a Geometric Analysis of Machine Unlearning in Second-Order Optimizers
- 通过特征分解模拟损失模型记忆，对比一阶与二阶优化器
- 二阶优化器虽性能对齐理想值，但状态波动显著异常
- 仅在几何信息被主动擦除时，才能恢复稳定与信息删除
我们指出当前机器遗忘的定义对二阶优化器而言仍不充分。通过不同程度的特征分解来模拟损失模型的记忆，比较一阶与二阶学习器在数据删除任务中的表现。尽管两者在性能和梯度上均能接近理想反事实结果，但二阶优化器在优化器状态上表现出显著波动，表明存在无法通过一阶分析检测到的残余信息。多种特征衰减处理表明，只有在受控的状态扰动下，即几何信息（或记忆）被擦除时，系统才恢复稳定并实现有效信息删除。
原文摘要 · Abstract (English)
We argue that current definitions of machine unlearning are underspecified for second-order optimizers. We compare first-order and second-order learners for their ability to handle the data deletion task with varying degrees of eigendecomposition to mimic the loss model memory. While both first and second-order methods realign with the ideal counterfactul in terms of performance and gradient, the second-order optimizer shows significant volatility in the optimizer state. This indicates residual information, supposedly deleted, that isn't detectable by first-order analysis. Various eigendecay treatments show that stability and information loss is regained only under controlled state pertubation where geometric information (or memory) is erased.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。