让大模型删除特定数据后完全复现原训练轨迹，实现状态精确还原。
Unlearning at Scale: State-Exact Trace-Preserving Deletion in Billion-Parameter Language Models
- 通过保留执行记录并置零请求数据贡献，实现可追溯的精确删除。
- 在多个模型上验证了状态完全一致，包括Pythia 2.8B的超27亿参数元素。
- 适合关注模型可删除性与数据控制权的研究者使用。
能否在预设训练续跑环境下，使被删除示例离开重放数据集后，精确复现其删除反事实？我们研究了一种保持执行痕迹的反事实方法：固定已记录的执行控制，将请求标识符的贡献设为零。该保证是前瞻性的：原始运行必须记录执行溯源，并保留未污染的检查点。在单卡固定环境中，从标记存储中移除指定行进行重放，可逐比特重建独立执行的轨迹真值。Pythia 160M 在四种删除几何下均精确；Pythia 2.8B 在随机删除5%请求时，匹配全部2,775,208,960个模型状态元素；Llama 3.2 1B 在从4,000个TOFU样本中移除400个后仍保持精确。这些结果确立了十亿参数级别的状态精确性。但该方法不廉价，因分散请求可能迫使近乎全量重放。我们公开标准化的TOFU/OpenUnlearning测量作为描述性诊断工具，因冻结实验缺乏匹配对照，无法做出因果行为声明。
原文摘要 · Abstract (English)
Can a prospectively instrumented training continuation reproduce a deletion counterfactual exactly after selected examples leave its replay dataset? We study a trace-preserving counterfactual that fixes recorded execution controls while assigning requested identifiers zero contribution. The guarantee is prospective: the original run must record this execution provenance and retain an eligible uncontaminated checkpoint. Under pinned single-GPU environments, replay from a token store materialized without the requested rows reconstructs a separately executed trace oracle bit-for-bit in model and optimizer state. Pythia 160M is exact across four deletion geometries; Pythia 2.8B matches all 2,775,208,960 model-state elements for a random 5% request; and Llama 3.2 1B is exact after omitting 400 of 4,000 TOFU examples from replay storage. These results establish billion-parameter state exactness. They do not establish cheap deletion, because dispersed requests can force nearly full replay. We release standardized TOFU/OpenUnlearning measurements as descriptive diagnostics only because the frozen campaign lacks the matched controls required for a causal behavioral claim.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。