arXiv:2606.30788cs.LGcs.CL2026-06

提出可撤销记忆的动态修正方法,解决安全训练后记忆删除难题。

Revocable Learned State via Process Sidecars

论文配图:Revocable Learned State via Process Sidecars
图 1 · 摘自论文原文
  • 用双系数过程侧车模型动态追踪并修正安全训练对记忆方向的偏移
  • 在三个模型上均显著提升拒绝率,优于传统方法和同类变体
  • 适合关注大模型可撤销性与安全可控性的研究者

语言模型常分阶段适配:公开技能阶段、私有记忆阶段,以及后期学习拒绝关联实体输出的安全阶段。安全阶段后的记忆撤销不同于简单减去记忆更新——因安全优化器已改变记忆方向。本文提出过程侧车(process sidecars)方法,其形式为 $\hatθ(λ,γ)=θ_{\mathrm{AMS}}-λΔ_{\mathrm{M}}-γ\hat{R}_{\mathrm{S}←\mathrm{M}}$,其中 $\hat{R}_{\mathrm{S}←\mathrm{M}}=\hat{J}_{\mathrm{S},\varepsilon}(Δ_{\mathrm{M}})-Δ_{\mathrm{M}}$,$\hat{J}_{\mathrm{S},\varepsilon}$ 是通过实际未来 AdamW 安全训练过程的中心割线估计。实现中 $\varepsilon=1$,在自然记忆编辑尺度下复用 $θ_{\mathrm{AMS}}$ 作为正端点,并计算 $θ_{\mathrm{A}}-Δ_{\mathrm{M}}$ 处一次额外的安全轨迹。理论证明:使用真实运输方向 $R_{\mathrm{S}←\mathrm{M}}$ 的精确侧车,在 $(λ,γ)=(1,1)$ 下以二阶精度恢复反事实仅安全的 $θ_{\mathrm{AS}}$;若无此过程信息,所有标量任务算术编辑均存在一阶反事实误差,而侧车编辑可达二阶准确。在三个模型上,经验证集选择的二维编辑在所有试验中均优于朴素任务算术,且在所有成对试验中优于 $γ=λ$ 的过程 JVP 子族与缓存二维网格的对角切片。

原文摘要 · Abstract (English)

Language models are often adapted in stages: a public skill phase, a private memory phase, and a later safety phase that learns to refuse outputs tied to the remembered entities. Revoking the memory after the safety phase is not the same problem as subtracting the memory update: the later safety optimizer has transported the memory direction. We introduce process sidecars, a two-coefficient edit family $\hatθ(λ,γ)=θ_{\mathrm{AMS}}-λΔ_{\mathrm{M}}-γ\hat{R}_{\mathrm{S}\leftarrow\mathrm{M}}$, with $\hat{R}_{\mathrm{S}\leftarrow\mathrm{M}}=\hat{J}_{\mathrm{S},\varepsilon}(Δ_{\mathrm{M}})-Δ_{\mathrm{M}}$, where $\hat{J}_{\mathrm{S},\varepsilon}$ is a centered secant through the realized future AdamW safety-training process. The implementation uses $\varepsilon=1$ at the natural memory-edit scale; it reuses $θ_{\mathrm{AMS}}$ as the positive endpoint and computes one additional safety trace at $θ_{\mathrm{A}}-Δ_{\mathrm{M}}$. We prove two things. First, the exact sidecar, using the true transported direction $R_{\mathrm{S}\leftarrow\mathrm{M}}$ rather than the secant estimate, at $(λ,γ)=(1,1)$ recovers the counterfactual safety-only oracle $θ_{\mathrm{AS}}$ up to second order; the proof treats AdamW as an augmented-state map over parameters, first moments, and second moments. Second, this process information is necessary: whenever future safety training bends the memory direction, every scalar task-arithmetic edit leaves first-order counterfactual error, while the process-sidecar edit is second-order accurate. Across three models, the validation-selected 2D edit improves held-out refusal closure over naive task arithmetic in all trials, and over the $γ=λ$ process-JVP subfamily, the diagonal slice of the cached 2D grid, in all paired trials.

模型可撤销安全训练记忆编辑动态修正

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。