arXiv:2605.15138cs.LGcs.CL2026-05被引 1

提出新方法实现量化后仍能永久遗忘,解决模型压缩导致遗忘失效的问题。

Forgetting That Sticks: Quantization-Permanent Unlearning via Circuit Attribution

论文配图:Forgetting That Sticks: Quantization-Permanent Unlearning via Circuit Attribution
图 1 · 摘自论文原文
  • 基于电路归因定位最小遗忘子图,结合投影与保留约束。
  • 在多种模型上实现有意义遗忘且量化后性能不降反升。
  • 引入新评估指标区分结构擦除与行为抑制,适合可信安全场景。

标准的模型遗忘评估在全精度下立即进行,但所有部署的语言模型均需先量化。近期研究发现4比特后训练量化会逆转机器遗忘;我们证明这并非调参偏差,而是系统性双重失败:梯度方法虽能有效遗忘但在压缩下失效,而抗压缩方法几乎不改变模型。两者根源相同:所有基线中参数更新量仅为NF4量化桶宽的47-828倍,海量参数更新无法突破量化边界,我们将其形式化为稀疏性-持久性权衡。本文提出MANSU(机制对齐零空间遗忘),通过因果电路归因定位最小遗忘子图,结合电路受限零空间投影与对角费舍尔保留界,并设置每参数幅度下限以确保量化生存。此外引入电路归因分歧(CAD)作为机制验证指标,可区分结构擦除与行为抑制——现有指标无法做到。在多个模型族与危险基准上,MANSU是首个在四项属性(有意义遗忘、保留保持、非正PTQ间隙、结构擦除)上均显著达标的方法,而梯度基线在压缩下准确率最高回升0.05。

原文摘要 · Abstract (English)

Standard unlearning evaluations measure behavioral suppression in full precision, immediately after training, despite every deployed language model being quantized first. Recent work has shown that 4-bit post-training quantization can reverse machine unlearning; we show this is not a tuning artefact but a systematic dual failure: gradient-based methods that achieve meaningful forgetting lose it under compression, while methods that survive quantization barely change the model. Both failures trace to the same root cause: across all baselines, per-parameter updates lie 47-828x below the NF4 quantization bin width; updates diffused across billions of parameters cannot clear quantization bin boundaries, a consequence we formalize as a sparsity-permanence tradeoff. We present MANSU (Mechanistic-Aligned Null-Space Unlearning), which resolves both modes by combining causal circuit attribution to isolate the minimal forget-set subgraph, circuit-restricted null-space projection with a diagonal-Fisher retain bound, and a per-parameter magnitude floor guaranteeing quantization survival by construction. We additionally introduce Circuit Attribution Divergence (CAD), a mechanistic verification metric distinguishing structural erasure from behavioral suppression, a distinction existing metrics cannot make. Across multiple model families and hazard benchmarks, MANSU is the first method to jointly satisfy all four properties with margin on each (meaningful forgetting, retain preservation, non-positive PTQ gap, and structural erasure), while gradient-based baselines recover up to +0.05 accuracy under compression.

模型遗忘量化机制归因安全

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。