arXiv:2608.05783cs.LGcs.AI2026-08

无需反向传播,秒级删除大模型敏感知识。

GROM: Gradient-Free Rapid One-Shot Machine Unlearning

论文配图:GROM: Gradient-Free Rapid One-Shot Machine Unlearning
图 1 · 摘自论文原文
  • 直接求解权重更新公式,跳过迭代优化。
  • 在5个数据集上实现最佳遗忘效果,计算耗时降低数量级。
  • 适合需要快速、彻底清除敏感信息的场景。

机器遗忘对安全移除大型语言模型中的特定敏感知识至关重要。现有最先进方法主要依赖基于梯度的微调式迭代遗忘,即使使用LoRA等参数高效技术,仍需大量计算且可能仅隐藏而非真正删除目标知识——量化后常可恢复被遗忘内容。为此,我们提出一种新型单次遗忘方法GROM,摒弃迭代优化,采用闭式解析解。将遗忘过程建模为岭正则化最小二乘问题,导出针对目标层权重矩阵的显式加法更新,强制抑制有害内容同时严格保持对保留数据的行为。该更新仅需前向传播计算,无反向传播与迭代,可在数秒内完成,速度远超传统微调。大量实验表明,GROM在TOFU-5%、TOFU-10%、MUSE-Books、MUSE-News和WMDP上均实现最优遗忘-效用平衡,显著降低计算开销且不损害模型整体性能。因更新从权重中实际移除目标内容而非掩蔽,其能抵御低比特量化攻击,而梯度基基线在此攻击下会恢复大量被遗忘信息。代码已开源:https://github.com/Batorskq/GROM。

原文摘要 · Abstract (English)

Machine unlearning has become a critical capability for safely removing specific, sensitive knowledge from large language models (LLMs). Current state-of-the-art approaches primarily rely on iterative, training-time unlearning via fine-tuning. However, even when utilizing parameter-efficient dimensionality reduction techniques like LoRA, gradient-based optimization remains computationally expensive and lacks explicit analytical formulations. It can also leave the targeted knowledge merely hidden rather than removed, to the point that simply quantizing the unlearned model restores much of what it was supposed to have erased. To resolve this, we propose a novel one-shot unlearning approach, abandoning iterative optimization in favor of a direct, exact analytical solution. We frame the unlearning process as a ridge-regularized least-squares optimization problem, deriving a closed-form additive update for targeted weight matrices. This update forces the selected layer to suppress unwanted content while strictly preserving its behavior on retained data. Computed from gradient-free forward passes alone, with no backpropagation and no iteration to convergence, GROM applies the weight edit in mere seconds, which makes it orders of magnitude faster than traditional fine-tuning. Extensive evaluations demonstrate that GROM achieves state-of-the-art forgetting-utility trade-offs on TOFU-5%, TOFU-10%, MUSE-Books, MUSE-News and WMDP, significantly reducing computational overhead without sacrificing overall model performance. Because the update removes the targeted content from the weights instead of masking it, GROM also withstands the low-bit quantization attack that recovers much of the content a gradient-based baseline had appeared to forget. Our code is publicly available at https://github.com/Batorskq/GROM.

机器遗忘大模型高效算法权重更新

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。