提出几何约束方法,让专家模型真正删除知识而非仅绕开。
GRIP: Algorithm-Agnostic Machine Unlearning for Mixture-of-Experts via Geometric Router Constraints
- 通过几何约束强制路由稳定,让删知识压力作用于专家参数。
- 路由稳定性从0.21提升至0.94,保留准确率最高提升89%。
- 适合需要真正删知识的场景,如版权或敏感信息清除。
在混合专家(MoE)大模型中,机器遗忘仍是一个关键且未充分探索的挑战。现有方法常利用动态路由作为优化捷径:不真正删除专家参数中的知识,而是通过操控路由将查询引导至其他专家。这不仅导致性能严重下降,还使敏感知识残留在闲置专家中,攻击者可绕过路由直接恢复信息。本文提出几何路由不变性保持(GRIP),一种算法无关的框架,通过在路由更新上施加硬几何约束来解决这些问题。通过将路由梯度更新投影到保留集路由矩阵的零空间,GRIP抑制路由操纵,同时不完全冻结路由,从而将遗忘压力导向所有相关专家的参数本身。该方法提供两种互补变体:训练时随机投影与训练后闭式解析修正。在两个MoE模型上的实验表明,GRIP将路由稳定性从0.21提升至>0.94,保留准确率最高提升89%,白盒攻击下知识恢复率从11%降至3%,且在黑盒提示攻击下表现与密集架构遗忘一致,确立了几何约束在稀疏MoE架构中实现真正遗忘的原理性解决方案。
原文摘要 · Abstract (English)
Machine unlearning in Mixture-of-Experts (MoE) large language models presents a critical yet under-explored challenge. Current unlearning methods applied to MoE architectures often exploit dynamic routing as an optimization shortcut: rather than genuinely erasing knowledge from expert parameters, they manipulate routers to redirect queries away from the originally assigned experts. This not only causes severe utility degradation but also leaves hazardous knowledge intact. Consequently, adversaries can bypass the router to recover sensitive information directly from dormant experts. In this study, we propose Geometric Routing Invariance Preservation (GRIP), an algorithm-agnostic framework that resolves these failure modes by enforcing hard geometric constraints on router updates. By projecting router gradient updates into the null space of the retain set's routing matrix, GRIP suppresses routing manipulation without freezing the router entirely, thereby directing the unlearning pressure into the expert parameters themselves across all relevant experts. GRIP offers two complementary variants: training-time stochastic projection and a post-training closed-form analytical correction. Extensive experiments on two MoE models across hazardous knowledge removal and copyright unlearning benchmarks demonstrate that GRIP restores routing stability from 0.21 to >0.94, improves retain accuracy by up to 89%, and reduces white-box adversarial knowledge recovery from 11% to just 3% while in line with dense-architecture unlearning under black-box prompt attack, establishing geometric constraints as a principled solution for genuine unlearning in sparse MoE architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。