针对专家模型遗忘中路由不匹配问题，提出精准校准方法提升遗忘效果。
Routing-Aware Expert Calibration for Machine Unlearning in Mixture-of-Experts Language Models
- 基于激活统计识别关键遗忘专家，动态调整保留数据的损失权重。
- 在多个MoE模型上实现9%相对性能提升，遗忘效果与保留性能更平衡。
- 适合需要高效且安全删除敏感数据的大型语言模型应用。
机器遗忘对大语言模型日益重要，但混合专家（MoE）架构中的遗忘机制仍研究不足。与密集模型不同，MoE在每层通过路由机制将每个标记分配给稀疏专家子集。本文观察到，遗忘数据常过度激活少数专家，而这些专家在保留数据中激活较弱，导致遗忘-保留路由不匹配，使关键遗忘专家在遗忘过程中被欠正则化。为此，我们提出 extbf{TRACE}：目标导向的路由感知专家校准方法。首先通过离线激活统计检测关键遗忘专家，再通过重加权保留数据的样本级损失，使选定专家在保留侧的激活频率更接近遗忘侧。在WMDP和MUSE-BOOKS数据集上对多个MoE大模型的实验表明，TRACE持续改善遗忘-效用权衡，在遗忘质量相当的前提下相较最强基线实现9%相对性能提升，并在四个MUSE-BOOKS指标中有三项表现最优。
原文摘要 · Abstract (English)
Machine unlearning is increasingly important for large language models, yet unlearning in Mixture-of-Experts (MoE) architectures remains underexplored. Unlike dense models, MoE architectures employ a router at each layer to assign each token to a sparse subset of experts. In this work, we observe that forget data often activates a small subset of experts disproportionately, while these experts may receive much weaker activation from retain data. This forget--retain routing mismatch can leave forget-critical experts under-regularized during unlearning. To address this, we propose \textbf{TRACE}, Targeted Routing-Aware Calibration of Experts, for MoE unlearning. TRACE first detects forget-critical experts from offline activation statistics, and then calibrates retain regularization by reweighting token-level retain losses so that each selected expert's retain-side activation frequency better matches its forget-side counterpart. Experiments on WMDP and MUSE-BOOKS across multiple MoE LLMs show that TRACE consistently improves the forget-utility trade-off, yielding a 9\% relative utility improvement over the strongest baseline under comparable forgetting quality and the best performance on three out of four MUSE-BOOKS metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。