针对MoE模型微调后安全路由失效问题,提出安全路由对齐防御方法。
Defending MoE LLMs against Harmful Fine-Tuning via Safety Routing Alignment
- 通过惩罚微调前后路由权重差异,保持有害输入始终由安全专家处理。
- 在7B到141B模型上将有害性得分从62.0降至5.0,任务性能下降不足1%。
- 适用于大模型如gpt-oss和Llama 4,是首个专为MoE设计的安全微调方案。
近期大语言模型(LLMs)越来越多采用混合专家(MoE)架构以提升效率。然而,现有模型依赖表面化的安全机制:将有害输入路由至安全关键专家。我们分析发现,微调后有害输入的路由决策显著漂移,暴露于有害微调(HFT)攻击之下。现有防御方法主要针对单体模型,难以阻止路由漂移。为此,我们提出SafeMoE,一种专为MoE LLM设计的安全微调方法。该方法通过惩罚微调模型与初始安全对齐模型间路由权重的差距,直接缓解路由漂移,确保有害输入持续被分配至安全专家。在开源的7B至141B参数规模的MoE LLM上实验表明,SafeMoE有效抑制了HFT攻击,例如将OLMoE的有害性得分从62.0降至5.0,同时任务性能损失低于1%,额外开销仅2%。其性能显著优于现有最先进防御方法,并在gpt-oss和Llama 4等大规模最新MoE模型中依然有效。代码已公开于https://anonymous.4open.science/r/SafeMoE。
原文摘要 · Abstract (English)
Recent large language models (LLMs) have increasingly adopted the Mixture-of-Experts (MoE) architecture for efficiency. MoE-based LLMs heavily depend on a superficial safety mechanism in which harmful inputs are routed safety-critical experts. However, our analysis reveals that routing decisions for harmful inputs drift significantly after fine-tuning, exposing a critical vulnerability to harmful fine-tuning (HFT) attacks. Existing defenses, primarily designed for monolithic LLMs, are less effective for MoE LLMs as they fail to prevent drift in harmful input routing. To address this limitation, we propose SafeMoE, a safe fine-tuning method tailored to MoE LLMs. SafeMoE directly mitigates routing drift by penalizing the gap between the routing weights of a fine-tuned model and those of the initial safety-aligned model, thereby preserving the safety-aligned routing of harmful inputs to safety-critical experts. Experiments on open-source MoE LLMs ranging from 7B to 141B parameters demonstrate that SafeMoE effectively mitigates HFT attacks, reducing the harmfulness score of OLMoE from 62.0 to 5.0, for example, while maintaining task utility within 1% degradation and incurring only 2% overhead. It significantly outperforms state-of-the-art defense methods for safeguarding LLM fine-tuning and remains effective in recent large-scale MoE LLMs such as gpt-oss and Llama 4. Our implementation is available at https://anonymous.4open.science/r/SafeMoE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。