多头注意力导致MoE模型持续学习时严重遗忘，新方法提升路由精度。
Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers
- 将多头注意力输出按头拆分路由，避免特征组合冲突。
- 实验显示路由冲突越严重，旧任务损失上升越多，最高达11.2%。
- 适合研究持续学习与MoE架构优化的开发者参考。
Mixture-of-Experts（MoE）架构因稀疏路由被认为适合持续学习，但即使专家使用稀疏且均衡，Transformer仍会显著遗忘。我们发现根源在于路由前的瓶颈：多头注意力将各头信号拼接为统一输入，迫使路由处理共现的特征组合而非独立头通道。该输入同时编码多个可分离的语义与结构因素，且各头支持不均，导致不同特征组合引发弱对齐的参数梯度方向，使多种组合被映射到同一路径。我们通过路径级有效组合数 $N_{eff}$ 量化此碰撞效应，发现 $N_{eff}$ 越高，持续训练后旧任务损失增加越大。为此提出 MH-MoE，对子表示进行头级路由，提升路由粒度并减少组合碰撞。在 TRACE 数据集上，基于 Qwen3-0.6B/8B 的实验表明，MH-MoE 将 BWT 从 LoRAMoE 的 11.2% 降低至 4.5%。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) architectures are often considered a natural fit for continual learning because sparse routing should localize updates and reduce interference, yet MoE Transformers still forget substantially even with sparse, well-balanced expert utilization. We attribute this gap to a pre-routing bottleneck: multi-head attention concatenates head-specific signals into a single post-attention router input, forcing routing to act on co-occurring feature compositions rather than separable head channels. We show that this router input simultaneously encodes multiple separately decodable semantic and structural factors with uneven head support, and that different feature compositions induce weakly aligned parameter-gradient directions; as a result, routing maps many distinct compositions to the same route. We quantify this collision effect via a route-wise effective composition number $N_{eff}$ and find that higher $N_{eff}$ is associated with larger old-task loss increases after continual training. Motivated by these findings, we propose MH-MoE, which performs head-wise routing over sub-representations to increase routing granularity and reduce composition collisions. On TRACE with Qwen3-0.6B/8B, MH-MoE effectively mitigates forgetting, reducing BWT on Qwen3-0.6B from 11.2% (LoRAMoE) to 4.5%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。