MoE+LoRA微调中，专家内子空间竞争会导致负迁移，需动态扩展适配器结构。
Routing Is Not Enough: Diagnosing Intra-Adapter Subspace Contention in MoE+LoRA Fine-Tuning
- 引入路由重叠与梯度相似性诊断专家间共享和更新冲突
- 生物医学数据使代码困惑度显著上升，证实路由分离不足
- 新方法SpawnLoRA动态添加门控子适配器，适合多领域微调场景
多领域微调常将MoE路由与LoRA结合，假设令牌级路由可实现领域特定更新。本文通过生物医学文本与数学推理数据测试该假设，发现尽管领域路由近乎不重叠，加入生物医学数据仍显著提升代码困惑度，表明仅靠路由分离无法避免负迁移。为定位问题，提出Jaccard路由重叠与适配器梯度余弦相似性，分别衡量专家共享与更新兼容性，结果显示干扰主要源于近正交领域梯度在同低秩适配器子空间中的竞争。为此提出SpawnLoRA：当检测到适配器级冲突时，动态在MoE专家内添加门控子适配器，保持路由器不变。在Phi-tiny-MoE-instruct与OLMoE-1B-7B上评估多种混合设置，结果表明其有效降低负迁移，优于标准及秩自适应LoRA。说明专家内部结构分离比路由或秩扩展更具优势。
原文摘要 · Abstract (English)
Multi-domain fine-tuning often combines MoE routing with LoRA, assuming that token-level routing separates domain-specific updates. We test this assumption in MoE+LoRA using Python code paired with biomedical text and mathematical reasoning. Although these domains show near-disjoint expert routing, adding biomedical data substantially increases code perplexity, indicating that routing separation alone may not prevent negative transfer. To localize the failure, we introduce Jaccard routing overlap and adapter-gradient cosine similarity, which measure expert sharing and update compatibility, respectively. These diagnostics indicate that interference arises mostly from nearly orthogonal domain gradients competing within the same low-rank adapter subspace. We address this issue with SpawnLoRA, which dynamically adds gated sub-adapters inside MoE experts when adapter-level contention is detected, while keeping the router fixed. We evaluate SpawnLoRA on Phi-tiny-MoE-instruct and OLMoE-1B-7B across multiple mixture settings and find that it effectively reduces negative transfer compared with standard and rank-adaptive LoRA. These results demonstrate that structural separation inside experts provides benefits beyond routing or rank expansion alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。