提出SMAR路由策略，让多模态大模型在保持语言能力的同时提升模态区分度。
SMAR: Soft Modality-Aware Routing Strategy for MoE-based Multimodal Large Language Models Preserving Language Capabilities
- 用KL散度约束跨模态路由分布，促进专家分工而不改架构
- 仅用2.5%纯文本数据就实现86.6%语言能力保留率
- 适合需平衡多模态性能与语言能力的模型优化场景
混合专家（MoE）架构已成为扩展大语言模型的关键方法，越来越多研究尝试将其拓展至多模态任务。现有构建多模态MoE模型的方法要么训练成本高，要么在迁移预训练模型时导致语言能力下降。为此，我们提出软模态感知路由（SMAR），一种新颖的正则化技术，通过使用Kullback-Leibler散度控制跨模态的路由概率分布，促使专家专业化，且不修改模型架构或过度依赖文本数据。在视觉指令微调实验中，SMAR在仅使用2.5%纯文本的情况下，仍保持86.6%的语言能力保留率，优于基线方法，同时维持强大的多模态表现。该方法为多模态MoE模型中模态区分性与语言能力的平衡提供了高效实用的解决方案。
原文摘要 · Abstract (English)
Mixture of Experts (MoE) architectures have become a key approach for scaling large language models, with growing interest in extending them to multimodal tasks. Existing methods to build multimodal MoE models either incur high training costs or suffer from degraded language capabilities when adapting pretrained models. To address this, we propose Soft ModalityAware Routing (SMAR), a novel regularization technique that uses Kullback Leibler divergence to control routing probability distributions across modalities, encouraging expert specialization without modifying model architecture or heavily relying on textual data. Experiments on visual instruction tuning show that SMAR preserves language ability at 86.6% retention with only 2.5% pure text, outperforming baselines while maintaining strong multimodal performance. Our approach offers a practical and efficient solution to balance modality differentiation and language capabilities in multimodal MoE models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。