让图文生成与理解协同进化,解决大模型训练时的冲突问题。
Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding

- 通过任务特异专家分组与共享专家桥梁设计,实现生成与理解的共生。
- 在MMLU和OCRBench上提升理解能力,生成任务收敛更快。
- 无需额外参数,适合需要多模态协同训练的研究者使用。
为大型多模态模型(LMMs)赋予图像生成能力常导致理解任务出现灾难性遗忘,根源在于严重梯度冲突。现有方法如Mixture-of-Transformers(MoT)虽通过结构隔离缓解冲突,却切断了跨模态协同并造成容量碎片化。本文提出Symbiotic-MoE,一种统一预训练框架,在原生多专家(MoE)Transformer架构中以零参数开销解决任务干扰。我们发现标准MoE微调易引发路由坍塌,生成梯度主导专家使用。为此,引入模态感知专家解耦,将专家分为任务专属组,并用共享专家作为跨模态语义桥梁。关键在于,共享专家能吸收生成任务中的细粒度视觉语义,增强文本表征。同时提出渐进式训练策略,采用差异学习率与早期梯度屏蔽机制,既保护预训练知识免受初期波动影响,又逐步将生成信号转化为理解任务的正向反馈。大量实验表明,Symbiotic-MoE实现快速生成收敛的同时激活跨模态协同,显著提升内在理解能力,在MMLU和OCRBench上取得明显增益。
原文摘要 · Abstract (English)
Empowering Large Multimodal Models (LMMs) with image generation often leads to catastrophic forgetting in understanding tasks due to severe gradient conflicts. While existing paradigms like Mixture-of-Transformers (MoT) mitigate this conflict through structural isolation, they fundamentally sever cross-modal synergy and suffer from capacity fragmentation. In this work, we present Symbiotic-MoE, a unified pre-training framework that resolves task interference within a native multimodal Mixture-of-Experts (MoE) Transformers architecture with zero-parameter overhead. We first identify that standard MoE tuning leads to routing collapse, where generative gradients dominate expert utilization. To address this, we introduce Modality-Aware Expert Disentanglement, which partitions experts into task-specific groups while utilizing shared experts as a multimodal semantic bridge. Crucially, this design allows shared experts to absorb fine-grained visual semantics from generative tasks to enrich textual representations. To optimize this, we propose a Progressive Training Strategy featuring differential learning rates and early-stage gradient shielding. This mechanism not only shields pre-trained knowledge from early volatility but eventually transforms generative signals into constructive feedback for understanding. Extensive experiments demonstrate that Symbiotic-MoE achieves rapid generative convergence while unlocking cross-modal synergy, boosting inherent understanding with remarkable gains on MMLU and OCRBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。