通过任务感知专家混合机制，解决多模态大模型理解与生成的冲突问题。
Resolving Task Objective Conflicts in Unified Model via Task-Aware Mixture-of-Experts
- 引入任务感知专家混合层，分离理解与生成路径。
- 在多个基准上达到最优性能，显著减少任务干扰。
- 适合需要统一处理多任务的多模态系统开发者。
基于端到端自回归（AR）Transformer的统一多模态大语言模型（MLLM）能将理解和生成任务整合于单一框架中。然而，理解任务中的高层语义抽象与生成任务中的细粒度细节保留之间存在固有的目标冲突，导致次优权衡和任务干扰。现有方案如解耦共享视觉编码器，因受限于固有的AR架构，难以从根本上解决此类冲突。本文提出一种新方法，通过解耦AR内部组件来缓解任务目标冲突。具体地，设计了UTAMoE——一种统一的任务感知专家混合（MoE）框架，利用任务感知的MoE层解耦内部AR模块，形成任务特定的优化路径。为增强任务区分性同时保持整体协调性，引入两阶段训练策略。在多模态基准上的大量实验表明，UTAMoE有效缓解了任务目标冲突，在多种任务上达到当前最优性能。可视化与消融实验进一步验证了该方法的有效性。
原文摘要 · Abstract (English)
Unified multimodal large language models (MLLMs) based on end-to-end autoregressive (AR) transformers effectively integrate both understanding and generation tasks within a single framework. However, intrinsic Task Objective Conflicts between high-level semantic abstraction in understanding and fine-grained detail preservation in generation pose significant challenges, often leading to suboptimal trade-offs and task interference. Existing solutions, such as decoupling shared visual encoders, fall short of fundamentally resolving these conflicts due to inherent AR architecture. In this paper, we propose a novel approach that decouples internal components of AR to resolve task objective conflicts. Specifically, we design UTAMoE, a Unified Task-Aware Mixture-of-Experts (MoE) framework that decouples internal AR modules via a Task-Aware MoE Layer to create task-specific optimization subpaths. To enhance task differentiation while maintaining overall coordination, we introduce a novel Two-Stage Training Strategy. Extensive experiments on multimodal benchmarks demonstrate that UTAMoE mitigates task objective conflicts, achieving state-of-the-art performance across various tasks. Visualizations and ablation studies further validate the effectiveness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。