提出AIA损失函数,让统一多模态模型在不拆解架构下同时提升生成与理解能力。
AIA: Rethinking Architecture Decoupling Strategy In Unified Multimodal Model
- 通过分析跨模态注意力行为,发现拆解架构本质是模仿任务专用模型
- 引入AIA损失显式学习任务特定的多模态交互模式,提升性能
- 适用于Emu3和Janus-Pro,无需额外组件,兼顾生成与理解
统一多模态模型在图像生成与理解任务中代表了迈向通用人工智能的重要方向,但其核心挑战在于理解与生成任务存在内在目标冲突,难以建立最优训练范式。现有方法常采用不同程度的架构解耦(如双图像编码器、MOE/MOT结构或冻结多模态大模型),但过度解耦会削弱模型的跨任务协同生成能力,违背统一模型初衷。本文通过分析跨模态注意力行为,发现解耦并非解决冲突,而是使模型趋向于任务专用模型的交互模式(如Qwen3-VL、HunyuanImage-3.0),且解耦越彻底,行为越一致。基于此,我们提出注意力交互对齐(AIA)损失,训练中显式学习任务特定的多模态交互模式。在Emu3和Janus-Pro上分别应用于SFT与后训练阶段,实验表明:无需额外模块,AIA不仅优化了跨模态注意力结构,还显著提升生成与理解性能。
原文摘要 · Abstract (English)
Unified multimodal models for image generation and understanding represent a significant step toward AGI and have attracted widespread attention from researchers. The main challenge of this task lies in the difficulty in establishing an optimal training paradigm due to inherent conflicting targets in understanding and generation tasks. To alleviate these conflicts and pursue higher performance, many researchers adopt varying degrees of architecture decoupling (e.g., Double image encoders, MOE/MOT architecture, or frozen MLLM). However, excessive model decoupling can lead to the loss of interleave generation ability, undermining the original intent of unified models. In this work, we aim to explore how to mitigate task conflicts without resorting to model decoupling. Firstly, we analyze why decoupling boosts performance by studying the cross-modal attention behavior of models. We observe that architecture decoupling does not solve task conflicts, but essentially drives models toward cross-modal interaction patterns of task-specific models, as seen in Qwen3-VL and HunyuanImage-3.0, and that the more thorough the decoupling, the more consistent the behavior becomes. Motivated by this observation, we propose Attention Interaction Alignment (AIA) loss, which explicitly learns task-specific multimodal interaction patterns during training. To demonstrate the generalizability of our AIA loss, we apply it to Emu3 and Janus-Pro during SFT and post-training stage respectively. Without bells and whistles, AIA not only refines cross-modal attention patterns, but also boosts both generation and understanding performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。