解决统一多模态模型中理解与生成任务的相互干扰问题
DIVA: Harnessing the Representation Divergence in Unified Multimodal Models for Mutual Reinforcement

- 通过分解视觉表征为共享与专属成分,实现双分支协同增强
- 在理解任务上提升7.82%,生成任务上提升8.46%
- 适用于各类统一多模态模型的后训练优化
统一多模态模型(UMMs)采用单一架构在理解和生成任务上均表现优异。我们发现其根本挑战源于不同监督信号带来的归纳偏置:生成分支偏好高保真、细粒度的可重建表征,而理解分支则倾向于对任务无关因素不变的语义区分性嵌入。因此,在单体主干网络中同时优化互补但不等价的目标,反而导致相互抑制。本文首先分析了该干扰的根源,并揭示了内部表征中存在的互补结构。受此启发,我们提出DIVA——一种自改进的后训练框架,将表征差异转化为内在协同。通过基于两种互补信息流显式分解视觉表征为共享与专属成分,DIVA使理解和生成分支实现有益迁移,同时通过互信息估计保护专属信息免受跨流干扰。尽管方法具通用性,仍能在视觉理解(+7.82%)和生成(+8.46%)任务上持续提升。官方代码已公开:https://github.com/Jayyy-H/DIVA。
原文摘要 · Abstract (English)
Unified Multimodal models (UMMs) built on a single architecture have shown impressive performance in both understanding and generation. We identify a fundamental challenge that lies in inductive biases induced by distinct supervision signals: generation branch prefers high-fidelity, fine-grained representations capable of reconstruction, while the understanding favours semantically discriminative embeddings that remain invariant to task-irrelevant factors. Consequently, optimizing these complementary but non-equivalent objectives within a monolithic backbone leads to mutual impairment instead of enhancement. In this paper, we first analyze the root cause of this interference in unified backbones and reveal a complementary structure in their internal representations. Motivated by the observation, we propose DIVA, a self-improved post-training framework that transforms the representation divergence into interior synergy. By explicitly factorizing the visual representation into shared and unique components based on two complementary information flow, DIVA enables both the understanding and generation branches to achieve beneficial transferring while preserving the integrity of unique information from cross-flow interference via mutual information estimation. Despite its generality, our method consistently achieves improvements across visual understanding (+7.82%) and generation (+8.46%). The official code is available at: https://github.com/Jayyy-H/DIVA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。