arXiv:2607.04423cs.CVcs.AI2026-07中稿 · ECCV被引 1

发现图文统一模型中理解与生成能力可相互迁移,提升生成效果。

Transferability Between Understanding and Generation in Unified Multimodal Models

论文配图:Transferability Between Understanding and Generation in Unified Multimodal Models
图 1 · 摘自论文原文
  • 通过共享骨干网络实现跨任务能力迁移,松散结构则无效。
  • 用理解任务训练提升生成性能,避免图像质量下降。
  • 在计数、空间关系、文本识别等任务上验证有效,适合多模态开发。

统一多模态模型(UMMs)将图像理解与生成整合于同一架构,但二者交互机制尚不明确。本文研究其跨任务迁移性:在一项任务上训练的能力是否能在另一任务上自发提升,无需显式监督。通过控制实验发现,采用完全共享的Transformer主干和统一视觉编码器的模型表现出稳定的跨任务迁移,而松散耦合设计则基本无迁移效果。基于此,提出实用训练策略:提升生成能力(如计数)时,先训练对应的理解任务,让能力自然迁移到生成端,从而改善特定生成表现,同时最小化分布偏移。在计数、空间关系及文本识别/生成三类能力上验证了该方法的有效性,证明可在统一多模态模型中系统性利用跨任务迁移性。

原文摘要 · Abstract (English)

Unified Multimodal Models (UMMs) integrate image understanding and generation within a single architecture, yet how the two tasks interact remains understudied. We investigate $\boldsymbol{\mathsf{transferability}}$ in UMMs: whether training a capability on one task improves the same capability on the other without explicit supervision. Through controlled experiments, we empirically find that transferability depends on architecture-models with fully shared transformer backbone and a unified visual encoder exhibit consistent cross-task transfer, while loosely coupled designs show little or none. Leveraging this transferability, we propose a practical training strategy. The most straightforward way to improve a target generative capability (e.g., counting) is to fine-tune generation directly, but this can degrade visual quality due to distribution shift. Instead, we train the corresponding understanding task and let it transfer into generation, which improves capability-specific generative performance while minimizing distribution shift. We validate this across three capabilities-counting, spatial relation, and text recognition/generation-showing that cross-task transferability can be systematically exploited in UMMs.

多模态迁移学习生成模型统一架构

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。