arXiv:2608.17564cs.CVcs.AI2026-08

揭示统一多模态模型中理解与生成的入口差异,找到提升跨任务可用性的关键。

Where a New Concept Must Enter: Entry Point Gates Cross-Task Usability in Unified Multimodal Models

论文配图:Where a New Concept Must Enter: Entry Point Gates Cross-Task Usability in Unified Multimodal Models
图 1 · 摘自论文原文
  • 通过分离理解与生成路径,发现两者在概念绑定方式上本质不同
  • 在第7层注入概念可实现可绘制性,且对整体能力影响极小(仅0.1%损失)
  • 适合研究多模态模型架构设计或高效概念注入的从业者

统一多模态模型(UMMs)旨在让理解和生成相互促进,但实验证明加入生成目标后理解性能未提升。为厘清二者关系,本文通过构建一个仅绑定于单一任务方向的3D视觉实体与伪词,分离两个方向进行测量。结果表明:生成训练仅使模型能从候选中匹配名称;理解训练则使其能主动生成该名称。跨任务可用性取决于概念绑定进入共享计算的位置。对36种配置的对齐探测显示相关性ρ=+0.68。在冻结权重下,基于激活的对齐项在第7层注入时可实现概念可绘制性,且自第14层起与原模型无异;而基于权重的编辑峰值出现在第10-14层。四组观察模型显示,该窗口仅出现在理解路径为语义视觉编码器时,说明共享权重不足以保证可用性,必须在入口处共享语义格式。据此设计的中间层对齐目标,仅以0.1%相对损失获得概念,远优于标准生成路径的41%损失。

原文摘要 · Abstract (English)

Unified multimodal models (UMMs) are motivated by the hope that understanding and generation reinforce each other but controlled ablations repeatedly find that adding a generation objective leaves understanding flat. Joint-training studies cannot settle the disagreement: with overlapping supervision, a gain cannot be attributed to the architecture rather than the data. To further investigate the relationship between the two directions in UMMs, we separate them by construction. A novel visual entity, a rendered 3D asset paired with a pseudo-word screened for absence from the frozen model's behavior, is bound through exactly one task direction, and the untrained direction is then measured. We find that the channel is real in both directions, but the directions differ in kind: generation training installs a name the model can only match among candidates; understanding training installs one it can also produce. What governs cross-task usability is where the binding enters the shared computation. An alignment probe predicts export across 36 configurations (Spearman $ρ= +0.68$). That objective's alignment term, maximized in closed form over activations with every weight frozen, makes a concept drawable when injected at layer 7 of 28 and is indistinguishable from the base model from layer 14 on, while the weight-based version of the same edit peaks at layers 10-14. In an observational series of four models, this window appears only where the understanding pathway is a semantic vision encoder, suggesting that unified weights are not enough: the two directions must share a semantic format at the entry point. Exploiting the rule, a mid-stack alignment objective acquires the concept for a $0.1\%$ relative loss of the model's general text-to-image ability, against $41\%$ for the standard generative route. Our code is at https://github.com/Zane-ZYQiu/entry-point-umm.

多模态模型概念注入架构设计生成理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。