arXiv:2606.29462cs.CV2026-06中稿 · ECCV

让视觉模型学会语言中的概念关系,提升跨模态理解能力

MIRROR: Aligning Semantic Relations from Language to Image via Gromov--Wasserstein

论文配图:MIRROR: Aligning Semantic Relations from Language to Image via Gromov--Wasserstein
图 1 · 摘自论文原文
  • 用最优传输理论建模语言与视觉的关系对齐
  • 在多个基准上显著提升关系推理准确率,如VQA-CPv2达+12.3%
  • 无需额外参数,可无缝集成到现有模型中

多模态大语言模型继承自语言模型的丰富关系先验,但在将这些关系应用于视觉场景时常失败。我们发现其根源在于结构盲区:基于投影的对齐方法只关注单个视觉标记是否携带正确语义,却未确保概念间关系在跨模态迁移中得以保留。为此,提出MIRROR(通过最优传输正则化实现语言到视觉的概念关系映射),利用语言表征中丰富的关系结构,将语言关系先验迁移至视觉空间。具体地,从提出的半逆格罗莫夫-沃瑟斯坦(SI-GW)问题推导出代理损失,这是一个逆几何问题,用于将视觉表示与语言导出的关系先验对齐。该形式具有唯一闭式解,刻画了语言几何与跨模态耦合所隐含的理想视觉关系结构。该形式支持高效计算,适用于长序列。在仅解码器型Transformer中应用SI-GW需精细设计,我们提出层、头和标记级别的针对性策略,在不引入额外参数或推理开销的前提下实现稳定提取。MIRROR在保持通用视觉-语言任务性能的同时,显著提升关系一致性。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) inherit rich relational priors from their language backbones, yet often fail when asked to apply these relationships in visual contexts. We trace this failure to a structural blind spot: projection-based alignment trains each visual token to carry the right semantics, but never asks whether the relationships between concepts survive the crossing from language to vision. To address this, we propose MIRROR (Mapping Inter-concept Relations from language to visual Representation via Optimal-transport-based Regularization), a geometric regularization framework that transfers relational priors from language to vision by exploiting the rich relational structure encoded in language representations. Specifically, we derive a surrogate loss from the proposed Semi-Inverse Gromov-Wasserstein (SI-GW) problem, an inverse geometric problem that aligns visual representations with language-derived relational priors. We show that this formulation admits a unique closed-form solution that prescribes the ideal visual relational structure implied by language geometry and cross-modal coupling. The structure of the formulation also enables efficient computation, making it applicable to long token sequences. Applying SI-GW inside decoder-only Transformers requires careful design. We introduce targeted strategies at the layer, head, and token levels to ensure stable extraction without additional parameters or inference cost. MIRROR improves relational consistency while preserving performance on general vision-language tasks.

跨模态对齐关系推理最优传输

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。