arXiv:2511.11526cs.CV2025-11被引 2

让视觉与语言模型的隐藏状态直接对齐,提升多模态理解能力

Bridging Hidden States in Vision-Language Models

  • 在编码器顶部加入轻量级跨模态注意力层,直接对齐视觉与语言隐藏状态
  • 在检索、VQA和视觉推理任务上超越同类模型,保持对比学习的高效性
  • 结构简洁可插拔,适合需要高效多模态理解的场景

视觉-语言模型(VLMs)通过将图像内容与自然语言对齐实现多模态理解。现有方法通常采用早期融合(在编码器内混合令牌/特征)或晚期融合(比较池化嵌入),许多方法还将融合与自回归解码器绑定。然而,两种模态的隐藏状态已包含丰富的特定结构(视觉的空间布局;文本的语法与语义),因此直接对齐这些状态是匹配两者“思考”方式的自然途径。我们提出一种轻量级融合模块:在两个编码器顶部添加若干仅跨模态、双向注意力层。每层将视觉与文本编码器的隐藏状态序列投影到共享空间,跨模态注意力后,通过门控残差更新回原路径,并使用简单稳定器增强对齐效果。编码器保持非因果性以强化理解能力,生成任务则通过可选解码器清晰解耦。在标准检索、VQA及视觉推理基准测试中,BRIDGE优于同类VLM,同时保留对比模型的双编码器效率。代码公开于 https://github.com/jfeinashley/BRIDGE。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) are a new family of models that align image content with natural language. Existing approaches typically fuse either (a) early: by mixing tokens/features inside the encoders, or (b) late: by comparing pooled embeddings. Many methods also tie fusion to an autoregressive decoder. However, the hidden states of both modalities already carry rich, modality-specific structure (spatial layout in vision; syntax and semantics in text), so directly aligning these states is a natural way to match what the two modalities "think". We propose a lightweight fusion module: a few cross-only, bidirectional attention layers placed near the top of both encoders. Each layer projects the vision and text encoder hidden-state sequences into a shared space, attends across modalities, and sends gated residual updates back, with simple stabilizers to improve alignment. The encoders remain non-causal and strong for understanding, while generation stays cleanly decoupled via an optional decoder. Across standard retrieval, VQA, and visual reasoning benchmarks, BRIDGE outperforms comparable VLMs while preserving the bi-encoder efficiency of contrastive models. We make our code publicly available at https://github.com/jfeinashley/BRIDGE.

多模态视觉语言隐状态对齐轻量级融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。