arXiv:2605.15300cs.CV2026-05中稿 · ICML

用小模型提前对齐视觉与文本特征,提升大模型多模态理解能力。

Deep Pre-Alignment for VLMs

论文配图:Deep Pre-Alignment for VLMs
图 1 · 摘自论文原文
  • 用小型VLM替代ViT作为感知器,实现视觉特征与文本空间的深层对齐。
  • 在4B模型上提升1.9分,32B模型上提升3.0分,跨多个基准表现更优。
  • 减少语言能力遗忘32.9%,兼容Qwen3、LLaMA 3.2等主流大模型。

多数视觉语言模型(VLM)通过轻量投影器将ViT编码器输出映射到大语言模型(LLM)中。尽管有效,近期研究表明该架构存在对齐难题:视觉特征在LLM初始层仍远离文本空间,导致模型浪费关键深度用于表面模态对齐,而非深层理解与复杂推理。本文提出深度预对齐(DPA),用小型VLM作为感知器替代标准ViT编码器,确保视觉特征在目标大语言模型的文本空间中实现深层对齐。全面实验表明,DPA在4B参数规模下于8个多模态基准上超越基线1.9分,32B规模下提升达3.0分。此外,通过将对齐任务移至感知器,DPA在3个文本基准上实现语言能力遗忘降低32.9%。这些增益在Qwen3和LLaMA 3.2等多种大模型家族中均保持一致,凸显方法普适性。除性能提升外,DPA还为现有VLM开发提供无缝升级路径,仅需模块化替换视觉编码器,计算开销几乎不变。

原文摘要 · Abstract (English)

Most Vision Language Models (VLMs) directly map outputs from ViT encoders to the LLM via a lightweight projector. While effective, recent analysis suggests this architecture suffers from an alignment challenge: visual features remain distant from the text space in the initial layers of the LLM, forcing the model to waste critical depth~\cite{zhang-etal-2024-investigating,artzy-schwartz-2024-attend} on superficial modality alignment rather than deep understanding and complex reasoning. In this work, we propose Deep Pre-Alignment (DPA), a novel architecture that replaces the standard ViT encoder with a small VLM as perceiver, ensuring visual features are deeply aligned with the text space of the target large language model. Comprehensive experiments demonstrate the effectiveness of DPA. On the 4B parameter scale, DPA outperforms baselines by 1.9 points across 8 multimodal benchmarks, with gains widening to 3.0 points at the 32B scale. Moreover, by offloading alignment to the perceiver, DPA achieves a 32.9\% reduction in language capability forgetting over 3 text benchmarks. We further demonstrate that these gains are consistent across different LLM families including Qwen3 and LLaMA 3.2, highlighting the generality of our approach. Beyond performance, DPA also offers a seamless upgrade path for current VLM development, requiring only a modular replacement for the visual encoder with marginal computation overhead.

多模态视觉语言模型对齐大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。