提出GAP框架,让多模态大模型视觉推理更稳定准确。
Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models

- 通过三重对齐机制,修正视觉隐变量的特征与上下文差异。
- 在Qwen2.5-VL 7B上实现最优感知与推理综合性能。
- 适合关注视觉推理稳定性与可解释性的研究者。
视觉隐变量推理使多模态大语言模型(MLLM)以连续标记生成中间视觉证据,无需外部工具或图像生成器。然而,现有方法多采用输出作为输入的隐变量范式,导致性能不稳定。我们发现其根源在于特征空间不匹配:主流视觉隐变量模型基于预归一化MLLM,复用解码器隐藏状态作为预测隐变量输入,但这些状态的范数与模型训练时接收的输入嵌入存在显著差异(Xie et al., 2025; Li et al., 2026; Team et al., 2026)。这种不匹配使直接的隐变量反馈不可靠。针对此问题,我们提出GAP——一种粒度对齐范式,从特征、上下文和容量三个层面实现对齐:特征级对齐通过轻量级PCA对齐的隐变量头将解码器输出映射为与输入兼容的视觉隐变量;上下文级对齐引入可检视的辅助视觉监督来锚定隐变量目标;容量引导对齐则仅对基础MLLM难以处理的样本施加隐变量监督。在Qwen2.5-VL 7B上,该模型在所有受监督变体中达到最优平均综合感知与推理性能。推理时干预探针进一步表明,生成的隐变量提供了超越单纯增加标记槽的任务相关视觉信号。
原文摘要 · Abstract (English)
Visual latent reasoning lets a multimodal large language model (MLLM) create intermediate visual evidence as continuous tokens, avoiding external tools or image generators. However, existing methods usually follow an output-as-input latent paradigm and yield unstable gains. We identify evidence for a feature-space mismatch that can contribute to this instability: dominant visual-latent models build on pre-norm MLLMs and reuse decoder hidden states as predicted latent inputs, even though these states occupy a substantially different norm regime from the input embeddings the model was trained to consume (Xie et al., 2025; Li et al., 2026; Team et al., 2026). This mismatch can make direct latent feedback unreliable. Motivated by this diagnosis, we propose GAP, a Granular Alignment Paradigm for visual latent modeling. GAP aligns visual latent reasoning at three levels: feature-level alignment maps decoder outputs into input-compatible visual latents through a lightweight PCA-aligned latent head; context-level alignment grounds latent targets with inspectable auxiliary visual supervision; and capacity-guided alignment assigns latent supervision selectively to examples where the base MLLM struggles. On Qwen2.5-VL 7B, the resulting model achieves the best mean aggregate perception and reasoning performance among our supervised variants. Inference-time intervention probing further suggests that generated latents provide task-relevant visual signal beyond merely adding token slots.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。