让大模型在视觉嵌入空间直接推理,提升细粒度视觉理解能力。
Latent Visual Reasoning
- 在视觉嵌入空间中进行自回归推理,突破语言空间限制。
- 在MMVP上达71.67%准确率,优于Qwen2.5-VL的66.67%。
- 适合需要精细视觉分析的多模态问答任务研究者。
多模态大语言模型通过在语言空间中引入思维链(CoT)推理取得了显著进展。近期工作通过外部工具进行视觉编辑,增强推理过程中的视觉信号,但仍受限于推理始终局限于语言空间,视觉信息被视为静态前提。本文提出潜空间视觉推理(LVR),一种直接在视觉嵌入空间中进行自回归推理的新范式。首先,视觉编码器将图像投影到与语言模型共享语义空间的视觉标记;随后,语言模型被训练生成能重构回答关键问题的视觉标记,构成潜空间视觉推理过程。通过与标准文本生成交替进行,模型在感知密集型视觉问答任务上取得显著提升。此外,我们采用GRPO算法对潜推理进行强化学习,进一步平衡潜推理与文本生成。实验表明,LVR大幅提升了细粒度视觉理解和感知能力,在MMVP上达到71.67%准确率,相较Qwen2.5-VL的66.67%有明显优势。代码与模型权重将在后续发布。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have achieved notable gains in various tasks by incorporating Chain-of-Thought (CoT) reasoning in language spaces. Recent work extends this direction by leveraging external tools for visual editing, thereby enhancing the visual signal along the reasoning trajectories. Nevertheless, these approaches remain fundamentally constrained: reasoning is still confined to the language space, with visual information treated as static preconditions. We introduce Latent Visual Reasoning (LVR), a new paradigm that enables autoregressive reasoning directly in the visual embedding space. A visual encoder first projects images into visual tokens within a joint semantic space shared with the language model. The language model is then trained to generate latent states that reconstruct key visual tokens critical for answering the query, constituting the process of latent visual reasoning. By interleaving LVR with standard text generation, our model achieves substantial gains on perception-intensive visual question answering tasks. In addition, we adapt the GRPO algorithm to conduct reinforcement learning on latent reasoning, further balancing LVR and textual generation. We show that LVR substantially improves fine-grained visual understanding and perception, achieving 71.67% on MMVP compared to 66.67% with Qwen2.5-VL. Code base and model weights will be released later.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。